
\documentclass[11pt]{article}

\usepackage[dvips]{graphicx}      %% FOR INCLUDING GRAPHICS

%% ----------------------------------------------------------------------
%% Add specials here
%% ----------------------------------------------------------------------

\def\ion#1#2{#1\,{\sc #2}\relax}  %% 

%% ----------------------------------------------------------------------
%% Modify: 
%%   Version number and Date
%%   Title
%%   Author and affiliation 
%% ----------------------------------------------------------------------

\setlength{\oddsidemargin}{0in}         %  1 inch (0 + 1) from left.
\setlength{\topmargin}{-0.5in}          %  1 inch (1.5 - 0.5) from top.
\setlength{\textwidth}{6.5in}           %  1 inch (8.5 - 1 - 6.5) from right.
\setlength{\textheight}{9in}            %  1 inch (11 - 1 - 9) from bottom.

\addtolength{\parskip}{0.5\baselineskip}

%  The environment centerpage centers text vertically on the page.  This is
%  useful within a titlepage environment.

\newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{
        \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}}

\begin{document}                
\bibliographystyle{plain}               % Define bibliography style.

\begin{titlepage}
\hrule
\noindent\textbf{EUV IMAGING SPECTROMETER}\\
\vspace{-0.7\baselineskip}\\
{\noindent\Huge\bf Hinode}
\vspace{2mm}
\hrule
\vspace{3mm}
\centerline{\bf EIS SOFTWARE NOTE No. 11}
\vspace{3mm}
\hrule
\noindent Version 1 \hfill  5 January 2010
\vspace{2mm}
\hrule

\begin{centerpage}

\begin{center}
{\Large\bf Notes on the Compression of EIS Spectral Data}\\
\mbox{}\\
\mbox{}\\
Harry Warren\\
Naval Research Laboratory\\
Code 7673HW\\
Washington, DC 20375, USA\\
\mbox{}\\
\verb+harry.warren@nrl.navy.mil+\\
\mbox{}\\
\end{center}

\end{centerpage}

\end{titlepage}

%% ----------------------------------------------------------------------
%% The document!
%% ----------------------------------------------------------------------

\section{Summary}

This document provides a very brief overview of the impact of JPEG
compression on EIS spectral line profiles. JPEG compression provides
for higher levels of data volume than the standard ``lossless'' DPCM
compression scheme. JPEG90, for example, provides almost twice the
data volume as DPCM.

The increased data volume, however, comes at the expense of
introducing additional noise into the observation. Since many
interesting spectral have relatively low levels of signal to noise,
lossey compression is potentially very dangerous. The purpose of this
study is to investigate some of the effects of more aggressive JPEG
compression on EIS observations. From this initial study we conclude
that relatively high levels of compression, such as JPEG90, are not
catastrophic for spectroscopy and should be considered as a means of
increasing the data volume. This study, however, is not comprehensive
and further work is needed.

\section{Procedure}

To understand the impact of JPEG compression on EIS spectra we use
existing data and the J-side mdpjpeg software provided by Hiro Hara.
This software mimics the compression processing done on board the
spacecraft. Note that the use of data that has already been compressed
with DPCM is not ideal since this algorithm is not entirely
lossless. For the relatively high levels of JPEG compression we will
be considering this is a small effect. After the data has been
``recompressed'' a new FITS file is written so that the analysis can
proceed in the usual way: running \verb+eis_prep+, which includes
background subtraction, hot pixel removal, and despiking, followed by
fitting the line profiles with Gaussians.

\section{Test Data}

For this test we use EIS observations of an active region. The file is
\begin{verbatim}
  eis_l1_20070202_104212.fits.
\end{verbatim}
Rasters generated from fits to the DPCM line profiles are shown in
Figure~\ref{fig:eis_rasters}. FITS files have been constructed for
JPEG 98, 95, 92, 90, 85, 75, and 65.

\begin{figure*}[t!]
\centerline{%
  \includegraphics[clip,scale=0.7]{figures/plot_rasters.ps}}
\caption{EIS rasters of an active region taken 02-FEB-2007. These are
  from the original data that were compressed with DPCM.}
\label{fig:eis_rasters}
\end{figure*}

\subsection{Intensity Variations}

The simplest point of comparison is the difference between the data
number (DN) values at each level of JPEG compression. To compute this we simply compute
\begin{equation}
DN_{DPCM} - DN_{JPEG}
\end{equation}
and construct the histogram of the differences. The histograms are
shown in Figure~\ref{fig:histogram}. These results suggest that using
higher levels of JPEG compression is similar to adding ``white noise''
to the original data. The magnitude of the noise is proportional to
the level of the compression, consistent with what one would
expect. 

\begin{figure*}[t!]
\centerline{%
  \includegraphics[clip,scale=0.8]{figures/compare_data_195.119.ps}}
  \caption{The distribution of differences between the data numbers
    returned from DPCM and various levels of JPEG compression. The
    differences are normally distributed. The legend givens the
    standard deviation in each Gaussian distribution. }
\label{fig:histogram}
\end{figure*}

\begin{figure*}[t!]
\centerline{%
  \includegraphics[clip,scale=0.8]{figures/compare_data_f_195.119.ps}}
  \caption{The fractional differences between the data numbers
    returned from DPCM and JPEG 90 as a function of DPCM data
    number. These values have had the pedestal removed.}
\label{fig:diff}
\end{figure*}

The distributions of the differences suggest that the JPEG compression
will be important in the fainter regions of the rasters while the
brightest features will be largely unaffected. Thus the impact of the
JPEG compression on the data is linked to the distribution of
intensities on the Sun.

\begin{figure*}[t!]
\centerline{%
  \includegraphics[clip,scale=0.5]{figures/compare_rasters_275.354.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_rasters_195.119.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_rasters_284.150.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_rasters_262.984.ps}}
  \caption{EIS \ion{Si}{vii} 275.354, \ion{Fe}{12} 195.119,
    \ion{Fe}{xv} 284.150, and \ion{Fe}{xvi} 262.984\,\AA\ rasters at
    various levels of compression.}
\label{fig:rasters}
\end{figure*}

\begin{figure*}[t!]
\centerline{%
  \includegraphics[clip,scale=0.5]{figures/compare_moments_195.119.centroid.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_moments_284.150.centroid.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_moments_195.119.width.ps}
  \includegraphics[clip,scale=0.5]{figures/compare_moments_284.150.width.ps}}
  \caption{Dopple shifts and line widths for \ion{Fe}{12} 195.119 and 
    \ion{Fe}{xv} 284.150 at various levels of compression.}
\label{fig:moments}
\end{figure*}

To see how this applies to these active region observations we have
calculated the fractional difference for each pixel
\begin{equation}
(DN_{DPCM} - DN_{JPEG})/DN_{DPCM}.
\end{equation}
Since there is an pedestal of approximately 500 added to each data
number the fractional difference needs to be corrected to account for
this. We simply use the minimum DPCM data number for this. The
relationship between the fractional difference and the DPCM data
number is shown in Figure~\ref{fig:diff} for JPEG90. Ini this example
the brightest pixels are all within a few percent of their DPCM
values. Approximately 85\% of the pixels vary by less than 10\%.

The EIS rasters for several emission lines at 4 different levels of
compression are shown in Figure~\ref{fig:rasters}.  It is clear that
the noise level rises with higher levels of compression. It is also
clear that the horizontal banding in the images is increased. This
banding is caused by ``stuck'' or ``warm'' pixels, which the JPEG
compression does not handle well.

The higher order moments of the line profiles are potentially more
susceptible the noise introduced by the compression. The Doppler
shifts and line widths for two of the strongest lines are shown in
Figure~\ref{fig:moments}. The variations in the width and Doppler
shift with compression level are follow the variations seen in the
intensity: weaker profiles are noiser and the banding is enhanced.

We have also calculated the distributions of Doppler shifts and line
widths and they are largely independent of the level of JPEG compression.

\section{Recommendations}

It is clear from comparing these rasters that the higher levels of
JPEG compression are not catastrophic for EIS spectroscopy. We stress,
however, that many diagnostics (densities from line ratios, weak
lines) have not been studied. The acceptable level of compression will
depend on the science goal and needs to be considered carefully. From
this initial survey it appears that compression levels as high as JPEG
90 will be able to meet many science objectives and is a useful
starting point.

Since the impact of compression is dependent on the intensity level
one application of the higher compression schemes may be for large
context rasters. One could imagine taking large area context rasters
in the strongest lines using relatively short exposures and relatively
high levels of compression to conserve data volume. Then smaller
regions could be observed with smaller rasters and lower levels of
compression.

To evaluate the impact of compression on a particular science goal it
is probably best to first obtain some data with very little
compression (e.g., DPCM). These data can then be modified to mimic the
effects of JPEG compression. Because of issues with intellectual
property rights the mdpjpeg software cannot be distributed. It is
possible to contact the author to have individual files processed in
this way. 

An alternative is to add some noise to the data directly and
investigate the potential impact of compression. Conceptually, this
could be done as follows
\begin{verbatim}
data = obj_new('eis_data',InputFile)
nwin = data->getnwin()
for iwin=0,nwin-1 do begin
  d = data-> getvar(iwin)
  d_new = d_new + normal_random_number
  data -> setvar,n_new,iwin
endfor
data->save,file=OutputFile
\end{verbatim}
This procedure is unlikely to reproduce the impact of the warm pixels
on the compressed data.

Also, one should note that the errors calculated in \verb+eis_prep+ do
not account for the higher noise levels introduced by the compression.

\end{document}
