\documentclass[11pt]{article} \usepackage[dvips]{graphicx} %% FOR INCLUDING GRAPHICS %% ---------------------------------------------------------------------- %% Add specials here %% ---------------------------------------------------------------------- \def\ion#1#2{#1\,{\sc #2}\relax} %% %% ---------------------------------------------------------------------- %% Modify: %% Version number and Date %% Title %% Author and affiliation %% ---------------------------------------------------------------------- \setlength{\oddsidemargin}{0in} % 1 inch (0 + 1) from left. \setlength{\topmargin}{-0.5in} % 1 inch (1.5 - 0.5) from top. \setlength{\textwidth}{6.5in} % 1 inch (8.5 - 1 - 6.5) from right. \setlength{\textheight}{9in} % 1 inch (11 - 1 - 9) from bottom. \addtolength{\parskip}{0.5\baselineskip} % The environment centerpage centers text vertically on the page. This is % useful within a titlepage environment. \newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{ \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}} \begin{document} \bibliographystyle{plain} % Define bibliography style. \begin{titlepage} \hrule \noindent\textbf{EUV IMAGING SPECTROMETER}\\ \vspace{-0.7\baselineskip}\\ {\noindent\Huge\bf Hinode} \vspace{2mm} \hrule \vspace{3mm} \centerline{\bf EIS SOFTWARE NOTE No. 11} \vspace{3mm} \hrule \noindent Version 1 \hfill 5 January 2010 \vspace{2mm} \hrule \begin{centerpage} \begin{center} {\Large\bf Notes on the Compression of EIS Spectral Data}\\ \mbox{}\\ \mbox{}\\ Harry Warren\\ Naval Research Laboratory\\ Code 7673HW\\ Washington, DC 20375, USA\\ \mbox{}\\ \verb+harry.warren@nrl.navy.mil+\\ \mbox{}\\ \end{center} \end{centerpage} \end{titlepage} %% ---------------------------------------------------------------------- %% The document! %% ---------------------------------------------------------------------- \section{Summary} This document provides a very brief overview of the impact of JPEG compression on EIS spectral line profiles. JPEG compression provides for higher levels of data volume than the standard ``lossless'' DPCM compression scheme. JPEG90, for example, provides almost twice the data volume as DPCM. The increased data volume, however, comes at the expense of introducing additional noise into the observation. Since many interesting spectral have relatively low levels of signal to noise, lossey compression is potentially very dangerous. The purpose of this study is to investigate some of the effects of more aggressive JPEG compression on EIS observations. From this initial study we conclude that relatively high levels of compression, such as JPEG90, are not catastrophic for spectroscopy and should be considered as a means of increasing the data volume. This study, however, is not comprehensive and further work is needed. \section{Procedure} To understand the impact of JPEG compression on EIS spectra we use existing data and the J-side mdpjpeg software provided by Hiro Hara. This software mimics the compression processing done on board the spacecraft. Note that the use of data that has already been compressed with DPCM is not ideal since this algorithm is not entirely lossless. For the relatively high levels of JPEG compression we will be considering this is a small effect. After the data has been ``recompressed'' a new FITS file is written so that the analysis can proceed in the usual way: running \verb+eis_prep+, which includes background subtraction, hot pixel removal, and despiking, followed by fitting the line profiles with Gaussians. \section{Test Data} For this test we use EIS observations of an active region. The file is \begin{verbatim} eis_l1_20070202_104212.fits. \end{verbatim} Rasters generated from fits to the DPCM line profiles are shown in Figure~\ref{fig:eis_rasters}. FITS files have been constructed for JPEG 98, 95, 92, 90, 85, 75, and 65. \begin{figure*}[t!] \centerline{% \includegraphics[clip,scale=0.7]{figures/plot_rasters.ps}} \caption{EIS rasters of an active region taken 02-FEB-2007. These are from the original data that were compressed with DPCM.} \label{fig:eis_rasters} \end{figure*} \subsection{Intensity Variations} The simplest point of comparison is the difference between the data number (DN) values at each level of JPEG compression. To compute this we simply compute \begin{equation} DN_{DPCM} - DN_{JPEG} \end{equation} and construct the histogram of the differences. The histograms are shown in Figure~\ref{fig:histogram}. These results suggest that using higher levels of JPEG compression is similar to adding ``white noise'' to the original data. The magnitude of the noise is proportional to the level of the compression, consistent with what one would expect. \begin{figure*}[t!] \centerline{% \includegraphics[clip,scale=0.8]{figures/compare_data_195.119.ps}} \caption{The distribution of differences between the data numbers returned from DPCM and various levels of JPEG compression. The differences are normally distributed. The legend givens the standard deviation in each Gaussian distribution. } \label{fig:histogram} \end{figure*} \begin{figure*}[t!] \centerline{% \includegraphics[clip,scale=0.8]{figures/compare_data_f_195.119.ps}} \caption{The fractional differences between the data numbers returned from DPCM and JPEG 90 as a function of DPCM data number. These values have had the pedestal removed.} \label{fig:diff} \end{figure*} The distributions of the differences suggest that the JPEG compression will be important in the fainter regions of the rasters while the brightest features will be largely unaffected. Thus the impact of the JPEG compression on the data is linked to the distribution of intensities on the Sun. \begin{figure*}[t!] \centerline{% \includegraphics[clip,scale=0.5]{figures/compare_rasters_275.354.ps} \includegraphics[clip,scale=0.5]{figures/compare_rasters_195.119.ps} \includegraphics[clip,scale=0.5]{figures/compare_rasters_284.150.ps} \includegraphics[clip,scale=0.5]{figures/compare_rasters_262.984.ps}} \caption{EIS \ion{Si}{vii} 275.354, \ion{Fe}{12} 195.119, \ion{Fe}{xv} 284.150, and \ion{Fe}{xvi} 262.984\,\AA\ rasters at various levels of compression.} \label{fig:rasters} \end{figure*} \begin{figure*}[t!] \centerline{% \includegraphics[clip,scale=0.5]{figures/compare_moments_195.119.centroid.ps} \includegraphics[clip,scale=0.5]{figures/compare_moments_284.150.centroid.ps} \includegraphics[clip,scale=0.5]{figures/compare_moments_195.119.width.ps} \includegraphics[clip,scale=0.5]{figures/compare_moments_284.150.width.ps}} \caption{Dopple shifts and line widths for \ion{Fe}{12} 195.119 and \ion{Fe}{xv} 284.150 at various levels of compression.} \label{fig:moments} \end{figure*} To see how this applies to these active region observations we have calculated the fractional difference for each pixel \begin{equation} (DN_{DPCM} - DN_{JPEG})/DN_{DPCM}. \end{equation} Since there is an pedestal of approximately 500 added to each data number the fractional difference needs to be corrected to account for this. We simply use the minimum DPCM data number for this. The relationship between the fractional difference and the DPCM data number is shown in Figure~\ref{fig:diff} for JPEG90. Ini this example the brightest pixels are all within a few percent of their DPCM values. Approximately 85\% of the pixels vary by less than 10\%. The EIS rasters for several emission lines at 4 different levels of compression are shown in Figure~\ref{fig:rasters}. It is clear that the noise level rises with higher levels of compression. It is also clear that the horizontal banding in the images is increased. This banding is caused by ``stuck'' or ``warm'' pixels, which the JPEG compression does not handle well. The higher order moments of the line profiles are potentially more susceptible the noise introduced by the compression. The Doppler shifts and line widths for two of the strongest lines are shown in Figure~\ref{fig:moments}. The variations in the width and Doppler shift with compression level are follow the variations seen in the intensity: weaker profiles are noiser and the banding is enhanced. We have also calculated the distributions of Doppler shifts and line widths and they are largely independent of the level of JPEG compression. \section{Recommendations} It is clear from comparing these rasters that the higher levels of JPEG compression are not catastrophic for EIS spectroscopy. We stress, however, that many diagnostics (densities from line ratios, weak lines) have not been studied. The acceptable level of compression will depend on the science goal and needs to be considered carefully. From this initial survey it appears that compression levels as high as JPEG 90 will be able to meet many science objectives and is a useful starting point. Since the impact of compression is dependent on the intensity level one application of the higher compression schemes may be for large context rasters. One could imagine taking large area context rasters in the strongest lines using relatively short exposures and relatively high levels of compression to conserve data volume. Then smaller regions could be observed with smaller rasters and lower levels of compression. To evaluate the impact of compression on a particular science goal it is probably best to first obtain some data with very little compression (e.g., DPCM). These data can then be modified to mimic the effects of JPEG compression. Because of issues with intellectual property rights the mdpjpeg software cannot be distributed. It is possible to contact the author to have individual files processed in this way. An alternative is to add some noise to the data directly and investigate the potential impact of compression. Conceptually, this could be done as follows \begin{verbatim} data = obj_new('eis_data',InputFile) nwin = data->getnwin() for iwin=0,nwin-1 do begin d = data-> getvar(iwin) d_new = d_new + normal_random_number data -> setvar,n_new,iwin endfor data->save,file=OutputFile \end{verbatim} This procedure is unlikely to reproduce the impact of the warm pixels on the compressed data. Also, one should note that the errors calculated in \verb+eis_prep+ do not account for the higher noise levels introduced by the compression. \end{document}