
\documentclass[11pt]{article}

\usepackage[dvips]{graphicx}      %% FOR INCLUDING GRAPHICS
\usepackage{epsf}

%% ----------------------------------------------------------------------
%% Add specials here
%% ----------------------------------------------------------------------

\def\ion#1#2{#1\,{\sc #2}\relax}  %% 

%% ----------------------------------------------------------------------
%% Modify: 
%%   Note number, version number and date
%%   Title
%%   Author and affiliation 
%% ----------------------------------------------------------------------

\setlength{\oddsidemargin}{0in}         %  1 inch (0 + 1) from left.
\setlength{\topmargin}{-0.5in}          %  1 inch (1.5 - 0.5) from top.
\setlength{\textwidth}{6.5in}           %  1 inch (8.5 - 1 - 6.5) from right.
\setlength{\textheight}{9in}            %  1 inch (11 - 1 - 9) from bottom.

\addtolength{\parskip}{0.5\baselineskip}

\newcommand{\lam}{$\lambda$}
\newcommand{\arcsec}{$^{\prime\prime}$}

%  The environment centerpage centers text vertically on the page.  This is
%  useful within a titlepage environment.

\newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{
        \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}}

\begin{document}                
\bibliographystyle{plain}               % Define bibliography style.

\begin{titlepage}
\hrule
\noindent\textbf{EUV IMAGING SPECTROMETER}\\
\vspace{-0.7\baselineskip}\\
{\noindent\Huge\bf Hinode}
\vspace{2mm}
\hrule
\vspace{3mm}
\centerline{\bf EIS SOFTWARE NOTE No. 24}
\vspace{3mm}
\hrule
\noindent Version 1.2 \hfill 17 December 2019
\vspace{2mm}
\hrule

\begin{centerpage}

\begin{center}
{\Large\bf EIS\_PREP: removing residual warm pixels and cosmic rays}\\
\mbox{}\\
\mbox{}\\
Peter Young \\ 
NASA Goddard Space Flight Center \\ 
Code 671 \\
Greenbelt, MD 20771, USA
\end{center}

\end{centerpage}

\end{titlepage}

%% ----------------------------------------------------------------------
%% The document!
%% ----------------------------------------------------------------------

\section{Overview}

This document describes procedures to remove residual warm pixels and
cosmic rays from EIS data as part of the calibration process
implemented in the routine EIS\_PREP. These procedures were introduced
in 2019, and they required some changes to how EIS\_PREP operates.

The new procedures apply to narrow slit data
only. Sect.~\ref{sect.slot} discusses the slot data.


\section{EIS\_PREP procedures prior to 2019}\label{sect.intro}

The default procedure for EIS\_PREP prior to 2019 was to do the
following sequence:
%
\begin{enumerate}
\item Flag cosmic rays using the routine EIS\_DESPIKE (/nocr).
\item Flag hot pixels using hot pixel maps stored in Solarsoft (/nohp).
\item Flag warm pixels using warm pixel maps stored in Solarsoft (/nowp).
\item Flag dusty pixels using dust pixel maps stored in Solarsoft (/nodp).
\end{enumerate}
%
Further details are given in EIS Software Notes Nos.~1 and 6. The
keywords indicated in the brackets above are used to switch off the
steps when calling EIS\_PREP. 

Inspection of the level-1 data produced by EIS\_PREP shows that many
warm pixels are not removed, either because they fall below the
threshold used in creating the warm pixel maps, or because the warm
pixels were not present in the maps (which are obtained on a weekly
basis). I refer to these as \emph{residual warm pixels}
(RWPs). Examples from a data-set
beginning at 12:07~UT on 19-Mar-2016 are shown in
Figure~\ref{fig.residual}. Time--Y images from the level-1 \ion{Fe}{ix}
\lam197.86 window are shown; the left panel is from wavelength pixel
11 (within the line), and the right panel from pixel 20 (in the
continuum). Significant resdiual warm pixels can be seen in each. 

Potentially the RWPs could be removed by the EIS despiking routine,
but since this routine is optimized to remove spikes that occur in
individual exposures then it is not ideal. In addition, RWPs are
often adjacent to regular, stronger WPs, and since the despiking
routine is applied prior to warm pixel removal then it is not able to
flag the RWP in this case.

In addition to the RWP problem, it was also noticed that the despiking
routine often flags many pixels that are clearly just
noise. Figure~\ref{fig.despike} shows an example from the data-set
beginning at 12:07~UT on 19-Mar-2016. Wavelength pixel 17 from the
\ion{Fe}{ix} \lam197.86 window was chosen, and the images show the
time--Y slice prior to cosmic ray cleaning (left panel) and the
locations of the flagged cosmic rays (right panel). 

\begin{figure}[t]
\centerline{\epsfxsize=5.5in\epsfbox{plot_residual_example.eps}}
\caption{Time--Y images for two wavelength pixels in the \ion{Fe}{ix}
  \lam197.86 window are shown. Residual warm pixels are the narrow
  horizontal lines in the images.}
\label{fig.residual}
\end{figure}

\begin{figure}[h]
\centerline{\epsfxsize=5.5in\epsfbox{plot_despike_example.eps}}
\caption{The left panel shows an image from an EIS data-set prior to
  cosmic ray cleaning. The right panel shows the pixels (black) that
  were flagged as cosmic rays by EIS\_DESPIKE.}
\label{fig.despike}
\end{figure}


\section{New procedures for EIS\_PREP}

The revised operating sequence for EIS\_PREP is as follows:
%
\begin{enumerate}
\item Flag dusty pixels using dust pixel maps stored in Solarsoft
  (/nodp).
\item Flag hot pixels using hot pixel maps stored in Solarsoft (/nohp).
\item Flag warm pixels using warm pixel maps stored in Solarsoft (/nowp).
\item Flag residual warm pixels with EIS\_PREP\_RESIDUAL\_WPS (/norwp).
\item Flag cosmic rays using the routine EIS\_PREP\_CLEAN\_CRS (/nocr).
\end{enumerate}
%
An important distinction between dusty pixels and the other types is
that EIS\_PREP does not try to interpolate the intensity over the
dusty regions from neighboring regions. For this reason it is now
moved to the first step.

The other four steps are now performed
consecutively on each data window in one block of code, whereas
previously each had a separate block of code. The reason for this is
the data interpolation. For example, consider a hot pixel directly
adjacent in the Y-direction to a warm pixel. After the hot pixel has
been flagged, the software tries to replace the intensity with a
median of the two neighboring pixels. Since the warm pixel has not yet
been flagged (since this is a later step in the code) then it is
possible that the hot pixel is replaced with the warm pixel
intensity. With the new code, the positions of the four types of
missing data get stored in a single array, and then the interpolation
is applied to this array. This greatly reduces the number of data
artifacts seen in the level-1 data.

(It is worth highlighting that the
interpolated intensity array produced by EIS\_PREP should not be used
for data analysis, only quicklook purposes. The flagged pixels should
either be ignored, i.e., treated as missing, or replaced using the
/refill keyword.)


\section{How to call EIS\_PREP with the new method}\label{sect.call}

EIS\_PREP is called in exactly the same manner as before, with the
recommended calling sequence (see also the EIS Wiki):
%
\begin{verbatim}
IDL> eis_prep, l0name, /refill, /quiet, /save, /default
\end{verbatim}

A new keyword, /original, has been added which will run EIS\_PREP
exactly how it was prior to the implementation of the new method
(which was introduced on 2019 December 4). The earlier code was
renamed EIS\_PREP\_ORIGINAL and thus /original switches to calling
this routine.

If you want to swich off the RWP or CR removal steps from the new
EIS\_PREP, then use the keywords /norwp and /nocr.


\section{How to check the RWP and CR removal}\label{sect.check}

The key difference with the previous version of EIS\_PREP is that the
cosmic ray removing routine, EIS\_DESPIKE, is no longer used, being
replaced by EIS\_PREP\_CLEAN\_CRS, and an additional routine,
EIS\_RESIDUAL\_WPS, is used to flag residual warm pixels that were not
previously removed.

To test how well the EIS\_RESIDUAL\_WPS  and EIS\_PREP\_CLEAN\_CRS
routines work for a given data-set, the following procedure can be run.

First, EIS\_PREP is run but with the /norwp, /nocr and /noabs keywords
set:
%
\begin{verbatim}
IDL> eis_prep,l0name,/quiet,/save,/default,/retain,/norwp,/nocr,/noabs
\end{verbatim}

Now read a data window from the level-1 file:
%
\begin{verbatim}
IDL> wd=eis_getwindata(l1name,wvl)
\end{verbatim}

The windata structure can then be ``cleaned'' with:
%
\begin{verbatim}
IDL> wdout=eis_clean_windata(wd,swtch=swtch)
\end{verbatim}
%
where SWTCH is an array containing the locations of the pixels that
have been cleaned, i.e., residual warm pixels and cosmic rays removed.

A GUI-based routine can then be used to browse the before-and-after
arrays, with the flagged pixels shown:
%
\begin{verbatim}
IDL> eis_spike_viewer,wd,wdout,swtch
\end{verbatim}
%
The routine shows a slice through the intensity arrays of WD (left)
and WDOUT (middle), and the SWTCH array (right). The latter is an
array of 0's and 1's of same size as the intensity arrays, with 1
indicating a flagged pixel.

There are options to show exposures ($\lambda$ vs.\ Y) or images (X
vs.\ Y). The `Zoom' and `Pixel' options control the behavior of the
left mouse button on the images. For the former, clicking-and-dragging
creates a rubber-band box allowing you to zoom in to the images. The
latter allows you to click on individual pixels, and the three bottom
panels show cuts through this pixel in the $\lambda$, X and Y
directions.


\section{Slot data}\label{sect.slot}

The slot data (i.e., the 40\arcsec\ and 266\arcsec\ slits) are not
processed for residual warm pixels, nor are cosmic rays removed with
EIS\_PREP\_CLEAN\_CRS. Instead, the old routine EIS\_DESPIKE is used
but a new keyword /CLEAN\_EXP has been added to force the routine to
use CLEAN\_EXPOSURE.


\section{Removing residual warm pixels}

The criteria for flagging a warm pixel are applied to a 2D
image of wavelength $\times$ solar-Y (i.e., an EIS exposure). Any warm, hot,
dusty or saturated pixels previously identified by EIS\_PREP are set to
the missing value ($-100$). A comparison of the pixel's intensity with
the median of surrounding pixels is the basis of the method. A feature
of warm pixels is that they affect every exposure in the raster, and
so the routine looks for pixels that are consistently flagged in a
majority of the exposures. If one is found, then the pixel will be
flagged in all of the exposures, irrespective of whether the pixel in
an individual exposure was flagged.

The method is
complicated by the missing pixels, which potentially
introduce a bias in the median value that can affect whether a pixel
is flagged or not.

To illustrate the potential problems, consider the case where a pixel
lies in the wing of a strong line, say at the full-width at
half-maximum intensity level. The basic check is to compare the pixel
with the median of the surrounding 3x7 pixel block. Because of the
strong slope in the line intensity across the three wavelength pixels,
the median would essentially come from the central 1x7 block. However,
consider the case where there are, e.g., 5 missing pixels in the 1x7
block on the high intensity side. Potentially the median pixel would
then come from the low intensity 1x7 block, which is much weaker than
the central 1x7 block. The pixel then has a much higher intensity than
this median pixel and so it gets flagged, even though it may have a
normal intensity.

This specific problem is dealt with by, in addition to checking the
3x7 block, the routine also checks against the median of the central 1x7
block. A pixel is flagged if it satisfies both criteria. However,
there is then potentially a further problem. Suppose the intensity in
the Y-direction is monotonically increasing from bottom to
top. Further suppose that the top 3 pixels are all missing. The median
is obtained from the three low intensity pixels 
and there is a good chance that the central pixel is flagged by
accident. The routine checks for this by requiring there be at least
one pixel on both the top and bottom side of the pixel.

(It should be noted that if the median is computed from an even number
of elements, then IDL always returns the higher number rather than the
lower number. For example, for [1,2,3,4] the median is 3, not 2. This
helps reduce the number of spurious flagged pixels when there are
missing data.)

Three checks are done on RWPs based on the pixel intensity, $I$,
(given in data numbers, DN). A limit of $L$~DN separates
weak and strong pixels, and a further limit of 1000~DN separates
strong from very strong pixels.
%% Three categories of warm pixel are considered: (i) intensity $< L$~DN
%% (weak), (ii) intensity $\ge L$~DN (strong), and (iii) intensity $\ge 1000$~DN
%% (very strong).

The limit, $L$, is defined by first taking a histogram of the image with
values between $-30$ and 50~DN. We then set $MVL$ (\verb|max_var_low|
in the code) to be the DN value at which the histogram peaks. If this
value is less than 6, then it is set to 6 (this value is a heritage
value from EIS\_DESPIKE). $L$ is then set to $(MLV/1.5)^2$.

For further details we refer the reader to the IDL code, which is
extensively commented. The various parameters and checks were arrived
at by extensive study of a number of data-sets. If you use
EIS\_SPIKE\_VIEWER to look at dark areas of the detector then you will
find the routine usually flags the strongest RWPs, but there often
remain many very weak RWPs. Although the eye can pick these out fairly
easily they are typically only a few DN in strength, and trying to
flag them will often result in false positives.

RWPs occurring in line profiles are more
difficult to flag due to the risk of removing real data, and so less
are flagged compared to dark areas.
The routine can have problems when multiple warm or hot pixels are
found in adjacent pixels. If these are present in the WP/HP maps, then
they are flagged as missing and there may not be enough good pixels
remaining to identify any RWPs.



\section{Removing cosmic rays}

The procedure for CRs is similar to that for RWPs in that pixels are
identified based on comparisons with the median value of pixels in the
neighborhood of a single exposure. Note that EIS\_DESPIKE performed
checks in the exposure dimension (time) in addition to the 2D detector
image. This method is generally more effective for accurately
identifying CRs, but was not used here as there are many EIS data-sets
for which the raster step size is larger than the slit width
(so-called ``picket-fence'' rasters). There is then the risk of
false-positive identifications due to large, real intensity variations
between slit positions with a large spatial separation.

The checks performed for flagging CRs are performed in
EIS\_CR\_CHECKER, and there are five checks in all. One is performed
on weak intensity pixels, the rest on strong intensity pixels. The
value of the pixel's 3x7 median is used to determine if the pixel lies
within an emission line. If yes, then the routine also checks the 1x7
median as described in the RWP section above. 

The locations of the weak and strong intensity CRs are stored
separately by EIS\_CR\_CHECKER. EIS\_PREP\_CLEAN\_CRS then flags
neighbors of the strong CRs, but not for the weak CRs (which generally
do not extend into neighboring pixels). EIS\_PREP\_CLEAN\_CRS then
performs another pass over the data using  EIS\_CR\_CHECKER, which
picks up residual CRs from the large events. 





\section{Known issues}

\begin{itemize}
\item Warm pixels and cosmic rays occurring in line profiles are more
  difficult to flag due to the risk of removing real data.
\item Warm and hot pixels can be found in adjacent pixels, sometimes
  3, 4 or 5 together. If some of these are removed using the WP and HP
  maps, leaving missing data, then the routine will often fail to
  remove a RWP simply because there are no good data nearby.
\item Using the spike viewer software, many very weak warm pixels
  can be seen in dark areas with the eye. Generally these are not
  flagged because the signal is too weak.
\end{itemize}

\begin{figure}[t]
\centerline{\epsfxsize=5.5in\epsfbox{plot_noresidual_example.eps}}
\caption{The same data as shown in Figure~\ref{fig.residual} are
  displayed, but after processing with the new version of EIS\_PREP.}
\label{fig.noresidual}
\end{figure}

\section{Results}\label{sect.results}

In Sect.~\ref{sect.intro}, data from 2016 was shown that illustrated
the problems with residual warm pixels and cosmic ray
removal. Figure~\ref{fig.noresidual} shows the same plots as
Figure~\ref{fig.residual} but after applying the new version of
EIS\_PREP. It can be seen that the strong WP in the left panel
is no longer present, and also several of the stronger WPs in the
right panel have been removed.

Figure~\ref{fig.despike} showed that the EIS\_DESPIKE routine
originally used by EIS\_PREP flagged many low intensity pixels as
CRs. With the new version, the region shown in
Figure~\ref{fig.despike} has no pixels flagged as CRs.

\begin{figure}[h]
\centerline{\epsfxsize=\textwidth\epsfbox{20191125_image_comparison.eps}}
\caption{Images in three emission lines from the 25-Nov-2019
  12:13~UT dataset. The top row shows results from the old EIS\_PREP
  data and the bottom row shows results from the new EIS\_PREP
  data.}
\label{fig.20191125}
\end{figure}


Figure~\ref{fig.20191125} compares images formed in three emission
lines (\ion{Fe}{xii} \lam195.12, \ion{Fe}{ix} \lam197.86 and
\ion{Si}{vii} \lam275.36) using the previous version of ES\_PREP, and
the new version. The data-set is from 2019 November 25 at 12:13~UT,
and the images were formed by averaging the seven
wavelength pixels centered on the lines' rest wavelengths. Only a
fraction of the Y field-of-view is shown. The data
were processed using the /refill option, and any remaining missing
pixels were filled using EIS\_PREP\_MED\_FILL. The lines were chosen
as they are examples of strong, medium and weak strength lines. The
\lam195.12 images are almost identical. The \lam197.86 images show one
quite strong WP that is mostly removed with the new method,
and other weaker WPs that have been removed (e.g., the two around
Y=180--185). The \lam275.36 images show one strong WP at Y=170 that is
mostly removed, and a number of cosmic rays around Y=230 (these are
probably remnants of a WP that were partially removed with the
previous method.

The \lam275.36 image shows that many weak warm pixels remain in the
data with the new method. The number is exaggerated in this plot since
the data are averaged over seven images. Potentially more of these WPs
could be removed by adjusting the detection threshold. 

Figure~\ref{fig.20191125-gauss} compares Gaussian fits to \ion{Fe}{ix}
\lam197.86, with the top row again showing the results from the old
version of EIS\_PREP and the bottom row showing the new version. The
fits were performed with EIS\_AUTO\_FIT, and the left column shows
integrated intensity, and the right column shows Doppler shifts. It can be seen
that the WP highlighted in Figure~\ref{fig.20191125} makes an effect
on the Gaussian fits. Otherwise the intensity and velocity images are
very similar, with slightly less artifacts in the velocity image.

\begin{figure}[h]
\centerline{\epsfxsize=\textwidth\epsfbox{20191125_fe9_fits.eps}}
\caption{Fit results for \ion{Fe}{ix} \lam197.86 from the 25-Nov-2019
  12:13~UT dataset. The top row shows results from the old EIS\_PREP
  data and the bottom row shows results from the new EIS\_PREP
  data. The left column shows integrated intensity, and the right
  column shows line-of-sight velocity scaled between
  $\pm$~20~km~s$^{-1}$.}
\label{fig.20191125-gauss}
\end{figure}

\newpage

\appendix

\section{Document history}

\noindent\emph{Ver.~1.2, 17-Dec-2019}. Added Sect.~\ref{sect.slot} to mention
slot data.

\noindent\emph{Ver.~1.1, 11-Dec-2019.} Added further examples to
Sect.~\ref{sect.results}. Added Sect.~\ref{sect.call}, and renamed Sect.~\ref{sect.check}.

\noindent\emph{Ver.~1.0, 5-Dec-2019.} First release.

\end{document}
