\documentclass[11pt]{article}

\usepackage{epsf}
\usepackage{natbib}
\usepackage{aas_macros}
\usepackage{hyperref}

\newcommand{\lam}{$\lambda$}
\newcommand{\kms}{km~s$^{-1}$}
\newcommand{\cm}{cm$^{-3}$}
\newcommand{\ion}[2]{#1\,{\sc #2}}
\newcommand{\ecssa}{erg cm$^{-2}$ s$^{-1}$ sr$^{-1}$ \AA$^{-1}$} % erg/cm2/s/angstrom
\newcommand{\ecss}{erg cm$^{-2}$ s$^{-1}$ sr$^{-1}$} % erg/cm2/s/sr

\setlength{\oddsidemargin}{0cm}
\setlength{\evensidemargin}{0cm}
\setlength{\topmargin}{0cm}
\setlength{\textwidth}{16cm}
\setlength{\textheight}{23cm}
\setlength{\parindent}{0cm}
\setlength{\parskip}{0.1cm}
 
\newcommand{\eisprep}{\textsc{EIS\_PREP}}
\newcommand{\arcsec}{$^{\prime\prime}$}

\newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{
        \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}}

\begin{document}                

\begin{titlepage}
\hrule
\noindent\textbf{EUV IMAGING SPECTROMETER}\\
\vspace{-0.7\baselineskip}\\
{\noindent\Huge\bf Hinode}
\vspace{2mm}
\hrule
\vspace{3mm}
\centerline{\bf EIS SOFTWARE NOTE No. 1}
\vspace{3mm}
\hrule
\noindent Version 3.8 \hfill 26 Oct 2022
\vspace{2mm}
\hrule

\begin{centerpage}

\begin{center}
{\Large\bf Calibrating EIS data: the EIS\_PREP routine}\\
\mbox{}\\
\mbox{}\\
Peter Young\\
NASA Goddard Space Flight Center \\ 
Code 671 \\
Greenbelt, MD 20771, USA
\end{center}

\end{centerpage}

\end{titlepage}

\section{Calling \eisprep}\label{sect.call}

\eisprep\ takes as input the name of a level-0 EIS FITS file and
creates a level-1 data-set with the data arrays containing calibrated
intensities. The standard calling sequence, and the one recommended
for beginners, is:

\begin{verbatim}
IDL> eis_prep, l0name, /default, /quiet, /save, /retain
\end{verbatim}

The five keywords do the following:

\begin{description}
\item[/default] Tells \eisprep\ to use the standard, simple dark
  current subtraction method rather than use dark frames stored in
  Solarsoft. 
\item[/quiet] Stops \eisprep\ from popping up widget information
  messages during operation.
\item[/save] Tells \eisprep\ to save the level-1 data into a FITS file
  in the current working directory, and also save the error arrays
  into a FITS file. These FITS files have the same name as the level-0
  FITS file, except `l0' is replaced with `l1' and `er' for the
  intensity and error arrays, respectively.
\item[/retain] Retains pixels with DN values with negative or zero
  values -- see Sect.~\ref{sect.dc} for more details.
%% \item[/correct\_sensitivity] Applies a correction to the output
%%  intensities due to the decreasing EIS sensitivity with time. 
\end{description}

\emph{Please read Sect.~\ref{sect.radcal} for information on how to
implement the latest radiometric calibration.}

The structure of the level-1 FITS file is identical to the level-0
FITS file except that the arrays that originally contained data
numbers are modified to contain calibrated intensities. In addition,
\eisprep\ also computes the 1$\sigma$ error on each intensity value
and the arrays for these are stored in a separate FITS file.

A complete set of keywords for \eisprep\ is given in Appendix~\ref{sect.keywords}


\section{Missing data and eis\_prep}

An important issue that users of \eisprep\ need to be aware of is the
concept of `missing data'. For any EIS data-set there will be pixels
within the data arrays that will not be usable for science
purposes. They arise from a variety of causes, including saturated
data, detector artifacts (dust, warm pixels, hot pixels), cosmic ray
hits and missing data packets. One of the key jobs of \eisprep\ is to
flag these pixels for the user. All of the flagged pixels are referred
to as `missing data'.

After \eisprep\ has completed its processing, each pixel in the data
arrays will have an intensity value and an error value. The
intensities and errors and stored in two separate files (see the next
section). All missing data are given a value of -100 in the error file
\emph{only}. The intensities of missing pixels are interpolated from
neighboring pixels. At first glance, then, users may believe that
missing pixels when viewed in the intensity arrays produced by
\eisprep\ are normal and so suitable for science analysis. Since they
do not have realistic errors assigned to them, however, then this is
not the case.

Some routines in the EIS software tree (for example eis\_auto\_fit)
automatically check for pixels with errors set to -100 and will not
include them in the processing.

An alternative interpolation method (described in EIS Software
Note \#13) for EIS data is available that
assigns a realistic error bar to missing data so that the pixels can
be used in science analysis.  The method is 
not the default option in \eisprep\ as it erases the `memory' of where
the missing pixels are (since they are no longer flagged with -100
values), but it can be initiated by using the keyword /refill. The
interpolation method can also implemented as a post-\eisprep\ step by
using the routine eis\_getwindata. For example:
%
\begin{verbatim}
windata=eis_get_windata(l1name, 195.12, /refill)
\end{verbatim}
%
where l1name is the name of the level-1 FITS file and 195.12 is the
selected wavelength.

\section{What does \eisprep\ do?}\label{sect.whatdo}

The eis\_prep routine in Solarsoft takes the EIS level-0 FITS files and
produces as output a level-1 FITS file. In addition to converting the
measured CCD signal into calibrated intensity units, a key part of
eis\_prep is to flag bad data points. These can arise through pixel 
saturation, cosmic ray hits, or simply defective pixels on the
CCD. The bad points are referred to as missing pixels. The central
outputs of EIS PREP are two level-1 FITS files, one containing
calibrated intensities at each pixel, and the other containing error
bars on these intensities. Note that the missing pixels are assigned a
value of -100 in the error file, not the intensity file. The bad
points in the intensity file are replaced with the median of the
neighbouring pixels. 

The sequential steps performed by \eisprep\ are as follows.

UPDATE: on 2019 December 4 an update to \eisprep\ was performed that
modified the order of the steps below, introducing a new step for
removing residual warm pixels, and modified the cosmic ray removal
method. Please Check EIS Software Note No.~24 for further details. The
sections below describe the \eisprep\ steps \emph{prior} to the
change. The previous version of \eisprep\ can be run by giving the
/original keyword.

\subsection{Step 1: flagging saturated data, 0~DN and 2048~DN values}\label{sect.sat}

There are three intensity values that need to be flagged as missing. The EIS CCDs
have a 14 bit dynamic range and so saturation occurs at 16,383 data
numbers (DN). Missing data packets can result in values of 0~DN, which
are not real because normal pixels will always have a pedestal DN
value of around 500~DN even if there are no counts. A further bad
pixel value is 2048~DN. It is not known how this value arises but,
when seen, an entire block of data will be set to this value. Since
2048~DN is potentially a real data value, then \eisprep\ will only
flag this value if the entire Y-column on the detector takes the value.



\subsection{Step 2: dark current and pedestal subtraction}\label{sect.dc}

In the raw data, the spectra are found to sit on a background of
around 500 DN that arises principally from the CCD bias (or pedestal),
and secondly from the CCD dark current. Before launch it was expected
that the bias and dark current could be subtracted together from the
science data by using a
dark frame (i.e., an exposure with the shutter closed), however tests
showed that this was not successful mainly because the dark current is
found to vary through the satellite's orbit (due to the CCD
temperature varying). The most commonly-used method for estimating the
bias and dark current is to use the science exposures themselves and
this is enabled by the /default keyword.
If /default is not specified, then the user must choose a dark frame
either through the dc\_file= keyword, or via a widget selection window.

For EIS data obtained with the 1\arcsec\ and 2\arcsec\ slits, there
are 
two methods for estimating the bias and dark current 
that are employed in \eisprep, and the one used depends on the size of the wavelength
windows used for the observation. Most rasters take narrow wavelength
windows of size 16--32 pixels centered on specific emission lines. For
these windows, each exposure is extracted and the 2\%\ of the detector
pixels that have the lowest DN values are isolated. The median DN
value of these 2\% pixels is then set to be the background level and
it is subtracted from the DN values of each pixel. 

The maximum size of an EIS wavelength window is 1024 pixels and in
this case 
a different method is applied. The reason for this lies in the
properties of the EIS CCDs. EIS has two CCDs (for the two wavelength
channels) and each has a  width of 
2048 pixels. The electronics method of reading the CCDs is such that
each CCD can be
effectively treated
as if it is two CCDs of 1024 pixels stuck together. This means that
the maximum size of a wavelength window is 1024 pixels, and in this
case the window occupies the complete width of the ``effective
CCD'' (one half of the actual CCD). Therefore if a window of 1024 pixels is selected to
observe one half of the EIS SW channel, then it will always observe
the complete spectrum in this half of the channel -- its position can
not be adjusted. For this reason,
the method for determining the pedestal in this case is to identify
a narrow wavelength region within the window that is free of emission lines, and then take the
median intensity value in this region. This value is then the
pedestal value for the wavelength window. 

Table~\ref{tbl.pedestal} shows the spectral regions that are used to
define the pedestal value for the four sets of 1024 wide pixel windows
that are currently implemented in \eisprep\ (as of October 2012). Note
that the 1\arcsec\ and 2\arcsec\ spectra are slightly offset from each
other on the EIS detector, hence the fixed pixel ranges correspond to
slightly different wavelength ranges for the two slits.

\begin{table}[h]
\caption{Spectral regions used to derive CCD pedestal values for 1024
  pixel windows.}
\begin{center}
\begin{tabular}{cccc}
\noalign{\hrule}
\noalign{\smallskip}
\noalign{\hrule}
\noalign{\smallskip}
%% table headings here
&&\multicolumn{2}{c}{Wavelengths / \AA} \\
\cline{3-4}
\noalign{\smallskip}
Channel & Pixels & 1\arcsec & 2\arcsec \\
\noalign{\hrule}
\noalign{\smallskip}
%% table entries here
SW 1  & 39--84 & 167.10--168.10 & 167.27--168.28 \\
SW 2  & 944--989 & 210.10--211.10 & 210.27--211.28 \\
LW 1  & 39--84 & 246.59--247.59 & 246.77--247.77 \\
LW 2  & 926--971 & 289.11--290.11 & 289.29--290.29 \\
\noalign{\hrule}
\end{tabular}
\end{center}
\label{tbl.pedestal}
\end{table}


Generally, when users select a window of 1024 pixels they will do so
for each of the four ``effective CCDs'', and such data-sets are
usually referred to as full CCD data-sets. However, it is possible for
users to combine a 1024 pixel window with narrower windows. In this
case a potential anomaly can occur whereby the 1024 pixel window may
appear to have a significantly different continuum level compared to
the other windows after running \eisprep\ -- this is because of the
different methods of estimating the pedestal.


% For full CCD spectra a different approach is used. Sections of the CCD have been identified where there are no (or very weak) emission lines and so it can be assumed that the background here represents a measure of the CCD background. The regions are then averaged to yield the background level which is subtracted from the spectra.

It is to be noted that both of the pedestal removal methods will yield
some pixels with negative DN values. For window data it will only be
1\% of pixels because of the method used, but for full CCD spectra it
can be up to 50\% of pixels if the raster contains very weak emission
(e.g., coronal holes or off-limb spectra). The default mode for
\eisprep\ sets pixels with zero or negative DN values to be missing
data. This is because it is not possible to assign a photon noise
error to the data points. (Note that negative DN values are expected
since, if there are zero solar counts, then the uncertainty in the CCD
counts will be 0 $\pm$ read noise.) 

By setting the /RETAIN keyword in \eisprep, pixels with zero or negative DN values will not be flagged as missing. The errors for the pixels will be set to the dark current error estimate (see Step 6).

For EIS slot data (obtained with the 40\arcsec\ and 266\arcsec\
slits), fixed values for the pedestal and dark current are set at 549,
500, 556 and 547~DN for the four CCD sectors identified in Table~\ref{tbl.pedestal}.



\subsection{Step 3: cosmic ray removal}

Anomalously bright pixels are found on the EIS CCD images that arise
from ‘hot pixels’, ‘warm pixels’ and cosmic rays. The cosmic ray
removal is performed by EIS DESPIKE, a wrapper routine that calls NEW
SPIKE, a routine developed for removing cosmic rays from SOHO/Coronal
Diagnostic Spectrometer (CDS) data-sets \citep{cds_sw46,2000A&A...362L..21P}. For
CDS data processing it was typical for not only 
the identified CCD pixels to be flagged, but also the
nearest-neighbour pixels on the CCD. This is because there is often
residual signal from the cosmic ray next to the brightest pixels. EIS
sees significantly less cosmic rays than CDS apart from during the
approximately 5 minute passes through the South Atlantic Anomaly. As
the most useful function of EIS DESPIKE was actually to flag warm
pixels, and since warm pixels are only single pixel events, then the
nearest-neighbour option is usually switched off for EIS. It is to be
noted that the NEW SPIKE routine was designed to be cautious when
removing cosmic rays from line profiles, thus possibly artificially
enhancing the emission line intensities at these locations. 

\subsection{Step 4: flagging hot and warm pixels}\label{sect.hotwarm}

Both hot pixels and warm pixels are single pixels that have
anomalously high DN values. A hot pixel is defined to be one that
yields 25,000 electrons pixel-1 s-1 at room temperature (a
specification from the CCD manufacturer). Pixels that fall below this
threshold but are still clearly identified as being anomalous when
inspecting the data are referred to as ‘warm’ pixels. Separate maps of
the locations of hot and warm pixels are generated by the EIS team
approximately every week following inspection of 100 s dark exposures and they
are stored in Solarsoft. The pixel maps that are closest in time to
the science observation are used by EIS\_PREP to mark the hot and warm
pixels as missing data.

From 4 October 2022 onwards, the EIS team are now using 150~s dark exposures rather than 100~s exposures, as these enable 10\%\ more warm pixels to be identified. The 100~s exposures are continued but on a monthly basis in order to maintain continuity with the original dataset.

Warm pixels have a significant impact on the analysis of EIS data and
users are recommended to read EIS Software Note \#6, `Warm and hot
pixels on the EIS CCDs', and EIS Software Note \#13, `Interpolating
missing pixels in EIS data'.


\subsection{Step 5: flagging dusty pixels}

The next step for EIS PREP is to flag the pixels affected by dust on
the CCD. Several small pieces of dust accumulated on the CCD before
launch and are found to completely block the solar signal on the CCD
at their locations. They are fixed in position and cover less than
0.1\% of the CCD, however two of the pieces do affect the strong lines
Fe XI 188.23, 188.30 and Fe XII 193.51 such that the lines can not be
used over 15--30" spatial ranges in solar-Y.  

\subsection{Step 6: radiometric calibration}\label{sect.radcal}

The final step of EIS PREP is to convert DN values into intensities in
units \ecssa. The errors on the intensities are
computed assuming photon statistics together with an error estimate
for the dark current. This varies from 2.24 to 2.37 DN for the four
CCD quadrants. 

To return units of photons~cm$^{-2}$~s$^{-1}$~sr$^{-1}$, the keyword \textbf{/photons} should be set. To not perform radiometric calibration at all, then the keyword \textbf{/noabs} should be used. This leaves the intensity in DN units.  \citet{2016ApJ...820...63B} found that leaving the intensities in DN units results in slightly narrower widths when fitting emission lines with Gaussians. Please check Appendix~\ref{sect.noabs} for further information about the \textbf{/noabs} keyword. The intensity units can be checked by using the \textsf{getunit} object method.

The call recommended in Sect.~\ref{sect.call} results in the original
laboratory calibration to be applied to the data. However, it is known
that the calibration has changed since launch and two alternative
prescriptions were presented by \citet{2013A&A...555A..47D} and
\citet{2014ApJS..213...11W}. \emph{It is generally recommended to apply one of
  these corrections during the data analysis rather than at the
  eis\_prep stage.}

One example is when using the eis\_auto\_fit suite of software (EIS
Software Note \#16). After fitting the lines, the intensity array is
generated with:
%
\begin{verbatim}
IDL> int=eis_get_fitdata(fitdata,/int,calib=calib)
\end{verbatim}
%
where calib=1 implements the \citet{2013A&A...555A..47D} calibration, and calib=3
implements the \citet{2014ApJS..213...11W} calibration. The EIS Wiki page at
%
\begin{verbatim}
  http://solarb.mssl.ucl.ac.uk/eiswiki/Wiki.jsp?page=EISCalibration
\end{verbatim}
%
has further information on implementing the calibration.


Prior to the \citet{2013A&A...555A..47D} and
\citet{2014ApJS..213...11W} work, users were recommended to use the
/correct\_sensitivity keyword when calling eis\_prep. Please see
Appendix~\ref{sect.corrsens} for more information on this keyword.



\section{Calculating intensities and errors}

The level-0 FITS file contains intensities stored as `data numbers',
or DN, and the main job of \eisprep\ is to convert these to calibrated
intensities with associated 1$\sigma$ errors. The steps in this
process are as follows.

The DN value, $D$, for a given pixel consists of three components:

\begin{equation}
D=D_{\rm ped}+D_{\rm dc}+D_{\rm photon}
\end{equation}

where $D_{\rm ped}$ is the CCD pedestal, $D_{\rm dc}$ is due to the
dark current, and $D_{\rm photon}$ is the signal from the incident
photons. An average value of  $D_{\rm ped}+D_{\rm dc}$ is estimated
from the data themselves through one of the methods described in
Sect.~\ref{sect.dc}. We refer to this value as $D_{\rm sub}$.

The number of photons in any pixel on the detector is then given by:

\begin{equation}\label{p_defn.eq}
P = (D-D_{\rm sub}) g \left( {\lambda\over {\rm \AA}}\right) {3.65\over 12398.5}
\end{equation}

where $g$ is the CCD gain, 3.65 eV is the energy to produce an
electron-hole pair in silicon, and 12398.5 is the conversion factor
for eV into \AA.

The photon statistics noise is given by $\sqrt P$, however the error
on the dark current estimate must be added in quadrature to this:

\begin{equation}
\sigma_P= \sqrt{ P + \sigma_{\rm dc}^2}
\end{equation}

where $\sigma_{\rm dc}$ is the dark current uncertainty. The latter has
been estimated directly from EIS dark frames and takes a value
2.24--2.37~DN (as of 2009 January 5), depending on the CCD
quadrant. The value $\sigma_{\rm dc}$ is then derived from these values
using Eq.~\ref{p_defn.eq}.

To convert from photons to \ecssa\ requires a number of factors
including exposure time, telescope effective area and slit size. Note
that effective area itself consists of a number of factors including
filter transmissions, coating reflectivities, telescope size and CCD
quantum efficiency. The effective area for a given wavelength can be
obtained with the IDL routine EIS\_EFF\_AREA(). Writing these factors
as simply $f(\lambda)$, the calibrated intensities are
$I=f(\lambda)P$. The 1$\sigma$ error on $I$ is then given by

\begin{equation}
{\sigma_I \over I} = {\sigma_P \over P}
\end{equation}

or

\begin{equation}
\sigma_I=f(\lambda) \sigma_P
\end{equation}



% A pixel with associated wavelength, \lam, registers a data number
% value of $d_{\lambda}$. The data number value is converted to a photon
% value by the expression

% \begin{equation}
% p_{\lambda}=k\,d_{\lambda}\lambda
% \end{equation}

% where $k$ is a constant

\bibliographystyle{aasjournal}
\bibliography{eis_swnote_01}  %$




\newpage
\appendix

\section{Full set of \eisprep\ keywords}\label{sect.keywords}

\begin{description}
\item[noabs] Do not perform absolute calibration, i.e., the intensity
  arrays are left in data number units.
\item[photons] Instead of \ecssa\ the output intensities are given in
  units of photons~cm$^{-2}$~s$^{-1}$~sr$^{-1}$~\AA$^{-1}$.
\item[refill] This uses the method described in Software Note \#13 to
  replace missing pixels with a 'sensible' intensity value. Generally
  the user is recommended not to use this keyword with \eisprep\ but
  use it instead with one of the post-processing routines such as \textsc{EIS\_GETWINDATA}.
\item[nocr] Switch off cosmic ray detection routine.
\item[nohp] Switch off hot pixel detection routine.
\item[nowp] Switch off warm pixel detection routine.
\item[nodp] Switch off dusty pixel detection routine.
\end{description}

\section{Comment on /noabs keyword}\label{sect.noabs}

If the /noabs keyword is set, then the ``intensity'' arrays returned
by \eisprep\ have DN units. Technically a 1-sigma error can be
assigned to these values by computing photon statistic errors and
converting back to DN values, but \eisprep\ does not do this. However,
\eisprep\ \emph{does} produce an error FITS file when /noabs is
set. The error values in this FITS file are not correct and should not
be used, however the error arrays do contain the positions of missing
pixels by flagging them with -100.  This is important when using the
/refill keyword (either for \eisprep\ or \verb|eis_getwindata|).

The /refill keyword interpolates missing values, and this works
correctly if /noabs is set. However, the interpolated error values
should again not be used.

\section{The /correct\_sensitivity keyword}\label{sect.corrsens}

This is no longer recommended to be used, but we give details on it here for
archive purposes.

EIS Software
Note \#2 described changes in the EIS sensitivity with time and this
was implemented with the /correct\_sensitivity keyword for eis\_prep.


The sensitivity decay was modeled as an exponential decay of the form
$\exp(-t/\tau)$ where $t$ is the time of the observation in days since
the Hinode launch (2006 September 22 21:36~UT), and $\tau$ is the
decay time in days.  The  value of $\tau$ can be obtained by
doing:
%
\begin{verbatim}
IDL> cal=obj_new('eis_cal')
IDL> print,cal->gettau_sensitivity()
\end{verbatim}
%
Note that the sensitivity decay is  modeled as being the same
for all wavelengths. The crucial advances of the \citet{2013A&A...555A..47D} and
\citet{2014ApJS..213...11W} implementations were to derive wavelength-dependent
changes with time.




\section{Document modification history}

\emph{version 3.8}: Updated Sect.~\ref{sect.hotwarm} to add information about new 150~s dark exposures; updated Sect.~\ref{sect.radcal} to mention /photons and /noabs keywords.

\emph{version 3.7}: Updated Sect.~\ref{sect.sat} to mention 0 and
2048~DN values; updated Sect.~\ref{sect.whatdo} to mentioned EIS Note No.~24.

\emph{version 3.6}: Updated recommendation for radiometric calibration
(Sects.~\ref{sect.call}, \ref{sect.radcal} and App.~\ref{sect.corrsens}).

\emph{version 3.5}: Added Sect.~\ref{sect.noabs}.

\emph{version 3.4}: Sect.~\ref{sect.dc} has been expanded.

\emph{version 3.3}: The /correct\_sensitivity keyword has been
restored as the implementation is now working correctly.

\emph{Version 3.2}: Changed document title. Removed /correct\_sensitivity from the recommended
eis\_prep calling sequence (Sect.~\ref{sect.call}) as there is a bug in the current
implementation. Also changed l1name to l0name in the IDL call line.



\end{document}
