PRO xrt_id_ecids, $ t_strt, $ t_stop, $ ecid_list, $ ecid_im_codes, $ archive_prefix =archive_prefix, $ quicklook =quicklook, $ level0 =level0, $ fits_index =index_full, $ ss_keep =ss_keep, $ ss_outliers =ss_outliers, $ qabort =qabort, $ qstop =qstop, $ quiet =quiet ;================================================================== ;+ ; ; PROJECT: ; Solar-B / Hinode / XRT ; ; NAME: ; XRT_ID_ECIDS ; ; CATEGORY: ; ; XRT Data Verification ; ; PURPOSE: ; ; Given a pair of times (t0,t1) during a single XOB run, identify all ; EC_ID image numbers between (t0,t1), and determine (a) whether ; they are present or missing in the data archive; and (b) which ; of the following image types they are: Patrol, Regular (i.e., ; was sent to the main image buffer), Preflare, or Indeterminable. ; Uses the dr_XRT*.sim file to understand expected image looping. ; ; CALLING SEQUENCE: ; ; XRT_ID_ECIDS, T_STRT, T_STOP, ECID_LIST, ECID_IM_CODES, $ ; [,ARCHIVE_PREFIX=ARCHIVE_PREFIX] [,/QUICKLOOK] $ ; [,/LEVEL0] [,FITS_INDEX=FITS_INDEX] $ ; [,QABORT=QABORT] [,/QSTOP] ; ; INPUTS: ; ; T_STRT, T_STOP - A pair of timestamps that define the range of ; EC_IDs to be examined. Date and time may be ; in any format that is recognized by ANYTIM.PRO . ; ; ARCHIVE_PREFIX - [OPTIONAL] A string that specifies the directory ; location of the local XRT archive. E.g., at SAO, ; archive_prefix='/archive/hinode/xrt/'. ; (Default = '/archive/hinode/xrt/'.) ; ; /QUICKLOOK, /LEVEL0 - [OPTIONAL] Keyword switches to indicate ; which data archive should be examined to ; determine "missing" data. ; (Default = /LEVEL0 .) ; ; /QSTOP - [OPTIONAL] Tell program to stop before exiting. ; Just in here for debugging. ; ; /QUIET - [OPTIONAL] shut up output ; ; ; OUTPUTS: ; ; ECID_LIST - 1D list of every EC_ID *possible* between ; (t0,t1). Starting and finishing values are ; taken from data present in the data archive, ; but *every* contiguous EC_ID value in the middle ; is included, whether or not the corresponding ; image is actually present in the data archive. ; Values wrap around from 32767 to 0. ; ; ECID_IM_CODES - 1D string array matched to ECID_LIST. Each ; element is a code that indicates whether there ; is data for that EC_ID, and which image buffer ; it went to (and whether it is a Patrol image). ; ; EC_ID Image Code Format = AAAA/BBBC ; ----------------------------------- ; AAAA = 'PRST', image is PReSenT in the data archive ; = 'MISS', image is MISSing from the data archive ; ; BBB = 'Reg', image went to the main image buffer AND ; is NOT a Patrol image ; = 'Pat', image went to the main image buffer AND ; it IS a Patrol image ; = 'PFn', image went to Preflare buffer "n" ; = 'Flr', image was taken in flare mode. ; = '???', image type could not be determined ; (by THIS program) ; ; C = '.', this EC_ID is definitely matched to the ; indicated image type (BBB). ; = '?', for a series of missing images, the ; indicated image type (BBB) is somewhere ; in the gap, but may or may not actually ; belong to this EC_ID. In other words, ; this entry is a placeholder to facilitate ; the counting of the missing image types. ; (Also see Note #10.) ; ; FITS_INDEX - Returns the FITS header for the data in (t0,t1). ; However, the indices are sorted to be in EC_ID ; order rather than in time order. Furthermore, ; the length of FITS_INDEX may be longer than ; that of ECID_LIST and ECID_IM_CODES. Instead, ; match them with FITS_INDEX[SS_KEEP]. The remainders ; are identified as FITS_INDEX[SS_OUTLIERS], and ; are not included in the processing. See Note #11. ; ; SS_KEEP - These are addresses of FITS_INDEX that correspond ; to the ECID_LIST and ECID_IM_CODES outputs, ; i.e., compare them to FITS_INDEX[SS_KEEP]. ; See Note #11. ; ; SS_OUTLIERS - These are addresses of FITS_INDEX that were ; determined to have incorrect timestamps and ; are not part of the EC_ID sequence being ; resolved here. They are identified by their ; EC_ID number not occuring between the initial ; and final EC_ID values. See Note #11. ; ; ; I/O KEYWORDS: ; ; QABORT - [OPTIONAL] (As Input) A nonzero value tells the ; program to exit gracefully before doing anything. ; (As Output) Indicates the status of the program exit: ; 0: Ran to completion, no errors. ; 1: Did not run to completion, forced to exit gracefully ; by input value of QABORT. ; 2: Did not run to completion, forced to exit gracefully. ; No FITS files found between T_STRT and T_STOP. ; 3: Did not run to completion, forced to exit gracefully. ; Input timerange spans more than one XOB. See Note #12. ; ; EXAMPLES: ; ; This simple example refers to the Level-0 data archive. ; ; IDL> t_strt = '2010-sep-6 14:30' ; IDL> t_stop = '2010-sep-6 15:30' ; IDL> xrt_id_ecids, t_strt, t_stop, /L0, ecid_list, ecid_codes ; IDL> help, ecid_list, ecid_codes ; ECID_LIST LONG = Array[228] ; ECID_CODES STRING = Array[228] ; ; Here's a piece of the output. The codes are explained under ; the description of ECID_IM_CODES. ; ; IDL> for ii = 0,10 do print, ecid_list[ii], ' ', ecid_codes[ii] ; 18944 PRST/Pat. ; 18945 PRST/Pat. ; 18946 PRST/Pat. ; 18947 PRST/Pat. ; 18948 PRST/PF1. ; 18949 PRST/PF2. ; 18950 MISS/PF2. ; 18951 MISS/PF3. ; 18952 PRST/Pat. ; 18953 MISS/PF3. ; 18954 MISS/Reg. ; ; METHOD: ; ; This program attempts to identify image data as either missing ; or present in the local data archive, and further to determine the ; image type as {Patrol, image to main buffer, image to a Preflare ; buffer, or indeterminable}. The input times are used to extract ; the relevant FITS headers for the data that is present. The ; dr_XRT*.sim file is used to build an internal rendering of the ; XOB's table loop structure. For image EC_IDs that are not ; in the data archive (i.e., are missing), this program tries to ; interpolate the missing image types by looking at how the present ; data skip through the looping tables. When more EC_IDs are missing ; than table steps, the remainder are assumed to be Patrol images. ; The dr_XRT*.sim file has the information to associate the image types ; to particular positions in the XOB's table loop structure. Patrol ; images are identifiable because, in their FITS headers, all of ; the table positions and counters are reported as zero. ; ; See the NOTES for more information and caveats. ; ; ; NOTES: ; ; 1) The data represent the outcome of a complicated operating ; environment, including OP/OG commands through COT, other OP/OG ; commands, real-time commands, safehold responses, and other ; complications. This program only refers to the dr_XRT*.sim file ; created by the COT in order to understand missing images, ; and in that sense is an imperfect reflection of the expected data ; flow. The human operator MUST use their own judgement and analysis ; to interpret the outputs of this program. ; ; 2) This program assumes that there is a local copy of the XRT ; archive, including the "timelines/" subtree, and including one or ; more of the Level-0 and Quicklook data subtrees. ; ; 3) T_STRT and T_STOP *MUST* lie within the same XOB run, ; as understood by the dr_XRT*.sim files. For example, stopping ; and starting a single XOB will be treated as the beginning of a new ; XOB run. This is because an XOB run in the dr_XRT*.sim files is ; understood as the period from one XOB start to the next, regardless ; of the content of the two XOBs. ; ; 4) It is possible for one or more timelines to overwrite the ; XOB definition for a given time period. This program will use the ; most up-to-date timeline (ie, dr_XRT*.sim file) for the ; BEGINNING of the XOB that spans the input timerange. (But see ; Note #5.) ; ; 5) It is possible for an XOB to continue running beyond the end of ; the first dr_XRT*.sim file containing the XOB run. However, this ; program only uses the first file to understand the table loop ; positions. Therefore, it is possible that the first dr_XRT*.sim ; file will not "see" all of the XOB's table loops, and therefore ; this program would not be able to interpret later frames. The ; 2010-Oct-01 version of this program does not gracefully deal with ; this situation. (Please send a BUG REPORT if you encounter a ; definite dataset that invokes this failure.) (See related Note #4.) ; ; 6) "Regular" buffer and Preflare buffer images are recognized by ; the association of this information with table loop positions, ; as given in the dr_XRT*.sim file. Patrol images are identified ; either (a) by the FITS metadata for image data, where the ; table loop positions are all zero; or (b) when there are more missing ; EC_IDs in a sequence than can be explained by missing table ; loop steps. Note that, for the latter, there is no check anywhere ; that the XOB actually uses Patrol images. (See Note #8.) ; ; 7) When there are more missing table loop steps in a gap than ; missing EC_IDs, or the table loop positions otherwise skip or reset ; in an non-understandable way, then all of the missing images ; in the gap are listed as "indeterminable" (i.e., as code ; code = "MISS/????"). (See Note #7.) ; ; 8) In order for this program to evaluate a gap, or sequence of ; missing images, against the XOB table loop positions, there must ; be "PRST/Reg." or "PRST/PFn." images on either side of the ; gap, against which the loop counting can be anchored. In the case ; that a gap is not bounded by these image types, the entire gap is ; recorded as "indeterminable" (i.e., as code = "MISS/????"). This tends ; to occur at either end of the input timerange (t0,t1). It will also ; occur when the entire timerange (t0,t1) only includes zero or one ; image of the anchoring types. ; ; 9) In order for this program to establish what the full span ; of EC_IDs are, it must look at the first and last FITS images within ; the input timerange (t0,t1). As a consequence, it is possible that ; there are missing images that were taken at the very beginning or ; end of (t0,t1), but that their EC_IDs do not lie within the range ; spanned by the FITS data. Such missing images will therefore not be ; noticed by this program. ; ; 10) When the number of missing EC_IDs in a gap is equal to the ; number of skipped table loop positions, then it is presumed that ; the missing image types are definitely linked to their EC_IDs, and ; part C of their image codes are "." (see description of ECID_IM_CODES). ; When the number of missing EC_IDs in a gap is greater than the ; number of skipped table loop positions, then the remainder are ; presumed to be Patrol images, BUT it is not definite WHICH missing ; EC_IDs match to the table positions and the Patrol images, so part ; C of their image codes are "?". Under this approach, the ; missing image types in the gap can still be counted, even if their ; sequence is not precisely known. ; ; 11) It is difficult to correctly synch up all the different ; operations clocks. As a consequence, the level-0 reformatter ; occasionally assigns an incorrect timestamp to an image, ; although its EC_ID will be correct. For this reason, the code ; re-sorts the FITS data by EC_ID (rather than by time-order) ; before processing. Some problem images really were taken ; between (t0,t1), and this should put them back into the correct ; placement. Other problem images really were taken outside ; (t0,t1)--- these are identified because their EC_ID does not ; lie within the initial and final values, and are removed and ; tagged by SS_OUTLIERS. This is not a perfect catch, since a ; bogus image from a different XOB might happen to have an EC_ID ; between the initial and final values--- this will probably ; screw up this program. ; ; 12) This program requires the user to enter a timerange that is only ; overlapped by one XOB (or fraction thereof). There is a check ; against overlapping two or more XOBs, in which case the program ; will exit gracefully with this error message: ; 'Input timerange spans more than one XOB.' ; 'See Note #12 in header documentation for help.' ; 'Program aborting.' ; The user should re-run the program with the FITS_INDEX keyword, ; and look for a timerange that only includes a single value ; of FITS_INDEX.PROG_VER. (Note that when the program aborts ; like this, FITS_INDEX is still time-ordered, and has not yet ; been sorted by EC_ID values.) ; ; Here is an example of the process. On the first run, the ; user was not expecting to encounter this problem. ; IDL> qabort = 0b ; IDL> xrt_id_ecids, '08-Jan-2011 06:23:45','08-Jan-11 11:11:44', $ ; IDL> ecid_list, ecid_codes, /level0, qabort=qabort ; XRT_ID_ECIDS: Input timerange spans more than one XOB. ; See Note #12 in header documentation for help. ; Program aborting. ; IDL> help, qabort ; QABORT BYTE = 3 ; ; Now the user runs again to get FITS_INDEX for investigation. ; IDL> qabort = 0b ; IDL> xrt_id_ecids, '08-Jan-2011 06:23:45','08-Jan-11 11:11:44', $ ; IDL> ecid_list, ecid_codes, /level0, qabort=qabort, $ ; IDL> fits_index=fits_index ; XRT_ID_ECIDS: Input timerange spans more than one XOB. ; See Note #12 in header documentation for help. ; Program aborting. ; IDL> print, fits_index[uniq(fits_index.prog_ver, $ ; IDL> sort(fits_index.prog_ver))].prog_ver ; 0 6212 6214 ; ; PROG_VER = 0 are just the patrol images, and so are not a concern. ; However, PROG_VER = 6212 and 6214 show that there are two XOBs ; in this timerange. So now the user identifies a more suitable ; timerange to try. ; IDL> ss = where(fits_index.prog_ver eq 6212, cnt) ; IDL> hprint, fits_index[ss[[0,cnt-1]]].date_obs ; 2011-01-08T06:23:52.806 ; 2011-01-08T09:46:41.900 ; IDL> ss = where(fits_index.prog_ver eq 6214, cnt) ; IDL> hprint, fits_index[ss[[0,cnt-1]]].date_obs ; 2011-01-08T11:11:02.257 ; 2011-01-08T11:11:42.022 ; ; And runs again... ; IDL> qabort = 0b ; IDL> xrt_id_ecids, '08-Jan-2011 06:23:51','08-Jan-11 11:11:01', $ ; IDL> ecid_list, ecid_codes, /level0, qabort=qabort, $ ; IDL> fits_index=fits_index ; IDL> help, qabort ; QABORT INT = 0 ; ; So the program ran successfully to conclusion (QABORT=0). ; The user confirms the included PROG_VER values. ; IDL> print, fits_index[uniq(fits_index.prog_ver, $ ; sort(fits_index.prog_ver))].prog_ver ; 0 6212 ; ; Now there is only one XOB (with patrol images). ; ; ; CONTACTS & BUG REPORTS: ; ; Comments, feedback, and bug reports regarding this routine ; should be directed to this email address: ; xrt_manager ~at~ head.cfa.harvard.edu ; ; When sending a Bug Report, please be sure to include the ; following items: ; ; - Which program version used. (This is in the PROGVER variable.) ; ; - Values for T_STRT and T_STOP, as given to the program. ; ; - The formulation of the call to XRT_ID_ECIDS you used. ; (I.e., show exactly how you called the program.) ; ; - A copy of the entire error message given by IDL when the ; program crashed (if it crashed). ; ; - A clear example of erroneous outputs, if that is the problem. ; ; ; MODIFICATION HISTORY: ; progver = 'v2010-Oct-01' ;--- (M.Weber, SAO) Written. progver = 'v2010-Dec-30' ;--- (K.Reeves, SAO) Fixed a bug. progver = 'v2011-Feb-08' ;--- (M.Weber, SAO) Fixed a bug in response ; to email from Reeves on 2011-01-06. progver = 'v2011-Feb-09' ;--- (M.Weber, SAO) Fixed a bug in response ; to email from Reeves on 2011-02-09. progver = 'v2011-Feb-10' ;--- (K. Reeves, SAO) Added quiet keyword, changed ; 'QuickLook' directory to QLraw progver = 'v2011-Feb-10a' ;--- (M.Weber, SAO) Fixed a bug in response ; to email from Reeves on 2011-02-10. progver = 'v2011-Feb-11' ;--- (M.Weber, SAO) Fixed a bug in response ; to email from Reeves on 2011-02-11, 09:58. ; Added outputs for SS_KEEP and SS_OUTLIERS. ; Added logic to deal with incorrectly ; timestamped images. Added Note #11. progver = 'v2011-Feb-11a' ;--- (M.Weber, SAO) Fixed a bug in response ; to email from Reeves on 2011-02-11, 15:03. ; Added check and graceful exit when ; timerange spans more than one XOB. ; Added Note #12. Changed QABORT to have ; more values than (0,1). progver = 'v2011-Feb-14' ;--- (K. Reeves, SAO) Fixed method of defining ; val_ms, etc. variables to account for ; cases where PRG is greater than 99 or 999, ; which will add extra characters to lines ; read from dr_XRT*sim file progver = 'v2011-Feb-15' ;--- (K. Reeves, SAO) Added BBB code to account for ; images taken in flare mode. progver = 'v2011-Feb-16' ;--- (K. Reeves, SAO) Changed method of finding files ; from xrt_cat to xrt_find_files, fixed a ; bug with images in flare mode. progver = 'v2011-Feb-16a' ;--- (K. Reeves, SAO) Fixed bug in counting loops when ; XOB hasn't run to completion. progver = 'v2011-Feb-18' ;--- (K. Reeves, SAO) Changed sim_pos_id and ; data_pos_id variables to have three ; characters per value instead of two to ; account for subroutine loops > 99 progver = 'v2011-Feb-22' ;--- (K. Reeves, SAO);- changed logic to account for ; programs with only one step progver = 'v2011-Jul-22' ;--- (K. Reeves, SAO);- Added logic to quit with an error ; in case xrt_find_files can't find FITS directory progver = 'v2011-Sep-12' ;--- (K. Reeves, SAO);- Added entries for ecid_codes if there are ; actual images not predicted in sim file progver = 'v2011-Oct-06' ;--- (K. Reeves, SAO); - Added logic to check for files 2 minutes before ; start_time returned by xrt_tim2xobt. ; Sometimes the telescope starts taking data before this time (why?...) progver = 'v2011-Oct-07' ;--- (K. Reeves, SAO); Moved above logic to after the check for multiple XOBs, ; fixed flare mode reporting progver = 'v2012-Jul-11' ;--- (K. Reeves, SAO); Added check for patrol images in addition to check for ; obs_mode='FL' so that patrol images mislabeled as obs_image='FL' don't foul things up. progver = 'v2012-Jul-11a' ;--- (K. Reeves, SAO); Added a check for patrol images that was neglected in the above fix. ; progver = 'v2014-Apr-02' ;--- (K. Reeves, SAO); Added logic that checks for pre-flare buffer images after the last ; regular image by comparing times of PFB images in sim file to patrol images in data. ; progver = 'v2014-Apr-04' ;--- (K. Reeves, SAO); Fixed bug in above logic ; progver = 'v2015-May-08' ;--- (K. Reeves, SAO); When running an XOB that is all PFB images, assume all missing images are PFB ; Also fixed a bug where all missing images ; after a flare mode were assumed to be flare images progver = 'v2018-Dec-11' ;--- (K. Reeves, SAO); Fixed a bug that appended 'PRST/???' to ecid_im_codes when the ; running program is misidentified instead ; of overwriting the 'MISS/' string that is already there ;================================================================== prognam = 'XRT_ID_ECIDS' prognul = ' ' qstop = keyword_set(qstop) qabort = keyword_set(qabort) if (qabort ne 0b) then begin print, prognam+': QABORT = TRUE as input. Aborting.' qabort = 1b return endif default, archive_prefix, '/archive/hinode/xrt/' case 1 of keyword_set(level0): level0 = 1b keyword_set(quicklook): quicklook = 1b else: begin ;; default to level0 data level0 = 1b endelse endcase ;; First check that T_STRT and T_STOP are in the same XOB run. ;; Abort if they are not in the same run. xrt_tim2xobt, t_strt, xob_start=xob_start_A, xob_stop=xob_stop_A, $ xob_string=xob_string_A, start_file=xob_start_file, $ archive_prefix=archive_prefix, timeline_dir=timeline_dir,$ qerror=qerror if n_elements(qerror) gt 0 then return xrt_tim2xobt, t_stop, xob_start=xob_start_Z, xob_stop=xob_stop_Z, stop_file=xob_stop_file,$ archive_prefix=archive_prefix, timeline_dir=timeline_dir if ((xob_start_A ne xob_start_Z) or ((xob_stop_A ne xob_stop_Z) and strmatch(xob_start_file,xob_stop_file))) then begin print, prognam+': T_STRT and T_STOP are not part of the same XOB run,' print, prognul+' according to the dr_XRT*.sim files. Aborting.' qabort = 1b return endif ;; Need to use the dr_XRT*.sim file to figure out the *expected* ;; loop structure of the XOB. For reasons having to do with the ;; search logic in , it is sufficient to only ;; check START_FILE. ;; Read file. text = rd_text(xob_start_file) ;; Get timestamped lines and convert to a seconds array, ;; referenced to T_STRT. ss1 = where(strmid(text,0,2) eq '20') ex_times = strmid(text[ss1], 0, 15) ex_t_str = '_' + strmid(ex_times,0,8) + $ '_' + strmid(ex_times,9,6) ex_t_ints = anytim(fid2time(ex_t_str,'_'), /ints) t_strt_ints = anytim(xob_start_A, /ints) ex_t_secarr = int2secarr(ex_t_ints, t_strt_ints) t_stop_ints = anytim(xob_stop_A, /ints) t_stop_secarr = int2secarr(t_stop_ints, t_strt_ints) ;; Identify the lines between xob_start_A and xob_stop_A ;; fixed broken indexing - 2010-12-30 KKR ss2 = where( (ex_t_secarr ge 0) and $ (ex_t_secarr lt t_stop_secarr) ) ctr_text = text[ss1[ss2]] ;; changed method of populating val_* variables to account for a PRG ;; value greated than 99 - 2011-02-14 KKR N_lines = n_elements(ctr_text) val_ms = strarr(N_lines) val_mr = strarr(N_lines) val_ss = strarr(N_lines) val_sr = strarr(N_lines) val_sq = strarr(N_lines) val_date = strarr(N_lines) val_time = strarr(N_lines) val_buff = strarr(N_lines) for i=0,N_lines-1 do begin text_arr = strsplit(ctr_text[i], ' ', /extract) val_ms[i] = string(text_arr[3], format='(I3.3)') val_mr[i] = string(text_arr[4], format='(I3.3)') val_ss[i] = string(text_arr[5], format='(I3.3)') val_sr[i] = string(text_arr[6], format='(I3.3)') val_sq[i] = string(text_arr[7], format='(I3.3)') val_date[i] = string(text_arr[0]) val_time[i] = string(text_arr[1]) val_buff[i] = text_arr[29] endfor ;; We are going to make two string ID arrays from these loop ;; positions and counters. We first use only the *unique* positions ;; to identify preflare images versus regular images. Patrol ;; images are not in the dr_xrt*.sim file, so we identify those ;; with the FITS data. sim_pos_id = val_ms+val_ss+val_sq ;; Now find the unique values. ss_sort = sort(sim_pos_id) pos_id_sort = sim_pos_id[ss_sort] text_sort = ctr_text[ss_sort] ss_uniq = uniq(pos_id_sort) sim_pos_id_uniq = pos_id_sort[ss_uniq] text_uniq = text_sort[ss_uniq] N_uniq = n_elements(sim_pos_id_uniq) ;; Identify Pre-flare counter positions to string codes. ;; POS_IM_CODE is paired to CTR_ID_UNIQ. sim_pos_im_codes = strarr(N_uniq) + 'Reg' dummy = grep('Preflare-1', text_uniq, index=ss_PF) if (ss_PF[0] ne -1) then sim_pos_im_codes[ss_PF] = 'PF1' dummy = grep('Preflare-2', text_uniq, index=ss_PF) if (ss_PF[0] ne -1) then sim_pos_im_codes[ss_PF] = 'PF2' dummy = grep('Preflare-3', text_uniq, index=ss_PF) if (ss_PF[0] ne -1) then sim_pos_im_codes[ss_PF] = 'PF3' ;; Add a code for Patrol images. sim_pos_id_uniq = [sim_pos_id_uniq, '*********'] sim_pos_im_codes = [sim_pos_im_codes, 'Pat'] N_uniq = N_uniq + 1 ;; Now we make the second string ID array from all of the loop ;; position and counter values. This will be used to understand ;; where a given FITS image is in the loop tables. We also match ;; the image codes. ;; We make the simplifying assumption that the sim file XOB always ;; starts at the first image in the tables. ;; In order to account for all the loop repeats, we need to ;; find out how many times each element of CTR_ID_UNIQ gets ;; repeated in one program pass. This is equal to the max loop ;; repeat count at the Sequence and Subroutine levels. All we ;; really need is the sum over all unique images. ;; Goto N_uniq-2 because the last unique im_code is for the Patrol ;; image, which is not part of the program loop tables. N_sum_loops = 0L for ii = 0L,(N_uniq-2) do begin ss = where(sim_pos_id eq sim_pos_id_uniq[ii]) N_sum_loops = N_sum_loops + ((max(val_mr[ss])+1) * (max(val_sr[ss])+1)) endfor ;; If XOB doesn't run all the way through one loop, ;; N_sum_loops calculated above will be to high. Check for this. ;; KKR 2011-02-16 if N_sum_loops gt n_elements(sim_pos_id) $ then N_sum_loops = n_elements(sim_pos_id) ;; The sim file assumes ideal operation, with no missing or unknown ;; images, so we may assume that in N_SUM_LOOPS steps the entire ;; program is passed through once, exactly. ;; SIM_POS_CTR_ID is the string array with the loop positions and counters. ;; SIM_POS_CTR_IM_CODES are the matched image types. Both variables ;; are for one complete pass through the program, with no Patrols. ;; Goto N_uniq-2 because the last unique im_code is for the Patrol ;; image, which is not part of the program loop tables. sim_pos_ctr_id = val_ms[0:N_sum_loops-1] + val_mr[0:N_sum_loops-1] + $ val_ss[0:N_sum_loops-1] + val_sr[0:N_sum_loops-1] + $ val_sq[0:N_sum_loops-1] sim_pos_ctr_im_codes = strarr(N_sum_loops) for ii = 0L,(N_uniq-2) do begin ss = where(sim_pos_id[0:N_sum_loops-1] eq sim_pos_id_uniq[ii]) sim_pos_ctr_im_codes[ss] = sim_pos_im_codes[ii] endfor ;; Now we need to get the FITS files for the period from ;; T_STRT to T_STOP, and extract the relevant info. ;; First, look for files. Inputs specified QuickLook or Level0. xrt_find_files, t_strt, t_stop, files, level0=level0, quicklook=quicklook, $ archive_prefix=archive_prefix, qerror=qerror if n_elements(qerror) gt 0 then begin qabort = 2b return endif ;; If no files found, then abort. if (files[0] eq '') then begin if not keyword_set(quiet) then $ print, prognam+': No FITS files found between ' + T_STRT +' and '+ T_STOP+$ '. Aborting.' qabort = 2b return endif ;; If files found, then read them in. read_xrt, files, index_full N_fits = n_elements(index_full) ;; Before starting to process the data, there are a couple of ;; potential problems that need to be addressed. The first potential ;; problem is if the user has mistakenly provided a time range ;; that overlaps more than one XOB. This first block attempts to ;; check that and exit gracefully. ss1 = uniq(index_full.prog_ver, sort(index_full.prog_ver)) ss2 = where(index_full[ss1].prog_ver NE 0, cnt2) ;; ignore last couple of images - sometimes last few images in flare mode are ;; incorrectly labeled in the flflg keyword if (cnt2 GT 1) then begin flcnt = n_elements(uniq(index_full[0:N_fits-3].flflg)) ;; catches instances where QT XOB is PFB only if (flcnt GT 1) then begin msg0 = [prognam+': Input timerange spans more than one XOB.', $ prognul+' See Note #12 in header documentation for help.', $ prognul+' Program aborting.'] if not keyword_set(quiet) then print, transpose(msg0) if qstop then stop qabort = 3b return endif endif ;; Now check that we didn't miss a few files at the beginning xrt_find_files, atime(anytim(t_strt)-120), t_strt, files_back, level0=level0, quicklook=quicklook, $ archive_prefix=archive_prefix, qerror=qerror if files_back[0] ne '' then begin read_xrt, files_back, index_back if total(index_full.prog_no ne 0) ne 0 then begin ;; only do this if index_full is something other that FLD images i = n_elements(index_back) -1 while total(where(index_back[i].prog_no eq index_full.prog_no)) gt 0 and i gt 0 do begin ;; only include back FITS if prog_no matches ones already there index_full = concat_struct(index_back[i],index_full) i = i-1 endwhile endif endif N_fits = n_elements(index_full) ;; For the second potential problem, an error case has to be ;; dealt with. Sometimes an image has a correct EC_ID but an incorrect ;; timestamp. Since the FITS files are time-ordered, such an image ;; appears to have an EC_ID value that "spikes" out of order, if the ;; timeshift is large. SO, we have to look for down-shifts in EC_ID ;; value that are not associated with the rollover. There might be ;; several such images in a row. ;; IF the shifted EC_IDs are still bracketed by the first and ;; last images, then they are just sorted into order. If they are ;; outside the range, then they are discarded. ;; Also note that EC_ID rolls over at 32768: ;; ... 32766, 32767, 0, 1, ... data_ecid = index_full.ec_id thresh_lo = data_ecid[0] ss = where(data_ecid LT thresh_lo, cnt) if (cnt GE 1) then data_ecid[ss] = data_ecid[ss] + 32768L thresh_hi = data_ecid[n_elements(data_ecid)-1] ;; Re-sort. ss1 = sort(data_ecid) data_ecid = data_ecid[ss1] index_full = index_full[ss1] ;; Identify keepers and outliers. ss_keep = where(data_ecid LE thresh_hi, N_fits) ss_outliers = where(data_ecid GT thresh_hi) index = index_full[ss_keep] ;; Extract the string code for their loop counter positions. data_pos_ctr_id = string(index.main_pos-1, format='(I3.3)') + $ string(index.subr_cnt-1, format='(I3.3)') + $ string(index.subr_pos-1, format='(I3.3)') + $ string(index.seqn_cnt-1, format='(I3.3)') + $ string(index.seqn_pos-1, format='(I3.3)') data_pos_id = string(index.main_pos-1, format='(I3.3)') + $ string(index.subr_pos-1, format='(I3.3)') + $ string(index.seqn_pos-1, format='(I3.3)') ;; Start mapping the FITS images into a *continuous* list of EC_IDs. ;; We compare this list to the FITS ec_ids to identify gaps and to ;; ascertain how to fill them in. ;; Dealing with gaps will be a big chunk of the logic. ;; Establish ECID_LIST. This needs to start with the 1st FITS image ;; and run through every number until the last image. Note that ;; missing images will leave gaps in EC_ID that need to be spanned. ;; Also note that EC_ID rolls over at 32768: ;; ... 32766, 32767, 0, 1, ... ;; HOWEVER, there is an exceptional case that must be addressed. ;; Sometimes an image has a correct EC_ID but an incorrect timestamp. ;; Since the FITS files are time-ordered, such an image appears to ;; have an EC_ID value that "spikes" out of order, if the timeshift ;; is large. SO, we have to look for down-shifts in EC_ID value ;; that are not associated with the rollover. There might be ;; several such images in a row. These outliers are removed. N_list = (index[N_fits-1].ec_id - index[0].ec_id + 1 + 32768L) mod 32768L ecid_list = (lindgen(N_list) + index[0].ec_id) mod 32768L ;; Prep ECID_IM_CODES and ECID_MISS_CODES. ecid_im_codes = strarr(N_list) + 'MISS/' ecid_miss_codes = bytarr(N_list) ;; Prep the working loop by handling the first entry. i_fits = 0L ss2 = where(sim_pos_id_uniq eq data_pos_id[i_fits]) ;; check for flare mode - kludge to get around some mislabeled data ;; in april 2012 ;; added check for patrol images with 'FL' obs_mode to get around ;; mislabeled data from july 7 and 10 2012 if (strmatch(index[i_fits].obs_mode, 'FL') and (index[i_fits].img_mode ne 3)) or $ (anytim(index[i_fits].date_obs) gt anytim('2012-04-07T09:33:17') and $ anytim(index[i_fits].date_obs) lt anytim('2012-04-07T09:39:16')) then $ ecid_im_codes[0] = 'PRST/Flr' + '.' else $ ecid_im_codes[0] = 'PRST/' + sim_pos_im_codes[ss2] + '.' if (ecid_im_codes[0] eq 'PRST/Pat.') then begin ecid_miss_codes[0] = 2 endif else begin ecid_miss_codes[0] = 1 endelse ;; Now the real work begins. ;; We step through FITS list, looking for matches against the EC_IDs ;; and the loop positions and counters. We use ECID_MISS_CODES to keep ;; track of where the gaps are and which present images are patrols ;; (which cannot help us reconcile which images are missing). ;; There will be a second loop to reconcile the missing images. ;; By definition, we know that the first and last FITS images match ;; to the first and last EC_ID. for i_fits = 1L,(N_fits-1) do begin ;; Record the image-code for the EC_ID of the FITS image we ;; are currently pointing at. ss1 = (where(ecid_list eq index[i_fits].ec_id))[0] ss2 = (where(sim_pos_id_uniq eq data_pos_id[i_fits]))[0] ;; if ss2 = -1, then there is an extra image in the data that ;; was not simulated. This can happen with long synoptics, ;; since COT thinks they take more time than they actually do. ;; Extra data is not the concern, missing data is. So we will ;; soldier on...KKR 2011/09/12 ;; except we need ss2 = -1 for the special case of flare mode. if ss2[0] ne -1 or strmatch(index[i_fits].obs_mode, 'FL') eq 1 then begin ;; check for flare mode if (strmatch(index[i_fits].obs_mode, 'FL') and (index[i_fits].img_mode ne 3)) then $ ecid_im_codes[ss1] = 'PRST/Flr' + '.' else $ ecid_im_codes[ss1] = 'PRST/' + sim_pos_im_codes[ss2] + '.' ;; ECID_MISS_CODES = 0 --> Missing, unreconciled ;; = 1 --> Present, not Patrol ;; = 2 --> Present, Patrol ;; = 3 --> Missing, reconciled as indeterminable if (ecid_im_codes[ss1] eq 'PRST/Pat.') then begin ecid_miss_codes[ss1] = 2 endif else begin ecid_miss_codes[ss1] = 1 endelse endif else begin ;; append a code to the ecid list to indicate image is there ;; ecid_im_codes = [ecid_im_codes,'PRST/????'] ;; Ack, no, this shouldn't append, it should overwrite ;; the 'MISS/' in the appropriate place in the case that ;; it can't figure out what program is running. How ;; did this ever work? -KKR 2018/12/11 ecid_im_codes[ss1] = 'PRST/????' endelse endfor ;; First, if we are in flare mode, label all missing images as ;; MISS/FLR., since there are no preflare buffer images taken during ;; flare mode. if (strmatch(index[0].obs_mode, 'FL') and (index[0].img_mode ne 3)) then begin minfl = min( where(strmatch(index.obs_mode, 'FL'))) maxfl = max( where(strmatch(index.obs_mode, 'FL'))) min_fl_ecid= index[minfl].ec_id max_fl_ecid= index[maxfl].ec_id match = where(ecid_list ge min_fl_ecid and ecid_list le max_fl_ecid) ss0 = where(ecid_miss_codes[match] eq 0, cnt0) if (cnt0 ge 1) then begin ecid_im_codes[ss0] = ecid_im_codes[ss0] + 'Flr.' endif ; return endif ;; Now we go back through and deal with the missing images. ;; We will try to reconcile them, ie, try to match them to ;; possible Patrol images or to frames from the XOB program tables ;; (as expected in the dr_XRT*.sim file). To do this, we identify ;; the PRST images that bound a gap of missing images. PRST/Pat ;; images are not helpful; we need to use PRST/Reg or PRST/PFn ;; images, which establish where the EC_IDs are in the program tables. ;; In order to reconcile missing images, you need at least two ;; known PRST images bracketing the gap. So we can first dispose ;; of the case wherein there aren't two PRST images in the whole series. ;; No more processing is then required. ss1 = where(ecid_miss_codes eq 1, cnt1) if (cnt1 le 1) then begin ss0 = where(ecid_miss_codes eq 0, cnt0) if (cnt0 ge 1) then begin ecid_miss_codes[ss0] = 3 ;; check to see if there are any Regular images planned. If ;; not, assume missing images are PFB images. if total(strmatch(val_buff, 'Regular,')) eq 0 then $ ecid_im_codes[ss0] = ecid_im_codes[ss0] + 'PFn' else $ ecid_im_codes[ss0] = ecid_im_codes[ss0] + '????' endif return endif ;; By similar reasoning, we may dispose of any missing images ;; before the first PRST image. But we may still have missing ;; images to be handled later on. ss0 = where(ecid_miss_codes eq 0, cnt0) ss0a = where(ss0 lt ss1[0], cnt0a) if (cnt0a ge 1) then begin ss0a = ss0[ss0a] ecid_miss_codes[ss0a] = 3 if total(strmatch(val_buff, 'Regular,')) eq 0 then $ ecid_im_codes[ss0a] = ecid_im_codes[ss0a] + 'PFn' else $ ecid_im_codes[ss0a] = ecid_im_codes[ss0a] + '????' endif ;; Likewise for any missing images after the last PRST image. ;; updated to see if present patrol images fall range of sim file ;; that has only PFB images. If that's true, then all missing ;; images between patrol images are probably PFB images, and not ;; missing. Needed for programs where lots and lots of PFB images ;; are run, but not very many regular images. KKR 2014/04/02 ss0b = where(ss0 gt max(ss1), cnt0b) if (cnt0b ge 1) then begin ;; find last regular image in sim file sreg = where(strmatch(val_buff, 'Regular,') eq 1, cntreg) if (cntreg ge 1) then begin ;;find time of last regular image from sim file reg_date = strmid(val_date(max(sreg)),0,4)+'-'+strmid(val_date(max(sreg)),4,2)+'-'+strmid(val_date(max(sreg)),6,2) reg_time = strmid(val_time(max(sreg)),0,2)+':'+strmid(val_time(max(sreg)),2,2)+':'+strmid(val_time(max(sreg)),4,2) reg_time = reg_date + ' ' + reg_time ;; find end time from sim file end_date = strmid(val_date(n_elements(val_date)-1),0,4)+'-'+strmid(val_date(n_elements(val_date)-1),4,2)+'-'$ +strmid(val_date(n_elements(val_date)-1),6,2) end_time = strmid(val_time(n_elements(val_time)-1),0,2)+':'+strmid(val_time(n_elements(val_time)-1),2,2)+':'$ +strmid(val_time(n_elements(val_time)-1),4,2) end_time = end_date + ' ' + end_time reg_datss = where(index.prog_ver ne 0, reg_dat_cnt) ;; if last image is patrol image if index[n_elements(index)-1].prog_ver eq 0 then begin ;; start and end time of patrol images pat_start_time = index[max(reg_datss)+1].date_obs pat_end_time = index[n_elements(index)-1].date_obs ;; anything between reg_time and end_time in sim file is a ;; PFB image if anytim(pat_start_time) gt anytim(reg_time) and anytim(pat_end_time) lt anytim(end_time) then begin ss0b = ss0[ss0b] ecid_miss_codes[ss0b] = 3 ecid_im_codes[ss0b] = ecid_im_codes[ss0b] + 'PFn.' endif else begin ss0b = ss0[ss0b] ecid_miss_codes[ss0b] = 3 ecid_im_codes[ss0b] = ecid_im_codes[ss0b] + '????' endelse endif else begin ss0b = ss0[ss0b] ecid_miss_codes[ss0b] = 3 ecid_im_codes[ss0b] = ecid_im_codes[ss0b] + '????' endelse endif ss0b = ss0[ss0b] ecid_miss_codes[ss0b] = 3 ecid_im_codes[ss0b] = ecid_im_codes[ss0b] + 'PFn' endif ;; Now we proceed to dealing with gaps that are bracketed by "PRST/Reg." ;; images. Be careful about all possibilities here. Eg, two gaps divided ;; only by "PRST/Pat." images. In such a case, we need to expand ;; the gap range until the bracketing "PRST/Reg." images are known, ;; as well as handle any interior Patrol images that are PRST. ;; By the preceding steps, we already know that every missing ;; image is bounded by "PRST/Reg." images. ss0 = where(ecid_miss_codes eq 0, cnt0) ss2 = where(ecid_miss_codes eq 2, cnt2) while (cnt0 ge 1) do begin ;; Identify missing images to reconcile in this gap. ss1a = ss1[max(where(ss1 lt ss0[0]))] ss1z = ss1[min(where(ss1 gt ss0[0]))] ss_miss = ss0[where(((ss0 gt ss1a) and (ss0 lt ss1z)), cnt_miss)] ;; Identify "PRST/Pat." images interior to this gap. ss_pat = where(((ss2 gt ss1a) and (ss2 lt ss1z)), cnt_pat) ;; What is the number of EC_ID places between the "PRST/Reg." images? N_ecid_steps = ((ecid_list[ss1z] - ecid_list[ss1a] + 32768L) $ mod 32768L) - 1L ;; But we need to subtract out the "PRST/Pat." images. N_ecid_steps = N_ecid_steps - cnt_pat ;; What is the number of program loop table steps between ;; the "PRST/Reg." images? ss_dummy = (where(index.ec_id eq ecid_list[ss1a]))[0] ss1a_posctr = (where(sim_pos_ctr_id eq data_pos_ctr_id[ss_dummy]))[0] ss_dummy = (where(index.ec_id eq ecid_list[ss1z]))[0] ss1z_posctr = (where(sim_pos_ctr_id eq data_pos_ctr_id[ss_dummy]))[0] N_prg_steps = ((ss1z_posctr - ss1a_posctr + N_sum_loops) $ mod N_sum_loops) - 1L ;; There are two possibilities here. ;; (a) N_ecid_steps ge N_prg_steps. This means that we may ;; assume there were some "MISS/Pat" images, and we will be ;; able to reconcile the gap with table steps. ;; (b) N_ecid_steps lt N_prg_steps. The program tables somehow ;; got ahead of the EC_ID stepping, or the tables were reset. ;; Either way, this code will have to reconcile the missing images ;; as "indeterminable". if (N_ecid_steps ge N_prg_steps) then begin ;; The first N_prg_steps missing images are assigned the ;; program table steps. if (N_prg_steps GE 1) then begin ecid_miss_codes[ss_miss[0:N_prg_steps-1]] = 1 ;; The concatenation in this next step is in case the prog steps ;; need to roll over back to the beginning of sim_pos_ctr_im_codes. ecid_im_codes[ss_miss[0:N_prg_steps-1]] = $ ecid_im_codes[ss_miss[0:N_prg_steps-1]] + $ ([sim_pos_ctr_im_codes,sim_pos_ctr_im_codes])[ss1a_posctr+1: $ ss1a_posctr+1+N_prg_steps-1] endif ;; Any remaining steps are assigned as Patrol images. if (N_ecid_steps gt N_prg_steps) then begin ;; If the program takes only one image over & over, ;; N_prg_steps will be -1 - KKR 2011-02-22 if (N_prg_steps ge 1) then begin ecid_miss_codes[ss_miss[N_prg_steps:*]] = 2 ecid_im_codes[ss_miss[N_prg_steps:*]] = $ ecid_im_codes[ss_miss[N_prg_steps:*]] + 'Pat' endif else begin ecid_miss_codes[ss_miss] = 2 ecid_im_codes[ss_miss] = ecid_im_codes[ss_miss] + 'Pat' endelse ecid_im_codes[ss_miss] = ecid_im_codes[ss_miss] + '?' endif else begin ecid_im_codes[ss_miss] = ecid_im_codes[ss_miss] + '.' endelse endif else begin ecid_miss_codes[ss_miss] = 3 ecid_im_codes[ss_miss] = ecid_im_codes[ss_miss] + '????' endelse ;; Count remaining unreconciled images to determine whether to ;; loop through WHILE again. ss0 = where(ecid_miss_codes eq 0, cnt0) endwhile if qstop then stop return END