Notes on procedures for making AIA cutouts of RHESSI flares, Kim Tolbert, 14-Feb-2018

There are 4 separate cron jobs that run at least once per day on hesperia:
 1. run_aia_subm - find RHESSI flares > 12 keV and submit cutout requests 
 2. run_aia_copyfits - copy FITS files for for cutout requests to hesperia
 3. run_aia_update_browser - update the text file RHESSI Browser uses to link to AIA movies for flares
 4. run_aia_gen_movie - make movies of AIA cutouts on hesperia if not available at lmsal

These four jobs run asynchronously. The subm and copyfits jobs take unknown amounts of time, so both write temporary files recording what they
are running.  Every time the cron job starts up, it checks the progress of the jobs in the temporary files.  When they are done, they update 
the master file.  If any job has been running for too long without completing, it is aborted and sumitted again. If any new jobs need to be
run, another temporary file is created. At frequent intervals 
the text file used by Browser is generated from the master file. Because any of these programs could be rewriting the master file, each one 
places a lock on the master file first and then unlocks it when done (if it can't lock it because it's already locked, the program aborts) using
the hsi_aia_lock_cat routine.  Th hsi_aia_lock_cat routine controls locking simply through a text file called lockcat.txt that has one line saying
'LOCKED' or 'UNLOCKED' and the time.  If for some reason, it gets stuck in the locked state (e.g. a program crashed), it can be edited manually to 
say 'UNLOCKED' (just make sure no programs that might write the master cat are still running!)

Here is a summary of the master, temporary and final files.  With the exception of the final Browser file, all are stored in /home/softw/aia/cutout_cat/:

 1. master file: aia_cutout_master.txt keeps track of everything that has been done for the aia cutouts
   This is a sample line from the master file:
   2010/07/08 22:03:52, 2010/07/08 22:32:28, 10070827, 12, ssw_service_150403_122514_99977, yes_lmsal_movie, no_hesp_movie, 2010/07/08/20100708_2203-2232
   giving start/end of cutouts, RHESSI flare number, high edge of highest RHESSI energy bin, ssw job id, whether there's a movie at lmsal or hesperia, and
   the directory name where the FITS files are (if they haven't been copied yet, the last field will say no_fits_dir). Each time this file is written
   with new info, the old file is saved in a file called aia_cutout_master_yyyymmdd_hhmm.txt so we have a record.

 2. submitted temporary file: aia_cutout_submitted.txt lists the requests that have been submitted but not complete yet (except note that the last one is
    always left there, so there is always at least one line in the file. It's not done again).  Each time this file is written, the old version is saved in a file
    called aia_cutout_submitted_yyyymmdd_hhmm.txt so we have a record.

 3. copyfits temporary files: any number of aia_cutout_copyfits_*.txt files keeping track of jobs that are copying the FITS files to hesperia. These are
    deleted when all of the copy jobs in a file finish.

 4. /data/sdo/aia_cutout/rhessi_flare_aia_cutout_new.txt -  Final file read by Browser, and stored in /data/sdo/aia_cutout. Each time a new file is created
    the old version is saved in rhessi_flare_aia_cutout_new_yyyymmdd_hhmm.txt in /home/softw/cutout_cats


Here is a summary of the programs that are run in the cron jobs and the routines they call. They are in /home/softw/aia (and some may 
be in $SSW/hessi/idl/gen).

run_aia_subm script
 run_aia_subm.pro
   hsi_aia_check_subm
   hsi_read_aia_cutout_submitted
   hsi_aia_cutout_subm
   
run_aia_copyfits script 
  run_aia_copyfits.pro
    hsi_aia_check_copyfits
	hsi_read_aia_cutout_master
	hsi_aia_copyfits
	
run_aia_gen_movie script
  run_aia_gen_movie.pro
    hsi_aia_check_gen_movie
	hsi_read_aia_cutout_master
	hsi_aia_gen_movie
	
run_aia_update_browser script 
  run_aia_update_browser.pro
    hsi_aia_update_browser_cat
	  hsi_read_aia_cutout_master

Current cron entries (14-Feb-2018) are:
[softw:softw@gs671-hesperia.ndc.nasa.gov]# crontab -l | grep aia
09 02,14 * * * /home/softw/aia/run_aia_copyfits > /home/softw/aia/run_aia_copyfits.log 2>&1
18 11 * * * /home/softw/aia/run_aia_gen_movie > /home/softw/aia/run_aia_gen_movie.log 2>&1
09 17 * * * /home/softw/aia/run_aia_subm > /home/softw/aia/run_aia_subm.log 2>&1
29 04,12,20 * * * /home/softw/aia/run_aia_update_browser > /home/softw/aia/run_aia_update_browser.log 2>&1

When I manually request AIA cutouts for long period that covers several RHESSI orbits and multiple flares, this is what I do. Submit the job after figuring
out the best xy center and fov for the entire time period. Once request is done, make an output dir for FITS files, and submit job to copy them, then edit
the master file.  e.g.
	hsi_aia_cutout, '4-sep-2017 20:10:00', '4-sep-2017 23:59:00', xy=[230,-300], fov=[ 350, 400], url=url, email='kim.tolbert@nasa.gov', /aec, /hcr, retain=-1
	WAIT until request is finished  !!!  
	out_dir = '/data/sdo/aia/2017/09/04/20170904_2010-2359'
	file_mkdir, out_dir
	ssw_service_get_data, 'ssw_service_180209_145800_17437', out_dir=out_dir, /loud

Edit aia_cutout_master.txt to add a line for each flare in time interval done, putting the same full time range and job id for each line, but with the unique 
flare ids in each line.  Set last field to no_fits_dir.  Once files are copied, change no_fits_dir on all the new lines to the dir created above.