1
0
-1

I'd like to get hold of the SExtractor configuration files used for Y3. Both those used for the coadds, and those used for the single-epoch images. Where I can find these?

    CommentAdd your comment...

    1 answer

    1.  
      2
      1
      0

      Note the above question has a lot of facets.  First two problems came to mind.

      1. there are multiple SExtractor calls in both cases:
        • SExtractor for input for scamp (in single-epoch)
        • SExtractor for input to PSFex (in both single- and multi-epoch)
        • SExtractor for catalog (again in both single- and multi-epoch)
      2. there can be command line over-rides so really the config files are not enough.


      Starting from Year 2 production, the processing framework keeps track of most of the things you would need to recreate the processing that DESDM runs.  (There is an alternate source and that is to dig through log files).  Here is how to find what you were looking for using the DB.  For the queries I describe below to work you need to referencing tables in the operations database (DESOPER).  Some elements are present in DESSCI but those are mainly those that are needed to connect you to the end-products (for instance there are tables like Y3A1_FILE_ARCHIVE_INFO  in DESSCI that contain the main end-products).


      1)  First you want to identify a typical run.  For example (for the Y3A1 single-epoch processing):

      select pfw_attempt_id from proctag where tag='Y3A1_FINALCUT' and rownum<10;

      2) From there I would choose a single run (attempt) and dig through what was run:

      select count(*),name from pfw_exec where pfw_attempt_id=62145 group by name order by name;

      Aha!  I want the executions of SExtractor (i.e. sex).  However, in the single epoch processing of this exposure it is now clear that SExtractor was executed 240 times.  

      3) Identify the process you wanted

      The PFW_EXEC tables carries the full command lines that were executed so you can actually look through the executions to see what was run.   To provide a little order though, I am adding a join to the task table so that I actually get the executions in the order they occurred.

      select e.task_id,t.start_time,e.cmdargs
      from pfw_exec e, task t
      where e.pfw_attempt_id=62145
          and e.name='sex'
          and e.task_id=t.id 
      order by t.start_time;

      Perusing over the command line arguments (cmdargs) you will see that there were four distinct sets of call.  The first are those that generating inputs for the astrometric refinement (i.e. for scamp).  Then a call to build a sextractor background. Then a call to get the inputs for PSFex, and finally a set of call that built the single-epoch catalogs.  That last set had command lines that look like the following:

      red/immask/D00360719_Y_c62_r2152p01_nullweight-immask.fits[0]  -c config/2015100
      8_sex.config  -FILTER_NAME config/20151008_sex.conv  -STARNNW_NAME config/201510
      08_sex.nnw  -INTERP_TYPE VAR_ONLY  -WEIGHT_TYPE MAP_WEIGHT  -WEIGHT_IMAGE red/im
      mask/D00360719_Y_c62_r2152p01_nullweight-immask.fits[2],red/immask/D00360719_Y_c
      62_r2152p01_nullweight-immask.fits[2]  -SEEING_FWHM 1.06947885378  -FLAG_IMAGE r
      ed/immask/D00360719_Y_c62_r2152p01_nullweight-immask.fits[1]  -CATALOG_NAME cat/
      D00360719_Y_c62_r2152p01_red-fullcat.fits  -SATUR_KEY SATURATE  -PARAMETERS_NAME
       config/20151008_sex.param.finalcut.20130702  -PSF_NAME psf/D00360719_Y_c62_r215
      2p01_psfexcat.psf  -CHECKIMAGE_TYPE SEGMENTATION,BACKGROUND  -CHECKIMAGE_NAME se
      g/D00360719_Y_c62_r2152p01_segmap.fits,red/bkg/D00360719_Y_c62_r2152p01_bkg.fits
        -DETECT_THRESH 1.5  -INTERP_MAXXLAG 4  -INTERP_MAXYLAG 4

      4) Getting to the files:

      You may have noticed that I also asked for the task_id that accompanied each call.  There are two tables based on the Open Provenance Model that allow you to connect inputs, outputs and processes.  Those tables are OPM_USED and OPM_WAS_DERIVED_FROM.  In this case we can use OPM_USED to obtain links to all files that were "used" by this task.


      select u.desfile_id,d.filetype,d.filename
      from opm_used u, desfile d
      where u.task_id=660837487
          and u.desfile_id=d.id;

      Note some of the items used might have been intermediate files (i.e. not saved long term).  The files that existwithin the storage condominium can be further identied by joining to file_archive_info.  In this case the image file that SExtractor ran over is such a file (i.e. not saved).  The final image that is saved in the single-epoch pipeline preserves the weights for all pixels.  When SExtractor is run, the file runs through a pre-processor and changes some of the weights to zero based on flags/bits that are thrown in the MSK plane.  If you wanted to build a replica of that image you would need to recreate that step.

      5) Finding what you can actually retrieve:

      Fortunately this question wanted the configuration files.

      select fai.path,d.filename,fai.compression      
      from opm_used u, desfile d, file_archive_info fai
      where u.task_id=660837487
          and u.desfile_id=d.id
          and fai.desfile_id=d.id;

      Note you could have simple looked in file_archive_info for the entry for the specific filename you wanted.  If it exists in the main file store it would be present in that table (note the system distinguishes between compressed and uncompressed versions of the same file by keeping track of the compression separately (i.e. if you were not matching on DESFILE_ID you would need to match both FILENAME and COMPRESSION

      6) Retrieving files (through http)

      If for example you wanted the configuration file (20151008_sex.config) you could take that information and construct the URL with:

      https://desar2.cosmology.illinois.edu/DESFiles/desarchive/{fai.path}/{filename}/{fai.compression}

      or in this case you would specifically form the URL:

      https://desar2.cosmology.illinois.edu/DESFiles/desarchive/OPS/config/20151008/finalcut/20151008_sex.config



      The same reasoning can be used to trace the COADD (multi-epoch) production.  In the first step above you would start from the Y3A1_COADD processing tag.

        CommentAdd your comment...