Home

Documentation

Project Support

Changes in Version 9 of GraspExtrasSoftwareOtatool

Author:
isani
Timestamp:
Sat Apr 20 15:47:28 2013

Legend:

Unmodified
Added
Removed
Modified

  • GraspExtrasSoftwareOtatool

    v8 v9
    1 1  '''[wiki:GraspExtrasSoftware otatool]''' 
    2 2  An "OTA" is an Orthogonal Transfer Array CCD composed of 8x8 cells of (typ.) 500x500 or 600x600 each.  We have developed a tool to generate some quick image statistics and create a fixed color stretch quicklook JPEG from an OTA image saved as a standard Iraf Multi-Extension FITS (MEF) with 64 image extensions.  With the proper keywords, otatool may be useful on other MEF images as well. 
    3 3  {{{ 
    4 4  % otatool -h 
    5 5  }}} 
    6 6  Currently produces the following help summary: 
    7 7  {{{ 
    8 8  OTA FITS Manipulation Tool v1.27 (svn 4004, Sep 10 2009) 
    9 9  usage: otatool [options] < inputfile.fits > outputfile.fits 
    10 10  where [options] can be (default values shown in [...]) 
    11 11          open=   Specify an open-filter or default input FITS file 
    12 12          blue=   For making color images, specify a blue-filter FITS file 
    13 13           red=   For making color images, specify a red-filter FITS file 
    14 14          dark=   Specify a dark calibration FITS file to subtract 
    15 15         j*peg=   [false] Output full sized JPEGs by name of each extension 
    16 16       jmosaic=   [false] Generate a mosaic JPEG of all cells/extensions 
    17 17        jthumb=   [false] Generate thumbnail JPEGs (1/8 size) of everything 
    18 18        jnoise=   [false] Generate RMS noise diag JPEG mosaic of full OTA 
    19 19        jc*map=   [gamma] Colormap (gamma, rainbow, terra, or stoplight) 
    20 20        jgamma=   (Obsolete, ignored.) 
    21 21     jq*uality=   [95] JPEG quality factor (1 to 100) 
    22 22          jout=   JPEG file name format string (e.g. o5432g01234o%s.jpeg) 
    23 23      ext*name=   Specify a single extension to extract, by its EXTNAME 
    24 24            x0=   Corner pixel x-coord of a cropping box. 
    25 25            y0=   Corner pixel y-coord of a cropping box. 
    26 26            xs=   Size in x of a region to crop. 
    27 27            ys=   Size in y of a region to crop. 
    28 28       prescan=   Override any row-prescan value in the FITS files 
    29 29     bias*norm=   [true] Normalize all overscan bias levels at 250 ADU 
    30 30    fancy*bias=   [false] Custom bias filter (not valid for science!) 
    31 31       ped*sub=   [true] Subtract pedestal if present (true/false) 
    32 32      gaussian=   [0.0] If >0, apply a blur of that FWHM (before scaling) 
    33 33    cut*levels=   [auto,auto] Force low and high ADU cut levels 
    34 34     ramp*jpeg=   [off] Just output a color-bar intensity ramp 
    35 35          flip=   [false] Flip image top-to-bottom 
    36 36    comp*osite=   [false] Convert from a MEF to a basic, composite FITS 
    37 37        median=   [false] Display median values of all cells 
    38 38         stats=   [false] Generate pedestal, bias, sky level and noise stats 
    39 39  ponstats*file=   Text input file with XRAY (gain) and LED (linearity) stats 
    40 40  gain*correct=   [false] Apply a gain correction to JPEG, from ponstats XRAY 
    41 41  up*date_keywords=   [false] Attempt to update FITS keywords with stats 
    42 42     statsfile=   Text file to create which contains statistics 
    43 43     cplotfile=   Create color plot from existing stats file 
    44 44      cplotcam=   Name of the camera when plotting whole focalplane (e.g. "gpc1") 
    45 45      cplotccd=   Name of the ccd, such as "ota34" 
    46 46      cplotvar=   [rmsov] Variable to color-plot 
    47 47  (Do not type the *.  Letters after the * are optional.) 
    48 48  }}} 
    49 49  === Dark Subtraction === 
    50 50  Some day, '''otatool''' is supposed to be able to do a scaled DARK subtraction from the image before generating JPEGs so they look nicer, but this is '''untested and probably not working correctly.''' 
    51 51  ||'''option'''||'''description''' 
    52 52  ||dark=darkmef.fits||. 
    53 53  === True Color JPEG Generation === 
    54 54  {{{ 
    55 55  #!html 
    56 56  <A HREF=http://www.stargrasp.org/attachment/wiki/GraspExtrasSoftwareOtatool/gpc1_by_ucam.jpeg?format=raw><IMG ALIGN=RIGHT WIDTH=300 HEIGHT=300 BORDER=1 SRC=http://www.stargrasp.org/attachment/wiki/GraspExtrasSoftwareOtatool/gpc1_by_ucam_thumb.jpeg?format=raw></A> 
    57 57  }}} 
    58 58  The '''otatool''' utility can generate a realistic JPEG image of a scene if provided with FITS files with neutral filter, red, and blue (without anything in the scene changing.)  This functionality has '''not been used or tested recently and probably does not work correctly.'''  This just documents the options that belong with this mode of operation. 
    59 59  ||'''option'''||'''description''' 
    60 60  ||open=mef0.fits||. 
    61 61  ||blue=mef1.fits||. 
    62 62  ||red=mef2.fits||. 
    63 63  Click image to load full resolution, and then (in most browsers) click again to zoom in to 1:1. 
    64 64  The example shown here reveals a few pitfalls.  Using clear/blue/red instead of red/green/blue means the neutral filter image will always saturate first unless you reduce its exposure time.  We didn't do anything like that when generating this example, and one cell of the OTA showed non-linear behavior before the others, so that is why one cell appears greenish.  Also, nothing in the setting should move between the images.  Some papers on the table moved slightly, resulting in the pink areas. 
    65 65  === False Color JPEG Generation === 
    66 66  More commonly, we use '''otatool''' to generate a false color image of a single FITS frame.  That mode of operation produces JPEG files with fixed names, so we run it in a separate subdirectory and then rename those output files.  (In the future, we may add another command line option to control the output names.)  Typical invocation for this mode of operation is: 
    67 67  {{{ 
    68 68  % otatool jpeg=true < mef.fits 
    69 69  % ls 
    70 70  < ... produces *.jpeg where "*" is the EXTNAME found in the FITS file. > 
    71 71  }}} 
    72 72  ||'''option'''||'''description'''|| 
    73 73  ||jpeg=true   ||Cause full 1:1 JPEGS to be generated for each image extension|| 
    74 74  ||jout=basename||Select basename for JPEG files instead of "bin8.jpeg", "bin64.jpeg", bin64ns.jpeg", etc.|| 
    75 75  ||jmosaic=true||.|| 
    76 76  ||jthumb=true ||.|| 
    77 77  ||jcmap=      ||Choose between colormaps called '''gamma''' (the default) '''rainbow''', and '''terra'''.  The '''gamma''' colormap is special because it modulates the brightness (using the '''rainbow''' colormap as a start) so that the steepest changes in brightness occur at the median of the image (see cutlevels=, below.) || 
    78 78  If used, the '''jout''' option must be a format string containing exactly one "%s".  The %s will be replaced with a 2-letter code that is unique to each of the JPEG products otatool produces: 
    79 79  ||'''two-letter code'''||'''Description''' 
    80 80  ||b8||Binned by 8 false color image 
    81 81  ||b6||Binned by 64 false color image 
    82 82  ||ns||Overscan noise levels color plot 
    83 83  ||sm||DO_STREAKY mode only: Auto-generated mask image 
    84 84  ||sr||DO_STREAKY mode only: RMS-row noise color plot 
    85 85  ||sc||DO_STREAKY mode only: RMS-tilt noise color plot 
    86 86  ||st||DO_STREAKY mode only: RMS-plane noise color plot 
    87 87  === Statistics Generation === 
    88 88  You can use '''otatool''' to calculate the RMS of the overscan, or a part of the image in each extension and report the results to a "stats" text file.  This can happen at the same time as JPEG generation, or instead.  (Just specify all the relevant options and otatool will carry out the operations requested.)  Options related to statistics calculation are: 
    89 89  ||'''option'''||'''description''' 
    90 90  ||stats=true  ||Generate statistics 
    91 91  ||statsfile=foo.txt||Where to write the calculated numbers 
    92 92  ||prescan=    ||Normally, our practice is to remove prescan with the controller (using clv prescan=3 or 4 for OTAs, or prescan=50 for e2v for example.)  If the image still contains prescan pixels, you must tell otatool how many there are with this option so it can skip them. 
    93 93  Here is a description of the statistics values calculated in the stats files: 
    94 94  ||'''stats var'''||'''description''' 
    95 95  ||ped=           ||If a readout was performed with "bpp=32" and the FITS file was '''save'''d with "bpp=32" then otatool can report on the mean pedestal level in the pedestal image.  Usually, the pedestal image is subtracted from the data before saving but this can be a good diagnostic.  Pedestal levels should be within a few 100 ADU of 5000 (or whatever value you use for "readcal".) 
    96 96  ||bias=          ||Measures the bias level, in ADU.  '''This assumes otatool is looking in the right place for overscan.  See "prescan=" above, and notes below.''' 
    97 97  ||back=          ||Measures the mean level (above bias) in a portion of the image area.  The portion used is currently hardcoded into otatool.c 
    98 98  ||rmsped=        ||If you have a 32bpp STARGRASP FITS file (see ped= above) then this statistic reports the RMS in ADU of the pedestal.  Expect this to be quite high in many applications.  This is the uncorrelated sampling noise. 
    99 99  ||rmsvid=        ||If you have a 32bpp STARGRASP FITS file (see ped= above) then this statistic reports the RMS in ADU of the uncorrelated video sample.  Like rmsped=, it could also be quite high (100s of ADU sometimes) but should be similar to the rmsped value. 
    100 100  ||rmsov=         ||This is the RMS of the subtraction of (vid-ped) which is the 16-bit value normally delivered as the image, for pixels in the serial overscan region.  '''See notes below.''' 
    101 101  Additional RMS values for image regions (intended to be used on BIAS readouts only) are available if otatool has been compiled with "-DDO_STREAKY" in the Makefile.  These stats are: 
    102 102  ||'''stats var'''||'''description''' 
    103 103  ||rmsmask=       ||The RMS in ADU of part of the image (region used is hardcoded in otatool.c) after masking out any bright chip defects. 
    104 104  ||rmsrow=        ||The RMS in ADU of the same region, after applying a "row-by-row" bias correction to each pixel row individually. 
    105 105  ||rmsplane=      ||The RMS in ADU after the above treatment by rmsrow, but after fitting a plane to the image to take out any overall horizontal slope. 
    106 106  ||rmstilt=       ||The RMS in ADU after fitting a sloped bias correction to each individual row of pixels. 
    107 107  '''NOTE:''' The region which '''otatool''' uses for "overscan" statistics and the window where the rmsmask/row/plane/tilt are taken are fragile, and hardcoded into the program!  Most commonly, they can be confused by incorrect "prescan" setting.  '''otatool''' is usually only used on data that contains no serial prescan pixels!  The STARGRASP controller will read them and not transmit them if the '''clvset''' clocking pattern is set up with '''prescan=N'''. 
    108 108  === Diagnostic Color Plot ("cplot") JPEG Generation === 
    109 109  Fruity colored plots of OTAs and specific OTA-camera focal planes can also be generated by otatool.  Instead of requiring a FITS for input, any file in the format of otatool's own "stats" file can be used.  The following options are related to this function: 
    110 110  ||'''option'''||'''description'''|| 
    111 111  ||cplotfile=  ||This is the INPUT text file that otatool will read.  It is expected to have a minimum of "ccd=otaXY ext=xyXY ..." on each line.|| 
    112 112  ||cplotcam=   ||Currently, only a value of "gpc1" is supported here.  This generates plots in PS1-standard "camera coordinates" for a focalplane of 60 OTAs|| 
    113 113  ||cplotccd=   ||Omit this for entire focalplane plots.  Set it to something like "cplotccd=ota43" to plot just one OTA|| 
    114 114  ||cplotvar=   ||Set this to the variable in the cplotfile you want to plot.  If you were to plot bias levels, for example, use "cplotvar=bias" and make sure you feed otatool a cplotfile with (at least) ccd=... ext=... bias=... on every line.|| 
    115    ||cut=X,X,Y   ||The this to the lowest value, median value, and highest value of cplotvar you wish to be able to plot.  Normally, set median value to the same as lowest value for these types of plots unless you really think it is useful for otatool to modulate the brightness of the colors in a way to create more contrast around the median.  (Usually, that is only useful for rendering the image data itself, but you have to specify the median so set it to the same as the low cut level.)|| 
      115  ||cut=X,X,Y   ||Set this to the lowest value, median value, and highest value of cplotvar you wish to be able to plot.  Normally, set median value to the same as lowest value for these types of plots unless you really think it is useful for otatool to modulate the brightness of the colors in a way to create more contrast around the median.  (Usually, that is only useful for rendering the image data itself, but you have to specify the median so set it to the same as the low cut level.)|| 
    116 116  ||jcmap=      ||See above.  All the same colormaps are available as for image rendering.  Try "jcmap=rainbow"|| 
    117 117  Currently, there is no way to override the output file produced in this mode.  It is always '''"cplot.jpeg"''' in the current directory. 
    118 118  ==== Example of "cplot" ==== 
    119 119  Suppose there is a typical GPC1 stats file in /tmp/stats and we want to plot bias levels across the whole focalplane to see if there are any interesting trends.  We notice that bias levels fall between 5000 and 10000 ADU, so we'll fix the color stretch to this.  The command to produce a JPEG takes only about 1 second to execute, and looks like this: 
    120 120  {{{ 
    121 121  otatool cplotfile=/tmp/stats cplotcam=gpc1 cplotvar=bias cut=5000,5000,10000 jcmap=rainbow 
    122 122  mv cplot.jpeg biasstats.jpeg 
    123 123  }}} 
    124 124  The output is a binned-by-64 view of the GPC1 focalplane with every cell colored a particular color based on its bias level.  As long as we use an appropriate scaling algorithm, such as !ImageMagick's Box filter, it can be reduced in size quite a bit and still convey the same information.  This is accomplished with this command: 
    125 125  {{{ 
    126 126  mogrify -filter Box -geometry 208x208 -quality 95 biasstats.jpeg 
    127 127  }}} 
    128 128  208x208 is a magic size that causes the Box filter to place nicely with the cell spacing the otatool produces for gpc1.  If you prefer a larger, sharp image, you can skip the scaling step, but it only takes a few milliseconds to execute.  The resulting image looks like this: 
    129 129  [[Image(biasstats.jpeg)]] 
    130 130  To the uninitiated, all the pretty colors may not mean very much, so it might be handy to produce a legend or color scale to go with the image.  You can use otatool for this as well: 
    131 131  {{{ 
    132 132  otatool rampjpeg=biasramp.jpeg cut=5000,5000,10000 jcmap=rainbow 
    133 133  }}} 
    134 134  and this produces the following image: 
    135 135  [[Image(biasramp.jpeg)]] 
    136 136  Finally, you can use !ImageMagick some more to apply text labels and tile the whole thing together.  Here are example commands that can be modified to do just about anything that !ImageMagick supports. 
    137 137  {{{ 
    138 138  mogrify -quality 95 -geometry '!208x16' biasramp.jpeg 
    139 139  montage -quality 95 -geometry 208x208+1+1 +frame +shadow +label -tile 3x3 -background '#226633' \ 
    140 140   -gravity North NULL: NULL: NULL: NULL: biasstats.jpeg NULL: NULL: biasramp.jpeg NULL: biasplot.jpeg 
    141 141  mogrify -quality 95 -shave 208x190 biasplot.jpeg 
    142 142  mogrify -quality 95 -fill '#bbbbbb' -pointsize 17 -draw "text 6,18 \"Title: BIAS Levels\"" \ 
    143 143   -fill '#ffffff' -pointsize 14 -draw "text 6,244 \"5000\"" -fill '#000000' \ 
    144 144   -draw "text 168,244 \"10000\"" biasplot.jpeg 
    145 145  }}} 
    146 146  This produces the resulting labelled map of the entire focal plane: 
    147 147  [[Image(biasplot.jpeg)]]