Next: Status and Goals of
Up: NICMOS Data Calibration and
Previous: NICMOS Data Calibration and
H. Bushouse, C. Skinner, J. MacKenty, and D. Axon
Space Telescope Science Institute, 3700 San Martin Drive,
Baltimore, MD 21218, Email: firstname.lastname@example.org
U. of Arizona, Steward Observatory, Tucson, AZ 85721
The Space Telescope Science Institute (STScI) data reduction and calibration
pipeline for the Near-Infrared Camera and Multi-Object Spectrometer (NICMOS)
is composed of two major tasks, CALNICA and CALNICB. CALNICA applies
instrumental calibration to all images, while CALNICB applies specialized
processing to sets of images associated with a single astronomical target.
This paper describes some of the technical aspects of the NICMOS data file
formats, the CALNICA and CALNICB software, as well as details and examples
of the processing that these tasks perform.
NICMOS, calibration, pipeline, software
The STScI NICMOS calibration pipeline is composed of two major tasks, CALNICA
and CALNICB. CALNICA applies instrumental calibration to all images and includes
basic corrections such as dark current subtraction, detector non-linearity
correction, and flat-fielding. CALNICA operates on one image at a time.
It takes as input the raw science data files produced by the Generic Conversion
process in the STScI OPUS environment. Its outputs are fully calibrated files
and, for MultiAccum observations only, an intermediate calibrated file.
CALNICB is only applied to associated NICMOS observations and is executed
after all images in an association have first been calibrated with CALNICA.
CALNICB combines images into mosaics and also removes background signal from
the images. Its inputs are an association table, which contains information
about the images in the association, and the calibrated images produced by
CALNICA. Its outputs are an updated version of the association table and
background-subtracted mosaic images, produced by combining the associated
Both CALNICA and CALNICB are built as host-level tasks and can therefore
be executed directly from the host operating system level (e.g. Unix or VMS),
or they can be executed from within the IRAF environment in the STSDAS
hst_calib.nicmos package. The tasks are completely data driven in that
they derive all information that is necessary to guide the processing from
values of the input image header keywords and, in the case of CALNICB, the
association table. Some user-setable parameters are available for tailoring
reprocessing efforts. These parameters use default values when operating in
the OPUS pipeline.
The run-time file format for all data I/O is FITS files with IMAGE and BINTABLE
extensions. All NICMOS images are represented by five data arrays, stored as
five IMAGE extensions in a single FITS file. The group of five data arrays is
known as an image set, or ``imset". MultiAccum-mode exposures, which produce
multiple intermediate detector readouts during the course of an exposure, have
multiple imsets stored in the FITS files. Each imset is comprised of the
science image (extension name ``SCI''), a statistical error image (``ERR''),
a data quality flag array (``DQ''), an array giving the number of data samples
at each pixel (``SAMP''), and an exposure time per pixel array (``TIME'').
All input and output files conform to the same file structure and organization,
which allows output files to be reused as input if desired. All data processing
steps involving reference data (e.g. dark and flat-field images) propagate
statistical uncertainties from the ERR arrays and data quality flags from the
DQ arrays of the reference data into the science data being processed.
Furthermore, all steps involving image combination propagate the number of
samples (SAMP) and their total exposure time (TIME) used to compute each
resulting science (SCI) image value.
The overall flow of CALNICA processing is shown in Figure 1.
CALNICA Processing Flow.
Included in this figure are the names of the image header keywords that
are read by CALNICA to determine the names of reference files (left column)
and the names of the keyword ``switches" used to turn each step on or off.
CALNICA applies basic instrumental calibrations to individual NICMOS
observations. For MultiAccum-mode observations, the processing steps are
repeated for each imset in the input file. The most significant calibration
steps are the subtraction of dark current, correction for non-linear
photometric response, and correction for pixel-to-pixel gain variations
(flat-fielding). MultiAccum-mode observations receive some specialized
processing, including correction for source signal present in the initial
or ``zeroth" detector readout, subtraction of the zeroth readout from all
subsequent readouts, and combination of the multiple readouts into a single
image, which includes cosmic-ray and bad pixel rejection. Details of some of
these steps are discussed below. For details regarding all of the individual
processing steps see Bushouse (1997a, 1997b).
The subtraction of dark current is a critical step in processing NICMOS data
because it actually includes corrections for several instrumental effects
(see Skinner, Bergeron, & Daou 1997; Bergeron & Skinner 1997).
The actual detector dark current is quite small for all three NICMOS cameras,
being at most 0.05 e-/sec. There are two other much larger instrumental
signatures superimposed on the darks. The first is a bias level or DC offset
which exhibits a signal gradient across each quadrant of an image. This is
the so-called ``shading", which is a noiseless signal than can reach levels
of tens or even hundreds of DN (data numbers). The other effect is known as
amp glow and appears as an elevated signal in the corners of NICMOS images.
This signal is due to a small amount of IR radiation emitted by the readout
amplifiers, which are situated near the detector corners.
Fortunately, both the shading and the amp glow signal levels are very
repeatable, and depend variously upon the number of readouts and the time
since the last detector reset. Hence a proper subtraction can be achieved by
applying dark current reference images having the exact same readout sequence
and exposure times. CALNICA automatically selects the appropriate imset from
the dark current reference file that has an exposure time matching that of
the science data being processed.
This step corrects for the non-linear response of the detectors. The response
can be conveniently divided into three regimes. In the low signal regime, the
detector response is linear, hence no correction is applied. In the mid-level
regime, the response deviates from true linearity in a way that is correctable
using a first-order polynomial of the form
SCI(corrected) = (c1 + c2*SCI) * SCI
where c1 and c2 are the polynomial coefficients.
The coefficients are stored, on a pixel-by-pixel basis, in the ``COEF" and
``ERR" image extensions in the NLINFILE reference file.
In the high signal regime, where pixels begin to saturate, no correction is
applied to the data, but a saturation flag value is set in the DQ image.
The DN levels defining the boundaries between the regimes are also stored
on a pixel-by-pixel basis in two ``NODE" image extensions in the NLINFILE.
The linear regime typically extends up to 14500 DN, and the non-linear
regime to 27500 DN, above which saturation begins to occur.
Because NICMOS does not have a shutter, signal from bright sources can
accumulate during the 0.2 second long period that elapses between
resetting the detector and taking the initial, or zeroth, readout that
begins an exposure. Any signal present in the zeroth read will be removed
when the zeroth read image is subtracted from all subsequent readouts by
CALNICA, but this can lead to an error in the linearity correction that is
applied to a given pixel, since the linearity correction depends on the total
amount of charge that accumulated in the pixel.
The ZSIGCORR step of CALNICA computes the count rate for each pixel by
differencing the zeroth and first readouts from one another and then
estimates the amount of signal that would have accumulated in the zeroth
read during its 0.203 second effective exposure time. This zeroth read signal
is carried through to the NLINCORR step, where it is added back into the
signal level for each pixel before determining its correction.
After applying the linearity correction, the zeroth read signal is again
Once all the individual readouts of a MultiAccum observation have received
all the necessary instrumental calibration steps, they are combined into a
single image, identifying and rejecting both cosmic-ray hits and bad pixels
in the process. The accumulated counts as a function of increasing exposure
time for each pixel are fitted with a first-order polynomial, using a standard
weighted least-squares algorithm. Readouts already flagged as bad in the DQ
array (e.g. due to saturation) are not used in the fit. The distance of each
readout value from the fit is computed and cosmic-rays are identified by their
characteristic negative-to-positive residual transition from one readout to
the next. Readouts containing a hit are rejected and the fit is recomputed.
This process is repeated until no new hits are identified. The slope of the fit
gives the final count rate for the pixel. The total number of non-rejected
readouts and their exposure time are recorded in the SAMP and TIME arrays of
the final output image.
Figure 2 shows some examples of the MultiAccum fitting process.
MultiAccum Ramp Fitting Examples.
The top panel shows the results for a pixel that had a cosmic-ray hit.
The open circles are the original data values, while the filled circles are
the result of removing the signal from the readout containing the CR hit.
The bottom panel shows a pixel that has the last two readouts flagged
as saturated and therefore not used in the fit.
The overall flow of CALNICB processing is shown in Figure 3.
CALNICB Processing Flow.
CALNICB is used exclusively for processing associated sets of images, all of
which must have already been calibrated using CALNICA. A NICMOS image
association is defined to be a logical grouping of multiple observations
of a single astronomical target. Associations are typically used for:
To make it easier for observers to plan dithered and chopped sets of
observations, STScI created several predefined observing patterns which
automatically dither or chop the telescope field of view through a grid of
observing positions on the sky (see MacKenty et al. 1997, chapter 10).
making multiple exposures at a single sky position so that cosmic-ray
rejection can be performed by later combining the images
dithering the telescope field of view to remove the effects of bad pixels or
flat field residuals
building image mosaics of large angular-size targets
chopping the field of view off of the target to measure the telescope thermal
The major processing steps of CALNICB include the combination of images taken
at individual sky positions, the measurement and subtraction of the thermal
background signal, and the mosaicing of images within an observing pattern.
The following sections describe some of these operations in a bit more detail.
For more information, see Bushouse (1997a, 1997c).
CALNICB, as run in the STScI pipeline, is completely data driven, getting all
the information it needs to guide the processing from the association table
that accompanies the individual images, and from header keywords in the images
Some user-setable task parameters are available in CALNICB versions 2.2 and
later, for use when running CALNICB interactively (see below).
The association table (denoted ``ASN") is a FITS file with a BINTABLE extension
containing the list of file names that make up the association to be processed
by CALNICB, as well as the file names of all output mosaic image files
(denoted ``MOS") to be produced.
CALNICB determines which names in the list are input files and which are output
files based on the MEMTYPE column in the ASN table. Input files have a
MEMTYPE prefix of ``EXP" (exposure), while output files have a prefix of
``PROD" (products). Furthermore, association members that are observations
of the target of interest have a MEMTYPE suffix of ``TARG" (target), while
observations of sky background locations (which are only produced by using
chop patterns) have a suffix of ``BCKn", where n is the background region
number in the pattern (see Table 1). One MOS image will always
be produced for the target, and chop patterns will result in one MOS image
being produced for each background region observed.
Once CALNICB has determined the list of input images from the ASN table, it
reads various keywords from the image headers to determine exactly
what processing needs to be performed. Certain keywords, such
as PATTERN (pattern type), NUMPOS (number of pattern positions), and NUMITER
(number of exposure iterations at each position) have the same value in all
images and are therefore only read from the first image in the association.
Others are image specific, such as PATT_POS (position number of each image in
the pattern) and World Coordinate System (WCS) keywords, have unique values in
each image and are therefore read from each image.
From this information CALNICB determines what type of pattern was used, how
many positions are in the pattern, how many images there are at each
pattern position, the relative positions of each image in the pattern, how
many output MOS images need to be produced, and which input images go into
which MOS image.
If there is more than one image at each pattern position, CALNICB will
combine the images at each position into a single image. The offsets
between each image at a given position are computed from their WCS information,
using the first image at each pattern position as a reference. These
offsets are then refined using a cross-correlation technique. The images are
then aligned with their reference image, using bilinear interpolation, and
then combined. The combining process computes the ERR-weighted mean
for each SCI image value, with rejection of DQ-flagged pixels and
iterative sigma clipping. The number of non-rejected samples, and their total
exposure time, are saved in the SAMP and TIME images.
This step flags (in the DQ images) pixels suspected of containing flux from
a source. This is accomplished using a 3-stage process. First, pixels that
are greater than 5 sigma above the median signal level in each image are
identified as candidate source pixels. Second, in order to filter out spurious
identifications, only those candidates having two or more neighboring pixels
also identified as containing a source are retained. Finally, the surviving
pixels have their source DQ flags ``grown" to their four immediate neighbors.
This step computes and subtracts a scalar (or ``DC") background signal from
all images in the association. This is accomplished as follows. First, the
median signal level in each BCK image in the association is computed. The
computation excludes bad and source-flagged pixels and uses iterative sigma
clipping to avoid any possible contamination. If there are not any BCK images
in the association (e.g. the observations are from a pure dither pattern),
the same computation is performed on the target images.
The resulting background levels for each image are then averaged to compute the
overall mean background level for the whole association. This overall mean
is then subtracted from all images. The mean background level is recorded in
the MEAN_BKG keyword in the output association (denoted ``ASC") table.
This step subtracts the ILLMFILE reference image from the associated images.
It is intended to remove any spatial variations in the telescope background
thermal pattern, but to date, little if any such spatial variations have
been seen. Thus the ILLMFILEs currently in use at STScI contain dummy data
(constant zero values). CALNICB recognizes that the files are dummy and skips
This step performs the final combination of images to produce the output
mosaics. The process is very similar to the step that combines the NUMITER
images at each pattern position. The relative offsets for each image in a
given mosaic are first computed from their WCS information, using the first
image in each mosaic as a reference. The offsets are refined using
cross-correlation, and then the images are aligned using bilinear interpolation
and combined. The combining process computes the ERR-weighted mean of
overlapping pixels, rejecting those with non-zero DQ values and applying
an iterative sigma clipping. The number of non-rejected samples and their
total exposure time are saved in the output SAMP and TIME images of the MOS
The final step of CALNICB creates a new association (ASC) table.
The ASC table is a copy of the input ASN table, with four new columns of
An example of an ASC table is shown in Table 1.
The new columns are BCKIMAGE, MEANBCK, XOFFSET, and YOFFSET.
The BCKIMAGE column indicates whether or not a given image was used in the
scalar background computation.
The example shown in Table 1 is for a TWO-CHOP pattern, with
NUMITER=2, and NUMPOS=5. So there are two images at each of the five pattern
positions and two of the positions are background (BCK) regions.
Thus, in this example, only the images at the BCK positions were used to
compute the scalar background level.
The MEANBCK column records the background signal computed for each individual
The XOFFSET and YOFFSET columns record the computed offsets for each image
relative to its reference image.
As mentioned above, beginning with CALNICB v2.2, there are a few parameters
available to interactive users for tailoring the CALNICB processing.
The subbkg parameter allows you to turn the scalar background subtraction
step on or off. The meanbkg parameter allows you to use the mean scalar
background value of all images, or subtract the value derived for each image
from itself. The readbkg and readoffsets parameters allow you to
use an ASC table as input to CALNICB and have it read either the image
background or offset (or both) values from the table, rather than computing
them. The crthresh parameter sets the cosmic-ray rejection threshold used
during image combining, and the xcwin parameter sets the size of the
cross-correlation search window box.
Next: Status and Goals of
Up: NICMOS Data Calibration and
Previous: NICMOS Data Calibration and