Merge pull request #1704 from pypeit/scattlight_model

Scattered light model

doc/api/pypeit.core.ref_index.rst

@@ -0,0 +1,8 @@
+pypeit.core.ref\_index module
+=============================
+
+.. automodule:: pypeit.core.ref_index
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:

doc/calibrations/align.rst

@@ -47,14 +47,14 @@ the spatial alignment.
 Trouble Shooting
 ================
 
-If your image appears to be in err, here are the things to consider:
+If your image appears to be in error, here are the things to consider:
 
  - Is one or more of your input alignment frames junk?
 
  - Check each of your input alignment frames. Some might be mislabeled.
 
  - If some alignments are apparently missing, you might want to check
-   that the slits are currently traced.
+   that the slits are correctly traced.
 
 Current Alignments Data Model
 =============================

doc/calibrations/scattlight.rst

@@ -0,0 +1,136 @@
+
+.. include:: ../include/links.rst
+
+.. _scattlight:
+
+===========================
+Scattered light subtraction
+===========================
+
+Overview
+========
+
+This document describes how PypeIt implements a scattered light subtraction.
+Scattered light usually affects data at the few percent level or even less.
+So, before you implement a scattered light correction for your data, convince
+yourself that this is really needed for your science case. There is currently
+one algorithm that can be applied to all spectrographs, although this model
+has only been tested with Keck/KCWI and Keck/ESI data, and may not be optimal
+for your instrument setup, so inspect the outputs to be sure that you are not
+adding artefacts to your data.
+
+Overview
+========
+
+This file describes the model fitting procedure and how to check the results of your
+scattered light fits. The input image used should generally be a bright object that
+does not fill the slit. A bright standard star or a series of alignment frames are
+a good choice.
+
+The image is written to disk as a multi-extension FITS file
+prefixed by ``ScatteredLight`` in the ``Calibrations/`` folder.
+See :ref:`calib-naming` for the naming convention.
+
+Model Implementation
+====================
+
+The scattered light model is derived from regions of the
+detector that are outside of the slit boundaries. The slit edges are padded (assuming 1x1 binning)
+by a default integer value so that bleeding light from the edges of the slits minimally
+impacts the scattered light modelling procedure. If you would like to change the default amount
+of padding, you can set the number of padded pixels by the following parameter (the following
+example will pad the slits by 10 detector pixels, assuming 1x1 binning):
+
+.. code-block:: ini
+
+    [calibrations]
+        [[scattlight]]
+            pad = 10
+
+Note: if your data are binned by 2 in the spatial direction, then setting ``pad=10`` would actually
+pad the slit edges by 5 pixels.
+
+The current scattered light model involves a 14+ parameter fit to the inter-slit regions. The model
+utilises the observed 2D (trimmed+oriented+debiased) frame to estimate the scattered light. The
+scattered light model is essentially a blurred, shifted, flux-scaled, rotated, and zoomed version of the
+detector image. The amount of blurring in the spatial and spectral direction include contributions from
+a Gaussian and a Lorentzian kernel.
+The relative importance of these kernels is controlled by scaling parameter (1 paramerer),
+and each kernel has two widths each (one for each direction on the detector; 4 parameters total),
+and the joint kernel is rotated (1 additional parameter).
+The scattered light image is than shifted in the spectral and spatial directions (2 additional parameters),
+and there is a parameter to estimate the zoom factor in the spectral and spatial directions (2 parameters).
+Finally, the scattered light flux is scaled by a low order 2D polynomial, including:
+(i) 2 parameters in the spatial direction (a linear term and a cross-term)
+``par1 * spatial + par2 * spatial*spectral``; and
+(ii) N (where N>=1) parameters for the spectral dependence.
+``par3 + par4 * spectral + par5 * spectral^2 + ...``.
+
+Once the best-fitting model parameters are determined, these can be applied to any
+other frame to estimate the scattered light contribution of this frame.
+Alternatively, the scattered light can be determined for each frame independently (i.e. you can
+derive a separate set of 14+ model parameters for each frame that you wish to perform a scattered
+light correction). To set this option, you can add the following arguments to your pypeit file:
+
+.. code-block:: ini
+
+    [calibrations]
+        [[FRAMETYPE]]
+            [[[process]]]
+            scattlight_method = frame
+
+You can also set ``scattlight_method = archive``, which will use pre-determined archival values
+for the spectrograph that you are trying to reduce data for. However, note that not all spectrographs
+have this feature implemented, so this is not a good idea in general.
+
+Inspecting
+==========
+
+There is a dedicated PypeIt script to check the scattered light model fit:
+
+.. code-block:: console
+
+    pypeit_chk_scattlight Calibrations/ScatteredLight_A_0_DET01.fits.gz Calibrations/Slits_A_0_DET01.fits.gz
+
+Where the first and second arguments are the ``ScatteredLight`` and ``Slits`` calibration frames.
+A three panel ginga window will be opened that shows:
+(1) the data frame used to determine the scattered light model parameters;
+(2) the model of the scattered light; and
+(3) The model subtracted from the data (i.e. data-model).
+All panels are re-oriented so that vertical is the spectral dimension with
+blue at the bottom, following the PypeIt :ref:`pypeit-orientation` convention.
+
+Here is an screen shot of a ``ginga`` view of an example from
+the ``keck_kcwi`` spectrograph (BL grating).
+The left image shows the raw data,
+the middle image shows the estimated scattered light, based on a model fit, and
+the right image shows the raw frame with the scattered light subtracted off.
+All images are shown on the same color scale.
+
+.. image:: ../figures/scatteredlight_image.png
+
+Trouble Shooting
+================
+
+If your scattered light model appears to be in error, here are some things to consider:
+
+ - Does the scattered light input frame have enough pixels between the
+   slits to accurately pin down the scattered light contribution?
+
+ - Check your chosen scattered light frame. It might be mislabeled. It's
+   often best to choose one really good scattered light frame, unless you
+   are confident that all frames labelled ``scattlight`` closely resemble
+   one another
+
+ - Check your slit tracing. If some slits are apparently missing, you might
+   want to check that the slits are correctly traced.
+
+Current Scattered Light Data Model
+==================================
+
+The datamodel written to disk is:
+
+.. include:: ../include/datamodel_scattlight.rst
+
+The `astropy.io.fits.BinTableHDU`_ contains all the data from the underlying
+:class:`~pypeit.scattlight.ScatteredLight` class.  For details see its datamodel.

doc/releases/1.14.1dev.rst

@@ -59,6 +59,8 @@ Script Changes
 - When making datacubes, the user can select a separate frame to use for the sky subtraction.
   In this case, it is the processed data that will be used for sky subtraction (akin to nodding).
 - Allow a list of files to be passed to ``pypeit_chk_wavecalib``.
+- A new script has been written (``chk_scattlight``) to check the generation of the scattered
+  light calibration frame model.
 
 Datamodel Changes
 -----------------

doc/scripts/build_datacontainer_datamodels.py

@@ -67,6 +67,7 @@ def build_datamodel_tbl(obj):
     from pypeit.flatfield import FlatImages
     from pypeit.manual_extract import ManualExtractionObj
     from pypeit.onespec import OneSpec
+    from pypeit.scattlight import ScatteredLight
     from pypeit.sensfunc import SensFunc
     from pypeit.slittrace import SlitTraceSet
     from pypeit.spec2dobj import Spec2DObj
@@ -86,7 +87,7 @@ def build_datamodel_tbl(obj):
 
     from pypeit.images import buildimage
 
-    datacontainers = [Alignments, EdgeTraceSet, FlatImages, ManualExtractionObj, OneSpec,
+    datacontainers = [Alignments, EdgeTraceSet, FlatImages, ManualExtractionObj, OneSpec, ScatteredLight,
                       SensFunc, SlitTraceSet, Spec2DObj, SpecObj, TracePCA, WaveCalib, WaveTilts,
                       bspline, DataCube, PypeItFit, MultiSlitFlexure, Telluric, DetectorContainer,
                       Mosaic, PypeItImage, WaveFit]

doc/spectrographs/keck_kcwi.rst

@@ -112,11 +112,16 @@ Scattered Light Removal
 KCWI suffers from mild scattered light (at the level of ~1 percent),
 and this appears to be worse near regions of the detector where there
 is brighter illumination. We are currently working towards building a
-full model of the scattered light. For the moment, PypeIt uses a robust
-piecewise polynomial to model the scattered light that is detected on
-the left of slice 1, the unilluminated region between slices 12 and 13,
-and the right of slice 24. The model is smooth and continuous, and is
-determined for each spectral pixel. By default, the scattered light is
+full model of the scattered light. For KCWI, the main contributor to
+the scattered light is referred to as the "narcissistic ghost" by
+Morrissey et al. (2018), ApJ, 864, 93. This scattered light is thought
+to be a reflection off the detector that travels back through the optical
+system. Some fraction gets sent back out to space, while the remainder
+comes back through the optical system and a fuzzy version of this is
+re-imaged onto the detector. The current KCWI scattered light model is
+designed to account for these effects. To generate a scattered light model,
+it's a good idea to use a frame that has a lot of counts (e.g. a flatfield
+frame, or a standard star). By default, the scattered light is
 subtracted from the science frame, the pixel flat, and the illumination
 flat. To turn off the scattered light subtraction, you can add the
 following lines to your :ref:`pypeit_file`: