From 545033e1ef0daf480b24f58448dc603e1df805b7 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Thu, 31 Oct 2024 07:33:35 -0700 Subject: [PATCH 1/8] doc update --- doc/pypeit_par.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/pypeit_par.rst b/doc/pypeit_par.rst index 31464764d..668bc3d32 100644 --- a/doc/pypeit_par.rst +++ b/doc/pypeit_par.rst @@ -482,8 +482,8 @@ Key Type Options ``echelle_pad`` int .. 3 Number of orders by which to pad the echellogram reference in the echelle method. Values > 0 allow for some error in the reddest order guess, but require sufficient reference orders. ``frac_rms_thresh`` float .. 1.5 For echelle spectrographs (i.e., ``echelle=True``), this is the fractional change in the RMS threshold used when a 1D fit is re-attempted for failed orders. ``func`` str .. ``legendre`` Function used for wavelength solution fits -``fwhm`` int, float .. 4.0 Spectral sampling of the arc lines. This is the FWHM of an arcline in binned pixels of the input arc image -``fwhm_fromlines`` bool .. True Estimate spectral resolution in each slit using the arc lines. If True, the estimated FWHM will override ``fwhm`` only in the determination of the wavelength solution (including the calculation of the threshold for the solution RMS, see ``rms_thresh_frac_fwhm``), but not for the wave tilts calibration. +``fwhm`` int, float .. 4.0 Spectral sampling of the arc lines. This is the FWHM of an arcline in binned pixels of the input arc image. Note that this is used also in the wave tilts calibration. +``fwhm_fromlines`` bool .. True Estimate spectral resolution in each slit using the arc lines. If True, the estimated FWHM will override ``fwhm`` in the determination of the wavelength solution (including the calculation of the threshold for the solution RMS, see ``rms_thresh_frac_fwhm``), and ALSO for the wave tilts calibration. ``fwhm_spat_order`` int .. 0 This parameter determines the spatial polynomial order to use in the 2D polynomial fit to the FWHM of the arc lines. See also, fwhm_spec_order. ``fwhm_spec_order`` int .. 1 This parameter determines the spectral polynomial order to use in the 2D polynomial fit to the FWHM of the arc lines. See also, fwhm_spat_order. ``lamps`` list .. .. Name of one or more ions used for the wavelength calibration. Use ``None`` for no calibration. Choose ``use_header`` to use the list of lamps recorded in the header of the arc frames (this is currently available only for Keck DEIMOS, Keck LRIS, MMT Blue Channel, and LDT DeVeny). From 9540496b9bce6618280528b05336ceba7111a7b9 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Fri, 1 Nov 2024 07:48:50 -0700 Subject: [PATCH 2/8] bump release notes --- doc/releases/1.16.1dev.rst | 158 --------------------------------- doc/releases/1.17.0.rst | 175 +++++++++++++++++++++++++++++++++++++ doc/whatsnew.rst | 2 +- 3 files changed, 176 insertions(+), 159 deletions(-) delete mode 100644 doc/releases/1.16.1dev.rst create mode 100644 doc/releases/1.17.0.rst diff --git a/doc/releases/1.16.1dev.rst b/doc/releases/1.16.1dev.rst deleted file mode 100644 index e578099ab..000000000 --- a/doc/releases/1.16.1dev.rst +++ /dev/null @@ -1,158 +0,0 @@ - -Version 1.16.1dev -================= - -Installation Changes --------------------- - -- Significant expansion of PypeIt's use of the cache system; see - :ref:`data_installation`. Important changes include that most cached files - are now version dependent. When upgrading to new versions of PypeIt, users - should delete their cache and start fresh. - -Dependency Changes ------------------- - -- Support added for numpy>=2.0.0; numpy<=2.0.0 should still be supported -- Deprecated support for python 3.10 -- General update to dependencies to be roughly consistent with release of python - 3.11 - -Functionality/Performance Improvements and Additions ----------------------------------------------------- - -- Added the ``max_overlap`` parameter, which limits the set of new order traces - added, to compensate for orders missed during automated edge-tracing, to those - that have less than a given fractional overlap with adjacent orders. -- Added the ``order_fitrej`` and ``order_outlier`` parameters used to set the - sigma-clipping threshold used when fitting Legendre functions to the order - widths and gaps. -- Added the possibility to decide if the extracted standard star spectrum should be - used as a crutch for tracing the object in the science frame (before it was done as default). - This is done by setting the parameter ``use_std_trace`` in FindObjPar. -- Now PypeIt can handle the case where "Standard star trace does not match the - number of orders in the echelle data" both in `run_pypeit` and in - `pypeit_coadd_1dspec`. -- Now PypeIt can handle the case where "Standard star trace does not match the number of orders in the echelle data" - both in `run_pypeit` and in `pypeit_coadd_1dspec`. -- Added the functionality to use slitless flats to create pixelflats. Note: new frametype - `slitless_pixflat` is added to the PypeIt frametype list. -- The created pixelflats are stored in the reduction directory and in the PypeIt cache directory - `data/pixelflats`. -- Added a functionality that allows, when multiple frames are combined, to scale each - frame to have the same mean value before combining. To use this - functionality, the new parameter ``scale_mean`` should be set to ``True``. -- Added the possibility to use the parameter ``fwhm_fromlines`` also for the tilts calibration. - -Instrument-specific Updates ---------------------------- - -- Improved LRIS frame typing, including the typing of slitless flats and sky flats. -- Improved HIRES frame typing and configuration setup. -- Added support for Keck/KCWI BH3 grating configuration. -- Added support for the (decommissioned) AAT/UHRF instrument -- Updated the requirements of a spectral flip for KCWI (blue) data. If all amplifiers are - used, the data will be flipped. If all amplifiers are used, the data will not be flipped. - Otherwise, the data will be flipped. -- Updated X-Shooter detector gain and read noise to come from header, and - updated plate scales to the most recent values from the manual. Detailed - changes are: - - - NIR arm: - - - Platescale updated from 0.197 to 0.245 arcsec/pixel - - Dark current updated from 0. to 72. e-/pixel/hr - - Gain updated from 2.12 to 2.29 e-/DN - - - VIS arm: - - - Platescale updated from an order-dependent value, to being 0.154 - arcsec/pixel for all orders - - - UVB arm: - - - Platescale updated from an order-dependent value, to being 0.164 - arcsec/pixel for all orders -- Add new P200/DBSP reid_arxiv template for 1200/7100 with D55 dichroic - -Script Changes --------------- - -- Modifications to the cache-related :ref:`install_scripts` to accommodate - expansion of and changes to the cache system. -- Added ``pypeit_clean_cache`` script to facilitate both viewing and removing - files in the cache. -- Changed the name of the multi-dimensional specdata to specdata_multi in pypeit_identify - and improved the robustness of the saving dialog when calibrating single trace spectra. -- Fixed a read-in error for the high resolution A0V PHOENIX model. -- A new script, called `pypeit_extract_datacube`, allows 1D spectra of point - sources to be extracted from datacubes. -- The sensitivity function is now generated outside of datacube generation. -- The `grating_corr` column is now used to select the correct grating - correction file for each spec2d file when generating the datacube. -- Added the ``--extr`` parameter in the ``pypeit_sensfunc`` script (also as a SensFuncPar) - to allow the user to specify the extraction method to use when computing the sensitivity - function (before only optimal extraction was used). -- Added ``pypeit_show_pixflat`` script to inspect the (slitless) pixel flat - generated during the reduction and stored in ``data/pixelflats``. -- Added ``pypeit_chk_flexure`` script to check both spatial and spectral flexure applied to - the reduced data. -- Treatment of file names is now more formal. Compression signatures are now - considered, and filename matching is now more strict. -- Removed `--spec_samp_fact` and `--spat_samp_fact` command line options from - `pypeit_coadd_2d`. These options are now parameters in `Coadd2dPar`. -- `pypeit_show_2dspec` now shows the first available detector in the 2D spectrum - by default. The user can specify the detector to show with the `--det` option. -- Added `--removetrace` command line option to `pypeit_ql` to not show the object - trace when displaying the 2D spectrum. -- Change the default value for `--skip_display` in `pypeit_ql` to `True`. - - -Datamodel Changes ------------------ - -- Adjusted spec1d datamodel to enable use with UVES_popler GUI tool - -Under-the-hood Improvements ---------------------------- - -- Introduced :class:`~pypeit.pypeitdata.PypeItDataPaths` to handle all - interactions with the ``pypeit/data`` directory, which provides a unified - interface for accessing on-disk and cached files. -- When adding missing orders, the full syncing procedure is no longer performed. - The code now only checks that the edges are still synced after the missed - orders are added. -- When detecting overlapping orders/slits, the code now forces each edge used to - have been directly detected; i.e., if an edge is inserted, the fact that the - resulting slit is abnormally short should not trigger the overlap detection. -- Improved the QA plot resulting from fitting order widths and gaps as a - function of spatial position. -- Updated general raw image reader so that it correctly accounts for - spectrographs that read the data and overscan sections directly from the file - headers. - -Bug Fixes ---------- - -- Fix "The system cannot find the file specified" errors when installing on Windows. -- Fixed a fault caused when all frames in a pypeit file are identified as being - part of ``all`` calibration groups. -- Allow for empty 2D wavecal solution in HDU extension of WaveCalib file -- Fixed a bug in the ginga display function, when the user doesn't provide the `trc_name` argument. -- Fix a MAJOR BUT SUBTLE bug in the use of ``numpy.argsort``. When using ``numpy.argsort`` - the parameter kind='stable' should be used to ensure that a sorting algorithm more robust - than "quicksort" is used. -- Fix error "ValueError: setting an array element with a sequence. The requested - array has an inhomogeneous shape after 1 dimensions..." occurring when unpacking - the SpecObj spectrum but having an attribute of the SpecObj object that is None. -- Fixed an hidden bug that was causing the spatial flexure to fail. The bug was in the - `SlitTraceBitMask` class, where the function `exclude_for_flexure()` was not returning - the 'BOXSLIT' flag. -- Fix a bug in `pypeit_coadd_2d` related to how the binning was taken into account - in the mask definition, and in the calculation of the offset between frames. -- Fix bug when trying to open mosaic data from previous versions; version - checking flag was not being propagated. - - - - diff --git a/doc/releases/1.17.0.rst b/doc/releases/1.17.0.rst new file mode 100644 index 000000000..849790fb7 --- /dev/null +++ b/doc/releases/1.17.0.rst @@ -0,0 +1,175 @@ + +Version 1.17.0 +============== + +Installation Changes +-------------------- + +- Significant expansion of PypeIt's use of the cache system; see + :ref:`data_installation`. Important changes include that most cached + files are now version dependent. When upgrading to new versions of + PypeIt, users should delete their cache and start fresh. + +Dependency Changes +------------------ + +- Support added for numpy>=2.0.0; numpy<=2.0.0 should still be supported +- Deprecated support for python 3.10 +- General update to dependencies to be roughly consistent with release + of python 3.11 + +Functionality/Performance Improvements and Additions +---------------------------------------------------- + +- Added the ``max_overlap`` parameter, which limits the set of new order + traces added, to compensate for orders missed during automated + edge-tracing, to those that have less than a given fractional overlap + with adjacent orders. +- Added the ``order_fitrej`` and ``order_outlier`` parameters used to + set the sigma-clipping threshold used when fitting Legendre functions + to the order widths and gaps. +- Added the possibility to decide if the extracted standard star + spectrum should be used as a crutch for tracing the object in the + science frame (before it was done as default). This is done by + setting the parameter ``use_std_trace`` in FindObjPar. +- Now PypeIt can handle the case where "Standard star trace does not + match the number of orders in the echelle data" both in ``run_pypeit`` + and in ``pypeit_coadd_1dspec``. +- Added the functionality to use slitless flats to create pixelflats. + Note: new frametype ``slitless_pixflat`` is added to the PypeIt + frametype list. +- The created pixelflats are stored in the reduction directory and in + the PypeIt cache directory ``data/pixelflats``. +- Added a functionality that allows, when multiple frames are combined, + to scale each frame to have the same mean value before combining. To + use this functionality, the new parameter ``scale_mean`` should be set + to ``True``. +- Added the possibility to use the parameter ``fwhm_fromlines`` also for + the tilts calibration. + +Instrument-specific Updates +--------------------------- + +- Improved LRIS frame typing, including the typing of slitless flats and + sky flats. +- Improved HIRES frame typing and configuration setup. +- Added support for Keck/KCWI BH3 grating configuration. +- Updated the requirements of a spectral flip for KCWI (blue) data. If + all amplifiers are used, the data will be flipped. If all amplifiers + are used, the data will not be flipped. Otherwise, the data will be + flipped. +- Added support for the (decommissioned) AAT/UHRF instrument +- Updated X-Shooter detector gain and read noise to come from header, + and updated plate scales to the most recent values from the manual. + Detailed changes are: + + - NIR arm: + + - Platescale updated from 0.197 to 0.245 arcsec/pixel + - Dark current updated from 0. to 72. e-/pixel/hr + - Gain updated from 2.12 to 2.29 e-/DN + + - VIS arm: + + - Platescale updated from an order-dependent value, to being + 0.154 arcsec/pixel for all orders + + - UVB arm: + + - Platescale updated from an order-dependent value, to being + 0.164 arcsec/pixel for all orders + +- Add new P200/DBSP reid_arxiv template for 1200/7100 with D55 dichroic + +Script Changes +-------------- + +- Modifications to the cache-related :ref:`install_scripts` to + accommodate expansion of and changes to the cache system. +- Added ``pypeit_clean_cache`` script to facilitate both viewing and + removing files in the cache. +- Changed the name of the multi-dimensional specdata to + ``specdata_multi`` in ``pypeit_identify`` and improved the robustness + of the saving dialog when calibrating single trace spectra. +- Fixed a read-in error for the high resolution A0V PHOENIX model. +- A new script, called ``pypeit_extract_datacube``, allows 1D spectra of + point sources to be extracted from datacubes. +- The sensitivity function is now generated outside of datacube + generation. +- The ``grating_corr`` column is now used to select the correct grating + correction file for each spec2d file when generating the datacube. +- Added the ``--extr`` parameter in the ``pypeit_sensfunc`` script (also + as a ``SensFuncPar``) to allow the user to specify the extraction + method to use when computing the sensitivity function (before only + optimal extraction was used). +- Added ``pypeit_show_pixflat`` script to inspect the (slitless) pixel + flat generated during the reduction and stored in ``data/pixelflats``. +- Added ``pypeit_chk_flexure`` script to check both spatial and spectral + flexure applied to the reduced data. +- Treatment of file names is now more formal. Compression signatures + are now considered, and filename matching is now more strict. +- Removed ``--spec_samp_fact`` and ``--spat_samp_fact`` command line + options from ``pypeit_coadd_2d``. These options are now parameters in + ``Coadd2dPar``. +- ``pypeit_show_2dspec`` now shows the first available detector in the + 2D spectrum by default. The user can specify the detector to show with + the ``--det`` option. +- Added ``--removetrace`` command line option to ``pypeit_ql`` to not + show the object trace when displaying the 2D spectrum. +- Change the default value for ``--skip_display`` in ``pypeit_ql`` to + ``True``. + +Datamodel Changes +----------------- + +- Adjusted spec1d datamodel to enable use with UVES_popler GUI tool + +Under-the-hood Improvements +--------------------------- + +- Introduced :class:`~pypeit.pypeitdata.PypeItDataPaths` to handle all + interactions with the ``pypeit/data`` directory, which provides a + unified interface for accessing on-disk and cached files. +- When adding missing orders, the full syncing procedure is no longer + performed. The code now only checks that the edges are still synced + after the missed orders are added. +- When detecting overlapping orders/slits, the code now forces each edge + used to have been directly detected; i.e., if an edge is inserted, the + fact that the resulting slit is abnormally short should not trigger + the overlap detection. +- Improved the QA plot resulting from fitting order widths and gaps as a + function of spatial position. +- Updated general raw image reader so that it correctly accounts for + spectrographs that read the data and overscan sections directly from + the file headers. + +Bug Fixes +--------- + +- Fix "The system cannot find the file specified" errors when installing + on Windows. +- Fixed a fault caused when all frames in a pypeit file are identified + as being part of ``all`` calibration groups. +- Allow for empty 2D wavecal solution in HDU extension of WaveCalib file +- Fixed a bug in the ginga display function, when the user doesn't + provide the ``trc_name`` argument. +- Fix a **MAJOR BUT SUBTLE** bug in the use of ``numpy.argsort``. When + using ``numpy.argsort`` the parameter ``kind='stable'`` should be used + to ensure that a sorting algorithm more robust than "quicksort" is + used. +- Fix error "ValueError: setting an array element with a sequence. The + requested array has an inhomogeneous shape after 1 dimensions..." + occurring when unpacking the ``SpecObj`` spectrum but having an + attribute of the ``SpecObj`` object that is ``None``. +- Fixed a hidden bug that was causing the spatial flexure to fail. The + bug was in the ``SlitTraceBitMask`` class, where the function + ``exclude_for_flexure()`` was not returning the ``'BOXSLIT'`` flag. +- Fix a bug in ``pypeit_coadd_2d`` related to how the binning was taken + into account in the mask definition, and in the calculation of the + offset between frames. +- Fix bug when trying to open mosaic data from previous versions; + version checking flag was not being propagated. + + + + diff --git a/doc/whatsnew.rst b/doc/whatsnew.rst index 66935e6ae..85de8cae7 100644 --- a/doc/whatsnew.rst +++ b/doc/whatsnew.rst @@ -11,7 +11,7 @@ What's New in PypeIt ---- -.. include:: releases/1.16.1dev.rst +.. include:: releases/1.17.0.rst ---- From d8ebb1deea36ff675bfad61695f37c55b44d2e00 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Fri, 1 Nov 2024 09:00:37 -0700 Subject: [PATCH 3/8] exclude pixelflats from package distribution --- MANIFEST.in | 1 + 1 file changed, 1 insertion(+) diff --git a/MANIFEST.in b/MANIFEST.in index 8f6c6502c..e7a7ee976 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -17,6 +17,7 @@ global-exclude *.pyc *.o *.so *.DS_Store *.ipynb # update the defined_paths dictionary in pypeit.pypeitdata.PypeItDataPaths! recursive-exclude pypeit/data/arc_lines/reid_arxiv *.fits *.json *.pdf *.tar.gz recursive-exclude pypeit/data/arc_lines/NIST *.ascii +recursive-exclude pypeit/data/pixelflats *.fits *.fits.gz recursive-exclude pypeit/data/sensfuncs *.fits recursive-exclude pypeit/data/skisim *.dat recursive-exclude pypeit/data/standards *.gz *.fits *.dat From e6a95e24bc3b7c1441aad53b1f303fc32ef47d23 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Fri, 1 Nov 2024 09:07:25 -0700 Subject: [PATCH 4/8] only zipped files --- MANIFEST.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MANIFEST.in b/MANIFEST.in index e7a7ee976..c884b33e2 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -17,7 +17,7 @@ global-exclude *.pyc *.o *.so *.DS_Store *.ipynb # update the defined_paths dictionary in pypeit.pypeitdata.PypeItDataPaths! recursive-exclude pypeit/data/arc_lines/reid_arxiv *.fits *.json *.pdf *.tar.gz recursive-exclude pypeit/data/arc_lines/NIST *.ascii -recursive-exclude pypeit/data/pixelflats *.fits *.fits.gz +recursive-exclude pypeit/data/pixelflats *.fits.gz recursive-exclude pypeit/data/sensfuncs *.fits recursive-exclude pypeit/data/skisim *.dat recursive-exclude pypeit/data/standards *.gz *.fits *.dat From bb1c12e360af6eeaff4d62c4f90d1bb2ef091aa9 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Fri, 1 Nov 2024 09:19:02 -0700 Subject: [PATCH 5/8] release doc fix --- doc/releases/1.17.0.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/doc/releases/1.17.0.rst b/doc/releases/1.17.0.rst index 849790fb7..26d719074 100644 --- a/doc/releases/1.17.0.rst +++ b/doc/releases/1.17.0.rst @@ -54,10 +54,9 @@ Instrument-specific Updates sky flats. - Improved HIRES frame typing and configuration setup. - Added support for Keck/KCWI BH3 grating configuration. -- Updated the requirements of a spectral flip for KCWI (blue) data. If - all amplifiers are used, the data will be flipped. If all amplifiers - are used, the data will not be flipped. Otherwise, the data will be - flipped. +- Updated the requirements of a spectral flip for KCWI (blue) data. If + all amplifiers are used, the data will not be flipped in the spectral + direction. Otherwise, the data will be flipped. - Added support for the (decommissioned) AAT/UHRF instrument - Updated X-Shooter detector gain and read noise to come from header, and updated plate scales to the most recent values from the manual. From 5bcfae699ab838c52a0668836f6cb8a4046e696e Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Mon, 4 Nov 2024 14:50:10 -0800 Subject: [PATCH 6/8] doc update --- doc/help/run_pypeit.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/help/run_pypeit.rst b/doc/help/run_pypeit.rst index 77ed59f2f..b0262023a 100644 --- a/doc/help/run_pypeit.rst +++ b/doc/help/run_pypeit.rst @@ -4,7 +4,7 @@ usage: run_pypeit [-h] [-v VERBOSITY] [-r REDUX_PATH] [-m] [-s] [-o] [-c] pypeit_file - ## PypeIt : The Python Spectroscopic Data Reduction Pipeline v1.16.1.dev95+g35e103c10 + ## PypeIt : The Python Spectroscopic Data Reduction Pipeline v1.16.1.dev635+g9540496b9 ## ## Available spectrographs include: ## aat_uhrf, bok_bc, gemini_flamingos1, gemini_flamingos2, From 1d3ec28730ca5648e095009193b01005568ca483 Mon Sep 17 00:00:00 2001 From: Kyle Westfall Date: Mon, 4 Nov 2024 14:52:18 -0800 Subject: [PATCH 7/8] changes --- doc/releases/1.17.0.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/releases/1.17.0.rst b/doc/releases/1.17.0.rst index 26d719074..774daf53a 100644 --- a/doc/releases/1.17.0.rst +++ b/doc/releases/1.17.0.rst @@ -79,6 +79,7 @@ Instrument-specific Updates 0.164 arcsec/pixel for all orders - Add new P200/DBSP reid_arxiv template for 1200/7100 with D55 dichroic +- Add B480 as a supported option for Gemini-S/GMOS Script Changes -------------- From b985656705b2ebcdb66e7b49cff2a71db4ae8f04 Mon Sep 17 00:00:00 2001 From: rcooke Date: Sat, 16 Nov 2024 21:00:16 +0000 Subject: [PATCH 8/8] add new release file --- doc/releases/1.17.1dev.rst | 40 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/releases/1.17.1dev.rst diff --git a/doc/releases/1.17.1dev.rst b/doc/releases/1.17.1dev.rst new file mode 100644 index 000000000..38705d546 --- /dev/null +++ b/doc/releases/1.17.1dev.rst @@ -0,0 +1,40 @@ + +Version 1.17.1dev +================= + +Installation Changes +-------------------- + + + +Dependency Changes +------------------ + + + +Functionality/Performance Improvements and Additions +---------------------------------------------------- + + +Instrument-specific Updates +--------------------------- + + + +Script Changes +-------------- + + + +Datamodel Changes +----------------- + + + +Under-the-hood Improvements +--------------------------- + + + +Bug Fixes +---------