GCRCatalogs v1.3.3

Add new v2 object, truth and object_with_truth_match catalogs for Data Portal; mark v1 catalogs as deprecated.

GCRCatalogs v1.3.2

Bug fix: Make code version consistent with package version.

GCRCatalogs v1.3.1

GCRCatalogs v1.3.1 includes a new redmapper catalog (v0.7.5) for cosmoDC2 v1.1.4 (thanks for @erykoff).

GCRCatalogs v1.3.0

GCRCatalogs v1.3.0 includes the addition of the photo-z catalog for Run 2.2i DR6 (thanks to @sschmidt23) and the truth catalogs for Run 3.1i (thanks to @JoanneBogart). See below for catalog names.

New and updated catalogs

The photo-z catalog for Run 2.2i DR6 is now available in dc2_object_run2.2i_dr6_v2_with_addons. In addition, a bug related to the metacal add-on path is also fixed.

The truth catalogs for Run 3.1i include:

• dc2_truth_run3.1i_agn_auxiliary_info: Useful derived information for AGN.
• dc2_truth_run3.1i_agn_truth_summary: AGN truth summary table.
• dc2_truth_run3.1i_agn_variability_truth: AGN variable truth information
• dc2_truth_run3.1i_lensed_agn_truth_summary: Truth summary table for lensed AGN.
• dc2_truth_run3.1i_lensed_agn_variability_truth: Variable truth information for lensed AGN.
• dc2_truth_run3.1i_lensed_host_truth_summary: Truth summary table for lensed host galaxies.
• dc2_truth_run3.1i_lensed_sne_truth_summary: Truth summary table for lensed SNe.
• dc2_truth_run3.1i_lensed_sn_variability_truth: Variable truth information for lensed SNe.

This version of GCRCatalogs will be available in desc environments and on conda-forge shortly.

GCRCatalogs v1.2.2

Update to Catalogs

• Catalogs dc2_object_run2.2i_dr6 and dc2_object_run2.2i_dr6_with_addons have both been updated to v2, which fixes a faulty patch in v1.

Improvements

• CosmoDC2 and SkySim5000 catalogs (both hdf5 and parquet versions) now have an available_healpix_pixels property that gives users a list of healpix pixels available in the catalog.

GCRCatalogs v1.2.1

New catalogs

skysim5000_v1.1.1_parquet and cosmoDC2_v1.1.4_parquet are now available. They are the parquet version of skysim5000_v1.1.1 and cosmoDC2_v1.1.4_image, respectively. The parquet version provides a better I/O performance and contains all GCR quantities (but not native quantities). The underlying files for these parquet version catalogs contain the same information as those loaded from GCRCatalogs (i.e., no translation needed).

Improvements

The non-parquet version of cosmoDC2 and skysim5000 catalogs (e.g., cosmoDC2_v1.1.4_image and skysim5000_v1.1.1) now has a much shorter initial load time (the quantity load time is still the same). For details about this improvement, please see #549.

Thanks to @evevkovacs and @JoanneBogart for their help in the preparation of this release. This version will be available on conda-forge shortly. It will also be installed in DESC environments (thanks to @heather999).

GCRCatalogs v1.2.0

Below is the change log since v1.2.0rc2. Please also see the change logs of v1.2.0rc1 and v1.2.0rc2 for changes since v1.1.0.

New catalogs

• Public catalog desc_cosmodc2 is available. Similar to the DC2 public catalogs (i.e., desc_dc2_run2.2i_dr6_*), this catalog is intended for non-DESC members who obtain data from the DESC Data Portal. DESC members should continue to use catalogs listed here.

This version of GCRCatalogs will appear in conda-forge and DESC environments shortly.

GCRCatalogs v1.2.0rc2

Bug fix: the GCRCatalogs.helpers submodule will now be installed.

GCRCatalogs v1.2.0rc1

This is a pre-release. It includes new catalogs and features needed for the first public release of DESC data as well as other updates.

New Catalogs

• desc_dc2_run2.2i_dr6_object DR6 object table for the public release
• desc_dc2_run2.2i_dr6_truth DR6 truth table for the public release
• desc_dc2_run2.2i_dr6_object_with_truth_match DR6 truth-match table for the public release

NOTE: These catalogs are meant for public use and are not accessible using default settings at NERSC or IN2P3 sites. However they are available to DESC members under the following names, respectively:

• dc2_object_run2.2i_dr6
• dc2_object_run2.2i_dr6_with_addons
• dc2_truth_run2.2i_summary_tract_partition

Schema changes

• in schema for dc2_dia_source, dc2_forced_source fields and dc2_source fields Ixx, Iyy, Ixy, IxxPSF, IyyPSF and IxyPSF have been renamed to Ixx_pixel, Iyy_pixel, etc. Units for the new fields are pixel^2 (had been incorrectly labeled asec2 in the old fields)
• in dc2_forced_source and dc2_source the field psf_fwhm_pixels has been changed to psf_fwhm_pixel.

Updates to Documentation

The Schema documentation has an updated object table description (column names and in some cases units) to be consistent with new catalogs and a description of truth summary catalogs has been added.

Improvements for Catalog Users

The following helper functions in the new module helpers.tract_catalogs return GCRQuery objects used in accomplishing tasks commonly appearing in example noteboooks (and user code):

• tract_filter(tracts, tract_high=None)
• sample_filter(frac, random_state=None)

Many thanks to @yymao for these updates and to @yymao and @heather999 for work on the plan for public release.

GCRCatalogs v1.1.0

The GCRCatalogs v1.1.0 release introduces the Run2.2i DR2 WFD and SkySim5000 catalogs. In addition, the Run2.2i DR6 catalogs have been renamed (postfix _wfd was dropped) for clarity. There is also schema change in the object catalogs. GCRCatalogs v1.1.0 now requires GCR v0.9.2.

See below for a complete list of changes. The "DC2 Runs Data Products and Add-ons" section of the README has also been updated to reflect the current list of recommended catalogs.

A code of conduct has been added to this repository.

New Catalogs

• dc2_object_run2.2i_dr2_wfd: static object catalog for Run2.2i DR2 (up to Year 1; WFD visits only)
• dc2_object_run2.2i_dr2_wfd_with_addons: same as dc2_object_run2.2i_dr2 but with all available add-ons (currently including only truth-match; metacal/photo-z not yet available)
• skysim5000_v1.1.1: 5000 sq deg of extragalactic catalogs (also available are skysim5000_v1.1.1_image and skysim5000_v1.1.1_small for subsets of sky; we are also working on a parquet version of SkySim5000 for faster access)

Changes to DR6 Catalog Names

Please use the following catalogs for Run 2.2i DR6:

• dc2_object_run2.2i_dr6: static object catalog for Run2.2i DR6 (up to Year 5; WFD+DDF visits; a small sky region was excluded)
• dc2_object_run2.2i_dr6_with_addons: same as dc2_object_run2.2i_dr6 but with metacal and truth-match add-ons. Photo-z not yet available.

All DR6 catalogs whose names contain dr6_wfd, dr6a, dr6b, or dr6c have been deprecated.

Schema change

In DC2 object catalogs, the second moment columns have been renamed so that their unit is clear. In particular, Ixx, Ixy, Iyy, IxxPSF, IxyPSF, and IxyPSF have been renamed to Ixx_pixel, Ixy_pixel, Iyy_pixel, IxxPSF_pixel, IxyPSF_pixel, and IxyPSF_pixel, respectively.

Similarly, Ixx_<band>, Ixy_<band>, Iyy_<band>, IxxPSF_<band>, IxyPSF_<band>, and IxyPSF_<band> have been renamed to Ixx_pixel_<band>, Ixy_pixel_<band>, Iyy_pixel_<band>, IxxPSF_pixel_<band>, IxyPSF_pixel_<band>, and IxyPSF_pixel_<band>, respectively.

A similar schema change will be applied to the DC2 source catalogs in the next release.

Improvements for Catalog Users

• The cosmoDC2 catalog reader now works with h5py v3+ (the version installed in desc-python-bleed)
• Performance and memory efficiency for accessing parquet-based catalogs have been improved when the users use arrow >=0.16
• New function get_available_catalog_names(...) was introduced as a shorthand for get_available_catalogs(names_only=True, ...)
• Users can now set default root dir in their user configs (Note: most DESC members would not need to do this)
• Upstream improvements from GCR v0.9.2 allow get_quantity_into() to work better with composite catalogs and quantity aliases

Improvements for Catalog Developers

• Composite catalogs now can be configured with more specific options thanks to GCR v0.9.0. See the GCR v0.9.0 release note for details.

Many thanks to the whole DC2 team for preparing the data products, and to @JoanneBogart @heather999 @evevkovacs for help in preparing this release. This version should be available in conda-forge soon and in DESC environments next week.