Merge pull request #556 from ngageoint/integration/1.3.59-rc

Merge integration/1.3.59-rc into master

CHANGELOG.md

@@ -4,6 +4,64 @@ SarPy follows a continuous release process, so there are fairly frequent release
 Since essentially every (squash merge) commit corresponds to a release, specific 
 release points are not being annotated in GitHub.
 
+## [1.3.59] - 2024-10-03
+### Added
+- `noxfile.py`
+- TOA visualization to `sarpy/visualization/cphd_kmz_product_creation.py`
+- Unit tests for `sarpy/tests/io/complex/sicd_elements`
+- `sarpy/tests/io/phase_history/cphd1_elements/conftest.py`
+- Unit tests for `sarpy/tests/io/phase_history/cphd1_elements`
+- CPHD 1.0.1 xml examples to `sarpy/tests/data`
+- `sarpy/processing/sicd/spectral_taper.py` and `sarpy/utils/sicd_sidelobe_control.py`
+- `--remap` argument to `sarpy/utils/create_product.py`
+- `GDM` to `sarpy/visualization/remap.py`
+- Unit tests for `sarpy/consistency/sicd_consistency.py`
+- Support reading CPHDs with an AmpSF PVP whose Data/SignalArrayFormat is CF8
+- Unit tests for `sarpy/consistency/sidd_consistency.py`
+- Support for MATESA TRE
+- Support reading/writing CPHDs with compressed signal arrays
+- Support for numpy 2.0
+### Fixed
+- `sarpy.io.kml.add_polygon` coordinate conditioning for older numpy versions
+- Replace unsupported `pillow` constant `Image.ANTIALIAS` with `Image.LANCZOS`
+- `sarpy/io/phase_history/cphd1_elements/GeoInfo.py` setters
+- SquintAngle calculation in `sarpy/io/complex/sicd_elements/SCPCOA.py`
+- Incorrectly assigned Graze in SIDD 2.0.0 and SIDD 3.0.0 ExploitationFeatures
+- SIDD `TimeCOAPoly` calculation
+- Set SIDD Display/Interpolation/Operation values to CORRELATION
+- `sarpy.io.phase_history.cphd1_elements.PVP.PVPType.get_size()`
+- SICD file reading in `sarpy/consistency/sicd_consistency.py`
+- Protect waveform validation from `waveform.TxFreqStart == None` in `sarpy/io/complex/sicd_elements/RadarCollection.py`
+- Fix `sarpy/io/complex/sicd_elements/Timeline.py` validation code to allow IPP T1End == T2Start
+- Properly close file objects in NITF and CPHD writers
+- SIDD file reading in `sarpy/consistency/sidd_consistency.py`
+- Application of adjustable parameter offsets in RIC frames during projection
+- Overflow bug in `ComplexFormatFunction` magnitude/phase -> real/imag
+- NITF image subheader parsing when there are more than 9 bands
+- Population of SIDD ExploitationFeatures resolution metadata when processed from a SICD
+- Fix BANDSB implementation to parse correctly
+- SingleLUTFormatFunction application for LUT with more than one dimension
+- SIDD NITF IALVL/IDLVL for NITFs consisting of multiple image segments and/or product images
+- Reading/writing of uncompressed NITF image segments with two complex-component bands interleaved by block/row
+- Replace deprecated `matplotlib.cm.get_cmap` with `matplotlib.pyplot.get_cmap`
+- ReferencePoint attribute in SIDD MeasurementType objects
+- Sentinel clutter radiometric polynomials are no longer transposed
+- NITFWriter no longer closes passed in file objects
+- Various SICD and SIDD elements fixed to better match NGA standard
+- Fixed case where DTEDInterpolator applied geoid offset incorrectly
+- Improved DTEDInterpolator handling of missing DEMs
+- SIDD 3.0.0 point projection
+- Restored missing antenna beam footprints in some KMZs
+- DTED parsing for tiles with null values
+- Improved mapping of SICD -> SIDD polarizations
+- Incorrect SIDD ISM.compliesWith definition
+- SIO reading/writing
+- SIDD 2.0+ FilterType handling
+- Erroneous SIDD consistency error re: NITF NBPP when PixelType=RGB24I
+- Properly recompute SCPCOA metadata when updating SCP using a DEM in `sarpy.io.complex.converter.conversion_utility`
+- SIDD GeoInfo namespace handling
+- Cache glob results in `GeoTIFF1DegList` to alleviate excessive filesystem load
+
 ## [1.3.58] - 2023-08-07
 ### Added
 - Added additional tests to test_sicd_elements_geodata.py

README.md

@@ -88,7 +88,7 @@ MIT license.
 
 Dependencies
 ------------
-The core library functionality depends only on `numpy >= 1.11.0` and `scipy`. 
+The core library functionality depends only on `numpy` and `scipy`. 
 
 Optional Dependencies and Behavior
 ----------------------------------
@@ -114,8 +114,9 @@ message indicating the missing optional dependency.
   data in hdf5 format require the `h5py` package, this includes Cosmo-Skymed, ICEYE, 
   and NISAR data.
 
-- Reading an image segment in a NITF file using jpeg or jpeg 2000 compression 
-  and/or writing a kmz image overlay requires the `pillow` package.
+- Reading an image segment in a NITF file using jpeg or jpeg 2000 compression,
+  reading a GeoTIFF DEM, and/or writing a kmz image overlay requires the `pillow` 
+  package.
 
 - CPHD consistency checks, presented in the `sarpy.consistency` module, depend on 
   `lxml>=4.1.1`, `networkx>=2.5`, `shapely>=1.6.4`, and `pytest>=3.3.2`. Note that these

docs/io/DEM/index.rst

@@ -6,5 +6,6 @@ DEM reading elements (sarpy.io.DEM)
     :caption: Contents:
 
     geoid
+    geotiff1deg
     DEM
     DTED

docs/processing/sicd/index.rst

@@ -9,5 +9,6 @@ SICD data processing (sarpy.processing.sicd)
     normalize_sicd
     csi
     fft_base
+    spectral_taper
     subaperture
     windows

docs/processing/sicd/spectral_taper.rst

@@ -0,0 +1,8 @@
+Spectral Taper methods (sarpy.processing.sicd.spectral_taper)
+=============================================================
+
+.. automodule:: sarpy.processing.sicd.spectral_taper
+    :members:
+    :show-inheritance:
+    :inherited-members:
+

docs/utils/index.rst

@@ -12,3 +12,4 @@ Command-line Utilities (sarpy.utils)
     nitf_utils
     cphd_utils
     nominal_sicd_noise
+    sicd_sidelobe_control

docs/utils/sicd_sidelobe_control.rst

@@ -0,0 +1,7 @@
+Apply/Remove a spectral taper window to/from a SICD (sarpy.utils.sicd_sidelobe_control)
+=======================================================================================
+
+.. automodule:: sarpy.utils.sicd_sidelobe_control
+    :members:
+    :show-inheritance:
+    :inherited-members:

noxfile.py

@@ -0,0 +1,26 @@
+#
+# Copyright 2023 Valkyrie Systems Corporation
+#
+# Licensed under MIT License.  See LICENSE.
+#
+
+import os
+
+import nox
+
+_PYTHON_VERSIONS = ['3.6', '3.11']
+_LOCATIONS = ["tests"]
+
+
+# Run only test session when no arguments are specified
+nox.options.sessions = ["test"]
+
+
+@nox.session(venv_backend="conda")
+@nox.parametrize('version', _PYTHON_VERSIONS)
+def test(session, version):
+    assert 'SARPY_TEST_PATH' in os.environ
+    args = session.posargs or _LOCATIONS
+    session.conda_install(f'python={version}')
+    session.install('.[all]')
+    session.run("pytest", *args)

sarpy/__about__.py

@@ -27,7 +27,7 @@
            '__license__', '__copyright__']
 
 from sarpy.__details__ import __classification__, _post_identifier
-_version_number = '1.3.58'
+_version_number = '1.3.59'
 
 __version__ = _version_number + _post_identifier
 

sarpy/consistency/cphd_consistency.py

@@ -1184,6 +1184,7 @@ def check_channel_signal_data(self, channel_id, channel_node):
 
         with self.precondition():
             assert self.header is not None
+            assert self.xml.find('./Data/SignalCompressionID') is None
             format_string = self.xml.findtext('./Data/SignalArrayFormat')
             signal_dtype = cphd1_utils.binary_format_string_to_dtype(format_string)
 

sarpy/consistency/sicd_consistency.py

@@ -47,7 +47,6 @@ def evaluate_xml_versus_schema(xml_string, urn_string):
     -------
     None|bool
     """
-
     try:
         the_schema = get_schema_path(urn_string)
     except KeyError:
@@ -370,19 +369,20 @@ def check_file(file_name):
         with open(file_name, 'rb') as fi:
             initial_bits = fi.read(30)
             if initial_bits.startswith(b'<?xml') or initial_bits.startswith(b'<SICD'):
+                fi.seek(0)
                 sicd_xml = fi.read()
                 return _evaluate_xml_string_validity(sicd_xml)[0]
 
     return check_sicd_file(file_name)
 
 
-if __name__ == '__main__':
+def main(args=None):
     parser = argparse.ArgumentParser('SICD Consistency')
     parser.add_argument('file_name')
     parser.add_argument(
         '-l', '--level', default='WARNING',
         choices=['INFO', 'WARNING', 'ERROR'], help="Logging level")
-    config = parser.parse_args()
+    config = parser.parse_args(args)
 
     logging.basicConfig(level=config.level)
     logger.setLevel(config.level)
@@ -391,4 +391,8 @@ def check_file(file_name):
         logger.info('\nSICD: {} has been validated with no errors'.format(config.file_name))
     else:
         logger.error('\nSICD: {} has apparent errors'.format(config.file_name))
-    sys.exit(int(validity))
+    return int(validity)
+
+
+if __name__ == '__main__':
+    sys.exit(main())    # pragma: no cover

sarpy/consistency/sidd_consistency.py

@@ -133,7 +133,6 @@ def check_sidd_data_extension(nitf_details, des_header, xml_string):
 
     def check_des_header_fields():
         # type: () -> bool
-
         des_id = des_header.DESID.strip() if nitf_details.nitf_version == '02.10' else des_header.DESTAG.strip()
 
         if des_id != 'XML_DATA_CONTENT':
@@ -148,7 +147,7 @@ def check_des_header_fields():
             logger.exception('SIDD: The SIDD DES.DESSHTN must be a recognized urn')
             return False
 
-        # make sure that the NITF urn and SICD urn actually agree
+        # make sure that the NITF urn and SIDD urn actually agree
         header_good = True
         if nitf_urn != xml_urn:
             logger.error('SIDD: The SIDD DES.DESSHTN ({}) and urn ({}) must agree'.format(nitf_urn, xml_urn))
@@ -197,18 +196,18 @@ def compare_sidd_class():
                     des_header.Security.CLAS.strip(), nitf_details.nitf_header.Security.CLAS.strip()))
         return True
 
-    # check sicd xml structure for validity
-    valid_sicd, xml_urn, the_sidd = _evaluate_xml_string_validity(xml_string)
-    # check that the sicd information and header information appropriately match
+    # check sidd xml structure for validity
+    valid_sidd, xml_urn, the_sidd = _evaluate_xml_string_validity(xml_string)
+    # check that the sidd information and header information appropriately match
     valid_header = check_des_header_fields()
     # check that the classification seems to make sense
     valid_class = compare_sidd_class()
-    return valid_sicd & valid_header & valid_class, the_sidd
+    return valid_sidd & valid_header & valid_class, the_sidd
 
 
 def check_sidd_file(nitf_details):
     """
-    Check the validity of the given NITF file as a SICD file.
+    Check the validity of the given NITF file as a SIDD file.
 
     Parameters
     ----------
@@ -261,6 +260,7 @@ def find_des():
                     logger.exception('SIDD: Old-style SICD DES header at index {}, but failed parsing'.format(i))
                     continue
 
+
     def check_image_data():
         valid_images = True
         # verify that all images have the correct pixel type
@@ -347,7 +347,8 @@ def check_image_data():
         elif the_sidd.Display.PixelType == 'MONO16I':
             sidd_nitf_details.append({'nbpp': 16, 'pvtype': 'INT', 'pixel_type': the_sidd.Display.PixelType})
         elif the_sidd.Display.PixelType == 'RGB24I':
-            sidd_nitf_details.append({'nbpp': 24, 'pvtype': 'INT', 'pixel_type': the_sidd.Display.PixelType})
+            # RGB24I is 3 channels of 8 bpp, see: https://github.com/ngageoint/sarpy/issues/544
+            sidd_nitf_details.append({'nbpp': 8, 'pvtype': 'INT', 'pixel_type': the_sidd.Display.PixelType})
         else:
             raise ValueError('Got unhandled pixel type {}'.format(the_sidd.Display.PixelType))
 
@@ -382,19 +383,20 @@ def check_file(file_name):
         with open(file_name, 'rb') as fi:
             initial_bits = fi.read(30)
             if initial_bits.startswith(b'<?xml') or initial_bits.startswith(b'<SIDD'):
-                sicd_xml = fi.read().decode('utf-8')
-                return _evaluate_xml_string_validity(sicd_xml)[0]
+                fi.seek(0)
+                sidd_xml = fi.read()
+                return _evaluate_xml_string_validity(sidd_xml)[0]
 
     return check_sidd_file(file_name)
 
 
-if __name__ == '__main__':
+def main(args=None):
     parser = argparse.ArgumentParser('SIDD Consistency')
     parser.add_argument('file_name')
     parser.add_argument(
         '-l', '--level', default='WARNING',
         choices=['INFO', 'WARNING', 'ERROR'], help="Logging level")
-    config = parser.parse_args()
+    config = parser.parse_args(args)
 
     logging.basicConfig(level=config.level)
     logger.setLevel(config.level)
@@ -403,4 +405,8 @@ def check_file(file_name):
         logger.info('\nSIDD: {} has been validated with no errors'.format(config.file_name))
     else:
         logger.error('\nSIDD: {} has apparent errors'.format(config.file_name))
-    sys.exit(int(validity))
+    return int(validity)
+
+
+if __name__ == '__main__':
+    sys.exit(main())    # pragma: no cover