RELEASE_NOTES.txt

 _ _ _

13/12/2023
Matthieu Ravet

Comments:
    - First official release notes : Implementation of MOSAIC (Multimodal framework + likelihood functions).
    See /ForMoSA/DEMO/inputs/MOSAIC/ folder for more info

_ _ _

20/12/2023
Matthieu Ravet

Comments:
    - Correction of the os import in adapt/extraction_functions.py

_ _ _

21/12/2023
Matthieu Ravet

Comments:
    - No more covariance matrix if not provided in the impute observation file or if the covariance likelihood (chi2_covariance) is not used
    - Cleaner version of the covariances treatment during both the importation and the inversion
    - Correction of the importation of 'res_spectro' in adapt/extraction_functions.py
    - New function in main_utilities to compute the diagonal merging of covariance matrix (~ 100x faster)
    - Sorting of the glob.glob when using the MOSAIC (i.e. you can now number your observation files, ex: 1_obs1, 2_obs2, ...)

_ _ _

01/02/2024
Matthieu Ravet

Comments:
    - Addition of checkups for the likelihoods (continuum checkups for the CCF and iterative checkups for the logL loop)
    - Correction of the continuum substraction function to make it run with custom ranges
    - Changes of the num_sigma parameter in extraction_functions to not smooth to much the models during the resolution decrease

_ _ _

15/02/2024
Matthieu Ravet

Comments:
    - TEST BRANCH

_ _ _

19/02/2024
Matthieu Ravet

Comments:
    - Small correction in the rv and vsini loops (len(flx) != 0, if you run with photometry only, usefull for MOSAIC)

_ _ _

08/03/2024
Paulina Palma-Bifani

Comments:
    - I've modified the readme to include PyMultinest installation instructions.
    I've modified the authors list and requirements too.
    - I've added the lines Matthieu wrote about PyMultinest run.
    - I've completed the docstrings for many functions.
    - I've modified the prints to have a consistent style over the code.
    - I'm working on adapting the plotting function to be consistent for: MOSAIC, pymultinest, have a fat-check plot generator, ect...

_ _ _

08/03/2024
Simon Petrus

Comments:
    - adapt/extraction_functions.py -> I've corrected 2 bugs
    - nested_sampling/nested_sampling.py -> I've reformated the result file provided by nestle and PyMultinest.
    The output of these modules are saved with the extension RAW but a new dictionary is created with the name:
    "result_PACKAGE.pic" containing 5 arguments:
        - "samples": sample of the living points kept at each iteration.
        - "weights": weights of the living points kept at each iteration.
        - "logl": log(L) of the living points kept at each iteration.
        - "logvol": log(prior volume) of the living points kept at each iteration.
        - "logz": the evidence and its error [log(Z), log(Z)_err]
        I notice that the result "h" is not saved anymore.
    This dictionary is used in the plotting Class.
    - plotting/plotting_class.py: I have adapted each function to make them enable to call/read the dictionary
    "result_PACKAGE.pic".

_ _ _

15/03/2024
Matthieu Ravet

Comments:
    - Correction of photometry importation in extraction_functions by putting it out of the selection windows
    - Parameter name changes :
            * obs_cut >> obs_spectro
            * obs_pho >> obs_photo
            * obs_cut_ins >> obs_spectro_ins
            * obs_pho_ins >> obs_photo_ins

            * obs_merge >> obs_spectro_merge
    > These name changes have been applied in extraction_functions, nested_sampling and plotting_class

    - Replacement of X = [] and X = np.zeros(len(Y)) by X = np.asaray([]) when possible to homogenize the code and speed-up the inversion
    - obs_spectro now store the optional parameters inv_cov, trans_obs and star_flx for more clarity
    - Removal of obs_merge_ins during when saving the spectrum_obs as it is not needed (and the instrument can still be access via obs_spectro_ins)
    - Use of X = np.asarray(X, dtype=object) for each list X containing multiple objects of different sizes for storage (usefull when we'll want
    to add more optional inputs)

Test that have been done to checkup the changes:
    - With/without spectral windows ✓
    - With/without continuum substraction ✓
    - With/without spectral covariances ✓
    - With/without photometry ✓
    - With/without NANs ✓
    - With/without MOSAIC ✓
    - With/without trans_obs and star_flx ✓

_ _ _

18/03/2024
Matthieu Ravet

Comments:
    - Keeping the optional arrays separated for storage to not slow down the grid extraction and adaptation by creating
    a separate array:
            * "obs_opt" for the optional observational arrays (cov, star_flx and trans_obs at the moment)
    - Correction of the functions names and imports in nested_sampling, plotting_class

_ _ _

05/04/2024
Matthieu Ravet

Comments:
    - NEW: addition of a new extrat-grid parameter, alpha. This parameter is usefull when you assume your datasets are poorly calibrated. It
    has the effect of an additionnal scaling factor ck = alpha * (R/d)**2 that can be set differently with different observations.
        FORMAT: * alpha = 'NA' # If you don't want to use it
                * alpha = 'prior', float, float # If you want to use it with a given 'prior' and range
                * alpha = 'prior_1', float_1, float_1, 'prior_2', float_2, float_2, ..., 'prior_n', float_n, float_n when using n observation in MOSAIC
    => updates of the plotting functions to take this new parameter into account
    - NEW: adaptation of the plotting functions:
                * possibility of plotting MOSAIC outputs
                * new arguments in plot_fit :
                            trans:      'yes' or 'no' to plot transmision curves for photometry
                            logx:       'yes' or 'no' to plot the wavelength in log scale
                            logy:       'yes' or 'no' to plot the flux in log scale
                            norm:       'yes' or 'no' to plot the normalized spectra (usefull for MOSAIC)
    - NEW: the DEMO file has been updated with the recent changes

Test that have been done to checkup the changes:
    - Classical with/without alpha ✓
    - MOSAIC with/without alpha ✓
    - Classical and MOSAIC with/without photometry ✓
    - With/without all new plotting options ✓


Note: alpha parameter is strongly correlated to R and d => need to be carefull about the physical interpretations. Inversion time is greatly lengthen
with this new parameter when using MOSAIC

_ _ _

30/04/2024
Matthieu Ravet

Comments:
    - Note : be carefull when using a small wavelength range to estimate the continuum (it is impossible if less than 2 points end up in the
    range for the interpolation...)
    - Corrections :
                * The 'wav_for_adapt' parameter did not work due to numpy not accepting different size arrays when storing them with
                'np.savez'. This has been corrected by replacing the importation list by array in 'extraction_functions' and adding lines
                to transform them in 'dtype=object' when saving.
                * The output grid were still called grid_merge and grid_phot. This has been replaced by 'grid_spectro' and 'grid_photo'
                accordingly to match the file names in the code.
    - NEWS :
                * MOSAIC is now fully integrated in the classical mode => no more 'observation_format' parameter. The only change when using
                the classical (only one obs) is that you need to specify the path where this observation is (i.e. /home/mravet/Documents/These/FORMOSA/INPUTS/GRAVITY/).
                The 'multiply_transmission' and 'star_data' have been removed from the config file as they will be automatically imported if
                provided in the .fits file (see below). All other parameters should work as usuall.
                * The radial velocity can now be fitted individually for different obs in the loop. The format is the same as for 'alpha' (see above notes).
                * The extra-grid parameters that can be fitted individually for each obs have 2 options :
                    1) If you put 1 set of values (either 'NA' or 'prior', x, y), it will be the same for all datasets
                    2) If you put N sets of values ('NA' or priors according to the obs), it will use separate ones for each datasets
                * The input format has be normalized to :
                        > 'WAV'
                        > 'FLX'
                        > 'ERR (or COV, optional)
                        > 'INS'
                        > 'RES'
                        > ('TRANSM', optional)
                        > ('STAR FLX', optional)
                * The DEMO config.ini as been updated accordingly


Test that have been done to checkup the changes:
    - 1 obs with/without alpha ✓
    - 1 obs with/without rv ✓
    - 1 obs with/without continuum substraction (multi/single continuum/adapt windows) ✓
    - 1 obs with/without custom reso ✓
    - 1 obs with/without wav_for_adapt windows ✓
    - 1 obs with/without wav_fit windows ✓
    - 1 obs with cov and chi2 classic logL ✓
    - HiRISE ✓
    - photometry ✓
    - all plotting options ✓
    - multiple observations (with the above tests) ✓
    - multiple observations with multiple alpha and rv ✓

_ _ _

12/05/2024

Paulina Palma-Bifani

Comments:
    - This is the official date of the release 2.0.0 in Pipy in preparation for the ForM-X workshop.

_ _ _

11/06/2024

Matthieu Ravet

Comments:
    - Changes in variables names in "adapt_grid.py", "nested_sampling.py", "modif_spec.py" and "plotting_class.py".
    Name changes are listed below :

        (In adapt_grid)
        - wav_obs_spec -> wav_obs_spectro
        - wav_obs_phot -> wav_obs_photo
        - grid_new_np -> grid_spectro_np
        - grid_phot_new_np -> grid_photo_np
        - ds_new -> ds_spectro_new
        - ds_new_photo -> ds_photo_new
        - obs_new_phot -> obs_photo_new
        - mod_phot -> mod_photo

        (In nested_sampling, modif_spec and plotting_class)
        - wav_obs_merge -> wav_obs_spectro
        - flx_obs_merge -> flx_obs_spectro
        - err_obs_merge -> err_obs_spectro
        - wav_obs_phot -> wav_obs_photo
        - flx_obs_phot -> flx_obs_photo
        - err_obs_phot -> err_obs_photo
        - grid_m -> grid_spectro
        - grid_p -> grid_photo
        - grid_merge -> grid_spectro
        - grid_phot -> grid_photo
        - inv_cov_obs_merge -> inv_cov_obs
        - transm_obs_merge -> transm_obs
        - star_flx_obs_merge -> star_flx_obs
        - system_obs_merge -> system_obs
        - ind_merge -> ind_spectro
        - ind_phot -> ind_photo
        - flx_obs -> flx_obs_spectro_modif
        - err -> err_obs_spectro_modif
        - flx_obs_phot -> flx_obs_photo_modif
        - err_phot -> err_obs_photo_modif
        - flx_mod -> flx_mod_spectro_modif
        - flx_mod_phot -> flx_mod_photo_modif
        - logL_spec -> logL_spectro
        - logL_phot -> logL_photo

    - Possibility to fit separate vsini (and ld) per observation (MOSAIC). Be carefull to put the same number/order
    of vsini and ld !

Correction :
    - Correction of the systematics export in modif_spec to be able to run non-HiRISE runs

Test that have been done to checkup the changes:
    - 1 obs with/without multiple vsini ✓
    - HiRISE ✓
    - all plotting options ✓
    - multiple observations (with the above tests) ✓

_ _ _

27/06/2024

Matthieu Ravet

Comments:
    - Optimization of the continuum removal and resolution functions:
        - The interpolated resolution array is now computed outside the adapt_grid loop to save time.
        - The model is no longer reinterpolated during the continuum extraction phase, as it already is with the resolution decreasing step.
        - Simplification of the nomenclature of the two functions (continuum removal + resolution)
        - Small name change in adapt_grid: flx_mod_spectro >> mod_spectro to match all functions inputs

Test that have been done to checkup the changes:
    - 1 obs with/without windows for extraction  ✓
    - 1 obs with/without windows for continuum  ✓
    - with and without continuum removal
    - multiple observations (with the above tests) ✓

_ _ _

08/07/2024

Allan Denis

Comment:
    - Adding a fourth parameter on the 'vsini' prior to indicate the type of broadening we want to apply. The possibles values of this parameter are 'RotBroad', 'Accurate', 'FastRotBroad'.
        - 'RotBroad' applies the classical PyAstronomy rotational broadening
        - 'FastRotBroad' applies the PyAstronomy fast version  of rotational broadening (applicable only for small wavelength range i.e high resolution data)
        - 'Accurate' applies an alternative version, faster than the PyAstronomy rotational broadening, slower than 'FastRotBroad' but applicable even for high wavelength ranges and more accurate than 'FastRotBroad'

    - Adding the possibility of having multiple components for the starlight contamintation, often used for high resolution data

    - New plotting functions in order to compare the planet model estimated by the least square function and the data substracted by the 'reference model'.
    In high resolution data, the 'reference model' refers to the estimated starlight contamination and the systematics (if they exist)

Tests that have been done to checkup the changes:
    - 1 obs with the 3 possible values for the fourth parameter of the vsini prior ✓
    - a few obs with 5 components for the starlight contamintation (Rico data and HiRISE data) ✓


- - -

18/07/2024

Allan Denis

Comments:
     - Fixing a small bug when using both photometric and spectroscopic data
     - Adding the possibility to plot the planetary model and the data (substracted by the starlight contamination and the systematics if they exist) by decreasing their resolution, with the function plot_HiRes_comp_model (used for high resolution data)
     The user must define in the function the resolution of the instrument (default to 0). is the default value is selected then the function doesn't decrease the resolution of the data. If the resolution of the instrument is defined then the instrument decrease the resolution of the data to the resolution given by the v.sini (= 3.0*10⁵ / v.sini)
     - Adding a ccf function that computes the ccf of the data with the best model. A model can also be defined by the user and given as the input of the fonction (if the user wants to to the ccf with a molecular template for exemple)

Tests that have been done to checkup the changes:
    - Different high resolution datasets
    - Datasets combining photometric and spectroscopic data points




- - -

28/07/2024

Paulina Palma-Bifani, Matthieu Ravet

Comments:
     - Commenting all functions
     - Replacing physical constants with astropy ones

Tests:
    None


- - -

02/08/2024

Paulina Palma-Bifani

Comments:
     - Adapted docstrings returns in some functions (not all)
     - Generated documentation for the webpage
     - This is a test. I will do a second commit afterwards with the complete updated docs.

Tests:
    Webpage API is generating without issues locally (make html)



- - -

08/08/2024

Matthieu Ravet

Comments:
     - Correcting for the HiRISE type data merging when using cuts (i.e. remplacing axis=0 by axis=1). In the future, it would be
     nice to create a more simple version of the star and systematic matrix (for now its (1, lambda, sys/star) it would be best to
     have (lambda, sys/star) to avoid keeping large matrices accross the inversion)

Tests:
    - Test with CRIRES+ datasets with cuts in the adapt and in the fit


- - -

29/08/2024

Allan Denis

Comments:
      - New approach in the adaptation of the data in the case where the user defined separated windows in 'wave_for_adapt' (eg : '1.4, 1.5 / 1.6, 1.7 / 1.8, 1.9 / ... '). Instead of making a loop on each sub-interval, we now combine the data in one array and do the adaptation on one array
      - New approach in the lsq function, we can now use the lsq in each sub wavelength separatly (eg. '1.4, 1.5', '1.6, 1.7', '1.8, 1.9' ...) and merge the results into a single array to compute the loglikelihood on one array only. This gives the advantage of limiting flux error calibration that can arrise between order for data such as CRIRES, HiRISE ... and being fast at the same time. Previously, the use had to define separated windows in 'wave_fit' (eg : '1.4, 1.5', '1.6, 1.7', '1.8, 1.9' ...) and one loglikelihood one computed for each of these windows. Now the user has to define separated windows in the format '1.4, 1.5 / 1.6, 1.7 / 1.8, 1.9 / ...'

Tests that have been done to checkup the changes:
     - High resolution datasets (HiRISE, CRIRES)


- - -

26/09/2024

Matthieu Ravet

Comments:
      - Updating / correting the readthedocs
      - Correction of the rv input loop if you want to set it to constant during the ns
      - Correction of the plotting function (for constant extra grid and for the custom resolution grid)


Tests that have been done to checkup the changes:
     - Test with constant priors on the rv


_ _ _

25/10/2024

Matthieu Ravet

Comments:
    Huge optimisation and correction update with multiple changes

    In extraction_functions / adapt_obs_mod and adapt_grid :
    - Updating the import name of STAR FLX to STAR_FLX1 to match what is done for SYSTEM1 (i.e. when only one is used)
    - Updating the extraction of the covariance matrix (to extract off-diagonal terms if multiple wav_for_adapt)
    - Separated spectral obs are not stored separatly anymore (not needed)
    - Spectral instrument are not stored anymore (not needed)
    - Removing the step np.load() during the adapation of the grid to speed up the code
    - Correcting the convolution function : removing factor 2 in fwhm = 2 * wav / res and reseting num_sigma = 1
    - Updating function names in adapt_grid

    In nested_sampling
    - Extraction the loop on each spectral window out of the inversion loop to speed up the code
    - Updating the extraction of the covariance matrix (to extract off-diagonal terms if multiple wav_fit)

    In plotting_class
    - Updating the function according to the previous changes
    - Updating to use astropy units to compute the Luminosity

Tests that have been done to checkup the changes:
     - Run with photometry / LRS / MRS and HRS
     - Run with/without MOSAIC
     - Run with/without multiple wav_for_adapt and multiple wav_fit
     - Run with/without continuum



_ _ _

30/10/2024

Matthieu Ravet

Comments:
    - Removing the Nyquist checkup for the data (HRS tends to have jumps inside orders which creates weird
    artefacts during inversion)
    - Adding new parameter in convolve_and_sample to give back an interpolated version of the model when its
    resolution is lower than that of the data

_ _ _

09/11/2024

Arthur Vigan

Comments:
    - Implement accurate and fast rotational broadening based on the method of Carvalho & Johns-Krull (2023)
    - Fix a bug that caused the code to continue running when specifying a rotational broadening method that is not in the list of available methods!
    - Implement better output of the time spent running
    - Automatically create ouput directory if it does not exist
    - Make a copy of the config file in the output directory (to go back to original parameters of a ForMoSA run)
    - Fix import to allow importation of plotting module outside of the ForMoSA package context
    - Update .gitignore file with more general patterns for a python package
    - Remove files that should not have been committed
    - Small improvements to plot_fit()
    
- - - 

13/12/2024

Allan Denis

Comments: Several modifications, including :
     - Adaptation of the model on a slightly larger wavelength range when we want to apply RV corrections to avoid losing data on the edges of the model
     - Removing of the lsq function which has been replaced by a script forward_model.py containing different possible functions to forward model the planet and star speckles for the high-contrast planets
     - Addition of 3 new parameters in the config file (fm_type, fm_continuum_res and bounds_lsq) which are related to the high-contrast planets, where the data are contaminated by the star speckles
     - Cleaning a big the plotting_class script
     
Tests that have been done to checkup the changes:
     - Runs with HiRISE data in the high-contrast scenario to check for the forward model scripts
     - Different runs to check for the gain associated to the adaptation of the model on a larger wavelength grid to avoid losing data on the edges during the RV application