Skip to content

Commit

Permalink
Merge pull request #287 from MeteoSwiss/develop
Browse files Browse the repository at this point in the history
v1.0.0
  • Loading branch information
fpavogt authored Jul 24, 2023
2 parents 7f24cac + f591d37 commit 7c5231c
Show file tree
Hide file tree
Showing 105 changed files with 4,548 additions and 2,298 deletions.
57 changes: 57 additions & 0 deletions .github/workflows/CI_pypi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# This workflow will push the code onto pypi.
# It assumes that TESTPYPI_API_TOKEN and PYPI_API_TOKEN secrets from GITHUB have been defined
# at the repo or organization levels to upload the package via API authentification.
#
# It will trigger the moment a new release or pre-release is being published.
#
# Copyright (c) 2023 fpavogt; [email protected]

name: CI_pypi

on:
release:
types: [published]

jobs:
pypi:

runs-on: ubuntu-latest
steps:

- name: Checkout current repository
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10'

- name: Install dependancies
run: |
python -m pip install --upgrade pip
pip install setuptools
pip install wheel
pip install twine
shell: bash

- name: Build the wheels
run: |
python setup.py sdist bdist_wheel
shell: bash

- name: Deploy to testpypi
# Let's make use of Github secrets to avoid spelling out secret stuff
env:
TESTPYPI_TOKEN: ${{ secrets.TESTPYPI_API_TOKEN }}
# We first go to testpypi to make sure nothing blows up.
run: |
twine upload -r testpypi dist/* --verbose --skip-existing -u __token__ -p "$TESTPYPI_TOKEN"
shell: bash

- name: Deploy to pypi
# Let's make use of Github secrets to avoid spelling out secret stuff
env:
PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
run: |
twine upload dist/* --verbose --skip-existing -u __token__ -p "$PYPI_TOKEN"
shell: bash
34 changes: 19 additions & 15 deletions .github/workflows/pylinter.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@
import glob
import os
import re
from pylint import epylint as lint
from io import StringIO
from contextlib import redirect_stderr
from pylint.lint import Run
from pylint.reporters.text import TextReporter


def main():
Expand Down Expand Up @@ -50,16 +53,15 @@ def main():
if args.restrict is not None:

error_codes = ','.join(args.restrict)
pylint_command = '--disable=all --enable='+error_codes
pylint_command = ['--disable=all', '--enable=' + error_codes]

# or do I rather want to simply exclude some errors ?
elif args.exclude is not None:
error_codes = ','.join(args.exclude)
pylint_command = '--disable='+error_codes
pylint_command = ['--disable=' + error_codes]

else: # just run pylint without tweaks

pylint_command = ''
pylint_command = []

# Get a list of all the .py files here and in all the subfolders.
fn_list = glob.glob(os.path.join('.', '**', '*.py'), recursive=True)
Expand All @@ -68,13 +70,13 @@ def main():
for bad_item in [os.path.join('.', 'build'), os.path.join('.', 'docs')]:
fn_list = [item for item in fn_list if bad_item not in item]

# Turn this into a string to feed pylint
fn_list = ' '.join(fn_list)

# Launch pylint with the appropriate options
(pylint_stdout, pylint_stderr) = lint.py_run(fn_list + ' ' + pylint_command, return_std=True)

# Extract the actual messages
# Launch pylint with the appropriate options, and get the results back out
# https://pylint.pycqa.org/en/v2.16.1/development_guide/api/pylint.html
# Let's also make sure to catch the stder
pylint_stdout = StringIO() # Custom open stream
reporter = TextReporter(pylint_stdout)
with redirect_stderr(StringIO()) as pylint_stderr:
Run(pylint_command + fn_list, reporter=reporter, exit=False)
msgs = pylint_stdout.getvalue()

# fpavogt, 2020-12-09: let's check the error messages, in case something went very wrong ...
Expand All @@ -85,7 +87,7 @@ def main():
print('pylint stderr:')
print(err_msgs)

raise Exception('Ouch! The linting crashed ?!')
raise RuntimeError('The linting crashed ?!')

# Extract the score ...
score = re.search(r'\s([\+\-\d\.]+)/10', msgs)[1]
Expand All @@ -96,7 +98,8 @@ def main():
if args.restrict is not None and score < 10:
# Display the output, so we can learn something from it if needed
print(msgs)
raise Exception('Ouch! Some forbidden pylint error codes are present!')
# pylint: disable=broad-exception-raised
raise Exception('Some forbidden pylint error codes are present!')

# If I do not have any restricted errors, then simply show the pylint errors without failing.
print(msgs)
Expand All @@ -105,7 +108,8 @@ def main():
# by a Github Action.
if args.min_score is not None:
if score < args.min_score:
raise Exception('''Ouch! pylint final score of %.2f is smaller than the specified
# pylint: disable=broad-exception-raised
raise Exception('''pylint final score of %.2f is smaller than the specified
threshold of %.2f !''' % (float(score), args.min_score))


Expand Down
81 changes: 80 additions & 1 deletion CHANGELOG
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,93 @@ The format is inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0
This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


## [v0.8.0.dev0]
## [v1.x.y]
### Added:
### Fixed:
### Changed:
### Deprecated:
### Removed:
### Security:

## [v1.0.0]
### Added:
- [fpavogt, 24.07.2023] Add new pypi CI action for automated release
- [fpavogt, 19.05.2023] Ingest SHC info into the profile metadata
- [fpavogt, 16.06.2023] Update docs for v1.0.0

## [v0.9.5.dev0]
### Added:
- [fpavogt, 20.04.2023] Ingest and carry lat/lon data for GDPs and CWS (no uncertainties)
- [fpavogt, 18.04.2023] Introduce and deploy "wvec", a new variable for the Wind vector calculations
### Fixed:
- [fpavogt, 14.04.2023] Fix a bug associated to pandas 2.0.0 in fancy_bitwise_or()
### Changed:
- [fpavogt, 14.04.2023] Parameter names are now proper abbreviations (fixes #282)
- [fpavogt, 14.03.2023] Units of wdir are now typset properly in LaTeX as $^{\circ}$
- [fpavogt, 13.03.2023] Units of RH are now %RH

## [v0.9.4.dev0]
### Added:
- [fpavogt, 23.02.2023] Add first_timestep(.uncertainty) to the CWS netCDFs
- [fpavogt, 22.02.2023] Add new "comment" field to dvas prms (fix #277 wdir info)
### Fixed:
- [fpavogt, 22.02.2023] Fix #280 (optimize DB queries to avoid superfluous conditions)
- [fpavogt, 22.02.2023] Fix #278 and #275
- [fpavogt, 22.02.2023] Fix #276 (correct UTLS, add MUS, rm troposphere)
- [fpavogt, 22.02.2023] Fix #277 (gph AMLS)
- [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
- [fpavogt, 08.02.2023] Break-up dot products to keep memory use low

## [v0.9.3.dev0]
### Fixed:
- [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
- [fpavogt, 08.02.2023] Break-up dot products to keep memory use low

## [v0.9.2.dev0]
### Fixed:
- [fpavogt, 31.01.2023] Ignore NaNs when unwrapping angles (fix #273)

## [v0.9.1.dev0]
### Added:
- [fpavogt, 26.01.2023] Add option to store profile data on disk instead of the db
- [fpavogt, 23.01.2023] Add choice of mid to sync from val
### Fixed:
- [fpavogt, 17.01.2023] Left align ylabels accross suplots (fix #272)

## [v0.9.0.dev0]
### Added:
- [fpavogt, 2023-01-12] Add synop cloud code to GDP and CWS metadata
- [fpavogt, 2023-01-06] Add clean TOD option to biglambda recipe step (fixes #249)
- [fpavogt, 2023-01-05] Sync via global match can be restricted to a specific range of values (fix #269)
- [fpavogt, 2022-11-18] Add/connect UTLS, PBL, and Free Tropopause region flags
- [fpavogt, 2022-11-10] Add a WAS_INVALID flag to keep track of fixed GDP points, e.g. via #205
### Fixed:
- [fpavogt, 2022-11-22] Fix #266 by keeping track of KS alpha level in metadata of CWS
- [fpavogt, 2022-11-14] Include fid in plots and filenames (fix #261)
### Changed:
- [fpavogt, 2023-01-11] Refactor the corr. coeffs. routine to improve performances
- [fpavogt, 2022-11-10] Cleanup filename of KS plots
### Removed:
- [fpavogt, 2021-01-05] Removed 'ucr' in favor of 'ucu' (fixes #268)

## [v0.8.1.dev0]
### Fixed:
- [fpavogt, 2022-11-09] Fix #262 - handle angular wrapping in resample
- [fpavogt, 2022-11-09] Fix Runtime warning caused by uc_tot=0 cases in gdp_vs_cws plots
- [fpavogt, 2022-11-08] Fix #235 and #236 (wdir wrap-around)

## [v0.8.0.dev0]
### Added:
- [fpavogt, 2022-10-28] Add netCDF export recipe step
### Fixed:
- [fpavogt, 2022-10-31] Fix #253 (no more Int64 flags) and #259 (resampled points now inherit the anchor flags)
- [fpavogt, 2022-10-31] Fix #252 and #256
- [fpavogt, 2022-10-28] Fix #254
### Changed:
- [fpavogt, 2022-10-28] Clean-up TAGS and FLAGS, incl. rename of "raw" to "original" (fix #255)

## [v0.7.1]
### Added:
- [fpavogt, 2022-09-13] Add new INVALID flag for GDP points with NaN uncertainties (see #244)
Expand Down
26 changes: 10 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
# Contributing to dvas

|:warning: it is not forseen that dvas will be developed or actively maintained beyond 2023-11-30. |
| --- |

If you have a question about dvas, [jump here](#asking-a-question).
If you want to report a bug with dvas, [jump here](#reporting-a-bug).

If you are still reading this, you may actually be considering contributing to the development of dvas. :heart_eyes: :tada:

There are many ways that you can do so, including by:
- [reporting a bug](#reporting-a-bug)
- fixing an [known issue](https://github.com/MeteoSwiss-MDA/dvas/issues?q=is%3Aissue+),
- implementing a new functionality, and/or
- improving the documentation:
* in the code, with better docstrings
* in this repository (for example this very file !)
* in the website, via the docs `.rst` files
If you are still reading this, you may actually be considering contributing to the development of dvas. At this stage, the development of dvas is not forseen beyond 2021-11-30. The project will
likely be frozen on that date, and kept as-is for legacy purposes.

All these contributions are welcome, and what follows should help you get started. Note that contributing to dvas does *not* necessarily require an advanced knowledge of Python and/or Github. Helping us fix typos in the docs, for example, could be an excellent first contribution. Plus, :anger: typos :anger: are the worst !
Below, we provide some basic elements of the project for the benefit of anyone interested to know
more about it.

## Table of contents

Expand All @@ -37,7 +33,7 @@ The [dvas Github Discussions page](https://github.com/MeteoSwiss-MDA/dvas/discus
If you find something odd/wrong/broken with dvas, first check if it is a [known issue](https://github.com/MeteoSwiss-MDA/dvas/issues?q=is%3Aissue+). If not, please create a new [Github Issue](https://github.com/MeteoSwiss-MDA/dvas/issues). This is the best way for everyone to keep track of new problems and past solutions.

## Essential things to know about dvas
dvas is a Python package composed of two sub-packages: `dvas` lies at its core, with `dvas_recipes` wrapped around it. In practice, dvas also includes a series of parameter and
dvas is a Python package composed of two sub-packages: `dvas` lies at its core, with `dvas_recipes` wrapped around it. In practice, dvas also includes a series of parameters and
utilitarian files related to its Github repository, and a dedicated documentation hosted using Github pages.

For the sake of clarity, and to facilitate the code maintenance, we list here (succinctly) a series of key facts about the dvas code and its repository:
Expand Down Expand Up @@ -70,7 +66,7 @@ For the sake of clarity, and to facilitate the code maintenance, we list here (s
[plantuml server](http://www.plantuml.com/plantuml).

4. **Development utilities:**
* On Windows, linter and tests can be run locally from terminal with `sh .\.dev_utils\linter_bash.bat`
* On Windows, linter and tests can be run locally from a terminal with `sh .\.dev_utils\linter_bash.bat`
resp. `sh .\.dev_utils\test_bash.bat` commands.
* There is a script to update the copyright years in all the `.py` files in the repository, that
can be run as follows:
Expand Down Expand Up @@ -171,9 +167,7 @@ For the sake of clarity, and to facilitate the code maintenance, we list here (s
## Step-by-step guide to contributing
We are currently in the early stages of development of dvas. If you would like to contribute to the code, please contact [[email protected]](mailto:[email protected]).
Until its release, the dvas repository will remain private: branching will thus remain the only way to contribute to the code. To get a local copy of dvas and contribute to its improvement, one can follow the following steps:
Until its release, the dvas repository remained private: branching was the only way to contribute to the code. To get a local copy of dvas and contribute to its improvement, one could then follow the following steps:
0. Make sure you have git installed. Check that the setup is correct:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/acknowledge.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Acknowledging dvas

dvas |version| (DOI:)

2. If dvas was useful for your research, please cite the dedicated article:
2. If dvas was useful for your research, please cite the UAII FInal Report:

.. todo::

When the time comes, include here the link to the dedicated dvas article.
When the time comes, include here the link to the report.

3. dvas relies on external Python libraries that require & deserve to be acknowledged in their own
right. The following LaTeX blurb is one way to do so:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
# -- Project information -----------------------------------------------------

project = 'dvas'
copyright = '2020-2022, MeteoSwiss'
author = 'Luca Modolo, Frédéric P.A. Vogt'
copyright = '2020-2023, MeteoSwiss'
author = 'Frédéric P.A. Vogt, Luca Modolo'
version = vers

# -- General configuration ---------------------------------------------------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ Creating dvas recipes
This page contains information for advanced dvas users that would like to alter existing
processing recipes, or create their own.

.. warning::
The creation of dvas recipes from scratch will not be actively supported beyond 2023-11-30.
The information on this page is provided for legacy purposes only. The devs are very much aware
that this only scratches the surface of what actually lives under the hood of the core dvas
module.

.. hint::

If you plan on coding with dvas, we strongly suggest that you begin by taking a look at the
Expand All @@ -18,26 +24,14 @@ point, take a look at ``demo_script.py``. This Python script, which we suggest y
a Python shell (using ``run demo_script.py``), illustrates the main functions and tools of the core
dvas sub-package. It is therefore an excellent place to familiarize yourself with the dvas spirit.


** AND NOW FOR A BIG PILE OF IMPORTANT INFO THAT NEEDS TO BE SORTED ...**

Naming conventions
------------------

Protected tag names
...................

The following tag names are used by dvas, and thus protected. They are defined in
`dvas/config/definitions/tag.py`:

* ``raw``:
* ``gdp``:
* ``derived``:
* ``empty``:
* ``sync``:
* ``1s``:
* ``resampled``:
* ``delta``:
`dvas/config/definitions/tag.py`.

Default database variable names
...............................
Expand All @@ -52,10 +46,9 @@ implemented by defaults. They follow MeteoSwiss conventions:
* ``pres``: pressure
* ``wdir``: wind direction
* ``wspeed``: wind speed
* ``xxx_ucr``: rig-uncorrelated uncertainty of xxx
* ``xxx_ucs``: spatial-correlated uncertainty of xxx
* ``xxx_uct``: temporal-correlated uncertainty of xxx
* ``xxx_ucu``: true uncorrelated uncertainty of xxx
* ``xxx_ucu``: un-correlated uncertainty of xxx

Parameter names
...............
Expand All @@ -66,10 +59,9 @@ names are applicable (as defined in `dvas.hardcoded`):
* ``val``: primary Profile value
* ``alt``: altitude
* ``tdt``: time delta
* ``ucr``: rig-uncorrelated uncertainty
* ``ucs``: spatial-correlated uncertainty
* ``uct``: temporal-correlated uncertainty
* ``ucu``: true uncorrelated uncertainty
* ``ucu``: un-correlated uncertainty
* ``flg``: flags

The following parameters are applicable to their event metadata:
Expand Down
Loading

0 comments on commit 7c5231c

Please sign in to comment.