Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CONTRIBUTING.md #102

Merged
merged 8 commits into from
Jan 17, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 34 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,35 +19,53 @@ or

6. Merge to main **and pull to your local machine** (since pulumi will operate on the locally-held version of your code).

## Pre-requisites
## Pulumi pre-requisites

Either set up a new virtual environment:
1. Clone the repo locally and `cd` to this folder.

2. Set up a new virtual environment:
`python3 -m venv <env_name>`
or activate the existing one:
We'll call our environment `env` from here onwards.

3. Activate it:

`source ./env/bin/activate`
Install the packages. If `which pip` returns an error message about pip not found, use `pip3` instead of `pip` in the followng line:
`pip install -r requirements.txt && pip install -r requirements-dev.txt`. Use the `--force-reinstall` flag to replace an existing version if necessary.

Then install the pre-commit hooks with `pre-commit install`.
Check that it is activated:

`which python3`

should return `<path_to_this_repo>/env/bin/python3` instead of your OS's global python executable (which will be something like `/opt/bin/python3`).

4. Install the packages:

`pip3 install -r requirements.txt && pip3 install -r requirements-dev.txt`

Use the `--force-reinstall` flag to replace an existing version if necessary. This may take several minutes depending upon your internet connection.

5. Then install the pre-commit hooks with `pre-commit install`.

## Access the Pulumi stack
## Previewing changes to the Pulumi stack

1. Activate the `restricted-admin-data` AWS role (see [instructions](https://dsdmoj.atlassian.net/wiki/spaces/DE/pages/3862331895/Set+up+AWS+access#Config-file-for-prisons-and-probation) if this isn't already configured).
2. Log in to the Pulumi backend with `pulumi login s3://data-engineering-pulumi.analytics.justice.gov.uk`.
3. Run `pulumi stack select` and pick `data-engineering-exports`.
4. Run `pulumi stack` to check you can see what's currently deployed.
5. Run `pulumi preview` to check the resources look correct. Use the `--diff` flag to see details.
1. Activate the AWS SSO role you use to access `analytical-platform-data-production`:
`aws-vault exec <name_of_your_role>`
If you're not sure which profile to use, consult `aws-vault list` or for even more detail, your `~/.aws/config` file.
3. Log in to the Pulumi backend with `pulumi login --cloud-url s3://data-engineering-pulumi.analytics.justice.gov.uk`.
4. Run `pulumi stack select` and pick `data-engineering-exports`(which may be out-of-sight, just keep hitting the up arrow).
5. Run `pulumi stack` to check you can see what's currently deployed. If asked for a passphrase, hit return.
6. Run `pulumi preview` to check the resources look correct. Use the `--diff` flag to see details.

You may see changes to update the local archive path, which can be ignored. If you are using a different version of `pulumi-aws` to the current deplyment you may see changes relating to the provider, you can avoid these by installing the specific version curently in use, for example, `pip install --force-reinstall pulumi-aws==5.40.0`.

## Deploying changes

6. Deploy the changes with `pulumi up` (there's a ticket to [automate the deployment](https://dsdmoj.atlassian.net/browse/PDE-1441)).
Pre-SSO, data engineers had the permissions to deploy changes. Now you will need to ask someone from the Analytical Platform team to do so in `#ask-analytical-platform` on Slack. As usual, they will deploy the changes with `pulumi up` (there's a ticket to [automate the deployment](https://dsdmoj.atlassian.net/browse/PDE-1441)).

You may also need to set `export PULUMI_CONFIG_PASSPHRASE=""` if you've changed this for other projects.
## Testing deployment

7. Ask the user to test the export. This should include making sure the destination system gets the test file, as we can't see the destination buckets ourselves.
After the stack is live, ask the user to test the export. This should include making sure the destination system gets the test file, as we can't see the destination buckets ourselves.

## Running tests
## Running Pulumi tests

There are normal unit tests that mock Pulumi resources. There is also an end-to-end test that uses Localstack, which creates a mock AWS environment. The test infrastructure should behave like real resources, but none of it needs access to a real AWS account.

Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ Only use lower case and underscores in your dataset name.

5. Commit the file and push it to GitHub
6. Create a new pull request and request a review from the data engineering team. Once this is approved, you can merge your PR: this doesn't happen automatically, so don't forget.
7. Once your changes are in the `main` branch, request a data engineer to `pulumi up` which deploys your changes to the infrastructure. They will tell you when it's ready. If you can't see your new role in IAM (in our example it's `export_<<new_project>>-move`) then your changes haven't been deployed.
7. Once your changes are in the `main` branch, request a data engineer to `pulumi up` which deploys your changes to the infrastructure. They will tell you when it's ready. If you have access to the data engineering SSO role, then you can do this yourself [following the instructions](./CONTRIBUTING.md).
8. If you can't see your new role in IAM (in our example it's `export_new_project-move`) then your changes haven't been deployed. You may need to wait 24 hours.

## Exporting from your bucket

Expand Down
Loading