Skip to content

Latest commit

 

History

History
79 lines (62 loc) · 2.63 KB

README.md

File metadata and controls

79 lines (62 loc) · 2.63 KB

Metadata Schema Publisher

Automatically publishes metadata schemas to schema.humacellatlas.org when changes are made to the GitHub repo.

Architecture

Architecture of schema.humancellatlas.org metadata publisher

Setup

Install NodeJS dependencies

Run:

npm install

To install required node libraries. Even though the Lambdas are written in Python the framework is in NodeJS and it needs extra plugins.

Install Python dependencies

Run:

pip install -r requirements.txt

Run Locally

Run a function locally

Run with a simulated GitHub event:

serverless invoke local --function onGithubPush --path ./tests/files/events/mock-develop-github-push-event.json

Invoke on AWS

serverless invoke --function onGithubPush --path ./tests/files/events/mock-develop-github-push-event.json

Deploy

  1. cd to deployment dir
cd deployment
  1. Build image
docker build -t lambda-deployment .

docker-compose up -d
  1. SSH into the container
docker exec -it deployment_deployment_1 /bin/bash
  1. Inside the container run, go to the publisher directory and run the deploy command
$ cd code/metadata-schema-publisher/
$ serverless deploy -v

AWS Permissions

If you encounter any permissions in running serverless cli commands manually, you could setup a new user with the same permissions as metadata-schema-publisher-serverless-cli user in AWS. which has adequate permissions to deploy and invoke the publisher lambda function. Alternatively, create a new access token for that user and set it up in your credentials and aws profile config.

Current Deployment

metadata-schema-publisher-prod-onGithubPush `

###Logic

  • On GitHub push event from WebHook
  • Go through all files ending with .json in the json_schema folder in the rep and update content to reflect latest versions in version.json
  • Check the id field in each schema and use that as the key of S3
  • Check if the S3 key exists and if not upload the file
  • Send notifications for when the process starts and a summary when it finishes

Setting up new schema environments

See notes: https://docs.google.com/document/d/1FQo9ofrFTSqLKx2a6eCHqTIpJxhAtShlCVkx6j6yA6s/edit