Skip to content

Latest commit

 

History

History
94 lines (55 loc) · 2.47 KB

README.md

File metadata and controls

94 lines (55 loc) · 2.47 KB

AidData's Geo Framework

Core components of AidData's Geo Framework running on William and Mary's SciClone High Permance Computing cluster.

http://geo.aiddata.wm.edu

Setup

core hpc config

  • make sure HPC account being used is set as priority user on for vortex-alpha nodes (HPC staff can do this)
  • make sure HPC servers have necessary ports open for mongodb, gmail

prepare database server

  • have IT open mongodb ports for geo.aiddata.wm.edu and all HPC servers (prod and dev servers, where applicable)
  • update mongod.conf
  • copy db_backup_script.sh and add cron (see comments in script for details)

prepare sciclone environment

  • install home directory environment scripts (copy from sciclone in this repo to your sciclone account home directory, or extract from the home_backups dir in /sciclone/aiddata10/geo)
  • load necessary modules by logging out and back in to sciclone or using source ~/.cshrc and source ~/.cshrc.rhel6-opteron
  • install python packages: pip install --user -r pip_list.txt
  • add ssh key from your sciclone account to aiddatageo github (approve rsa key first time manually)

initialize framework

  • run bash setup.sh <branch>

Ingest Datasets

  • utilizes data and ingest files from asdf-datasets repo
  • see ingest dir for specifics on ingesting datasets (related resources in asdf-datasets repo)

Components

Tasks

todo

Tools

todo

Utils

todo

Assets

static files/resources used by other scripts (e.g., images, text templates, pdfs)

Scr

Scratch - misc scripts and tools not critical to running geo framework

Tests

py-test based tests for testing geo utiltiies and components

extract-scripts

Tools for manually running batch extract jobs

Related Repos / Resources

Wiki - Additional details about Geo Framework and Sciclone:

Dataset preparation and ingest:

Website source and related config files (Private repos):

Related:

Previous repos for components of the Geo Framework which were combined in this repo: