Data Management

[NewGraphEnvironment]

we can look to download all the data that Laura put together mirroring data directory of what is there. If you don't have room on your computer to put it on onedrive I will put an external hard drive on the chair just inside the door that you can grab whenever. You can update Laura's file with column for the package-name for things that we can just grab from skt directly. Although we may want to just have it all on the drive anyway - it will be good to know what is on skt vs on the harddrive.

understand and run https://github.com/NewGraphEnvironment/restoration_wedzin_kwa_2024/blob/main/scripts/api_skt.R and fetch_package from https://github.com/NewGraphEnvironment/restoration_wedzin_kwa_2024/blob/main/scripts/functions.R . We can look to do blogs to document and sharef how to understand the API and downloading and uploading eventually

Remember that api_skt.R is just one way to look at it and fetch_package is just one way for us to grab things. We can definitely evolve and there could even be issues with what we have resulting in us not seeing everything (doubt it but maybe)..

There is a bunch of background work that was done in https://github.com/NewGraphEnvironment/restoration_wedzin_kwa_2024/blob/main/scripts/archived/api_skt_orgs_groups.R which resulted in some dead ends due to hard coded download limits on skt's side and within the ckan framework itself. Good to be aware but pulled them out to simplify.

[lucy-schick]

Sweet. I will pick up the hardrive this evening and try downloading tomorrow. I just tried downloading now and I am also seeing VERY slow download speeds, it only managed to download 3.02 GB in ~1h, and looks like it has a long way to go (see screenshot). I see that the 2021 folder is already in Onedrive so I will try downloading the folders one at a time. I will work through the other scripts in the meantime. Thanks

[lucy-schick]

I have reviewed and run api_skt.R and fetch_package. Very cool

[NewGraphEnvironment]

right? good times. ckans are amazing and are everywhere. Living Lakes has one too and actually because I showed them SKT a bunch of years ago! https://livinglakescanada.ca/project/columbia-basin-water-hub/

This began years ago with this little repo that really doesn't do very much except test downloads and uploads https://github.com/NewGraphEnvironment/skt_sern

Thoughts right now (things always change once the details are exposed and input from the team will be key - ha) :

1. Through this project - move absolutely all project specific data (or links to data) to SKT? (ie. upload any project data that is not currently there and below a threshold size and uploadlinks or descriptions of where it is to data that is above a certain size - and store it on the cloud somewhere cheap like AWS ). I am thinkin gof things like thousands of 4mb images gathered by UAV to be used to make orthomosaics - or past video footage used for project public relations and education.
2. If we want to migrate outstanding data to SKT we will want to discuss access permissions and organization specifics. We currently have a SERNbc organization authorization key but we may want to get one to upload as the OW organization if that is something their internal team is interested in seeing happen.

[lucy-schick]

I think having everything on skt would be a great idea.

[lucy-schick]

• All of Laura's data is now in OneDrive/Projects/2024-069-ow-wedzin-kwa-restoration
• I've added 2 new columns to Lauras's spreadsheet to include the package_name as well as the package id which is more specific. Still a couple I need to add most have a package_name and id now if its in the skt database.