Embargoes not expiring

[simholt]

Descriptive summary

Example: https://ir.library.oregonstate.edu/concern/graduate_thesis_or_dissertations/1c18dq00d
was supposed to expire to public visibility 2024-12-31 but is still embargoed.

Has this been going on since Sept?
https://ir.library.oregonstate.edu/embargoes?locale=en#expired

Is the cron job running?

[simholt]

Would appreciate some documentation/info about how to keep this from happening again -- has been a regular occurrence.

[decimalator]

I can't get any logging output from sa-prod/jobs, the Deployment that runs crond and some of our scheduled jobs. It should be sending output from crond to the system log but there's nothing there. So, we can't verify whether or not the jobs are running, since they basically only output to STDOUT

I've tried restarting the Deployment a few times to see if we could get output, but no effect.

AFAICT there is no glaring cause. Without logs there's not much to do to troubleshoot.

The solution is to remove the sa-prod/jobs Deployment and re-implement each of the jobs as k8s CronJob objects. This will remove the dependency on crond as a job scheduler and utilize Kubernetes' scheduler to launch Jobs when there are CronJob jobs scheduled to run.

This will leave a number of Pods from previous CronJob runs along with output from the job for analysis. The log entries don't need to be emitted by crond and streamed through the container STDOUT to make it into the container's logs, the container will be able to capture output directly from rake. And we'll get logging and events output so there won't be any ambiguity about whether or not a job ran, or whether it was successful or not.

[simholt]

Okay, that sounds good!

[decimalator]

New CronJob are in place and functioning, I think we're good to close this