Create a CI pipeline in github workflows #477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix opensearch plugin removal on latest 2.x - move all docker-related files into root of the project - fix config mounting issue - Update README to make each command individually run inside the app container - Add app files into Dockerfile so that we can start building and publishing it
scottaubrey
changed the title
scottaubrey
added a commit
that referenced
this pull request
Sep 12, 2024
* Improve docker (#476) These changes primarily are to improve the experience of using docker to run the application, though they also start the process of being able to build a docker container for search. Improvements: - Docker-related files are now moved to the top-level directory of the project so we don't need to specify config files when running docker commands. - A SQS queue watcher service and a gearman worker service are now run in docker compose without dev intervention. - Included healthchecks for services and proper dependency resolution for services. - Auto-setup of the queue and KV store indexes. - The application files are copied into the docker image when built - Add missing tools to docker image so that tests can be run in docker (rather than currently only on the search--ci instance) * Create a CI pipeline in github workflows (#477) Create a CI pipeline running in github workflows, that tests and then builds an image * Run CI on develop too (#479) * run CI on develop too * fix syntax for or
scottaubrey
added a commit
that referenced
this pull request
Oct 8, 2024
* WIP: remove gearman * Refactor QueueWatchCommand, Workflow.php, and WorkflowInterface and code improvements * More refactoring Remove Gearman * fix cs * fix tests * Remove unnecessary files, refactor Workflow.php * Fix the test * Remove more Gearman * Remove Gearman from ci tests, docker, and configs * update README.md * add aws-sdk * Refactor * Refactor Workflows * Refactor * added retry mechanism * Remove gearman from bin scripts * running queue:watch in the background in bin/reindex script * redirect outputs to /dev/null * Use localstack v2 and restart opensearch container on failure * Watching queue before running the import and refactoring import by requesting 100 result for each api call * fix cs * Revert to use iterator * Revert "Revert to use iterator" This reverts commit e1f8788. * Lazy loading items to prevent memory exhaustion * enable xdebug in docker * fix a few bugs, and some improvements * remove queue:watch from bin/reindex * fix a minor bug * Add a delay after each indexing * Fix a bug * Clean up * Fix minor bugs and a few improvements * Dockerise Remove gearman (#478) * Improve docker (#476) These changes primarily are to improve the experience of using docker to run the application, though they also start the process of being able to build a docker container for search. Improvements: - Docker-related files are now moved to the top-level directory of the project so we don't need to specify config files when running docker commands. - A SQS queue watcher service and a gearman worker service are now run in docker compose without dev intervention. - Included healthchecks for services and proper dependency resolution for services. - Auto-setup of the queue and KV store indexes. - The application files are copied into the docker image when built - Add missing tools to docker image so that tests can be run in docker (rather than currently only on the search--ci instance) * Create a CI pipeline in github workflows (#477) Create a CI pipeline running in github workflows, that tests and then builds an image * Run CI on develop too (#479) * run CI on develop too * fix syntax for or * reenable building on remove-gearman branch * add missing capabilities for connecting to opensearch * Add limit option to restrict number of items imported * Tidy up * remove superflous transform * move most logic up to AbstractWorkflow * move retry logic up a level to run() * Unify debugging output into AbstractWorkflow * remove further unused code * move most of the constructors to the AbstractWorkflow * Expand tests (#493) * rename method, and add extra debug line. Fix test typing. * remove unused const * remove unused imports * add new test to test behaviours now stored in AbstractWorkflowTest * create a new Indexer component, with ModelIndexers, and Changeset to represent the changes the ModelIndexers want to make * port across the BlogArticleWorkflow logic to ModelIndexer * add CollectionIndexer and test. refactor Indexers and Helpers to be more useful and less coupled * add InterviewIndexer and test * add LabsPostIndexer and test * add PodcastEpisodeIndexer and test * refactor ModelProvider to allow more than one Model definition * add ResearchArticleIndexer and test * fix, called AbstractModelIndexer constructor instead of assuming we know what it does * add ReviewedPreprintIndexer and test * remove old Workflows and wire up new Indexer. Various bits of cleanup along the way * add back in removed tests --------- Co-authored-by: Saeed Moghadamzadeh <[email protected]> Co-authored-by: Saeed <[email protected]> Co-authored-by: Ali Amin <[email protected]> Co-authored-by: Nathan Lisgo <[email protected]> Co-authored-by: Nathan Lisgo <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Create a CI pipeline running in github workflows, that tests and then builds an image