Problems with "environment setup" in JePL

[AJRubio-Montero]

Good Morning,

I'm trying replicate my pipeline, that only runs simple tests:

https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/onedataSim/job/dev-ajrubio-montero/

Into the "dev" branch of my repo:

https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/onedataSim/job/dev/

However, the "Environment Setup" stops now in this command:

The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Avoid second fetch
Cloning repository https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git init /jenkins/workspace/eosc-synergy-org_onedataSim_dev/onedatasim # timeout=10
Fetching upstream changes from https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > /usr/bin/git fetch --tags --progress -- https://github.com/EOSC-synergy/onedataSim +refs/heads/*:refs/remotes/origin/* # timeout=10
 > /usr/bin/git config remote.origin.url https://github.com/EOSC-synergy/onedataSim # timeout=10
 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/build-S0
Seen branch in repository origin/build-S1
Seen branch in repository origin/dev
Seen branch in repository origin/dev-ajrubio-montero
Seen 4 remote branches
 > /usr/bin/git show-ref --tags -d # timeout=10

and Jenkins is showing this error:
"Check out from version control | Couldn't find any revision to build. Verify the repository and branch configuration for this job".

It could seem the last command is not returning any repository TAG, but I tested it by hand.

On the other hand, actually the step are checking out a new revision ("aee4..."), which is used in the pipeline.

For example, for my branch in the pileline 44:

The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git init /jenkins/workspace/g_onedataSim_dev-ajrubio-montero/onedatasim # timeout=10
Fetching upstream changes from https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > /usr/bin/git fetch --tags --progress -- https://github.com/EOSC-synergy/onedataSim +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision aee44a894a636f62b245bafcca3a67447c4ee82f (origin/dev)
Enabling Git LFS pull
Commit message: "Update config.yml"
Cleaning workspace
 > /usr/bin/git config remote.origin.url https://github.com/EOSC-synergy/onedataSim # timeout=10
 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/build-S0
Seen branch in repository origin/build-S1
Seen branch in repository origin/build-S2
Seen branch in repository origin/dev
Seen branch in repository origin/dev-ajrubio-montero
Seen branch in repository origin/dev-ajrubio-montero-backup
Seen branch in repository origin/dev-asoreyh
Seen branch in repository origin/dev-backup
Seen branch in repository origin/dev-rpaganmunoz
Seen 9 remote branches
 > /usr/bin/git show-ref --tags -d # timeout=10
 > /usr/bin/git config core.sparsecheckout # timeout=10
 > /usr/bin/git checkout -f aee44a894a636f62b245bafcca3a67447c4ee82f # timeout=10
 > /usr/bin/git config --get remote.origin.url # timeout=10
 > /usr/bin/git lfs pull origin # timeout=10
 > /usr/bin/git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > /usr/bin/git reset --hard # timeout=10
 > /usr/bin/git clean -ffdx # timeout=10

Afther this, in the pipeline 45, the step detects the previous checkout "aee44...":

The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Avoid second fetch
Checking out Revision aee44a894a636f62b245bafcca3a67447c4ee82f (origin/dev)
Enabling Git LFS pull
Cloning repository https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git init /jenkins/workspace/g_onedataSim_dev-ajrubio-montero/onedatasim # timeout=10
Fetching upstream changes from https://github.com/EOSC-synergy/onedataSim
 > /usr/bin/git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > /usr/bin/git fetch --tags --progress -- https://github.com/EOSC-synergy/onedataSim +refs/heads/*:refs/remotes/origin/* # timeout=10
 > /usr/bin/git config remote.origin.url https://github.com/EOSC-synergy/onedataSim # timeout=10
 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/build-S0
Seen branch in repository origin/build-S1
Seen branch in repository origin/dev
Seen branch in repository origin/dev-ajrubio-montero
Seen 4 remote branches
 > /usr/bin/git show-ref --tags -d # timeout=10
 > /usr/bin/git config core.sparsecheckout # timeout=10
 > /usr/bin/git checkout -f aee44a894a636f62b245bafcca3a67447c4ee82f # timeout=10
 > /usr/bin/git config --get remote.origin.url # timeout=10
 > /usr/bin/git lfs pull origin # timeout=10
Commit message: "Update config.yml"
Cleaning workspace
 > /usr/bin/git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > /usr/bin/git reset --hard # timeout=10
 > /usr/bin/git clean -ffdx # timeout=10

Thank you very much in advance.
Cheers.

[AJRubio-Montero]

Magically when I modified my branch variable to a static name in .sqa/config.yml

      #branch: ${GIT_LOCAL_BRANCH##*/} 
      branch: 'dev'

The step works fine, but it is not my required behaviour.

Do you know a recipe to enable current branch as an environmental variable in JePL? Why does it work with my branch (dev-ajrubio-montero) but does it not in "dev"?

Thank you in advance,
Cheers.

[orviz]

Morning @AJRubio-Montero if you just want to check out the current version of the current branch (the one that triggered the pipeline execution, so last dev when the dev pipeline runs or last dev-ajrubio-montero when dev-ajrubio-montero pipeline runs) you can remove the definition of your repo under project_repos.

This will check it out to a detached head, so if you want it to have the same branch name (i.e. dev, dev-ajrubio-montero, ..) you need to pass the localBranch flag when calling to pipelineConfig() method

(example) https://github.com/EOSC-synergy/sqaaas-api-spec/blob/prototype/1.1-no-polymorphism/Jenkinsfile#L14
(localBranch docs) https://javadoc.jenkins.io/plugin/git/hudson/plugins/git/extensions/impl/LocalBranch.html

[AJRubio-Montero]

Thanks Pablo.

It is not working for me.

If I'm removing the project_repos definition, Jenkins does not bind the repository with the MV that executes the checks.

If I'm maintaining the definition, choosing among GIT_LOCAL_BRANCH, GIT_BRANCH, etc, the Jenkins cannot find the commit at the repository:

´´

/usr/bin/git config remote.origin.url https://github.com/EOSC-synergy/onedataSim # timeout=10
/usr/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
/usr/bin/git rev-parse origin/${GIT_BRANCH}^{commit} # timeout=10
/usr/bin/git rev-parse ${GIT_BRANCH}^{commit} # timeout=10
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // script
[Pipeline] }
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
´´

(I will write back to the original configuration)

Cheers.

[orviz]

Hi @AJRubio-Montero I just created a simple example of what I suggest using the SQAaaS platform. The pipeline execution did the checkout of dev-ajrubio-montero branch (commit fffa7f2bfe6b88b791c8f573d9326716f22b5db0) without having the onedataSim repo under the project_repos definition (only the sqaaas-tooling is defined there to point to the right Docker images for the checks being defined).

The same pipeline, without any modification, can be used in whatever branch you use and will (should) work. Not sure if it is what you are looking for..

[AJRubio-Montero]

Hello again.

Testing "dev" in production (the dev-ajrubio-montero branch is obsolete)...I tested several ways. I think Jenkins is not binding correctly the repo with the docker.

In your example, you are setting an external repo Is it for tricking Jenkings?

Thanks.
Cheers.

[orviz]

@AJRubio-Montero I see that you ran some test cases. Note that there are a few things to consider when composing the pipeline 'manually', such as the right combination of jpl-validator and JePL version (in jpl-validator-1.2.0 we changed the criteria codes to QC.xxx), the arguments passed to pipelineConfig() or the working dir & volume stuff in docker-compose.yml (that places the workdir in the right place). The SQAaaS platform handle all these settings for you as I showed with the odsim.sqaaas example above, so I definitely recommend you to start off from this one (only changing the command strings and for the flake8 you will need to keep the current Docker image you defined) or create a new one with the platform.

Also note that the pipeline example (odsim.sqaaas) was created in GitHub by the platform since I wanted to check how it was working, but this is not needed, you could just compose the pipeline and download it without actually creating a new repo.

Regarding you question, setting an external repo is not required at all. In the odsim.sqaaas, the definition of the sqaaas-tooling repo is required since it contains the pointers to the Docker images for the tools used in this pipeline, i.e pycodestyle and bandit. As an example, the JePL pipeline for the SQAaaS API specification does not use them.

[AJRubio-Montero]

Thanks for the explanations Pablo,

I thought updating to new jpl-validator will be easy...

I will explore the new jpl-validator with testing branches, and I will mantain the old-pipeline in production.

I will keep you informed about my advances.

Cheers.