Add support for Azure DevOps API-based datasource

[jessehouwing]

What would you like Renovate to be able to do?

Renovate has support for Azure-DevOps built-in tasks and the Azure DevOps Marketplace's public tasks.

But it doesn't support tasks that were installed directly into an Azure DevOps Organization (tfx build tasks upload) nor does it support extensions that are privately listed on the marketplace.

Renovate also won't be too helpful for people using Team Foundation Server or Azure DevOps Server, since these might not even have all tasks available, nor won't they have the same list of built-in tasks available to them.

Each instance of Azure DevOps however, has a set of APIs that one can query to provide grab the data needed directly from the instance. Below is a set of APIs to query to get all the information required to give proper advice

If you have any ideas on how this should be implemented, please tell us here.

• Have a way to configure the PAT to access the marketplace and the Azure DevOps Organization URI
• Potentially support Azure DevOps Server
• Query all tasks installed in an organization to get the task name's and ids (GET https://dev.azure.com/jessehouwing/_apis/distributedtask/tasks/)
  • Tasks can be identified by:
    • task.name
    • task.id
    • task.contributionIdentifier.task.name
    • task.contributionIdentifier.task.id
  • The built-in tasks can be identified by "serverOwned":true
• Query each task by id to get a list of versions (GET https://dev.azure.com/jessehouwing/_apis/distributedtask/tasks/{{task.Id}}?allversions=true
• Use the ResourceArea's API to find the ExtensionManagement API (GET https://dev.azure.com/jessehouwing/_apis/resourceareas/)
• Fetch the extention manifest from the marketplace API (GET https://extmgmt.dev.azure.com/jessehouwing/_apis/extensionmanagement/installedextensionsbyname/jessehouwing/jessehouwing-vsts-variable-tasks?api-version=7.1-preview.1&assetTypes=Microsoft.VisualStudio.Services.Manifest) using the contribution identifier task.contributionIdentifier.
• Grab the repository data (repository.uri) from the fetched manifest file (GET https://jessehouwing.gallerycdn.vsassets.io/extensions/jessehouwing/jessehouwing-vsts-variable-tasks/3.0.10/1670319234705/Microsoft.VisualStudio.Services.Manifest)

Ideally Renovate should also suggest:

• Tasks renames. These can be found by looking for a task with the same task.id, a higher task.version, but a different task.name.
• Removing deprecated tasks. These can be found by looking at the task.deprecated: true.
• Making it configurable to suggest preview versions task.preview: true.

If RenovateBot would support Private Extensions, then a few new issues pop-up, since private extensions can publish the same tasks (same task.name), even with the same version. In that case a task must be disambiguated by its task.id or by using the long name task.contributionIdentifier.task.name.

The current mechanism, with statically generated files, would allow suggestions for Azure DevOps Server in the case of a non-marketplace connected server instance where the admin would have to download and install the extension manually.

We should probably make the desired sources configurable, as renovatebot might expose the names of internal extensions and tasks through calling out to GitHub.

[jessehouwing]

@JamieMagee I found the old Azure Pipelines issue and for most of the problems with the current extension, I think I know the solution and have documented the correct APIs to use above.

• Related: Support for Azure Pipelines Tasks #15818

[bdovaz]

Related renovatebot/azure-devops-marketplace#42

If it is not fixed it is very confusing because it creates a PR for you to upgrade to a version that does not exist in your organization's instance yet since from what @jessehouwing comments not all instances are upgraded at the same time nor do you know when it will be upgraded.

That detail is beyond the user's control, i.e. Microsoft decides when to update each instance in the Azure DevOps Server (Cloud).

Is there at least some workaround @jessehouwing that can be done with package rules (consume the organization's json and check if that version exists or not as a precondition)? Or is it a too advanced case for package rules?

[mathisgauthey]

Looking for the same thing, have you found a workaround ?

[bdovaz]

@jessehouwing to advance this feature could not be divided into several iterations? Because otherwise I see that it will never be addressed?

The basic thing is that the data source changes from that static source in that repository to use the API you mention so that at least the PRs are already reliable in that sense and the version to which you are going to update really exists in your organization.

The manager is in beta so it makes sense to first cover the most common and basic cases and then extend to more advanced cases: Azure DevOps Server, GitHub + Azure DevOps, task renames, task deprecations, task previews, ...

[bdovaz]

@jessehouwing how can we get this issue moving? I wouldn't mind starting to work on at least the basics (use the API instead of the static source) through a PR but it doesn't even have its relevance with its corresponding issue yet, it's just a mere discussion....

[jessehouwing]

It was moved to a discussion, because the core team wasn't picking this up. I could build this in C# or Pwsh, but not easily in the right language for RenovateBot.

I'd say, get started, make us happy.

[bdovaz]

@jessehouwing PR created #32966

[RDalziel]

Am i right in following this that this will also be able to cover custom extensions which have been published internally?

[jessehouwing]

Yes, the above proposal would also work with private extensions.