[BUG] Extractor fails with error MethodNotAllowedInPricingTier when attempting to extract groups from consumption tier instances

[james-woodbridge]

Release version

6.0.1.2

Describe the bug

A similar issue was recently resolved by PR #681 to fix the issue #561 where the extractor fails when attempting to extract product groups from a consumption tier instance.

Product groups are now correctly skipped but now the extractor fails with the following error when attempting to extract groups.

Expected behavior

Extractor skips groups as they are not supported for consumption tier

Actual behavior

Extractor fails with the following error:

❯ .\extractor.exe
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Temp\extractor-win-x64
info: extractor[0]
      Running extractor...
info: extractor[0]
      Extracting named values...
info: extractor[0]
      Writing named value information file C:\Temp\apim_extractor_out\named values\66697167a41026c2bbeebebd\namedValueInformation.json...
info: extractor[0]
      Extracting tags...
info: extractor[0]
      Writing tag information file C:\Temp\apim_extractor_out\tags\t-bau-66f299a71f472afa18498b14\tagInformation.json...
info: extractor[0]
      Writing tag information file C:\Temp\apim_extractor_out\tags\t-poc-66f2990863dd3f06b3e82404\tagInformation.json...
info: extractor[0]
      Writing tag information file C:\Temp\apim_extractor_out\tags\t-vsat-66f298de360d62199249346a\tagInformation.json...
info: extractor[0]
      Extracting gateways...
info: extractor[0]
      Extracting version sets...
info: extractor[0]
      Extracting backends...
info: extractor[0]
      Extracting loggers...
info: extractor[0]
      Extracting diagnostics...
info: extractor[0]
      Extracting policy fragments...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetAuthentication-FlowToken\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetAuthentication-FlowToken\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_deleteStagedVsatTemplates\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_deleteStagedVsatTemplates\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_insertCashflow\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_insertCashflow\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_insertSubmission\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_insertSubmission\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_sharepoint_cloneVsat\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_sharepoint_cloneVsat\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_sharepoint_deleteVsat\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-poc_sharepoint_deleteVsat\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-sharepoint_cloneVsat\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-sharepoint_cloneVsat\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertDBAssetAllocation\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertDBAssetAllocation\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertDBMembership\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertDBMembership\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertIndices\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertIndices\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertLiquidityLeverage\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-Flow-vsatInsertLiquidityLeverage\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-dataPlatformVSAT-ENV-data-01\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-dataPlatformVSAT-ENV-data-01\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-dataverse-ENV-data-01\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-dataverse-ENV-data-01\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-out-ENV-data-01\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-out-ENV-data-01\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatbus-ENV-data-06\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatbus-ENV-data-06\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-01\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-01\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-02\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-02\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-03\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-03\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-04\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-04\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-05\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-LogicApp-dataPlatform-vsatout-ENV-data-05\policy.xml...
info: extractor[0]
      Writing policy fragment information file C:\Temp\apim_extractor_out\policy fragments\SetBackend-WebApi-dpapi-graph-uks-it-ENV-data-01\policyFragmentInformation.json...
info: extractor[0]
      Writing policy fragment policy file C:\Temp\apim_extractor_out\policy fragments\SetBackend-WebApi-dpapi-graph-uks-it-ENV-data-01\policy.xml...
info: extractor[0]
      Extracting service policies...
info: extractor[0]
      Writing service policy file C:\Temp\apim_extractor_out\policy.xml...
info: extractor[0]
      Extracting products...
info: extractor[0]
      Writing product information file C:\Temp\apim_extractor_out\products\de-framework\productInformation.json...
info: extractor[0]
      Extracting policies for product de-framework...
info: extractor[0]
      Writing product information file C:\Temp\apim_extractor_out\products\scratchpad\productInformation.json...
info: extractor[0]
      Extracting policies for product scratchpad...
info: extractor[0]
      Extracting groups for product de-framework...
info: extractor[0]
      Writing product information file C:\Temp\apim_extractor_out\products\vsat\productInformation.json...
info: extractor[0]
      Extracting policies for product vsat...
info: extractor[0]
      Extracting groups for product scratchpad...
info: extractor[0]
      Extracting tags for product de-framework...
info: extractor[0]
      Extracting tags for product scratchpad...
info: extractor[0]
      Extracting APIs for product de-framework...
info: extractor[0]
      Extracting APIs for product scratchpad...
info: extractor[0]
      Writing product policy file C:\Temp\apim_extractor_out\products\vsat\policy.xml
info: extractor[0]
      Extracting groups for product vsat...
info: extractor[0]
      Extracting tags for product vsat...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\de-framework\apis\framework\productApiInformation.json...
info: extractor[0]
      Extracting APIs for product vsat...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\authentication\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\cashflow\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\dataplatform\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\dataverse\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\environment\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\nservicebus\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\servicebus\productApiInformation.json...
info: extractor[0]
      Writing product API information file C:\Temp\apim_extractor_out\products\vsat\apis\sharepoint\productApiInformation.json...
info: extractor[0]
      Extracting groups...
crit: extractor[0]
      Application failed.
      System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/1107bbb5-28cf-46e6-8447-1f66008e8925/resourceGroups/rg-uks-it-de2-data-05/providers/Microsoft.ApiManagement/service/apim-uks-it-de2e-data-01/groups?api-version=2023-09-01-preview failed with status code 400. Content is '{"error":{"code":"MethodNotAllowedInPricingTier","message":"Method not allowed in Consumption pricing tier","details":null}}'.
         at common.HttpPipelineExtensions.<>c__DisplayClass4_0`1.<IfLeftThrow>b__0(Response response) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 77
         at LanguageExt.Either.Left`2.Match[B](Func`2 Left, Func`2 Right)
         at common.HttpPipelineExtensions.IfLeftThrow[T](Either`2 either, Uri requestUri) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 73
         at common.HttpPipelineExtensions.GetJsonObject(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 105
         at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+MoveNext() in D:\a\apiops\apiops\tools\code\common\Http.cs:line 87
         at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
         at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIteratorWithTask`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 392
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
         at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
         at System.Threading.Tasks.Parallel.<>c__57`1.<<ForEachAsync>b__57_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IAsyncEnumerableExtensions.IterParallel[T](IAsyncEnumerable`1 enumerable, Func`3 action, Int32 maxDegreeOfParallelism, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Enumerable.cs:line 178
         at common.IAsyncEnumerableExtensions.IterParallel[T](IAsyncEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Enumerable.cs:line 157
         at extractor.GroupModule.<>c__DisplayClass1_0.<<GetExtractGroups>b__0>d.MoveNext() in D:\a\apiops\apiops\tools\code\extractor\Group.cs:line 44
      --- End of stack trace from previous location ---
         at extractor.AppModule.<>c__DisplayClass1_0.<<GetRunApplication>b__0>d.MoveNext() in D:\a\apiops\apiops\tools\code\extractor\App.cs:line 71
      --- End of stack trace from previous location ---
         at common.HostingModule.RunApplication(IHost host) in D:\a\apiops\apiops\tools\code\common\Hosting.cs:line 89
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Unhandled exception. System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/1107bbb5-28cf-46e6-8447-1f66008e8925/resourceGroups/rg-uks-it-de2-data-05/providers/Microsoft.ApiManagement/service/apim-uks-it-de2e-data-01/groups?api-version=2023-09-01-preview failed with status code 400. Content is '{"error":{"code":"MethodNotAllowedInPricingTier","message":"Method not allowed in Consumption pricing tier","details":null}}'.
   at common.HttpPipelineExtensions.<>c__DisplayClass4_0`1.<IfLeftThrow>b__0(Response response) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 77
   at LanguageExt.Either.Left`2.Match[B](Func`2 Left, Func`2 Right)
   at common.HttpPipelineExtensions.IfLeftThrow[T](Either`2 either, Uri requestUri) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 73
   at common.HttpPipelineExtensions.GetJsonObject(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Http.cs:line 105
   at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+MoveNext() in D:\a\apiops\apiops\tools\code\common\Http.cs:line 87
   at common.HttpPipelineExtensions.ListJsonObjects(HttpPipeline pipeline, Uri uri, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
   at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
   at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
   at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIteratorWithTask`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 392
   at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
   at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
   at System.Threading.Tasks.Parallel.<>c__57`1.<<ForEachAsync>b__57_0>d.MoveNext()
--- End of stack trace from previous location ---
   at common.IAsyncEnumerableExtensions.IterParallel[T](IAsyncEnumerable`1 enumerable, Func`3 action, Int32 maxDegreeOfParallelism, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Enumerable.cs:line 178
   at common.IAsyncEnumerableExtensions.IterParallel[T](IAsyncEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in D:\a\apiops\apiops\tools\code\common\Enumerable.cs:line 157
   at extractor.GroupModule.<>c__DisplayClass1_0.<<GetExtractGroups>b__0>d.MoveNext() in D:\a\apiops\apiops\tools\code\extractor\Group.cs:line 44
--- End of stack trace from previous location ---
   at extractor.AppModule.<>c__DisplayClass1_0.<<GetRunApplication>b__0>d.MoveNext() in D:\a\apiops\apiops\tools\code\extractor\App.cs:line 71
--- End of stack trace from previous location ---
   at common.HostingModule.RunApplication(IHost host) in D:\a\apiops\apiops\tools\code\common\Hosting.cs:line 89
   at common.HostingModule.RunHost(String[] arguments, String applicationName, Action`1 configureRunApplication) in D:\a\apiops\apiops\tools\code\common\Hosting.cs:line 24
   at extractor.Program.Main(String[] arguments) in D:\a\apiops\apiops\tools\code\extractor\Program.cs:line 10
   at extractor.Program.<Main>(String[] arguments)

Reproduction Steps

• Download extractor-win-x64 application from release v6.0.1.2
• Set env vars for subscription ID, resource group name, APIM instance name, output folder, and API specification format
• Run extractor
• Extractor succeeds for all APIM components until groups, wherein it fails with the error MethodNotAllowedInPricingTier listed above.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.