PPMS: Verify updated Production endpoint API / key

[jilladams]

Prerequisites

• Recruit Devops help for potential weekend work, if not possible to complete ON Fri Aug 2
  • Verify where / how we can test the Prod key. Will Staging use the key?
• Receive API key / URL from PPMS

User Story or Problem Statement

Original thread from PPMS nprod update: https://dsva.slack.com/archives/C0FQSS30V/p1715982323485409 (#18205)
Timeline for PPMS prod update: https://dsva.slack.com/archives/C03LFSPGV16/p1719940170217209

PPMS is updating Prod API keys and URL that should be used for API calls.
Platform Devops manages API keys for us in AWS param store, AND where the vets-api forward proxy port points to, for those calls.

Announcement:

BLUF: Provider Profile Management System (PPMS) will be migrating it's Azure Environments to a dedicated Azure Subscription. Included in the subscription migration, PPMS will also be upgrading to the Azure Services Environment (ASE) version 3. INT/PPD will begin Testing May 20, 2024 through May 24, 2024. A new URL and the API Subscription Key for your product will be sent this week before testing begins. Please update your new key on 05/20/2024.
BACKGROUND: VAEC informed PPMS that Azure will be deprecating Azure Services Environment (ASE) version 1 & 2 on August 31, 2024. PPMS has taken advantage of this opportunity to upgrade the current ASE v.2, which requires a migration from our current Shared Subscription to a Dedicated Subscription. These improvements will enhance performance/ scalability, simplify management, and aide in futureproofing for ongoing improvements.

UPDATED KEY

Sent from Arlene Jarvis, 7/23, subject line "FW: PPMS Upcoming Migration to Azure Dedicated Subscription (Pre-Production/INT Environment)"

The keys are for the Southwest and East environments. Instructions: PPMS-APIM-KEYs-New-Subscription_Instructions.docx

Southwest: e0b7be49b3f44f80b29da9b69fbd5fa8
East: 0fee6ec934ec4ddb8795fa895e7396a1
PROD APIM Root URL: https://dws.pmt.vaec.va.gov/

Please Note: You may get similar API errors since the new environment mirror the existing production.

TO REPORT ANY ISSUES

Reply to the email chain from Arlene where the API key was provided. They will be monitoring the inbox over the weekend.

Prerequisites / tasks

• Obtain new Prod subscription key from PPMS Team (before August 2)
• Update the Key in the AWS parameter store (on August 2)
• Update the URL used for the forward-proxy to reach PPMS (on August 2)
• By Sunday August 4: Verify on va.gov (using ArgoCD?) that Facility Locator is able to:
  • Return typeahead suggestions for Service when Facility Type is Community Care provider, or Community Care pharmacy
  • Return Community care providers
  • Return Community care pharmacies

Acceptance Criteria

• Facility Locator can return PPMS data (community care) successfully after API key / URL update

[jilladams]

Re: testing from Eli:
For the prod key we cannot verify on dev.va.gov (bc dev uses its own PPMS key for nprod env.)

We need to figure out where we can test:

• Prod only?
• Could we test on Staging? This would require the ability to change the Staging forward proxy destination independent of prod. Devops will need to tell us. If so, we could:
  • Have Devops put API key in the AWS param store
  • Have Devops update Staging Fwdproxy
  • Test/ verify on Staging
  • Have Devops update Prod (and verify)

If Staging points to Dev, we will have to test on prod, and roll back if it fails.

[jilladams]

Started a thread with Devops in the COP channel to see about securing resources over the weekend of 8/2-4: https://dsva.slack.com/archives/C04CYC4LMU6/p1720040117841219

[jilladams]

1. Updated API key has been provided so we are unblocked from this work when the time comes.
2. @eselkin has moved PTO around, so will be on deck to make the necessary AWS changes to the key in param store, and endpoint.
3. Devops is aware we may need their help on that weekend, in case we need rollback, etc.

I think we are fully good to go here, but if anyone has other questions, holler. (cc @Agile6MSkinner @mmiddaugh )

[eselkin]

I drafted PRs to change to regular PRs on 8/2
The reason for 2 is, it is uncertain at that time if DevOps will have moved to the new repository or still using the old one at that time.

[jilladams]

Discussed risks with Michelle, and have sign off on rolling forward to test on Friday 8/2, to allow us easier access to Devops in case of needed rollback. Accepted risk that this may mean a little downtime for community care search, if the testing does not go well and we need to roll back.

[jilladams]

From PPMS office hours today:

• Friday starting at 6pm ET: they will cut over to their new Azure subscription. There was some discussion of some endpoints having downtime but they did not mention CCP, so I do not expect downtime for us.
• By Monday morning, they'll make the Go / No Go if launch was successful. If they have to rollback, they will let us know on Monday morning.

TO REPORT ANY ISSUES: Reply to the email chain from Arlene where the API key was provided. They will be monitoring the inbox over the weekend.

[jilladams]

Code has been deployed, testing is signed off, and PPMS has been notified. Leaving this ticket open until Monday when PPMS makes the final call on whether we are done, or they need to rollback for whatever reason. If PPMS emails that we're good, or if we hear that in PPMS office hours on Tuesday, this can close.

[eselkin]

We got a reply from Arlene Jarvis at PPMS that we are checked off. I don't think they will follow-up with more information unless we ask.

[eselkin]

There are a couple of things not related to the prod rollout
Staging was using (still is using) nprod. I asked for guidance on whether it should be updated to use the new dev or the new prod. I didn't find staging in the configs earlier because it has a different path than dev or prod.

I think it should use the new prod because then we'd have an upgrade path if PPMS changes prod in the future.

However, all updates for prod are done and PPMS signed off on this and decommissioned the former prod endpoint.

[eselkin]

forward proxy changes for sandbox and staging are merged and will go out in the forward proxy deploy tomorrow (missed today by 17 minutes after the review happened). Values are changed in staging parameter store. Staging vets-api will pick up the change at their deploy tomorrow and staging PPMS should resume functioning again at 4:30 ET tomorrow.

[eselkin]

Update fixes staging. Works on staging.va.gov

tested:
https://staging-api.va.gov/facilities_api/v2/ccp/specialties

Not sure why sandbox doesn't work... 503 error which is a different error than expected. There may be other problems with sandbox. I've never actually used it before.