[$250] Fix Proposal Police from commenting unexpected things on issues / Update to use JSON formatting

[thienlnam]

Example here: #54905 (comment)

The ProposalPolice bot commented something unexpected and this was due to the API call returning something unexpected which was not handled.

Solution:
Update the API to use structured responses which will ensure that the returned parameter is in a format we expect which should prevent situations like this in the future

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~021877133911642803604
• Upwork Job ID: 1877133911642803604
• Last Price Increase: 2025-01-08
• Automatic offers:
• ikevin127 | Contributor | 105617348

Issue Owner

Current Issue Owner: @ZhenjaHorbach

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~021877133911642803604

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ZhenjaHorbach (External)

[melvin-bot]

Triggered auto assignment to @JmillsExpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[melvin-bot]

Upwork job price has been updated to $125

[melvin-bot]

📣 @ikevin127 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

[thienlnam]

Kevin is working on this - slack discussion here

[melvin-bot]

Upwork job price has been updated to $250

[ikevin127]

♻️ Plan

• update AI assistant's instructions to account for recently updated proposal template (new automated tests section)
• update AI assistant's instructions to enforce returning JSON structured data response (see slack discussion for details)
• upgrade and test new AI model for proposal-police to reduce cost of API usage (see slack discussion for details)
• update proposal-police GH action logic to match updated AI assistant's structured data response

[ikevin127]

Updates

• ✅ update AI assistant's instructions to account for recently updated proposal template (new automated tests section)
• ✅ update AI assistant's instructions to enforce returning JSON structured data response (see slack discussion for details): updated the instructions and took advantage of OpenAI's new Assistant Model configuration - Response format to request explicit json structured schema as response
• ✅ upgrade and test new AI model for proposal-police to reduce cost of API usage (see slack discussion for details): changed to gpt-4o-mini (cheapest - see slack discussion for details)

Here's the test repository to get an Idea of action response after the 3 changes mentioned above were implemented: ikevin127/expensify-proposal-testing#1 (comment)

♻️ Tomorrow will proceed with update proposal-police GH action logic to match updated AI assistant's structured data response and perform final testing before opening E/App PR.

[ikevin127]

♻️ Did some more testing and looks like the o1-mini model is not available with AI assistants. I started testing the different cases and neither gpt-4o-mini nor gpt-4o seem to follow the instructions when it comes to wrong proposal template case, even though it does return structured format, it doesn't seem to understand the context when it comes to the proposal template and the mandatory / optional sections.

The only one that's working as expected is the gpt-4-1106-preview model, which is what I use on my side for testing and what proposal police was based on from the beginning.

↪️ I will continue to test on my side with gpt-4-1106-preview, then I'll move forward with to the GH action code changes part. When I'm done I'll report on the testing and we can decide whether :expensify: wants to stay on the gpt-4-1106-preview model given the pricing or explore other alternatives.

After further discussions I'll do some prompt engineering to try and have it work on gpt-4o-mini or gpt-4o because otherwise the costs would be too high.

[ikevin127]

cc @thienlnam @marcochavezf PR is ready for review! 🚀