Gemini Cog

This is a Crank Cog for Gemini, providing steps and assertions for you to validate the state and behavior of your Gemini instance.

• Installation
• Usage
• Development and Contributing

Installation

Ensure you have the crank CLI and docker installed and running locally, then run the following. You'll be prompted to enter your Gemini credentials once the Cog is successfully installed.

$ crank cog:install stackmoxie/gemini

Note: You can always re-authenticate later.

Usage

Authentication

You will be asked for the following authentication details on installation. To avoid prompts in a CI/CD context, you can provide the same details as environment variables.

Field	Install-Time Environment Variable	Description
apiKey	CRANK_STACKMOXIE_GEMINI__APIKEY	Gemini API Key

# Re-authenticate by running this
$ crank cog:auth stackmoxie/gemini

Steps

Once installed, the following steps will be available for use in any of your Scenario files.

Name (ID)	Expression	Expected Data
Compare Gemini model A and B prompt responses from completion (CompletionEqualsAb)	Gemini model (?<modela>[a-zA-Z0-9_-]+) and (?<modelb>[a-zA-Z0-9_-]+) responses to "(?<prompt>[a-zA-Z0-9_ -\p{P}]+)" should (?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<expectation>.+)?	- prompt: User Prompt to send to Gemini Model - modela: Gemini Model A to use for completion - modelb: Gemini Model B to use for completion - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, not be one of) - expectation: Expected Gemini model response value
Check Gemini prompt response from completion (CompletionEquals)	Gemini model (?<model>[a-zA-Z0-9_-]+) response to "(?<prompt>[a-zA-Z0-9_ -\p{P}]+)" should (?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<expectation>.+)?	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, not be one of) - expectation: Expected Gemini model response value
Check Gemini prompt response FRES reading ease evaluation (CompletionReadability)	Gemini model (?<model>[a-zA-Z0-9_-]+) school level of the response to "(?<prompt>[a-zA-Z0-9_ -\p{P}]+)" should (?<operator>be less than|be greater than|be one of|be|not be one of|not be) ?(?<schoollevel>.+)?	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - operator: Check Logic (be, not be, be greater than, be less than, be one of, not be one of) - schoollevel: Expected School Level
Check Gemini semantic similarity of response to provided text from completion (CompletionSemanticSimilarity)	Gemini model (?<model>[a-zA-Z0-9_-]+) response to "(?<prompt>[a-zA-Z0-9_ -\p{P}]+)" semantically compared with "(?<comparetext>[a-zA-Z0-9_ -\p{P}]+)" should (?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<semanticsimilarity>.+)?	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, or not be one of) - comparetext: Expected text to compare to Gemini response - semanticsimilarity: Expected Semantic Similarity Score
Check Gemini prompt response word count from completion (CompletionWordCount)	Gemini model (?<model>[a-zA-Z0-9_-]+) word count in a response to "(?<prompt>[a-zA-Z0-9_ -\p{P}]+)" should (?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<expectation>.+)?	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, or not be one of) - expectation: Expected Gemini word count
Check Gemini prompt token cost given a prompt and model (CompletionTokenCost)	Gemini model (?<model>[a-zA-Z0-9_-]+) ?(?<type>.+)? token cost in response to "(?<prompt>[a-zA-Z0-9_ -]+)" should ((?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<expectation>.+)? tokens	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - type: Specify which token output to show (prompt/completion/total) - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, or not be one of) - expectation: Expected Gemini prompt/response/total token cost
Check Gemini prompt response time from request to completion (CompletionResponseTime)	Gemini model (?<model>[a-zA-Z0-9_-]+) response time in response to "(?<prompt>[a-zA-Z0-9_ -]+)" should (?<operator>be set|not be set|be less than|be greater than|be one of|be|contain|not be one of|not be|not contain|match|not match) ?(?<expectation>.+)? ms	- prompt: User Prompt to send to Gemini Model - model: Gemini Model to use for completion - operator: Check Logic (be, not be, contain, not contain, be greater than, be less than, be set, not be set, be one of, or not be one of) - expectation: Expected Gemini response time in milliseconds

Development and Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to add or update tests as appropriate.

Setup

1. Install node.js (v12.x+ recommended)
2. Clone this repository.
3. Install dependencies via npm install
4. Run npm start to validate the Cog works locally (ctrl+c to kill it)
5. Run crank cog:install --source=local --local-start-command="npm start" to register your local instance of this Cog. You may need to append a --force flag or run crank cog:uninstall stackmoxie/gemini if you've already installed the distributed version of this Cog.

Adding/Modifying Steps

Modify code in src/steps and validate your changes by running crank cog:step stackmoxie/gemini and selecting your step.

To add new steps, create new step classes in src/steps. Use existing steps as a starting point for your new step(s). Note that you will need to run crank registry:rebuild in order for your new steps to be recognized.

Always add tests for your steps in the test/steps folder. Use existing tests as a guide.

Modifying the API Client or Authentication Details

Modify the ClientWrapper class at src/client/client-wrapper.ts.

• If you need to add or modify authentication details, see the expectedAuthFields static property.
• If you need to expose additional logic from the wrapped API client, add a new public method to the wrapper class or mixins, which can then be called in any step.
• It's also possible to swap out the wrapped API client completely. You should only have to modify code within this class or mixins to achieve that.

Note that you will need to run crank registry:rebuild in order for any changes to authentication fields to be reflected. Afterward, you can re-authenticate this Cog by running crank cog:auth stackmoxie/gemini

Tests and Housekeeping

Tests can be found in the test directory and run like this: npm test. Ensure your code meets standards by running npm run lint.