Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip files and folders that are unchanged when reindexing #19

Merged
merged 5 commits into from
Mar 31, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .autodoc/docs/data/docstore.json

Large diffs are not rendered by default.

Binary file modified .autodoc/docs/data/hnswlib.index
Binary file not shown.
9 changes: 5 additions & 4 deletions .autodoc/docs/json/src/cli/commands/estimate/index.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
{
"fileName": "index.ts",
"filePath": "src/cli/commands/estimate/index.ts",
"url": "https://github.com/context-labs/autodoc/src/cli/commands/estimate/index.ts",
"summary": "The `estimate` function in this code file is responsible for providing an estimated cost of indexing a given repository using the AutodocRepoConfig configuration. This function is particularly useful for users who want to get an idea of the cost involved in processing their repository before actually running the process.\n\nThe function takes an `AutodocRepoConfig` object as input, which contains various configuration options such as the repository name, URL, root directory, output directory, and other settings related to the processing of the repository.\n\nThe main steps involved in the function are:\n\n1. Set the output path for the JSON files generated during the process.\n2. Update the spinner text to display \"Estimating cost...\".\n3. Perform a dry run of the `processRepository` function with the given configuration options. The dry run does not actually process the repository but instead returns the details of the models that would be processed.\n4. Stop the spinner once the dry run is complete.\n5. Print the details of the models obtained from the dry run using the `printModelDetails` utility function.\n6. Calculate the total estimated cost using the `totalIndexCostEstimate` utility function.\n7. Display the estimated cost in a user-friendly format using the `chalk` library.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './autodoc/estimate';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output/',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example demonstrates how a user can call the `estimate` function with a specific configuration to get an estimated cost for processing their repository.",
"questions": "1. **What is the purpose of the `estimate` function and what parameters does it accept?**\n\n The `estimate` function is used to estimate the cost of processing a repository for indexing. It accepts an `AutodocRepoConfig` object as a parameter, which contains various configuration options such as repository URL, output path, and other settings.\n\n2. **How does the `estimate` function calculate the cost estimate?**\n\n The `estimate` function performs a dry run of the `processRepository` command to get the estimated price for indexing the repository. It then uses the `totalIndexCostEstimate` function to calculate the total cost based on the returned run details.\n\n3. **What is the purpose of the `printModelDetails` function and how is it used in the `estimate` function?**\n\n The `printModelDetails` function is used to display the details of the models used in the estimation process. In the `estimate` function, it is called with the values of the `runDetails` object to print the model details before displaying the total cost estimate."
"filePath": "src\\cli\\commands\\estimate\\index.ts",
"url": "https://github.com/context-labs/autodoc/src\\cli\\commands\\estimate\\index.ts",
"summary": "The `estimate` function in this code is responsible for providing an estimated cost of processing a given repository using the Autodoc project. It takes an `AutodocRepoConfig` object as input, which contains various configuration options such as the repository name, URL, root directory, output directory, and other settings related to the processing of the repository.\n\nThe function starts by constructing the path to the JSON output directory, which will be used to store the intermediate results of the processing. It then updates the spinner text to indicate that the cost estimation is in progress.\n\nNext, the `processRepository` function is called with the provided configuration options and a `true` flag to indicate that this is a dry run. This means that the repository will not actually be processed, but the function will return the details of what would happen if it were processed. This is used to calculate the estimated cost of processing the repository.\n\nOnce the dry run is complete, the spinner is updated to show success, and the results are printed using the `printModelDetails` function. The total estimated cost is then calculated using the `totalIndexCostEstimate` function, which takes the values of the `runDetails` object as input.\n\nFinally, the estimated cost is displayed in the console using the `chalk.redBright` function to format the text in a red color. The message also includes a disclaimer that the actual cost may vary and recommends setting a limit in the user's OpenAI account to prevent unexpected charges.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './path/to/this/file';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example would estimate the cost of processing the \"my-repo\" repository with the specified configuration options.",
"questions": "1. **What is the purpose of the `estimate` function?**\n\n The `estimate` function is used to perform a dry run of the `processRepository` command to get an estimated price for indexing the given repository. It then prints the model details and the total estimated cost.\n\n2. **What are the parameters passed to the `processRepository` function?**\n\n The `processRepository` function is called with an object containing the following properties: `name`, `repositoryUrl`, `root`, `output`, `llms`, `ignore`, `filePrompt`, `folderPrompt`, `chatPrompt`, `contentType`, `targetAudience`, and `linkHosted`. Additionally, a second argument `true` is passed to indicate that it's a dry run.\n\n3. **How is the total estimated cost calculated and displayed?**\n\n The total estimated cost is calculated using the `totalIndexCostEstimate` function, which takes an array of values from the `runDetails` object. The cost is then displayed using `console.log` with `chalk.redBright` for formatting, showing the cost with two decimal places and a note that the actual cost may vary.",
"checksum": "2b0b3903432ae423bbc597d04b052ecb"
}
18 changes: 10 additions & 8 deletions .autodoc/docs/json/src/cli/commands/estimate/summary.json
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
{
"folderName": "estimate",
"folderPath": ".autodoc/docs/json/src/cli/commands/estimate",
"url": "https://github.com/context-labs/autodoc/.autodoc/docs/json/src/cli/commands/estimate",
"folderPath": ".autodoc\\docs\\json\\src\\cli\\commands\\estimate",
"url": "https://github.com/context-labs/autodoc/.autodoc\\docs\\json\\src\\cli\\commands\\estimate",
"files": [
{
"fileName": "index.ts",
"filePath": "src/cli/commands/estimate/index.ts",
"url": "https://github.com/context-labs/autodoc/src/cli/commands/estimate/index.ts",
"summary": "The `estimate` function in this code file is responsible for providing an estimated cost of indexing a given repository using the AutodocRepoConfig configuration. This function is particularly useful for users who want to get an idea of the cost involved in processing their repository before actually running the process.\n\nThe function takes an `AutodocRepoConfig` object as input, which contains various configuration options such as the repository name, URL, root directory, output directory, and other settings related to the processing of the repository.\n\nThe main steps involved in the function are:\n\n1. Set the output path for the JSON files generated during the process.\n2. Update the spinner text to display \"Estimating cost...\".\n3. Perform a dry run of the `processRepository` function with the given configuration options. The dry run does not actually process the repository but instead returns the details of the models that would be processed.\n4. Stop the spinner once the dry run is complete.\n5. Print the details of the models obtained from the dry run using the `printModelDetails` utility function.\n6. Calculate the total estimated cost using the `totalIndexCostEstimate` utility function.\n7. Display the estimated cost in a user-friendly format using the `chalk` library.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './autodoc/estimate';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output/',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example demonstrates how a user can call the `estimate` function with a specific configuration to get an estimated cost for processing their repository.",
"questions": "1. **What is the purpose of the `estimate` function and what parameters does it accept?**\n\n The `estimate` function is used to estimate the cost of processing a repository for indexing. It accepts an `AutodocRepoConfig` object as a parameter, which contains various configuration options such as repository URL, output path, and other settings.\n\n2. **How does the `estimate` function calculate the cost estimate?**\n\n The `estimate` function performs a dry run of the `processRepository` command to get the estimated price for indexing the repository. It then uses the `totalIndexCostEstimate` function to calculate the total cost based on the returned run details.\n\n3. **What is the purpose of the `printModelDetails` function and how is it used in the `estimate` function?**\n\n The `printModelDetails` function is used to display the details of the models used in the estimation process. In the `estimate` function, it is called with the values of the `runDetails` object to print the model details before displaying the total cost estimate."
"filePath": "src\\cli\\commands\\estimate\\index.ts",
"url": "https://github.com/context-labs/autodoc/src\\cli\\commands\\estimate\\index.ts",
"summary": "The `estimate` function in this code is responsible for providing an estimated cost of processing a given repository using the Autodoc project. It takes an `AutodocRepoConfig` object as input, which contains various configuration options such as the repository name, URL, root directory, output directory, and other settings related to the processing of the repository.\n\nThe function starts by constructing the path to the JSON output directory, which will be used to store the intermediate results of the processing. It then updates the spinner text to indicate that the cost estimation is in progress.\n\nNext, the `processRepository` function is called with the provided configuration options and a `true` flag to indicate that this is a dry run. This means that the repository will not actually be processed, but the function will return the details of what would happen if it were processed. This is used to calculate the estimated cost of processing the repository.\n\nOnce the dry run is complete, the spinner is updated to show success, and the results are printed using the `printModelDetails` function. The total estimated cost is then calculated using the `totalIndexCostEstimate` function, which takes the values of the `runDetails` object as input.\n\nFinally, the estimated cost is displayed in the console using the `chalk.redBright` function to format the text in a red color. The message also includes a disclaimer that the actual cost may vary and recommends setting a limit in the user's OpenAI account to prevent unexpected charges.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './path/to/this/file';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example would estimate the cost of processing the \"my-repo\" repository with the specified configuration options.",
"questions": "1. **What is the purpose of the `estimate` function?**\n\n The `estimate` function is used to perform a dry run of the `processRepository` command to get an estimated price for indexing the given repository. It then prints the model details and the total estimated cost.\n\n2. **What are the parameters passed to the `processRepository` function?**\n\n The `processRepository` function is called with an object containing the following properties: `name`, `repositoryUrl`, `root`, `output`, `llms`, `ignore`, `filePrompt`, `folderPrompt`, `chatPrompt`, `contentType`, `targetAudience`, and `linkHosted`. Additionally, a second argument `true` is passed to indicate that it's a dry run.\n\n3. **How is the total estimated cost calculated and displayed?**\n\n The total estimated cost is calculated using the `totalIndexCostEstimate` function, which takes an array of values from the `runDetails` object. The cost is then displayed using `console.log` with `chalk.redBright` for formatting, showing the cost with two decimal places and a note that the actual cost may vary.",
"checksum": "2b0b3903432ae423bbc597d04b052ecb"
}
],
"folders": [],
"summary": "The `estimate` function in `index.ts` is a crucial part of the Autodoc project, as it allows users to estimate the cost of indexing a given repository before actually processing it. This function takes an `AutodocRepoConfig` object as input, which contains various configuration options for processing the repository.\n\nThe main steps involved in the `estimate` function are:\n\n1. Setting the output path for the JSON files generated during the process.\n2. Updating the spinner text to display \"Estimating cost...\".\n3. Performing a dry run of the `processRepository` function with the given configuration options. The dry run does not actually process the repository but instead returns the details of the models that would be processed.\n4. Stopping the spinner once the dry run is complete.\n5. Printing the details of the models obtained from the dry run using the `printModelDetails` utility function.\n6. Calculating the total estimated cost using the `totalIndexCostEstimate` utility function.\n7. Displaying the estimated cost in a user-friendly format using the `chalk` library.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './autodoc/estimate';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output/',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example demonstrates how a user can call the `estimate` function with a specific configuration to get an estimated cost for processing their repository. The function is designed to work seamlessly with other parts of the Autodoc project, such as the `processRepository` function, which is responsible for the actual processing of the repository.\n\nBy providing an estimated cost upfront, the `estimate` function helps users make informed decisions about whether to proceed with the indexing process or not. This can be particularly useful for users with large repositories or those who are working within a budget. Overall, the `estimate` function is an essential tool for users looking to leverage the power of Autodoc while managing their costs effectively.",
"questions": ""
"summary": "The `estimate` function in `index.ts` is a crucial part of the Autodoc project, as it provides an estimated cost of processing a given repository. It takes an `AutodocRepoConfig` object as input, containing various configuration options such as repository name, URL, root directory, output directory, and other settings related to the processing of the repository.\n\nThe function begins by constructing the path to the JSON output directory, which stores intermediate results of the processing. It then updates the spinner text to indicate that cost estimation is in progress. The `processRepository` function is called with the provided configuration options and a `true` flag, signifying a dry run. This dry run returns the details of what would happen if the repository were processed, which is used to calculate the estimated cost.\n\nUpon completion of the dry run, the spinner is updated to show success, and the results are printed using the `printModelDetails` function. The total estimated cost is calculated using the `totalIndexCostEstimate` function, which takes the values of the `runDetails` object as input.\n\nFinally, the estimated cost is displayed in the console using the `chalk.redBright` function to format the text in red. The message also includes a disclaimer that the actual cost may vary and recommends setting a limit in the user's OpenAI account to prevent unexpected charges.\n\nHere's an example of how the `estimate` function might be used in the larger project:\n\n```javascript\nimport { estimate } from './path/to/this/file';\n\nconst config = {\n name: 'my-repo',\n repositoryUrl: 'https://github.com/user/my-repo.git',\n root: './',\n output: './output',\n llms: ['en'],\n ignore: ['.git', 'node_modules'],\n filePrompt: true,\n folderPrompt: true,\n chatPrompt: true,\n contentType: 'code',\n targetAudience: 'developers',\n linkHosted: true,\n};\n\nestimate(config);\n```\n\nThis example would estimate the cost of processing the \"my-repo\" repository with the specified configuration options.",
"questions": "",
"checksum": "4b8fd2b2abaec4959873fc3396c414d8"
}
Loading