Skip to content

AITEAM-587: add run_evaluation_tests tool & run_pipeline w. custom config & job parallelism #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jun 3, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0f5d90f
update runPipeline to take arbitrary config
jvincent42 May 14, 2025
d89ac99
test
jvincent42 May 20, 2025
78c49c1
wip
jvincent42 May 22, 2025
3b65258
wip
jvincent42 May 28, 2025
49a788a
Merge branch 'main' into run-pipeline-custom-config
ryan-circleci May 29, 2025
ceaea47
fix: tests and remove `runEvaluationTests.old` files
ryan-circleci May 29, 2025
9e06b13
Merge branch 'main' into run-pipeline-custom-config
ryan-circleci May 30, 2025
9bdd6ab
fix import
jvincent42 Jun 2, 2025
28f0f15
fix import
jvincent42 Jun 2, 2025
9fe73af
wip
jvincent42 Jun 2, 2025
1edd219
update gist
jvincent42 Jun 2, 2025
b43ce05
refactor: `inputSchema` with `promptFile` object
ryan-circleci Jun 2, 2025
d66ea4a
refactor: job name - `evaluate-prompt-template-tests`
ryan-circleci Jun 2, 2025
0c084fd
refactor: convert YAML to JSON
ryan-circleci Jun 2, 2025
3645b58
refactor: Base64 encode file content
ryan-circleci Jun 2, 2025
03acdc0
chore: uninstall `js-yaml`
ryan-circleci Jun 2, 2025
f18f945
chore: attempt gzip & hex encoding
ryan-circleci Jun 3, 2025
13c49c5
chore: try gzip & base64 for shorter string
ryan-circleci Jun 3, 2025
5135e25
refactor: set up eval jobs w `paralellism` for all `promptFiles`
ryan-circleci Jun 3, 2025
be16645
chore: update `runEvaluationTests` unit tests
ryan-circleci Jun 3, 2025
2e08149
chore: remove `console.error`
ryan-circleci Jun 3, 2025
07eb354
chore: make `configContent` as compact as possible
ryan-circleci Jun 3, 2025
6e60a53
Revert "chore: make `configContent` as compact as possible"
ryan-circleci Jun 3, 2025
bf3b500
chore: handle if no prompt template files are provided
ryan-circleci Jun 3, 2025
e04106a
refactor: rename workflow - `mcp-triggered-workflow`
ryan-circleci Jun 3, 2025
70fd6e0
refactor: rename workflow - `mcp-run-evaluation-tests`
ryan-circleci Jun 3, 2025
0ae033a
chore: update `CHANGELOG` & `package.json` to `"version": "0.9.0"`
ryan-circleci Jun 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
update runPipeline to take arbitrary config
  • Loading branch information
jvincent42 committed May 20, 2025
commit 0f5d90f73e7a5b4fd75efcc1d68e535f21d73509
3 changes: 3 additions & 0 deletions src/clients/circleci/pipelines.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,17 +138,20 @@ export class PipelinesAPI {
projectSlug,
branch,
definitionId,
configContent = '',
}: {
projectSlug: string;
branch: string;
definitionId: string;
configContent?: string;
}): Promise<RunPipelineResponse> {
const rawResult = await this.client.post<unknown>(
`/project/${projectSlug}/pipeline/run`,
{
definition_id: definitionId,
config: {
branch,
content: configContent,
},
checkout: {
branch,
Expand Down
2 changes: 2 additions & 0 deletions src/tools/runPipeline/handler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const runPipeline: ToolCallback<{
workspaceRoot,
gitRemoteURL,
branch,
configContent,
projectURL,
pipelineChoiceName,
projectSlug: inputProjectSlug,
Expand Down Expand Up @@ -115,6 +116,7 @@ export const runPipeline: ToolCallback<{
projectSlug,
branch: foundBranch,
definitionId: runPipelineDefinitionId,
configContent,
});

return {
Expand Down
6 changes: 6 additions & 0 deletions src/tools/runPipeline/inputSchema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,10 @@ export const runPipelineInputSchema = z.object({
'If provided, it must exactly match one of the pipeline names returned by the tool.',
)
.optional(),
configContent: z
.string()
.describe(
'The content of the CircleCI YAML configuration file for the pipeline.',
)
.optional(),
});