Skip to content

Harden evals with retry logic + centralize logs on Docker host #4440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 7, 2025
Merged

Harden evals with retry logic + centralize logs on Docker host #4440

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fd43557
Harden evals with retry logic + centralize logs on Docker host
cte Jun 7, 2025
9fc7c65
Don't delete controller container on exit
cte Jun 7, 2025
826dd4d
Add randomness
cte Jun 7, 2025
715f8e6
More tweaks
cte Jun 7, 2025
a3c55bb
More tweaks
cte Jun 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
More tweaks
  • Loading branch information
@cte
cte committed Jun 7, 2025
commit a3c55bb824ffa0e6dc880b31fa6ff08c2d074321
6 changes: 3 additions & 3 deletions packages/evals/src/cli/runEvals.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@ export const runEvals = async (runId: number) => {
await queue.addAll(
tasks
.filter((task) => task.finishedAt === null)
.map((task) => () => {
.map((task) => async () => {
try {
if (containerized) {
processTaskInContainer({ taskId: task.id, logger })
await processTaskInContainer({ taskId: task.id, logger })
} else {
processTask({ taskId: task.id, logger })
await processTask({ taskId: task.id, logger })
}
} catch (error) {
logger.error("error processing task", error)
Expand Down
Loading