Skip to content

Harden evals with retry logic + centralize logs on Docker host #4440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 7, 2025

Conversation

cte
Copy link
Collaborator

@cte cte commented Jun 7, 2025

Description

Add retry logic for unexpected container failures (due to CPU or RAM spikes).


Important

Adds retry logic for container failures and centralizes logging to Docker host using a new FileLogger class.

  • Behavior:
    • Adds retry logic in processTaskInContainer() in processTask.ts for container failures, with exponential backoff up to 10 retries.
    • Centralizes logs to /var/log/evals on Docker host by modifying docker-compose.yml and runs.ts.
  • Logging:
    • Introduces FileLogger class in FileLogger.ts for logging with levels (INFO, ERROR, WARN, DEBUG).
    • Updates runEvals(), processTask(), and runTask() to use FileLogger for logging.
  • Misc:
    • Modifies createRun() in runs.ts to stop removing Docker containers automatically.
    • Updates docker-compose.yml to mount /tmp/evals to /var/log/evals for logging.

This description was created by Ellipsis for a3c55bb. You can customize this summary. It will automatically update as commits are pushed.

@cte cte requested review from mrubens and jr as code owners June 7, 2025 07:57
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jun 7, 2025
}
} else {
throw error
}
}

logger.close()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To ensure FileLogger cleanup even on non-timeout errors, consider moving logger.close() into a finally block so it always executes.

@daniel-lxs daniel-lxs moved this from Triage to PR [Draft / In Progress] in Roo Code Roadmap Jun 7, 2025
@cte cte merged commit 52673b3 into main Jun 7, 2025
9 checks passed
@cte cte deleted the cte/harden-evals branch June 7, 2025 15:58
@github-project-automation github-project-automation bot moved this from PR [Draft / In Progress] to Done in Roo Code Roadmap Jun 7, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Jun 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant