Run AI-generated code without the risk
Evaluate code produced by language models and agents by actually running it. CodeForge executes it in a sandbox with no network, no credentials and strict limits, then returns the results you need to score it.
No network, by design
Model-written code can’t exfiltrate data, download payloads or call external services.
Benchmark-friendly
Deterministic runtime versions and per-run limits make evaluation runs comparable over time.
Every failure mode, labelled
Compile errors, crashes and timeouts are distinct statuses, ideal for pass@k style scoring.
Retry-safe pipelines
Idempotency keys let long evaluation jobs resume without re-running or double-counting work.
Scoring model outputs against a test (Python)
import os, time, uuid, requests
API = "https://api.tatvastack.com"
HEADERS = {"Authorization": f"Bearer {os.environ['CODEFORGE_API_KEY']}"}
def execute(source, stdin):
eid = requests.post(f"{API}/v1/executions", headers={**HEADERS, "Idempotency-Key": str(uuid.uuid4())},
json={"language": "python", "source": source, "stdin": stdin}).json()["executionId"]
while (r := requests.get(f"{API}/v1/executions/{eid}", headers=HEADERS).json())["status"] in ("QUEUED", "RUNNING"):
time.sleep(0.5)
return r
candidates = generate_solutions(prompt, n=5) # your model call
passed = sum(execute(c, "4\n")["stdout"].strip() == "24" for c in candidates)
print(f"pass@5: {passed}/5")Works the same for Python, JavaScript, Java and C++: change the language field. Full reference in the API docs.
Questions from ai code evaluation teams
Is it safe to run code an LLM wrote?
CodeForge treats every program as hostile: isolated micro-VM-based sandboxes, no network route, no credentials, and strict time, memory and process limits.
Can model code install packages or call APIs?
No. Runs have the standard library only and no network access, which is exactly what makes them safe to evaluate.
How do I run large evaluation batches?
Submit runs concurrently up to your plan’s limit and use an Idempotency-Key per run so a restarted job never duplicates work. Contact us for higher limits.
Built for ai code evaluation
500 free runs a month during the beta. No servers to manage.