AI Agent Runs 187 FEM/CFD Solves: Accuracy Results

DEV Community

An AI Agent Ran 187 FEM/CFD Solves Over MCP. Here's How Many Matched the Textbook

From 6 to 12 November 2026, using dates as recorded in the logs, an AI agent (Claude Code) drove an open-source simulation app, NovaSolver Studio, through its MCP server. Structural, modal, buckling, thermal, flow, electrostatic and DC current problems. Every call and every reply was written to a log. The headline numbers, all from those logs: MCP calls 3,355 (49 log files) Solves started 187 Accepted comparisons with a reference 136 (from 118 analyses) Median difference 0.13% Within 1% / within 3% 105 / 130 Above 5% 2 The accepted set has 132 external-reference comparisons (formulas, series, correlations or independent 2D FEM) and 4 comparisons with another mesh of the same Studio model; 140 comparisons were recorded including reference-only results. The interesting part is not the median. It is what got thrown away before those 136, and two types of setup mistake that passed a mesh or reference check. What "the agent ran it" means here It was not a chat where the model clicked around freely. Claude Code wrote Python scripts that act as an MCP client. Each script starts Studio's MCP server over stdio, calls the tools in a fixed order, and appends every call and reply to a .jsonl file. Claude Code also wrote the STEP geometry, the scripts that compute the reference values, and the scripts that compare the two. The client is small. This is the core of mcpdrv.py, the shared driver: async def call(self, name, args):
t0 = time.perf_counter()
r = await self.s.call_tool(name, {"language": "ja", args})
...
with io.open(self.log, "a", encoding="utf-8") as f:
f.write(json.dumps({"tool": name, "args": args, "reply": text,
"s": round(time.perf_counter() - t0, 2),
"data": getattr(r, "structuredContent", None)},
ensure_ascii=False, default=str) + "\n")
return text

async def wait(self, text):
m = re.search(r"Running as (\S+?).", text)
if not m:
return text
while True:
await asyncio.sleep(5)
rep = await self.call("job_status", job_id=m.group(1))
if rep.startswith(("done", "failed", "stopped", "error")):
return rep
A typical case script then reads like a recipe: create_case or import_geometry, update_case with materials and boundary conditions, validate_case, check_convergence, get_results, verify_case, probe_results. 58% of the calls were waiting job_status was called 1,949 times out of 3,355. Solves run as background jobs. run_simulation and check_convergence return a job id if the job is not done in about 20 seconds, and job_status waits up to 20 seconds per call before it answers. So most of the traffic is the client asking "done yet?". The calls that carry engineering content are an almost flat band of about 170–200 each: set up, check, solve on three meshes, read results, compare. The sum of all call durations is 42,086 seconds (about 11.7 hours). That is not wall time: it does not include the scripts' own time, and jobs that queued behind each other are counted in parallel. From 206 cases to 171 results Of 187 solves started: Outcome Count Out of memory (Studio's pre-check or during the solve) 11 Iterative solver did not converge 3 Tool error (the disk filled up) 1 Result not recorded 1 Results obtained 171 ...with a completed three-mesh test 167 ...mesh tests passed / failed 145 / 22 ...single-mesh pipe calculations 4 Before any solve, validate_case returned RED 8 times. All eight were DC current cases where Joule heating was on by default and no face set a temperature. Four plate updates were rejected; the client continued, and one started job has no recorded completion. Eleven cases were only opened to read the geometry. The accepted set excludes the 800 mm column, which failed the mesh test. Re-solves often covered a whole set after one case failed, to keep the mesh settings consistent. Out of 23 topics, 13 needed at least one redo, design change or abandon...

Powered by Readybot.io