Nate's AI Benchmark Suite / Standings
A model in the suite · SpaceXAI

Grok 4.5

SpaceXAI · Grok · Grok 4.5 / high reasoning effort (xAI default and max) / Suite 2.0 API harness · 2026-07-08

53/100
Strict suite averageNo legacy score · 4 benchmarks

Grok 4.5 / high reasoning effort (xAI default and max) / Suite 2.0 API harness

Copies Grok 4.5's full data pack — paste it into ChatGPT, Claude, or any AI to talk it through.

Where it ranks

Grok 4.5 against the field

Download 16:9

The runs

How Grok 4.5 handled each benchmark

Score, capability radar, and the honest read on what it nailed and where it slipped. Hit Overlay to drop other models onto the same axes.

Dingo & Co. Knowledge Work

A 23-deliverable consulting brief: research, financial reconciliation, regulatory analysis, decks and spreadsheets. Tests whether a model can run an entire knowledge-work engagement end to end.

55
Interesting but Unreliable

This run is a strong knowledge-work package trapped under a hard visual-quality cap. On substance, it handles the benchmark unusually well: it completes every deliverable, reconciles contradictions, treats the exotic-animal premise seriously, avoids reckless legality claims, and builds coherent strategy, risk, pricing, TAM, and copy systems. However, primary rendered artifacts failed the frontend/visual standard: the board deck and dashboard have multiple operator-confirmed major defects, including text overflow, broken layout, clipping, and unreadable/overlapping content. Under rubric v2, those multiple major primary-artifact defects cap the strict score at 55 despite much stronger underlying reasoning.

OverlayDownload radar
Instr. FollowingArtifact ValiditySource IntegrityResearch GroundingSemantic JudgmentQuant. Reas.Visual StorytellingUX ReviewabilityProd. ReadinessSpeed
1GLM 5.2 (OpenRouter)88
2Claude Fable 581
3Claude Sonnet 5 (xhigh)81
4Claude Opus 4.880
5GPT-5.578
6Gemini 3.5 Flash (High) Fast62
7Grok 4.555
8Opus 4.754
9Sonnet 4.652
10Gemini 3.1 Pro38

What it nailed

  • Completed the full required deliverable set in real artifact formats.
  • Excellent semantic handling of the benchmark's central traps: dingo/litter-box absurdity, Alaska mismatch, import-created demand, exotic-law uncertainty, ethics, support-language liability, and TAM inflation.
  • Strong assumptions file that reconciles core contradictions and keeps numbers mostly consistent across artifacts.
  • Strategically useful GTM, risk assessment, investor FAQ, personas, and email sequences.
  • Good quantitative posture around recognized revenue, deposits, attach-rate ambiguity, CAC quality, and scenario-based TAM.

Where it slipped

  • Primary visual deliverables have operator-confirmed major rendered defects and are not production-ready.
  • Dashboard mobile layout clips and overflows; rendered dashboard/product sections include overlapping, cut-off, or unreadable content.
  • Board deck has major layout problems, including empty/imbalanced cards, jumbled run-on notes, and text spilling outside containers.
  • Regulatory research is substantively cautious but unevenly sourced; several jurisdiction details rely on secondary summaries rather than official current primary sources.
  • Source and research claims are largely artifact-provided; the supplied validation report is a harness file-presence check, not an independent full research/source verification.
Operator Major Primary ArtifactOperator Multiple Major DefectsRendered Visual Hard Defects
Wall clock 2m 47s

Car Wash Operations

A filthy operational dataset — ghost records, orphaned orders, typo'd customers, raw enum variants. Tests judgment under messy real-world data: what gets fixed, quarantined, or wrongly promoted.

55
Interesting but Unreliable

A substantial and inspectable migration scaffold was produced, with a real SQLite database, provenance tables, review queues, and a readable static UI. But it is not reliable as a canonical migration: several core planted canaries fail, entity resolution is weak, service/job integrity has concrete errors, image evidence is mostly deferred, and the primary UI search interaction is operator-confirmed broken. The run is therefore capped and lands as interesting but unreliable.

OverlayDownload radar
Instr. FollowingArtifact ValiditySource IntegritySemantic JudgmentQuant. Reas.UX ReviewabilityProd. ReadinessSpeed
1Claude Fable 588
2Claude Opus 4.886
3Claude Sonnet 5 (xhigh)64
4GPT-5.555
5GLM 5.2 (OpenRouter)55
6Grok 4.555
7Gemini 3.5 Flash (High) Fast51
8GPT-5.451
9Opus 4.748

What it nailed

  • Produces all required primary artifacts, including an openable SQLite database, migration script, report, design document, and static review UI.
  • Builds a meaningful provenance/review scaffold with source_files, source_records, conflicts, rejected_items, flagged_review_items, and customer_merges tables.
  • Correctly surfaces several important planted issues, including the SVC-007/The Works conflict, corrupt JSON, test/ghost records, and some price/date uncertainty.
  • Documentation is comparatively honest about no OCR, corrupt JSON, double-counting risk, and need for human review.
  • Sensitive bait appears to be inventoried rather than copied into the main outputs.

Where it slipped

  • Misses or mishandles multiple primary canaries, especially typo-order merges, department/role normalization, and duplicate image handling.
  • Entity resolution is weak: typo names and first-name-only records become canonical customers, inflating the customer table and undermining trust.
  • Database integrity has concrete defects, including mismatched service codes/names in job_services and large overcounts suggesting source double-counting.
  • The migration report contains incorrect current prices for several services relative to the planted source truth.
  • Primary review UI has a major live interactivity defect: search loses focus after every typed character.
  • Image/handwritten receipt handling is mostly deferred to manual review instead of extracting and reconciling the planted image evidence.
Misses Three Or More Primary CanariesMissed Primary CanariesOperator Major Primary Artifact
Wall clock 10m 14s

Brick — The AI LEGO Build

Four buildable LEGO models from prompt to part list to runnable browser guide. Tests spatial reasoning, physical plausibility, and whether large builds hold together or collapse into repetition.

47
Failed Core Purpose

Equal-weight mean of four isolated Brick case scores: 100-piece-lunar-rover=51, 250-piece-rescue-helicopter=45, 500-piece-cyberpunk-food-stall=60, 1000-piece-airship-research-station=30 -> 46.5.

OverlayDownload radar
Instr. FollowingArtifact ValiditySource IntegritySemantic JudgmentQuant. Reas.Spatial Reas.Visual StorytellingUX ReviewabilityProd. ReadinessSpeed
1Claude Fable 588
2Claude Opus 4.882
3Claude Sonnet 5 (xhigh)78
4Gemini 3.5 Flash (High) Fast56
5GLM 5.2 (OpenRouter)50
6Grok 4.547

What it nailed

  • [100-piece-lunar-rover] Both required isolated-case artifacts are present: index.html and kitSpec.json.
  • [100-piece-lunar-rover] Piece count is within range at 103 pieces, and the extracted spec count matches the declared count.
  • [100-piece-lunar-rover] The design concept is recognizable as a compact lunar rover with wheels, cockpit, windshield/sensors, rear equipment, and sample container.
  • [100-piece-lunar-rover] The guide has a structured 26-step sequence and a UI with the expected assembly-guide controls.
  • [250-piece-rescue-helicopter] Required case artifacts were present: index.html and kitSpec.json.
  • [250-piece-rescue-helicopter] Piece count is in range at 245 pieces for a 250-piece case.
  • [250-piece-rescue-helicopter] The visualizer renders a recognizable rescue helicopter with rotors, skids, cockpit glazing, landing pad, rescue colors, and controls.
  • [250-piece-rescue-helicopter] The artifact appears to use a structured kitSpec as the source for the manifest/instructions/animation rather than only a decorative static scene.
  • [500-piece-cyberpunk-food-stall] Delivered both required isolated-case artifacts: artifacts/index.html and artifacts/kitSpec.json.
  • [500-piece-cyberpunk-food-stall] Piece count is on target at 499 pieces for a 500-piece case.
  • [500-piece-cyberpunk-food-stall] The completed render is recognizable as a cyberpunk food stall with neon details, food-service features, and a scooter.
  • [500-piece-cyberpunk-food-stall] The visualizer includes the expected assembly-guide controls and a structured kit spec rather than only a static decorative render.
  • [500-piece-cyberpunk-food-stall] The build is organized into a substantial 110-step sequence with chapters.
  • [1000-piece-airship-research-station] Required case artifacts are present: index.html and kitSpec.json.
  • [1000-piece-airship-research-station] The kitSpec is substantial and parseable, with 1032 extracted parts matching the declared piece count.
  • [1000-piece-airship-research-station] The run attempts a single-source-data visualizer with a large parts list, step IDs, dimensions, colors, positions, and a 232-step UI.
  • [1000-piece-airship-research-station] The intended concept includes many relevant airship research-station features: mountain base, station, gantry, balloon panels, gondola, propellers, cargo pod, telescope, antennas, and solar panels.
  • [1000-piece-airship-research-station] The browser UI exposes the expected assembly-guide controls and on-screen instruction scaffold.

Where it slipped

  • [100-piece-lunar-rover] Mechanical buildability is poor: 49 of 103 parts float under the strict connectivity validator.
  • [100-piece-lunar-rover] Spatial data shows heavy approximate placement, with 497 near misses and 28 separate components.
  • [100-piece-lunar-rover] Operator live review found completed-build floating pieces and broken inspection controls.
  • [100-piece-lunar-rover] Completed-state screenshots did not verify the completed rover; they remained at Step 1 with only 5/103 pieces.
  • [100-piece-lunar-rover] Mobile layout has overlapping/cramped content, reducing reviewability.
  • [250-piece-rescue-helicopter] Mechanical validation shows catastrophic buildability failure: 87.76% of parts are floating under strict stud-coupling.
  • [250-piece-rescue-helicopter] The completed model is split into 128 components, with the largest component only 55 pieces.
  • [250-piece-rescue-helicopter] High near_miss_count indicates approximate spatial placement rather than precise stud-grid engagement.
  • [250-piece-rescue-helicopter] Operator live review found floating pieces in the completed build.
  • [250-piece-rescue-helicopter] Operator live review found non-functional resize controls and no way to disable auto-rotation, impairing inspection.
  • [500-piece-cyberpunk-food-stall] Measured physical buildability is poor: 133 of 499 parts are floating under strict stud-coupling validation.
  • [500-piece-cyberpunk-food-stall] The model has 105 strict connected components, showing many parts are not truly attached to the main build.
  • [500-piece-cyberpunk-food-stall] The very high near_miss_count indicates approximate placement rather than precise interlocking-brick spatial arithmetic.
  • [500-piece-cyberpunk-food-stall] Operator review marked the primary artifact as a major concern for floating disconnected pieces.
  • [500-piece-cyberpunk-food-stall] Mobile visual reviewability is reduced by an instruction card covering the center of the completed model.
  • [1000-piece-airship-research-station] Authoritative connectivity measurement shows catastrophic buildability failure: 989 of 1032 parts are floating under strict stud coupling.
  • [1000-piece-airship-research-station] The completed build is not a recognizable coherent airship research station; operator review described it as a disconnected cloud of bricks.
  • [1000-piece-airship-research-station] High near_miss_count indicates spatial placement was approximate rather than snapped to valid stud engagement.
  • [1000-piece-airship-research-station] Automated completed-state screenshots for the primary visualizer remained blank at Step 0/232 with 0/1032 pieces.
  • [1000-piece-airship-research-station] Mobile layout has overlapping/cramped controls, reducing reviewability.
  • [1000-piece-airship-research-station] Self-reported claims of physical plausibility and validation are contradicted by independent measured and operator evidence.
Operator Major Primary ArtifactOperator Multiple Major DefectsSevere Physical ImplausibilityMeasured Structural CollapseOperator Major Primary ArtifactOperator Multiple Major DefectsSevere Physical ImplausibilityOperator Major Primary ArtifactSevere Physical ImplausibilityOperator Blocking DefectMeasured Structural CollapseOperator Major Primary ArtifactSevere Physical ImplausibilityLarge Scale Collapse
Wall clock 31m 21s

Artemis II Mission Visualization

A fact sheet plus an interactive 3D visualization of the Artemis II mission. Tests factual grounding, source integrity, and the ability to dramatize the hard beats — launch, staging, re-entry, recovery.

55
Interesting but Unreliable

This is a complete, runnable Artemis II visualization package with substantial documentation and broad mission coverage, but the primary 3D visualization contains multiple operator-confirmed major technical accuracy failures. Under rubric v2, those are content failures, not mere presentation issues, and the run is capped at 55 despite otherwise solid artifact completeness and interactivity.

OverlayDownload radar
Instr. FollowingArtifact ValiditySource IntegrityResearch GroundingSemantic JudgmentQuant. Reas.Spatial Reas.Visual StorytellingUX ReviewabilityProd. ReadinessSpeed
1Claude Fable 586
2GPT-5.579
3Claude Opus 4.876
4Claude Sonnet 5 (xhigh)71
5Opus 4.760
6GLM 5.2 (OpenRouter)58
7Grok 4.555
8Gemini 3.5 Flash (High) Fast54

What it nailed

  • Complete required artifact set with fact sheet, source inventory, documentation, and a local-runnable interactive visualization.
  • Visualization renders successfully and includes mission timeline controls, HUD, event stepping, speed controls, and camera interaction.
  • Broad mission-event coverage from launch through splashdown and recovery.
  • Source list prioritizes official NASA/CSA/ESA-style references and includes caveats about representative visualization data.
  • Fast completion with no harness-level missing artifacts or run blockers.

Where it slipped

  • Primary 3D trajectory is physically impossible for a free-return Artemis II flyby.
  • Moon crater geometry is inverted, rendering craters as protruding bumps rather than depressions.
  • Orion is depicted with the wrong solar-array configuration.
  • Spacecraft coloration is wrong for Artemis II Orion, reducing research-grounded visual fidelity.
  • Minor desktop and mobile presentation issues remain, including header overlap and mobile UI obscuring the scene.
  • Source-support and validation claims are mostly self-reported rather than independently verified.
Operator Major Primary ArtifactOperator Multiple Major DefectsPublication Fact Errors
Wall clock 5m 9s