Parsing 149 GB of JSON Without Loading It Into Memory

  Part 9 of a series — a Dynamo tunnel geometry that was already finished, a 39-sheet Excel workbook it had never been wired to, the web viewer built to referee the two, and the Revit add-in at the end of it.

When the work stopped, it had left 149 GB of snapshots on a desktop drive. The obvious next task was to delete them. This post is about what happened when I tried to establish that deleting them was safe — which turned out to be a longer job than the deletion, and ended with a different answer than the one I started with.

What 45 days physically left behind

Part of what 45 days left behind: the reference images pulled out of the workbook, on one contact sheet with each source size printed underneath. 57 MB of this, against 149 GB of snapshots.

A single verification artefact: source tunnel section with the production candidate drawn over it, and the raw scale conflict stated in the header. Hundreds of files like this one are what the reports directory actually is.

First, the inventory. All of it measured on 2026-08-01, not recalled.

Artefact	Count	Size
Oracle snapshots (tunnel_mirror_oracle_*.json)	333 files — 188 in the live exports folder, 145 in a July archive	149.5 GB
Reference and verification images	495 PNGs across five categories	57.3 MB
Reports directory as it sits on disk	27,713 files	1.84 GB
… of which is actual work product	8,484 files — 6,442 report JSONs, 605 PNGs, 110 logs	0.42 GB
… the remainder	19,229 files of browser-profile debris from headless capture runs	1.42 GB
Archived logs and dumps	450 files	0.98 GB
Verification scripts	99 (67 validators, 32 producers)	1,361,141 B
Revit add-in source	11,818 lines of C#	DLL 360,448 B
Production payload (current)	66 frames × 33 sheets	35,786,750 B
Session records	49	118 KB of extracts

Two rows are worth pausing on before anything else.

The snapshots are 98.4 % of the total volume and none of them is a deliverable. The deliverable is a 35 MB payload and a 360 KB DLL. Everything else is process.

The reports directory is 69 % junk by file count and nobody noticed for six weeks. Headless-browser screenshot automation writes a fresh browser profile on every run — caches, fonts, dictionaries, local storage — into whatever working directory it is given. Nineteen thousand files accumulated in the same folder as the verification evidence. That is not a serious problem, but it is a good illustration of the general one: output directories accumulate things nobody chose to put there, and the accumulation is invisible until someone counts.

The question, and the answer I expected

The archive's own cleanup plan was written before any of this investigation, and it was unambiguous:

Deletion targets (execute after confirmation): 187 timestamped oracle snapshots, 93.6 GiB — unreferenced, regenerable, and the newest is MD5-identical to _latest. Plus the same class of dump in the July archive, 45.2 GiB. Total roughly 139 GiB.
Keep: tunnel_mirror_oracle_latest.json (0.53 GB) — six validators and the viewer's front-end read it.

Three claims hold that plan up: unreferenced, regenerable, and the newest one is a duplicate anyway. Each is checkable. I checked all three, in that order, and two of them survived.

Claim 1 — unreferenced. Mostly true, and the "mostly" matters

A grep across the codebase for anything that opens a snapshot found exactly one path in use: exports/tunnel_mirror_oracle_latest.json. Six validators declare it as EXPORT_PATH, and the viewer's front-end reads it at one place. No code anywhere reads a timestamped snapshot.

That distinction is the whole content of the claim, and it is easy to state carelessly. "Nothing reads them" is false — something reads _latest constantly. "Nothing reads the 333 timestamped files" is true. The difference between those two sentences is the difference between deleting 149 GB and breaking the entire verification suite.

Claim 2 — the newest is a duplicate. True, and only just

_latest.json hashes to db74d25a0601, byte-for-byte identical to the timestamped snapshot from 07-23 23:55:08. So the file the validators read is a copy of a file already in the deletion list, and keeping one of the pair is sufficient.

The second duplicate pair is the archive folder's own _latest, matching a 07-03 snapshot at 488 MB.

And that is the end of the duplicates. Of 333 files, 331 have distinct content. Which quietly demolishes an argument I had been carrying around without examining it — the sense that a directory of near-identically-named 500 MB files is mostly redundancy. It is not. Every one of those files is a different moment. They are large because the thing being snapshotted is large, and there are many of them because the work iterated many times, not because anything was copied.

Claim 3 — regenerable. This is where it fell apart

Held until the last check, and covered below.

What a snapshot actually is

Worth explaining, since half a gigabyte of JSON per save is not self-evidently reasonable. The snapshot is an oracle export: a single file containing the entire model state that any validator might need to cross-check against — the Excel drawing registry, the arc schema status, the original graph's node inventory, the detail requirements, and then, for every branch on every one of 66 frames, the complete world-coordinate and section-local polylines.

The design is deliberate and it is the reason six validators can read one path. Rather than each check re-deriving geometry from the server, they all read the same frozen artefact, so two validators can never disagree because they measured at different moments. It is a snapshot in the database sense: a consistent read.

The size follows arithmetic, not carelessness. Roughly 65 branches, 66 frames, and tens of thousands of polylines with hundreds of thousands of points, each written as a JSON number pair with full precision. One production payload from the same era holds 25,182 polylines and 238,596 points, and the oracle carries considerably more because it includes the review and audit branches the payload excludes. Half a gigabyte is what that costs in a text format. And the top-level structure grew too — the early generation has 8 top-level keys, the later one up to 13, so even the outline of the file changed under it.

Reading 149 GB without parsing it

Before the third claim could be tested, I needed to know what was actually in the 333 files, and that is not a trivial request. The largest single snapshot is 593.5 MB. A json.load() on one of those builds several gigabytes of Python objects and dies. Multiply by 333 and the naive approach is not slow, it is impossible.

So the extraction ran in two phases with different guarantees, and the split is the interesting engineering.

Phase A — the head buffer

Read only the first 8 MB of the file in binary, expanding 4× at a time up to 192 MB if the target is not inside it. Locate the opening brace of the registry's sections array, then find only its closing brace by brace-matching that is aware of string escapes — so a } inside a quoted string does not terminate the span early. Then json.loads that span alone, which is about 0.8 MB.

That span carries the entire per-sheet structure: sheet name, section number, drawing item count, image count, shape count, role counts and coverage, the image list with roles and hashes and cell coordinates, the shapes, and the exact label-to-value table.

Cost: 0.03 seconds per file. Ten seconds for all 333. Exact values, not estimates, because it is a real JSON parse — just of one one-thousandth of the document.

Phase B — the streaming sweep

Then every byte of every file, read in 16 MB chunks with a 400-byte overlap at the boundaries so that a key name split across two chunks is not lost. Collecting four things: the md5 of the whole file; every JSON key name and its occurrence count, via regex, wherever it appears; the complete set of string values for 23 named low-cardinality keys (role, branch, kind, side, status, schema and so on); and, for the older schema generation, the flat dimensions object parsed in full.

Measured throughput about 261 MB/s. Total 9.6 minutes for 149.5 GB.

It is worth being precise about which number belongs to which phase, because the tidy version of this story — "head-parsing extracted metrics from 333 files in 9.6 minutes" — is wrong in a way that flatters the method. Head-parsing took ten seconds. The 9.6 minutes was reading every byte of all 149.5 GB. Both numbers are real; they measure different things, and the cheap one only got the structured summary.

Was the cheap method accurate?

This is the part I would skip if I were not writing it down. A regex sweep is an approximation unless you prove it is not, so three files were fully parsed with json.load() as ground truth and compared against what the sweep produced:

File size	Keys by full parse	Keys by regex	Missing	Extra	Count mismatches
8.7 MB	68	68	0	0	0
36.0 MB	268	268	0	0	0
43.7 MB	494	494	0	0	0

Phase A was checked on the same files: 33 sections against 33, the registry summary an exact match, the label table an exact match across all 33 sheets, and per-sheet image/shape/item counts matching on every sheet.

So: exact, not approximate. With two limits the method states about itself rather than leaving for a reader to discover.

First, those three files total 88 MB — 0.06 % of the corpus by volume — and the same comparison could not be run on the 500 MB-class files, for exactly the reason the method exists. The accuracy claim is an extrapolation from the small end of the distribution. Second, a regex sweep knows a key's name and not its address: two identically-named keys at different nesting depths are summed together, and path information survives only inside Phase A's parsed span.

Neither limit invalidates the result. Both change what the result may be used for, which is the only useful thing a caveat can do.

What the sweep produced

149.5 GB in, 48.4 MB out — a compression of roughly 3,000 to 1, and the shape of what survived is the answer to "what was in there" as much as any prose summary:

Output	Size	Content
Per-snapshot metrics table	212,401 B	71 columns × 333 rows. One row per snapshot: size, timestamps, schema generation, content hash, duplicate group, registry counts, key-set ID, role-set ID, key occurrence totals, and per-schema fields
Key evolution document	557,196 B	3,829 lines. 189 individual change points — a row exists only where a set actually changed, so the absence of a row is itself information
Snapshot × sheet counts	5,128,360 B	10,238 rows
Distinct set library	3,868,198 B	The actual sets behind the ID columns: 116 JSON key sets, 32 role sets, 2 sheet-count sets, 1 label-key set
Raw per-file records	37,949,221 B	333 JSON lines, one per snapshot — the unreduced form
The extraction scripts themselves	50,390 B	Both extractors, the verification script, and the 39 KB run log, copied in alongside the results

That last row is the part I would insist on. The scripts sit next to their output, in the same folder, so that the method is recoverable from the artefact. Given what the rest of this post is about, storing an extraction alongside the extractor is not thoroughness — it is the minimum.

One internal check worth naming: the per-snapshot section counts sum to exactly 10,238, and the sheet-level table has exactly 10,238 data rows. Two independently produced files agreeing on a total is weak evidence, but it is evidence, and it costs nothing to look.

The set library's numbers are the most compact statement of what happened over 45 days. 116 distinct key sets among 333 snapshots means roughly one snapshot in three introduced a structure that had never existed before. 32 distinct role sets means the vocabulary of named parts was re-cut 32 times. And one label-key set — the Excel dimension labels — means the input never moved at all, which is the finding the next section is about.

What the sweep found: 68 keys becoming 1,593

The named-part vocabulary the key sweep was counting: purple opening, blue pocket, red D300, orange slab, teal trim. Thirty-two distinct role sets across 333 snapshots means this legend was re-cut thirty-two times.

The same region as a distance map — green under 2 px, amber under 8, red over 8. A snapshot is large because it carries every polyline behind pictures like this one, for 66 frames at once.

The first snapshot, 06-18 at 20:24:31, is 8.7 MB with 68 distinct JSON keys. The last, 07-23 at 23:55:08, is 558 MB with 1,593. That is 23.4× the vocabulary and 64× the volume over 35 calendar days — 21 of which had any activity at all.

Date	Snapshots	Size range (MB)	Keys	Δ new	Δ removed
06-18	2	8.7	68	—	—
06-19	19	23–26	185	+117	0
06-20	17	36–46	531	+348	2
06-21	3	84–111	692	+161	0
06-27	13	412–424	959	+267	0
06-28	82	447–459	1122	+164	1
06-29	1	469	1170	+49	1
07-01	1	469	1172	+2	0
07-03	7	486–488	1354	+182	0
07-04	19	487–491	1367	+13	0
07-05	18	491–501	1412	+45	0
07-07	4	501–514	1441	+29	0
07-08	13	510–517	1451	+16	6
07-09	18	517–523	1464	+13	0
07-10	25	522–551	1471	+7	0
07-11	18	547–585	1489	+27	9
07-12	11	584–593.5	1532	+44	1
07-13	21	508–582	1571	+41	2
07-14	11	547–552	1581	+10	0
07-15	13	549–551	1583	+4	2
07-23	17	551–558	1593	+10	0

Three days account for 797 of the 1,525 net new keys: 06-20 (+348, the Excel registry schema arriving), 06-27 (+267, the cut-line and source-trace vocabulary), and 07-03 (+182, the labelled-point and audit-contract layer). Those are the three days on which the concept of the model changed, and each is visible in a key count without reading a line of prose.

The removals are more interesting than the additions, because deletions from a schema are decisions.

• 07-08, six keys removed at once: arc_direction, arc_sweep_deg, start_gap_mm, end_gap_mm, tangent_gap_deg, max_radius_deviation_mm. That is an entire family of arc-fit diagnostics disappearing on one day — the R2/R3 endpoint contract being re-cut so that these measurements no longer described anything.
• 07-11, nine keys including score, exteriorBulgeMm, interiorIntrusionMm, and a key whose name is its own epitaph: p_r2_100_pass_through_legacy_audit.
• 07-15, two keys: DRAIN_SHAFT_LEFT and DRAIN_SHAFT_RIGHT — the connector part that had been added in response to Loop B, restructured out again.

Two shape facts complete the picture. 06-28 alone is 82 snapshots and 37.2 GB — a quarter of the entire corpus, by files and by bytes, in one day, and that is the day twelve verification report families were created (Part 5). And 07-13 is the only day the corpus shrinks: maximum file size falls from 593 to 582 MB, minimum drops to 508, and the role-value count goes down from 341 to 329. Something was removed on purpose that day, and the snapshots record it without explaining it.

Three other things the counts give away

The schema generation switched mid-project, inside a single day. 23 snapshots use an older flat structure organised around frames; 310 use the registry structure. The changeover happens inside 06-20 — the same day the Excel registry was built — and it is visible as that day's row carrying both generations at once. A schema replacement, mid-flight, with no ceremony, recoverable now only because the files on both sides of it still exist.

The role vocabulary tells a different story from the key count. Roles start at 5, climb to a peak of 341 on 07-11, and settle at 332. Unlike keys, which grow essentially monotonically, roles overshoot and come back — the sign of a naming scheme being tried, found too fine, and consolidated. There are 32 change points in that series, and the last third of the project changes it barely at all.

An anomaly that took thirty seconds to explain and would have taken an afternoon to guess at. Exactly 8 files report 34 sections instead of 33. They are all on 06-20, between 12:35:30 and 15:10:36, and they are the same 8 files that carry the old sheet-count signature and the old dimension table. So the 34 is not a stray sheet in the workbook; it is the pre-correction state of the registry, bounded to a 2.5-hour window and closed by the same edit that settled the dimension values at 15:32:36. The anomaly and its resolution are 157 minutes apart in the same corpus. Nothing outside these files records that it ever happened.

And the calendar itself carries information. There are gaps with zero snapshots on 06-22 through 06-26, 06-30, 07-02, 07-06, and 07-16 through 07-22 — an eight-day silence immediately before the final burst of 17 snapshots on 07-23. The work did not stop in those windows; it moved to things that do not produce oracle exports. A snapshot corpus records the shape of one activity, not the shape of the project, and reading it as an activity log would overstate several of those gaps.

The finding that changed the question

Road and slab, both sides, in one strip. Wide and shallow, almost unreadable at a glance — which is exactly why the numbers, not the picture, were the thing worth keeping.

Production reinforcement in red over the source in black and grey. Regenerating this today would produce today's code's answer, not the answer that was recorded on the day — which is where the deletion argument broke.

Having summarised what varied, the sweep answered a question nobody had asked: what did not vary?

Quantity	Distinct values across 310 snapshots	Value
Sheet count	1	39
Section sheet count	1	33
Embedded image count	1	503
Drawing item count	1	606
Shape count	1	103
Excel dimension label key set	1	32 keys, present in every snapshot without variation
Dimension values	2	changed exactly once, at 06-20 15:32:36

Read the right-hand column. The Excel source layer moved once, on the third working day, and never again. Eight early snapshots carry an older signature; the remaining 302 are identical in every one of these respects.

Which means the whole thing resolves into one sentence: the 149.5 GB of variation across 333 snapshots is entirely derived geometry. Not one byte of it is a change in the source. The input was settled by 20 June, and everything after that is 35 days of the same 32 dimensions being turned into shapes in progressively different ways.

One honest exception, found while checking rather than assumed away. The workbook's own byte size is recorded in each snapshot, and it changed once: at 07-23 23:27:56 it reads 1,678,568 instead of 1,685,149 — 6,581 bytes smaller. That snapshot's extracted drawing layer, dimension table, label keys and image count are all identical to every other. So the .xlsm binary was touched near the end while the content the pipeline consumes was unchanged. It does not disturb the conclusion, but a document that says the workbook size was constant "in all snapshots" is wrong in 1 case out of 333, and it is better to say so than to have someone else find it.

And then the .git was empty

Which brings back claim 3: regenerable. The check has two halves, and only the first passes.

Can today's code regenerate a snapshot? Yes, and the inputs are all present: the export endpoint exists in the server, the source workbook is on disk at the exact byte size the snapshots record, and the 505 extracted PNGs are present. Run it and you get a snapshot.

Can the code that produced a snapshot on 10 July be recovered? No. The repository's .git directory exists and contains nothing — zero entries, at two separate levels of the tree, verified directly rather than inferred from a tool's error message.

So the sentence "these are regenerable, therefore deletable" was doing something subtler than it appeared. It is true of a snapshot and false of that snapshot. Every one of the 333 was produced by a specific version of a generator that changed continuously — 189 of the 333 differ from their predecessor in their key set alone — and none of those versions exists anywhere. The output survived and the thing that made it did not.

Two pieces of honesty about that finding.

First, an empty .git is an observation; "the history was lost" is an inference. I do not know whether that directory ever held commits. It may have been initialised and never used. The measurable fact is that there is no version history available now, and the consequence is identical either way.

Second, and this is the stronger argument, the regeneration question is partly moot. The extraction deliberately did not capture geometry coordinates — worldPolylines, localPolylines, point coordinates, arc centres — because that is roughly 99 % of the file volume, and capturing it would have meant keeping the 149 GB in a different shape. So the manifest can tell you which branches, roles and keys existed at any moment. It cannot reproduce the section shape at any moment. Delete the originals and that ability is gone, generator history or not.

A third loss is smaller and, to me, the one that stings. The sweep captured values only for 23 low-cardinality keys. Long strings were excluded above 160 characters, which means the note, basis and reason fields did not survive. Those are the fields where a generator explained, in words, why it had done something. The numbers are summarised and the reasoning is gone — which is an exact inversion of what this whole series has argued is worth keeping.

The conclusion, and whose it is

The grouting pattern with the source masks stripped out, so what is on screen is only what the model built. Small, cheap, and reproducible — the opposite of the files under discussion.

The barrier distance map, with the source held in grey behind. Red is where the reconstruction and the drawing still disagree — a record of a state, not a deliverable, and that distinction is the whole question.

The archive's plan said: delete about 139 GiB, keep _latest. After the investigation I do not think that is right, and I want to be clear that this revision is my conclusion, not a decision recorded anywhere. The project's own standing note on the subject says only this:

Clean up snapshots periodically. The reason we chose not to add automatic deletion is that intermediate states are useful during detail work. Instead, repeat a cleanup like this one every few months.

No retention rule. So here is one, with its arithmetic:

Policy	Files kept	Volume	What survives
Delete everything except _latest	1	0.53 GB	The final state. No history at all
Keep the largest snapshot per active day	21	9.02 GB	Every day on which anything happened, including all three schema-explosion days and the one shrink
Keep everything	333	149.5 GB	Everything, including 82 near-identical files from one afternoon

21 files, 9.02 GB, a 94.0 % reduction. The reason the middle row is the right one is visible in the daily table above: the corpus is not evenly distributed in information. One day holds 82 snapshots and 37 GB, and the difference between the first and the last of those 82 is one day's work — but the difference between 06-20 and 06-27 is the model's entire vocabulary changing, and that transition is only observable if a file exists on both sides of it.

Per-day retention keeps every inflection and discards the intra-day repetition, which is the shape of the actual information. Deleting to a single file keeps the destination and throws away the route.

What is an asset, and what is bulk

Production target only, calibration body excluded from the score. This is the asset. The 149 GB is the record of getting here — and the two are not the same kind of thing.

Generalising past this project, the sort into two columns is not close:

Asset — not regenerable	Bulk — regenerable, or worthless
The reasoning: why a value was chosen, what was misread, what a correction cost	Derived geometry, at any resolution
Authority decisions — which source wins for which region	Intermediate payloads between two saved states
Adjudications from a human who is no longer available to re-adjudicate	Screen captures of a state that is fully described elsewhere
The verification scripts, and the thresholds' locking comments	Browser profiles, caches, temp dumps
Failure records: what looked correct and was not	Any file whose content is a pure function of files you still have and code you still have

The right-hand column's last row carries the whole argument, and it has two conjuncts. Most people check the first — do I still have the inputs? — and assume the second. In this project the inputs were fully intact and the second conjunct failed silently, at some unknown point, in a directory nobody looked in.

So the transferable rule is narrow and, I think, correct:

If the generator is not under version control, its outputs are not intermediate files. They are the only copy of a result that can never be produced again.

Everything about a generated artefact feels disposable. It has a timestamp in its name, it was written automatically, nobody chose its contents, and there are three hundred more like it. That feeling is a claim about reproducibility, and reproducibility is a property of the toolchain, not of the file. When the toolchain is not versioned, the arrow reverses: the generated output becomes the primary record and the code becomes the ephemeral thing.

This project produced its own proof of that, unrelated to the snapshots. A set of guide documents from the 07-24 work — versions 1 through 6.2 — was destroyed by repeated writes to the same filename. No commits stood behind them. They are confirmed unrecoverable, and they are gone for the ordinary reason: each overwrite was the correct action for that moment, and the sequence was not.

There is a second, smaller rule hiding in the reports directory. 19,229 of its 27,713 files were never chosen by anyone — browser profiles written as a side effect of automated screen capture, into the same folder as the evidence. Nobody decided to keep them; nobody decided anything about them at all. That is how most storage gets consumed in practice: not by deliberate hoarding, but by a tool defaulting its scratch space into a directory that happens to be the current one. Give automation its own scratch directory, or your evidence folder becomes indistinguishable from a cache — and the first casualty is anyone's ability to say what is in there without a script.

Three practical consequences follow, and all of them are cheaper than the investigation described in this post:

1. Version the generator before you generate at volume. A repository with commits would have made the entire question trivial: delete freely, regenerate on demand. The cost of that would have been one command on day one.
2. Write the reasoning somewhere the sweep would keep. The note, basis and reason fields died because they were long strings inside a huge file. The same sentences in a 4 KB markdown document would have survived every cleanup in this project's history and cost nothing to carry.
3. Decide the retention rule while the files are being produced, not afterwards. Per-day retention is trivially enforceable by the process that writes the file and requires a 149 GB forensic exercise to apply retroactively. The rule is identical in both cases; only the price of implementing it differs, by about four orders of magnitude.

For what it is worth, the arithmetic of doing it the expensive way: 9.6 minutes of machine time and most of a working day of analysis, to justify deleting files that had not been read by any program in five weeks. That is not an argument against doing it — the investigation reversed its own starting conclusion, which is the only real test of whether an investigation was worth running. It is an argument for not needing to.

Closing the series

Forty-five days. An Excel workbook turned into a coordinate contract with 69 dimension bindings and a six-level authority ladder. A web viewer used as a controlled mirror in front of the expensive tools. A raster gate with thresholds locked before the first run, which spent most of its life proving that its own failures were measurement artefacts — and was right to. A Revit add-in of 11,818 lines that gets two parts out of nine cut.

What is not done, stated plainly:

• Cross-passage opening subtraction fails on six of nine parts. Three of those carry a 21 % geometric distortion originating in my own server code, in a band-stitching function that never aligned its two loops' start points. The recommended fix has not been applied.
• The final add-in build has never run in Revit. Not once. The self-intersection removal it exists for is unverified, and no execution log carries its version string.
• All seven adjudication questions were answered the same night they were asked, and the summary table's answer column was never filled in — it still reads "(to be entered)" for every one of them. Two of the seven also came back answering a different question than the one asked: the construction rules arrived, the member names did not. A project history written seventeen days later read the empty column instead of the detailed document sitting beside it, and recorded that no reply had been received. Getting an answer and pinning it into the record turned out to be two different jobs; Part 8 reconstructs what was actually said.
• The viewer and the add-in now disagree about support spacing, because the pattern table found on the last day was wired into one and not the other.
• Cross-passage dimensions have no source. Zero of the 121 Excel dimension keys relate to it; the values are defaults from a reference tunnel's drawings.

Every item on that list was written down as it happened, in the archive, by the thing that made the mistake. That is the only part of this I would defend without qualification — not the geometry, not the gate, not the add-in, but the habit of recording the failure at the same resolution as the success.

The 149 GB will be cut to 9. The reasoning will be kept. And the next project starts with a repository that has commits in it.

---

Tunnel automation series — nine parts.
← Previous: AI Clarifying Questions: A Template From Seven Real Ones (not published yet)
Start of the series: Dynamo Built the Whole Tunnel and Never Read the Excel File