Parsing 149 GB of JSON Without Loading It Into Memory
Part 9 of a series — a Dynamo tunnel geometry that was already finished, a 39-sheet Excel workbook it had never been wired to, the web viewer built to referee the two, and the Revit add-in at the end of it. When the work stopped, it had left 149 GB of snapshots on a desktop drive. The obvious next task was to delete them. This post is about what happened when I tried to establish that deleting them was safe — which turned out to be a longer job than the deletion, and ended with a different answer than the one I started with. What 45 days physically left behind Part of what 45 days left behind: the reference images pulled out of the workbook, on one contact sheet with each source size printed underneath. 57 MB of this, against 149 GB of snapshots. A single verification artefact: source tunnel section with the production candidate drawn over it, and the raw scale conflict stated in the header. Hundreds of files like this one are what the reports directory actually is. Fi...