Large (~1 GB) local JSON file fails to display

Mæstro · Post by **Mæstro** » 2026-02-15, 21:47

Operating system: Debian 10
Browser version: 34·0·1
Problem URL: file:///home/maestro/Herunterladen/Aktivit%C3%A4t/tns/events-2026-00000-of-00001.json (local JSON file)
Browser theme (if not default): Fox 2 The Moon
Installed add-ons: Add As Search Engine 1·0, Aviary Add-ons Manager 1·0·1, Cookie Masters 3·2, Greasemonkey 3·31·4, Image Search Options 2·0·3·7, µBlock Origin 1·16·4·30, Pure URL 3·4, ScrapBook X 1·14·7, URL Rewriter 2·8·5
Installed plugins: (about:plugins): Clean Flash 34·0·0·137

Because I am withdrawing from Discord by month’s end, I deemed it wise to export my data package for my personal archives. Discord prepares these as a series of JSON files. It is easy for me to view small files in Pale Moon, but the largest of these JSON files, that associated with Trust & Safety, happens to be just under a gigabyte. I would like to view its contents, but if I open it through Pale Moon, the browser loads for a while and consumes an additional gigabyte of store*, but the display in the relevant tab is completely blank: none of the developer tools that would show when a smaller JSON file is open, but an appearance identical to about:blank but for the address bar. Closing the tab then frees the claimed gigabyte. Clearly, something is happening invisibly, but I need to see the contents. (I also tried opening in LibreOffice Calc, but it just freezes and possibly crashes.) In contrast, the file appears gradually to load and render correctly if opened using Waterfox, but goes on to consume all my computer’s available store until it freezes and I must perform a hard restart. Pale Moon’s behaviour is much tamer, and this browser is my best hope for exercising my right under the Gdpr to access my data. How can I do so?

*When I first attempted this, my computer froze as all my available store (<5 GB) was used up, forcing a hard restart. On trying again with nothing but Pale Moon and the system monitor open on boot, I could gain a clearer picture of what happened.

Post by **Moonchild** » 2026-02-15, 22:15

I suggest you use a dedicated viewer for it. Pale Moon's internal json viewer isn't designed to create such large data structures. alternatively you can disable the json viewer formatter in preferences which will display it as flat text.

Mæstro · Post by **Mæstro** » 2026-02-15, 22:34

Moonchild wrote: ↑
2026-02-15, 22:15
I suggest you use a dedicated viewer for it.

Before even attempting with Pale Moon, I tried searching my repository for json viewer (no results), json (nothing apparently relevant). Would you know some?

…alternatively you can disable the json viewer formatter in preferences which will display it as flat text.

How can I do that? I looked through all the Preferences tabs, including the Advanced submenus, and could not find anything even about JSON.

adoxa · Post by **adoxa** » 2026-02-16, 00:07

If you don't mind the terminal there's jq or fracjson (redirect these or pipe to a viewer) or fx.

Mæstro · Post by **Mæstro** » 2026-02-16, 00:41

JQ came up before, and is already installed on my system, but I cannot figure out how to use it. The ‘guide’ for jq --help is incomprehensible to me, and articles I find online about how to use it are too full of technical jargon for me to understand. A GUI viewer would be better for me, as a non-technical layman who just wishes to exercise the rights promised to us.

adoxa · Post by **adoxa** » 2026-02-16, 01:11

To just view a JSON file with jq it's simply jq . file.json | less (or whatever your viewer of choice is).

A quick search turned up these: Dadroit & Janice.

Mæstro · Post by **Mæstro** » 2026-02-16, 01:36

I could catch that much from somewhere or other, and pasted that in my terminal, but I had (and have) no idea how to navigate what I was looking at. (This is why I was so perplexed by the manual; I know what an identity map is!) Just now, I have finished waiting half an hour opening the events file in Xed. It worked at first, displaying various entries: all HTTP header data including my apparent IP address and location, which were actually my VPN locations, from early 2024. I tried to scroll through it and search for a few terms which I knew to be present (eg ‘Tokyo’), but the scroll bar disappeared and Xed crashed. This still qualifies as the best results I have yet got.

Dadroit came up in my search results looking for a JSON viewer, but the free version only handles files up to 50 megabytes, and paying to exercise my rights is absurd. I should try the Janice tarball tomorrow (it is 2.30a here).

I could see no sign, in what of my results I could scan, of the sex or age prediction mentioned as appearing under ‘analytics’. No analytics file exists for me, for I disabled this in the privacy settings. Elsewhere, under Werbung [advertising], Discord seems to believe I am Georgian (Tiflis, not Atlanta) and somehow both aged 13–17 and not underage!

adoxa · Post by **adoxa** » 2026-02-16, 02:23

This might have the same issue as PM itself, but there's Discord Data Package Explorer.

You could perhaps use jq to split the big file into smaller files; I'll provide more details if the above or Janice don't work out.

Mæstro · Post by **Mæstro** » 2026-02-16, 16:57

Splitting the big file will be needed. Janice has proven itself as able as any other attempt to build from source or AppImage I have ever attempted, and the DDPE renders scriptless as a blank slate,

Post by **Moonchild** » 2026-02-16, 23:19

Mæstro wrote: ↑
2026-02-15, 22:34
How can I do that? I looked through all the Preferences tabs, including the Advanced submenus, and could not find anything even about JSON.

it's an about:config preference. devtools.jsonview.enabled

adoxa · Post by **adoxa** » 2026-02-17, 01:09

Here's a script to create a file for each key. Not exactly sure how effective or efficient it will be, but without really knowing the structure it's the best I can offer.

Code: Select all

IN=/path/to/events.json
for key in `jq -r 'keys | join(" ")' $IN`; do jq -c .$key $IN > events-$key.json; done

Adjust IN to the path of your JSON; it will create events-*.json files in the current directory.

Mæstro · Post by **Mæstro** » 2026-02-17, 22:11

Moonchild wrote: ↑
2026-02-16, 23:19
it's an about:config preference. devtools.jsonview.enabled

This behaved about as you predicted. More precisely, the file opened in Pale Moon as plain text, consuming about 1 GB store, before several minutes later, possibly due to trying to scroll through it, it went on to consume all available store on the system, rather like what had happened in Waterfox. This confirms that an alternative JSON viewer would be proper if the file cannot be divided, as it appears it cannot (see below).

adoxa wrote: ↑
2026-02-17, 01:09
Here's a script to create a file for each key. Not exactly sure how effective or efficient it will be…

Bash goes on to draw steadily more store over a few minutes as the cursor blinks, without any files being deposited in the relevant directory, until it had got to 2½ GB, using up all available store, whereupon I needed to kill the process to avert total system freeze.

Incidentally, I have studied Discord Data Package Explorer (and its successor, Dumpus) through Waterfox, and it is unfit for purpose. It prepares a variety of usage statistics about one’s Discord habits, akin to the ‘Checkpoint 2025’ which Discord prepared (and out of which I had opted). My interest is instead of examine to determine to what degree and success Discord has attempted to infer or gather personal details about me.

adoxa · Post by **adoxa** » 2026-02-17, 23:58

Okay, I guess it wasn't terribly efficient, then. Did you try fx? I'll see if I can come up with a Python script to break it up...

Edit: here it is (3.6 is the minimum, I think). By default it will create 100MB files. It's possible you may still end up with a big file, in which case use 0 as the size to split by key (as the script should have done), then split the big one(s) of those by size.

adoxa · Post by **adoxa** » 2026-02-19, 06:12

The minimum Python version is 3.8 (I still think; I used 3.12), although it shouldn't take much to drop it down, or even use 2.

Mæstro · Post by **Mæstro** » 2026-02-20, 18:22

Apologies for the late delay! I am leaving my flat in a week, so preparing to move house (as well as academic affairs and migrating off Discord) have commanded more of my attention. My system has only Python 2·7·16, and any attempt at programming on my own over the years has uniformly resulted in giving me panic attacks.

Post by **Moonchild** » 2026-02-20, 19:20

Off-topic:

Mæstro wrote: ↑
2026-02-20, 18:22
2·7·16

I really do wonder why you insist on using the multiplication dot instead of regular periods.

Mæstro · Post by **Mæstro** » 2026-02-20, 20:26

Off-topic:
My stylistic choice to use the middle dot in English for version numbers has decimals in mind. Hence, in other European languages, I would use the comma instead. I would not consider using the full stop for version numbers wrong in English; numerical separators for times and paragraphs are written this way. My usage is doubtless influenced by how, as a child, I would read the Firefox version number as three and a half. I also think it simply looks prettier.

adoxa · Post by **adoxa** » 2026-02-21, 02:47

Here's a Python 2 version, with a few extra tweaks.

Mæstro · Post by **Mæstro** » 2026-02-21, 19:02

adoxa wrote: ↑
2026-02-21, 02:47
Here's a Python 2 version, with a few extra tweaks.

We have made some progress here, and for this, you have my sincere thanks. After pasting it in the proper directory and running python json-split.py events.json 5%, I receive the output remark 0% part 1 before being returned to prompt. In the folder, a single file, about five kilobytes, called events-1.json appears. This file contains my first single event from the logs, giving my apparent location at the time (15 Ⅱ 24, or two years before I requested the package) as Warsaw, and picking up on the fact that I was using Epiphany then as my browser for Discord. This suggests that it has managed to export just the first event in the log. (Judging by the relative sizes, I estimate some 200 thousand in all.) The instructions, which show with python json-split.py, were invaluable in assisting me. Some debugging is still needed for the script to work, but my impression is that the hardest part is behind us.

adoxa · Post by **adoxa** » 2026-02-22, 02:30

Here's an updated version I think will work, just add -l (lowercase L) to the command. (In which case you didn't need this at all, something like split -l10000 events.json would have done it, although my split files will be JSON arrays.)

Pale Moon forum

Large (~1 GB) local JSON file fails to display Topic is solved

Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display

Re: Large (~1 GB) local JSON file fails to display