# What to do with large data files?
**URL:** https://forum.makecode.com/t/what-to-do-with-large-data-files/12571
**Category:** Help
**Created:** [March 7, 2022, 6:00pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571 "2022-03-07T18:00:52Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@AlexK](https://forum.makecode.com/u/AlexK)
#### Post date: [March 7, 2022, 6:00pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/1 "2022-03-07T18:00:52Z")
Wanted to get some opinions, particularly from the devs, on a rather ridiculous idea that I have.
I’ve been thinking about word games lately, and I found myself down a rabbit hole this weekend. I thought, “what about a generic word lookup extension?” So, I created a proof of concept:
> **[YAWL](https://arcade.makecode.com/79223-24844-73289-05507)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.
The program works just fine in a browser. The data file contains the _entire_ YAWL (“Yet Another Word List”) with some 200,000+ entries. The data file itself is about 5 MB on my file system. In contrast, the word list that I used for _What’s My Word_, with 2,000+ entries, clocks in at around 30 kB. This clearly is ridiculous, and I really can’t think of a reason why I would need the entire YAWL loaded into a project. But, for giggles … what if I did?
The strings are already in hex format, so is it better to place these into buffers somehow and access the data that way? Or does the compiler already know to place constants into the game file and make them accessible to the runtime? Is the game file limit still 512 kB for hardware, or has that been expanded?
Would love your thoughts.
---
### Author:  [@kwx](https://forum.makecode.com/u/kwx)
#### Post date: [March 7, 2022, 7:03pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/2 "2022-03-07T19:03:24Z")
For a word list, if you’re OK with some non-words being accepted as words (false positives), you could use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) as a space-efficient alternative. It’s a one-way function where you can only check if a word is part of the set, so it would work for Wordle’s list of valid guesses but not for the list of target words.
According to an online calculator, a 2000-word list with a 0.1% false positive rate would need about [3.51 kiB](https://hur.st/bloomfilter/?n=2000&p=0.001&m=&k=). For 10000 words with 0.01% false positives, it would be [23.4 kiB](https://hur.st/bloomfilter/?n=10000&p=0.0001&m=&k=).
---
### Author:  [@AlexK](https://forum.makecode.com/u/AlexK)
#### Post date: [March 7, 2022, 7:03pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/3 "2022-03-07T19:03:33Z")
Well, I suppose this answers one of my questions. 😆
---
### Author:  [@richard](https://forum.makecode.com/u/richard)
#### Post date: [March 7, 2022, 7:05pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/4 "2022-03-07T19:05:39Z")
@kwx TIL! this is very cool
---
### Author:  [@richard](https://forum.makecode.com/u/richard)
#### Post date: [March 7, 2022, 7:07pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/5 "2022-03-07T19:07:46Z")
Also @AlexK, no need to pack strings into buffers! The compiler does indeed include strings in a space efficient way.
---
### Author:  [@UnsignedArduino](https://forum.makecode.com/u/UnsignedArduino)
#### Post date: [March 7, 2022, 7:16pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/6 "2022-03-07T19:16:08Z")
Maybe compress it and store in base85?
---
### Author:  [@AqeeAqee](https://forum.makecode.com/u/AqeeAqee)
#### Post date: [March 8, 2022, 1:41am UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/7 "2022-03-08T01:41:38Z")
I am doing prj using word list too, recently.
It’s a words reciting game. First vacabulary has about 1500 words, each word with 2 fields(spelling, meaning) stored in a 2d string array.
When I download to device(Meobit) got compile error. After I commented last 600+ words, about 900 left, it works. But could got 021 error in high chance, especially, when connecting USB cable.
(Only Arcade Text / Sprite Text ext imported)
> **[ReciteWords](https://arcade.makecode.com/33353-84281-48938-57768)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.
---
### Author:  [@AlexK](https://forum.makecode.com/u/AlexK)
#### Post date: [May 15, 2023, 5:02pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/8 "2023-05-15T17:02:41Z")
So, I’ve been thinking about this one for quite some time. @kwx piqued my interest in Bloom filters with his response, and I just kept returning to this thread every few weeks. I decided to take a break from Monopoly this weekend and take a deep dive on this.
I stumbled upon a great little write up on universal hash functions here,[1] which took me back to the original paper on them by Carter and Wegman.[2] Those two references gave me the information that I needed to give Bloom filters a try.
I wrote a C# program that ingests a word list and spits out a TypeScript array that I can copy-and-paste into MakeCode Arcade. I used the Game Words list by Dana Bell;[3] YAWL[4] would work just as well. This first pass worked just fine for words up to five characters in length. Beyond that, the numbers got too big to fit into TypeScript’s `number` type. This was my motivation to port `bigint` to MakeCode.
It works, too! The complete Game Words dictionary up to 12-letter words fits in ~400kB. Take a look!
> **[Game Words with Bloom Filters and BigInt](https://arcade.makecode.com/80812-71863-44835-42515)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.
I’ll come back to this again another time. I’ll write a proper interface to my C# program and put that in GitHub so that y’all can use it with any word list you like. I’ll also refine the TypeScript code a bit and put that up on GitHub for anyone who wants it. I could wrap this into an extension, but because of the size of the dictionary, it might not work on hardware. You will want to strip out the filters that you don’t need to make the code optimal.
I’ll be adding _Countdown_ to my list of future projects. That and _Lingo_. 🙂
I’m just stoked that it works! Have fun!
* * *
[1] Mount, Dave. (2019). _CSMC 420 Lecture 10: Hashing - Basic Concepts and Hash Functions._ [https://www.cs.umd.edu/class/fall2019/cmsc420-0201/Lects/lect10-hash-basics.pdf](https://www.cs.umd.edu/class/fall2019/cmsc420-0201/Lects/lect10-hash-basics.pdf)
[2] Carter, Larry; Wegman, Mark N. (1979). “Universal Classes of Hash Functions.” _Journal of Computer and System Sciences_ . **18** (2): 143–154. [https://doi.org/10.1016/0022-0000(79)90044-8](https://doi.org/10.1016/0022-0000(79)90044-8) Conference version in STOC’77.
[3] Bell, Dana. (2020). _Game Words 2020 (provisional)_. [https://www.tylerhosting.com/gamewords/](https://www.tylerhosting.com/gamewords/)
[4] Cooper, Mendel Leo. _Yet Another Word List (YAWL)_. Republished by Aaron Bull Schaefer. [https://github.com/elasticdog/yawl](https://github.com/elasticdog/yawl)
---
### Author:  [@KIKIvsIT](https://forum.makecode.com/u/KIKIvsIT)
#### Post date: [May 15, 2023, 5:03pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/9 "2023-05-15T17:03:25Z")
Super cool, Alex! It looks like the game file is missing at that URL, though. Can you relink the game?
---
### Author:  [@AlexK](https://forum.makecode.com/u/AlexK)
#### Post date: [May 15, 2023, 7:05pm UTC](https://forum.makecode.com/t/what-to-do-with-large-data-files/12571/10 "2023-05-15T19:05:12Z")
Aww bummer! Wonder where it went…
Here’s a new link!
> **[Game Words with Bloom Filters and BigInt](https://arcade.makecode.com/89838-58682-16064-27814)**
>
> Made with ❤️ in Microsoft MakeCode Arcade.