Help with proper use of image.ofBuffer() or creating image from string

[chembot]

Trying to make an image from a string, so I can store images with less text.

My first thought was to use something like img. 1 2 . , but I couldn’t find a way to substitute in a string (the string format was ". 1 2 . ", similarly)

My second approach was to attempt using image.ofBuffer, but no matter how I set up the buffer the image comes out entirely black. I tried generating a buffer using Buffer.fromHex with a single long line of space separated hex characters, and generating a buffer using Buffer.fromArray with array of numbers, which each pixel being one number. Neither worked.

Am I going about this wrong? I know I could use image.setPixel but that seemed inefficient. How can I quickly and efficiently set an entire image from a string (or byte array)?

Thanks.

[richard]

@chembot could I ask what you’re trying to achieve?

Is this string something you’re assembling programmatically, or something you’re pasting into your project?

Either way, the image.ofBuffer function does take in a Buffer but the data inside has to be formatted correctly. We have our own image format that we use that looks likes this:

Byte   Data
0:     Magic Number (0x87)
1:     Bits per pixel (either 1 or 4)
2-3:   Image width (2 bytes)
4-5:   Image height (2 bytes)
6:     0
7:     0
8-N:   Pixel data

The pixel data is stored in column-order using either 1 bit per pixel (monochrome) or 4 bits per pixel

[richard]

I’m guessing that’s not what you want, though. You probably want something that takes a string like this:

let myImage = `
 0 1 2 3
 4 5 6 7
 8 9 a b
 c d e f
`

and spits out an image. Is that right? If it is, we have no built in function for doing it but it’s easy enough to write a function that does just that:

function decodeImage(imageString: string) {
    const rows: number[][] = [];
    let width = 0;
    let currentRow: number[] = [];

    // These are all valid hexadecimal characters
    const hexCharacters = "0123456789abcdefABCDEF"

    // Loop over every character in the input string
    for (let i = 0; i < imageString.length; i++) {
        const character = imageString.charAt(i);

        // If the character is a newline, push a row
        if (character === "\n") {
            // Ignore empty rows
            if (!currentRow.length) continue;
            else {
                // Make sure we update the width as we go
                width = Math.max(width, currentRow.length);
                rows.push(currentRow);
                currentRow = [];
            }
        }
        else if (character === ".") {
            // "." isnt a hex character, so handle it separately
            currentRow.push(0);
        }
        else if (hexCharacters.indexOf(character) !== -1) {
            // Passing in a 16 to parseInt will parse the string as a hex value
            currentRow.push(parseInt(character, 16));
        }
    }

    // After we've finished parsing the string, check to see if we have one row left over
    if (currentRow.length) {
        width = Math.max(width, currentRow.length);
        rows.push(currentRow);
    }

    // Now we'll create a new image and fill it with our parsed values
    const out = image.create(width, rows.length);
    for (let y = 0; y < rows.length; y++) {
        currentRow = rows[y];
        for (let x = 0; x < currentRow.length; x++) {
            out.setPixel(x, y, currentRow[x]);
        }
    }
    return out;
}

[richard]

…and if you do want to mess around with using buffers for images, here’s a function for encoding them into our format:

function f4EncodeImg(width: number, height: number, bitsPerPixel: number, getPix: (x: number, y: number) => number) {
    let byteHeight: number;

    // If the bitsPerPixel is 1, we align to the byte. Otherwise we align
    // to the word
    if (bitsPerPixel === 1) {
        byteHeight = (height + 7) >> 3
    }
    else if (bitsPerPixel === 4) {
        byteHeight = (((height << 2) + 31) >> 5) << 2;
    }
    else {
        throw "Invalid byte height"
    }

    // Create our buffer; 8 is the size of our header
    const out = control.createBuffer(8 + (width * byteHeight))
    out[0] = 0x87;          // Magic number
    out[1] = bitsPerPixel;  // Must be either 1 or 4
    out[2] = width & 0xff;  // Width
    out[3] = width >> 8;
    out[4] = height & 0xff; // Height
    out[5] = height >> 8;
    out[6] = 0;             // Unused
    out[7] = 0;             // Unused

    let ptr = 4
    let curr = 0
    let shift = 0

    let pushBits = (n: number) => {
        curr |= n << shift
        if (shift == 8 - bitsPerPixel) {
            out[ptr + 4] = curr;
            ptr++
            curr = 0
            shift = 0
        } else {
            shift += bitsPerPixel
        }
    }

    for (let i = 0; i < width; ++i) {
        for (let j = 0; j < height; ++j)
            pushBits(getPix(i, j))
        while (shift != 0)
            pushBits(0)
        if (bitsPerPixel > 1) {
            while (ptr & 3)
                pushBits(0)
        }
    }

    return out;
}

However, I wouldn’t recommend using it unless you have a good reason. Creating an image and calling setPixel is probably faster.

[chembot]

Thanks for all the info! I was able to vastly improve my program using that f4EncodeImg method.

My application is storing a huge amount of images data. To be precise, I’m storing 3288 images each 160x120 pixels, though I’ve opted for a reduced color palette to save space. Instead of using the built in image initialization I decided to store the data more compactly in a seperate ts file.

Initially the size of the images.ts was ~75MB, but with f4EncodeImg and by reducing whitespace (also having one character encode two pixels, with 8 possible colors per pixel) I got it down to 37MB. With some basic RLE compression I reduced it further to 4.5MB, though it’s still too large to upload to a meowbit, so I’m now experimenting with reducing my color palette from 8 to 4 (each character out of 64 options encoding 3 pixels instead of 2).

I was also looking into storing the data as number[][][] ([imgIndex][y][]) where each entry is a byte representing the next 4 pixels (2 bits per pixel) instead of string[][] ([imgIndex][y]) where each string is an encoded pixel line, which is my current approach. I’m a little wary of typescript’s ‘number’ type though, since I’m unsure of how to control the number of bits and if it’s a floating point or not. Would something like let num = 0x1f stay a byte? I could also only record the changes between each image, but that approach seemed more difficult.

One other concern I had was vscode notified me saying “program too big by 4888568 bytes!” which is slightly larger than the full size of my project. I’m assuming it’s the autogenerated binaries that are too large, but would it be possible to alter (or fully create) those manually?

Thanks again for the help.

[richard]

@chembot numbers are always going to end up as 32bits. Also, I would recommend against using triple arrays as those are going to be very inefficient code-gen wise; instead use Buffers wherever you can.

If you want to really reduce things, you can also exclude the game library from compilation. We have a minimal game lib you can use in its place that excludes things like sprites and physics to reduce compile size. If you’re, say, displaying a movie then you don’t need any of that stuff anyhow. Here’s a template project you can use that does this:

game-lite

Can you share a link to your code? I might be able to advise a bit better once I get a look at it.

[chembot]

Good to know. I’ll try with the minimal game lib, see how much it improves.
Numbers being 32 bits is usable if I can set bits directly, but as I understand it uses floating points. I’ll try recoding it with a buffer. My current approach is to use one bit to encode color (black or white), then 15 bits (only up to 120*160 though) to encode number of times the color appears in a row.

One interesting thought is manual code generation, though I know its probably a poor idea. I just saw a binary.asm file and thought it would in theory be possible to compile from something like C++ or even C#.

Here are the links to my code. I split it into two projects since it’s easier to work on that way.

GitHub - ch3mbot/MakeCodeSimpleImage

I found that less than 4096 unique color-length combos existed, so instead of using 16 bits to represent a line of pixels, I use 12 bits for the index in the map, where unique color-length combos are stored in a separate array. I know this doesn’t make a difference right now though since it’s 32 bits to store a number either way, I just wanted to set it up well for the future, where three bytes could hold the indices of two color-lengths.

The two possibilities were halving the resolution to quarter storage use, and using a more advanced compression algorithm, though that might make it take too long to generate a frame.

GitHub - ch3mbot/MakeCodeSimpleSong

The midi data is stored in a number[][4], [startTime, pitch, length, instrument]. From what you said about triple arrays being inefficient, I’d imagine it would be better to have a unique array for each property, and just have them each be the same size. A buffer could also significantly reduce the size since I can store start time in 11 bits, pitch in 6, channel in 3 bits, and length in 3 bits, so 3 bytes per note, 14kb total. May also create premade segments of music that are common and repeat, like the drum track. Should reduce storage dramatically.

One other question, about game.onUpdateInterval(period: number, a: () => void). Does this work off of in game time, or real time? I’m assuming real time, like game.runtime() but I wanted to check just in case.

Hopefully those explain everything, but let me know if there’s anything I should clarify.

[richard]

@chembot you should absolutely use buffers for all of these use cases; no need for any arrays at all.

You can embed buffers in your code like so:

let myBuffer = hex`deadbeef`

…where the data inside the backticks is encoded in hexadecimal. Buffers have getNumber and setNumber methods that can be used to work with numbers that are larger than one byte. For example:

let value = myBuffer.getNumber(NumberFormat.UInt16LE, 0)

Doing that will greatly improve your code size.

As for other languages, we do indeed support C++ and asm, but not C# or anything like that. However, I strongly recommend against using them as they are really meant for extensions that need low-level hardware access for communicating with sensors/pins. C++ and asm code is only supported on hardware, so it won’t run at all in the browser. I think in general you won’t get that much gain out of it anyhow; best to stick to TypeScript.

Also, re: our number representation, you can absolutely use numbers as integers. Our numbers are stored as 31-bit integers until you do something that converts them into floats (e.g. division). If you’re just using bitwise operations, you don’t have to worry about floats. Likewise, doing any bitwise operation on a number will convert it into an int if it isn’t already (we use num | 0 in our codebase all over the place to convert numbers to integers).

BTW the reason our numbers are 31-bit and not 32-bit is because we use the top bit to indicate if it has been converted into a floating point or not.

[chembot]

The use of buffers significantly reduced project size, and being able to control each bit is a huge benefit. Using buffers I’ve added more data but taken less space. The 31-bit numbers are interesting, I always wondered how everything was handled with just ‘number’.

I also had luck with using game-lite, it’s good to be able to draw directly to screen and have more control there, as well as less overhead. Took your advice on using SetPixel instead of buffer creation. Did a some profiling and SetPixel was consistently over twice as fast.

Unfortunately the data is still too large, and generating a share link fails, though I’m not sure if its because it’s too much data or if having 4MB in one line is breaking it. I’m going to take a crack at JPEG/MPEG compression to reduce it further, but that’s another topic.

Thanks for all the help with this project! I’ve learned a lot about makecode functionality and TypeScript in general.