I tried using similar syntax to melody.addNote and calling queuePlayInstructions to play several sine waves (generated using a Fourier analysis tool, and a few others) but the results are a little shaky. I’m sure it can be refined further but it seems like a very roundabout way of getting things done. Voice Test Project
In another project I tried setting the frequency every 0.25ms, matching a 8khz wav file I put into makecode, but anything less than 1ms broke. I started looking through the pxt git and found queuePlayInstructions, so I’m thinking maybe U.delay(when) is the culprit. Found playInstructionAsync as well, but I couldn’t find any kind of definition for OscillatorNode, which seems like my best bet, since playing audio seems to be done with osc.frequency.setValueAtTime and gain.gain.setValueAtTime.
Not sure if this is even going to be possible or if sine wave layering is the limit for now, and wanted to ask about it before going any further. Almost all of my information is from testing or inference, so please correct me if I’m wrong. Thanks!
@chembot anything less than 1ms is going to be running up against limitations of the browser itself. iirc that’s the range where browsers start messing with things to avoid fingerprinting via high precision timestamps. I know firefox limits precision to 2ms
that being said, i’ve been playing around with adding some new experimental audio features that might help you out with this; specifically, wavetable synthesis which could probably be expanded to playing PCM buffers.
below is a demo link, but this is made with a beta version of the editor so please be sure to open it in an incognito window to avoid messing with your projects. also, this might not ship anytime soon so don’t do too much with it! very much subject to change
GASP!?
I’m flabbergasted. Have you HEARD my ‘Hello’? I mean it was literally just a whistle. This is stunning, simply. You have no idea how long I’ve researched using Fourier analysis for this!!
Please, you must tell me. What tool did you use!? I’ve been using PRAAT, but its data output rarely works.
Your research of queuePlayInstructions and melody.addNote are exactly what I was missing- my research stopped because I just don’t know as much about JS.
(By the way, have you seen my work on this? Did I inspire you at all?)
This is AMAZING!! And @richard , wavetable synthesis is exactly what I’ve been asking for… my dreams are coming true here!! : D
Thanks for all the kind words, I wasn’t expecting this kind of response
@richard that demo is interesting… Looking forwards to the eventual release! I also didn’t think about browser limits, I had assumed it was makecode. Thanks for the info.
@randomuser I glad to find out someone else was working on the same thing as me! I used a few programs to get this result:
First processed the recordings to .wav files with some online tool, so I could open them in SPEAR. I found I got decent results with a frequency resolution of around 30-90, and a minimum amplitude threshold of around -40dB. I also deleted anything above ~4000Hz, and any very small lines.
Exported it to a .txt, and then processed that with a C# console app I threw together. If you want to try something similar the data has a 5 line header, then each line starts with time, number of waves, then repeating wave data in groups of three: channel, pitch, volume. Not sure if volume is amplitude or actual volume in dB though.
The console app spits out a number[][][][] array, where number[n] represents a sine wave that changes frequency and amplitude over time, number[n][0]=[[start tick, end tick]] (SPEAR gave me 10ms/tick) and number[n][1] is an array of [frequency, amplitude] pairs.
Kept the start and end tick an extra array deep so the type stayed number[][][][].
I’ll post the code for the converter github tonight once I clean it up a little bit. I’m tempted to change it from an array to buffers, since 3 lines per ms of audio is a bit much, but that can wait.
I’m glad to know there’s so many people pushing the bounds of makecode! So many interesting ideas to try
Wow, yeah, that is complicated! I kept up with you until the C#, but I think I get it- you ran the audio through SPEAR which gave you the individual sine waves, and you used the C# to sort it out / get it into a workable format. I’m excited for that converter!!
This is SOO cool!! : D
Sorry, I realize now I was a bit vague with how SPEAR works. It outputs a bunch of sine waves that change in frequency and amplitude over time, which start and end at different times. It’s a bit more advanced than normal Fourier analysis, and works better than overlaying unchanging sine waves, although I have no idea how it actually works. But it is why the “Hello” is so audible despite only like 30 waves being overlayed.
In the test program going past around 30 waves sounded worse and worse. At a min length of 12 with 30 waves its mostly audible, but at a min length of 8 with 38 waves it starts to get muddy. Even going as far as min 25 and playing only 12 waves is still somewhat passable.
I’m not sure if it’s the data from spear being interpreted wrong, higher frequencies being played too loudly, or just a browser limitation, I’m going to do some testing and see what I can find out why more waves sounds worse.
I put the SPEAR to MakeCode program on git. It’s commented a little but feel free to message if anything is unclear!
