Computer Vision Gamepad Hack

Hey friends!
I was jealous of Scratch’s “Video Sensing”, so I wanted to see what it would take to do it with MakeCode Arcade. I cobbled together a bit of Python script that measures the optical flow of the entire camera view (the direction of most of the motion) and sends keypresses to whichever app has the operating system focus. You can see the window from the script on the right, labeling the direction detected.


My repo is here:

I don’t expect anyone to use this, I just wanted to share the fun. Would something like this be possible as an extension? Can anyone say if Microsoft is considering adding video sensing?
4 Likes