Raspberry Pi - A credit card sized, linux based mini-computer for $30

i am learning so, so much about linux, RPi and python. i always hear that the best way to learn is to have a project in mind as you do it, and it's absolutely true in this case. i have run into about one trillion problems, large and small, and have managed to figure them all out.

i think i also solved a common problem that doesn't have a solution posted online anywhere? when playing multiple videos using omxplayer, there is a roughly 1-second pause between each video, often accompanied by the sound cutting out for the first bit of the clip. i came up with a band-aid workaround but fuck, i think it works! i have a loop going where it opens up a new instance of omxplayer just before the previous clip ends, and positions it one render layer higher each time around so that it overlays on top of the previous instance, which closes as soon as it's done playing a moment later. the effect is a seamless transition, and it only uses about 10% of the CPU! i'm super happy because before i started on this i didn't know how to use bash or the terminal and instances and all of that.

i mentioned it a bit upthread, but i'm getting closer to completing this project, and it's changed quite a bit since i started as i learn more about what i can do with python and the rpi. i plan on creating a ton of very short clips that show a face tracking an object - a dozen clips of the face looking straight forward, a dozen to the center-center-left, a dozen to the center-left, etc etc, and then putting them in different buckets. then i'll play back the clips on a small display according to sensor/GPIO readings, so if you're standing directly in front of the display, you'll see clips of the face looking straight at you, but if you walk off to the side, the face and eyes will follow. i have other fun surprises in store as well (or i'm trying to, at least), but that's the gist of it.

i pretty much have everything working on the tech side except for the sensor stuff. but i'm hopeful that it won't be hard, and that i'll be able to easily feed the sensor readings to my python script so that it can determine which bucket of video clips should be playing at any moment. but the rest of it, the hard part - learning how to install python and packages and modules and use the command line via a python script, getting seamless video playback - is pretty much over. the rest is just fun stuff!

anyway sorry if i'm a little giddy but i feel like i just completed a basic, but real, irl conversation in a foreign language for the first time and it's exciting!

also i might have to finally register a stackexchange account to share the video playback workaround, because i've seen so many different people ask about it across multiple years and all the suggested solutions i ran across seem inadequate.

Good on ya, man. I barely got the thing to run RetroPi

Sounds pretty sweet!

that's a pretty cool solution!

programmer mind kicking in here -- if the render layers have any sort of sequence -- i.e. you're literally incrementing a variable by one, it might be worthwhile to have a check if there's an upper limit

the fix there would probably be to loop back to zero and you'd end up with a little pause again, but it'll be a little better than having someone restart your video art in the MoMA some day.

yep - i did set the render layers to increment by one with each cycle. the (scanty) documentation doesn't mention anything about an upper limit, but i reset it back to 1 whenever it reaches 999, so hopefully it won't cause any issues.

right now the biggest issue is performance. there's a tradeoff between the length of the videos, the dimensions of the video - if the length gets too short or the dimensions too big, it sometimes crashes. right now i have it cycling through 1-second long, 500x500 videos, and it's smooth! i want to make the videos as short as possible so that as the sensor tracks the human(s) observing it, they update as close to real-time as possible. the biggest constraint will probably be the sensor and how quickly it can send its new data to python via GPIO. i haven't even really looked into that yet, but i'm hoping that it can do it in near-real-time. i flailed my way through an afternoon arduino workshop a few years ago, and the sensor readings made it to the Serial almost instantaneously, so i'm hopeful that will be the case with this as well.

another interesting dilemma will be what to do if the sensor picks up more than one object on the physical stage. i guess i should just stop fucking around and buy a sensor and give it a shot.

is the ultrasonic beam collimated or focused somehow? how do you plan to arrange the sensors?

i...don't know yet.

a kind stranger in chicago contacted me a few months ago and offered to help me out with general programming stuff, and he went ahead and bought an ultrasonic sensor (i forget which one - an inexpensive one for sure) and started playing around with it. we met up a few weeks ago to talk about our progress and he said that the sensor was able to pick up humans in the room, up to about 10-12 feet away. i asked him if it was able to detect more than one at once, and he said that it did. but i haven't met with him since, and i haven't tried it myself.

so i'm not sure at the moment. ideally i would only need one sensor, but i might end up using two (one from the front, one from the side) in tandem? do you have any thoughts? i should probably have held off on mentioning all that until i had done even 10 minutes of research.

(that last sentence applies to everything i post on ilx btw)

eventually i may end up shifting to one master RPi equipped with the sensor(s) which sends out instructions to a set of subsidiary RPis with displays the videos accordingly. the ultimate evil plan is to go with a modular approach, so i can deploy individual RPIs with displays wherever i want in the room and they'll still be able to track the position of the observer.

i'm thinking each individual unit would only cost ~ $30 for the Rpi + another $30-60 for a small HDMI display. obviously i want to get one unit working well before buying any more, but having a set of 3 would be nice.

ok, after ~5 minutes of looking into i see that ultrasonic sensors only measure the distance to the closest object, so tracking more than one object would be tricky. that's fine though, i'm ok with it only identifying the closest object. that's almost better, in fact, because i'm planning on creating video clips that involve a human in a different dimension trying to reach across the void and make contact with the observer. almost like a frightened animal, so it would make sense that the video faces would only track the closest object, because the idea is that it's a struggle to clearly connect across the void in the first place. i have been watching a lot of unsolved mysteries lately.

as far as the positioning goes, i think i would place the sensor above the RPi + display, so that it's aiming down onto the stage at about a 45 degree angle. If I put it lower and aim it out horizontally, it could pick up objects that are 10-20 meters away, and i don't want that. i only want it to activate if someone actually approaches to take a closer look. (otherwise, the default/inactive state is looping videos of the faces going about their business, bored, or maybe trying and failing to establish contact across the void). if i place the sensor a little higher though, and aim it downward, i'll be able to define the sensor's field of vision.

does that sound right? i am making this up as i go but with the confidence that it should be possible.

So does the sensor return a reflection vs. distance or just a single distance value? The signal that the sensor emits will have some angular cone shape. The received signal will also have a cone-shape dependence away from the receiving aperture. "closest object" likely just means "largest reflection". It seems like it could get more complicated if you have multiple objects of different scattering cross-sections within these sensor cones. But it may also work out fine, as you said. You can probably do more with the signal processing if the sensor return something more than just a single distance value, but maybe that'd be overly complicated. Also, any signal processing might be too slow for your purpose.

I would also think that you'd direct the sensor along the axis you wish to detect movement. Otherwise it will be harder to distinguish movement along the axis you care about versus the orthogonal axis.

the downside is that you'd need somewhere to attach the sensor that doesn't look clunky. you could definitely solve that by buying a more expensive sensor with greater range, but that may not be in the cards. I suppose you'll learn the most from using the thing. Anyway, this all sounds very cool!

ultimate would probably be doing some kind of phased array or MIMO that allowed you to track different objects in the whole space. then maybe you could switch between them in a deliberately creepy way.

hey, i would never do anything in a deliberately creepy way

*rimshot*

what you're saying is really useful - i'm making some rough diagrams which i'll post in a bit if you're interested. i'm realizing that i definitely need more than one sensor!

I am breathlessly following this discussion in which I understand about every fourteenth word.

I'm also wondering if you couldn't get a camera + image processing to work well enough for this project. I know there are some cheaper raspberry pi cameras out there? I've never used one, though.

1) the sensors only have about a 30 degree cone of vision, so i would want more than one just to cover more of the space.

2) the sensor only returns the time that it took for the frequency to hit the object and reflect back to the sensor. that means that using a single sensor to determine an exact x/y coordinate is impossible because each measurement could mean that the object is anywhere along a certain arc:

http://i.imgur.com/cnFo6TF.jpg

3) but with two sensors and two signals, you would be able to solve some of the issues with both 1) and 2):

http://i.imgur.com/qg1UziR.jpg

the area where the two cones overlap (Zone 1 + 2) is the area where the exact coordinates could be determined. for objects in the other zones, i'd at least be able to know the general direction of the object. so if you only detect an object in Zone 1, you'd know that it must be to the left of center. Also, you would know that if the measurement is within a smaller range, the object is generally center-right, while if it the measurement is beyond a certain point the object is far-right:

http://i.imgur.com/2pvi7pe.jpg

i think that would be good enough for me. i'd be able to know the exact location of objects close to the display and trigger the video clips so that they follow the object closely. for objects in only zone 1 or zone 2, i would be able to trigger videos that show the faces looking in the appropriate general direction. for objects straight in front of the display that are out of range of both sensors, i could trigger a video or two of the face looking straight forward, searching for a signal, before reverting back to the original default/inactive state after a certain amount of time.

so if you only detect an object in Zone 1, you'd know that it must be to the left of center.

whoops, i meant right of center here, if you're referring to my shitting diagrams! (which are a lot of fun to make! being unemployed is awesome!)

I am breathlessly following this discussion in which I understand about every fourteenth word.

i am also clueless, but i'm learning! i keep waiting for someone to come in and be like ACTUALLY you can't do this and here's why you dum dum

(which shouldn't deter someone from telling me to abandon ship if this isn't going to work - that's why i'm posting the diagrams here! hopefully so that if there's an obvious better way to do it i can learn about it here, but also so that if it's a complete waste of time someone will let me know before i end up wandering on the mean streets muttering about sensor failures the rest of my life)

m also wondering if you couldn't get a camera + image processing to work well enough for this project. I know there are some cheaper raspberry pi cameras out there? I've never used one, though.

a distant stretch goal with all of this, if and when the physical version is completed, is to make it a website where the input is the user's webcam and it detects the user's face and adjusts the video clips accordingly. i have no idea how to do all of that but it also seems possible.

also, i know that i'd get more sensor coverage if i had one sensor that was located opposite of the display, like across the room, pointed back toward the display. but i wanted to conceal the sensors as much as possible, and i'm already a bit bummed that i might have these two sensors off to the side of the display, connected by cables to the RPi. so ugly. i don't want to add to that mess by having a sensor all the way across the room.

ok i'm going to stop posting but i just realized a fatal flaw in my logic is that it all might break down if more than one object/person is detected. it's fine if the object is the same for both sensors, but if one object is close to the sensor on the left, and another object is closer to the sensor on the right, it could be trouble. i will shut my trapper for a bit and think about it. i guess the good news is that i'm not a perfectionist and i'm very willing to implement some sort of backup plan if the two sensors present measurements that are in conflict with each other - like just pick the location that is closest to the display, or trigger a video clip that shows the face's eyes darting left and right and up and down, obviously searching and a bit confused - that fits in with the theme of unsettling attempts at connecting across dimensions, anyway!

yeah, I think your diagrams are probably otm for a single person in the room. I wonder what you can do when there are two people in Zone 1+2. I suppose you will have another special case when there are short reflections at sensors 1 and 2 such that the time contours cannot possibly intersect, and you will just choose to track one sensor's reading? Maybe that is not something that will happen often? I also worry about crosstalk between the sensors. Can you operate the two sensors at different frequencies or coordinate the pulses/readings?

oh, that's a great point about the crosstalk, but i think it could be addressed. in the python script i could just have the sensors search for objects at staggered intervals so they don't overlap.

i'm not sure about the multiple objects issue. i'm hopeful that it's something that could be mostly addressed with a bunch of if statements in the python script so that if the sensors are giving contradicting object locations the RPi will still have an idea of how to respond.

for example, one rare situation might be if sensor 1 detects an object outside of sensor 2's range, and sensor 2 detects an object outside of sensor 1's range. i wouldn't know the exact location of either object, but i would know that generally there's one off to the left somewhere, and one off the right, so i could trigger a video clip that shows the face scanning from left to right and right to left, acknowledging both of the objects? that's just one example but i think that would be the idea, just plotting out all the different possibilities and making sure that some sort of video clip is prepared for that situation, even if the video clip is just confusedface.mp4

confusedface.mp4 is my default face btw

Honestly I think the camera+image processing idea might be the best option. Not sure if the RPi has enough horsepower to do the image processing though.

lol. another slightly "dumb" option could be to place a few sensors in places that are best for the look of the thing. grid out the room. get some volunteers and measure every possible permutation you'd like to account for. Then you'd just have to calculate a "distance" between every measurement and one of your pre-measured cases and select the closest one. A display response would be associated with each pre-measured permutation. This could get messy if you find that different people have widely varying scattering cross-sections. Would also be a disaster if the sensors were moved or varied with time in any way.

oh man, i just went through a cycle of despair and hope. up until a few minutes ago, i thought any multi-sensor solution was hopeless because of the crosstalk sufjan mentioned above. i started thinking about the waves bouncing at obliuque angles off of objects (and walls) and getting picked up by the second sensor long after they were sent out by the first.

but i think it's ok as long as the two sensor's measurement windows are spaced a few tenths of a second apart. each measurement takes place in a very short amount of time. the speed of sound is 343 m/s at sea level. so if an object is 10 m away, the ultrasonic wave will reflect off the object in ~0.0145 seconds, and return to the sensor at ~0.029 seconds. if i understand correctly, each measurement from the ultrasonic sensor takes place in a brief window of time - it sends a wave out and listens for a defined amount of time (say, 0.1 seconds), then it closes up. so even if a wave hit an object and bounced around the room several times, it would be undetectable to the second sensor because the second sensor's measurement window wouldn't be open yet.

so you'd have the two sensors collecting measurements staggered by 0.2 seconds from each other, and the signals wouldn't interfere with each other, and you'd still be able to get two different readings on the same object as about the same moment in time.

i'm still not sure exactly which method to use to translate the readings into different buckets of videos, but i think it might be something that could be done in a messy/dumb fashion through trial and error.

yes, that time division multiplexing approach should work so long as there isn't a strong resonance in the room at the frequency of the sensor. You can also maybe get sensors operating at different frequencies that filter for that frequency if TDM doesn't work. I would not despair. I think you'll get something good working. You seem to have a good intuition for this stuff.

Raspberry PewDiePieBeret

this is silly, but i'm thinking about other ways to track people near the display. vibration sensors are super cheap. i'm wondering if i could lay down a rug in front of the display and then install a grid of vibration sensors. no matter where someone stood on the rug, they'd be setting off one or more of the sensors on the grid, and it would be easy to track multiple people - almost like an overhead view version of the rug area. downside - no one would ever step on the rug. i think people would think they weren't supposed to touch the rug. bummer. oh well, this could at least be fun just to have at home

i'm still going to order the ultrasonic sensors too, though, since they're really cheap as well

I'm at a loss here since I was digging around to find which Japanese video artist I saw a work from at the NY MoMA a couple weeks ago, but all of this motion sensor talk is reminding me of it: a room with a series of moving projectors in the center, triggered by the presence and movement of viewers in the room, that projected a group of people walking and running on the walls of the room, overlapping but never touching

webcam solution possible (and free)!
https://webgazer.cs.brown.edu/#examples

that is amazing!!! i think you'd have to require a calibration for every person that used it, though, and it wouldn't make sense for more than one person at once. i bet there's some other really cool stuff you could do with that, though!

even the basic demo game is kind of otherworldly, controlling it with only your eyes. holy shit!

controlling it with only your eyes. holy shit!

immediately makes me think of devices for people with cerebral palsy

I tried that just now and it took a few seconds before I realized that without a webcam on my end it didn't matter were I focused my eyes.

Kinect? Gives you colour image AND a depth map (per pixel distance measurement) and the software is capable of whizzy things like skeleton and face detection, can detect up to 6 bodies... Expensive though and I think the open source library has been bought by someone and and the new fork is lagging behind iirc.

oooo you could make the eyes widen in response to screams of terror

a kinect does seem like a good idea if you have the budget and can get it running fast enough on the pi

I'd never looked into how the kinect works. It seems that it has an illuminating IR LED and then an ASIC to quickly process what the camera is seeing reflected back using a trained machine learning algorithm. So it is like the deluxe version of my "stupid" idea for the ultrasound sensors.