Raspberry Pi - A credit card sized, linux based mini-computer for $30

what you're saying is really useful - i'm making some rough diagrams which i'll post in a bit if you're interested. i'm realizing that i definitely need more than one sensor!

I am breathlessly following this discussion in which I understand about every fourteenth word.

I'm also wondering if you couldn't get a camera + image processing to work well enough for this project. I know there are some cheaper raspberry pi cameras out there? I've never used one, though.

1) the sensors only have about a 30 degree cone of vision, so i would want more than one just to cover more of the space.

2) the sensor only returns the time that it took for the frequency to hit the object and reflect back to the sensor. that means that using a single sensor to determine an exact x/y coordinate is impossible because each measurement could mean that the object is anywhere along a certain arc:

http://i.imgur.com/cnFo6TF.jpg

3) but with two sensors and two signals, you would be able to solve some of the issues with both 1) and 2):

http://i.imgur.com/qg1UziR.jpg

the area where the two cones overlap (Zone 1 + 2) is the area where the exact coordinates could be determined. for objects in the other zones, i'd at least be able to know the general direction of the object. so if you only detect an object in Zone 1, you'd know that it must be to the left of center. Also, you would know that if the measurement is within a smaller range, the object is generally center-right, while if it the measurement is beyond a certain point the object is far-right:

http://i.imgur.com/2pvi7pe.jpg

i think that would be good enough for me. i'd be able to know the exact location of objects close to the display and trigger the video clips so that they follow the object closely. for objects in only zone 1 or zone 2, i would be able to trigger videos that show the faces looking in the appropriate general direction. for objects straight in front of the display that are out of range of both sensors, i could trigger a video or two of the face looking straight forward, searching for a signal, before reverting back to the original default/inactive state after a certain amount of time.

so if you only detect an object in Zone 1, you'd know that it must be to the left of center.

whoops, i meant right of center here, if you're referring to my shitting diagrams! (which are a lot of fun to make! being unemployed is awesome!)

I am breathlessly following this discussion in which I understand about every fourteenth word.

i am also clueless, but i'm learning! i keep waiting for someone to come in and be like ACTUALLY you can't do this and here's why you dum dum

(which shouldn't deter someone from telling me to abandon ship if this isn't going to work - that's why i'm posting the diagrams here! hopefully so that if there's an obvious better way to do it i can learn about it here, but also so that if it's a complete waste of time someone will let me know before i end up wandering on the mean streets muttering about sensor failures the rest of my life)

m also wondering if you couldn't get a camera + image processing to work well enough for this project. I know there are some cheaper raspberry pi cameras out there? I've never used one, though.

a distant stretch goal with all of this, if and when the physical version is completed, is to make it a website where the input is the user's webcam and it detects the user's face and adjusts the video clips accordingly. i have no idea how to do all of that but it also seems possible.

also, i know that i'd get more sensor coverage if i had one sensor that was located opposite of the display, like across the room, pointed back toward the display. but i wanted to conceal the sensors as much as possible, and i'm already a bit bummed that i might have these two sensors off to the side of the display, connected by cables to the RPi. so ugly. i don't want to add to that mess by having a sensor all the way across the room.

ok i'm going to stop posting but i just realized a fatal flaw in my logic is that it all might break down if more than one object/person is detected. it's fine if the object is the same for both sensors, but if one object is close to the sensor on the left, and another object is closer to the sensor on the right, it could be trouble. i will shut my trapper for a bit and think about it. i guess the good news is that i'm not a perfectionist and i'm very willing to implement some sort of backup plan if the two sensors present measurements that are in conflict with each other - like just pick the location that is closest to the display, or trigger a video clip that shows the face's eyes darting left and right and up and down, obviously searching and a bit confused - that fits in with the theme of unsettling attempts at connecting across dimensions, anyway!

yeah, I think your diagrams are probably otm for a single person in the room. I wonder what you can do when there are two people in Zone 1+2. I suppose you will have another special case when there are short reflections at sensors 1 and 2 such that the time contours cannot possibly intersect, and you will just choose to track one sensor's reading? Maybe that is not something that will happen often? I also worry about crosstalk between the sensors. Can you operate the two sensors at different frequencies or coordinate the pulses/readings?

oh, that's a great point about the crosstalk, but i think it could be addressed. in the python script i could just have the sensors search for objects at staggered intervals so they don't overlap.

i'm not sure about the multiple objects issue. i'm hopeful that it's something that could be mostly addressed with a bunch of if statements in the python script so that if the sensors are giving contradicting object locations the RPi will still have an idea of how to respond.

for example, one rare situation might be if sensor 1 detects an object outside of sensor 2's range, and sensor 2 detects an object outside of sensor 1's range. i wouldn't know the exact location of either object, but i would know that generally there's one off to the left somewhere, and one off the right, so i could trigger a video clip that shows the face scanning from left to right and right to left, acknowledging both of the objects? that's just one example but i think that would be the idea, just plotting out all the different possibilities and making sure that some sort of video clip is prepared for that situation, even if the video clip is just confusedface.mp4

confusedface.mp4 is my default face btw

Honestly I think the camera+image processing idea might be the best option. Not sure if the RPi has enough horsepower to do the image processing though.

lol. another slightly "dumb" option could be to place a few sensors in places that are best for the look of the thing. grid out the room. get some volunteers and measure every possible permutation you'd like to account for. Then you'd just have to calculate a "distance" between every measurement and one of your pre-measured cases and select the closest one. A display response would be associated with each pre-measured permutation. This could get messy if you find that different people have widely varying scattering cross-sections. Would also be a disaster if the sensors were moved or varied with time in any way.

oh man, i just went through a cycle of despair and hope. up until a few minutes ago, i thought any multi-sensor solution was hopeless because of the crosstalk sufjan mentioned above. i started thinking about the waves bouncing at obliuque angles off of objects (and walls) and getting picked up by the second sensor long after they were sent out by the first.

but i think it's ok as long as the two sensor's measurement windows are spaced a few tenths of a second apart. each measurement takes place in a very short amount of time. the speed of sound is 343 m/s at sea level. so if an object is 10 m away, the ultrasonic wave will reflect off the object in ~0.0145 seconds, and return to the sensor at ~0.029 seconds. if i understand correctly, each measurement from the ultrasonic sensor takes place in a brief window of time - it sends a wave out and listens for a defined amount of time (say, 0.1 seconds), then it closes up. so even if a wave hit an object and bounced around the room several times, it would be undetectable to the second sensor because the second sensor's measurement window wouldn't be open yet.

so you'd have the two sensors collecting measurements staggered by 0.2 seconds from each other, and the signals wouldn't interfere with each other, and you'd still be able to get two different readings on the same object as about the same moment in time.

i'm still not sure exactly which method to use to translate the readings into different buckets of videos, but i think it might be something that could be done in a messy/dumb fashion through trial and error.

yes, that time division multiplexing approach should work so long as there isn't a strong resonance in the room at the frequency of the sensor. You can also maybe get sensors operating at different frequencies that filter for that frequency if TDM doesn't work. I would not despair. I think you'll get something good working. You seem to have a good intuition for this stuff.

Raspberry PewDiePieBeret

this is silly, but i'm thinking about other ways to track people near the display. vibration sensors are super cheap. i'm wondering if i could lay down a rug in front of the display and then install a grid of vibration sensors. no matter where someone stood on the rug, they'd be setting off one or more of the sensors on the grid, and it would be easy to track multiple people - almost like an overhead view version of the rug area. downside - no one would ever step on the rug. i think people would think they weren't supposed to touch the rug. bummer. oh well, this could at least be fun just to have at home

i'm still going to order the ultrasonic sensors too, though, since they're really cheap as well

I'm at a loss here since I was digging around to find which Japanese video artist I saw a work from at the NY MoMA a couple weeks ago, but all of this motion sensor talk is reminding me of it: a room with a series of moving projectors in the center, triggered by the presence and movement of viewers in the room, that projected a group of people walking and running on the walls of the room, overlapping but never touching

webcam solution possible (and free)!
https://webgazer.cs.brown.edu/#examples

that is amazing!!! i think you'd have to require a calibration for every person that used it, though, and it wouldn't make sense for more than one person at once. i bet there's some other really cool stuff you could do with that, though!

even the basic demo game is kind of otherworldly, controlling it with only your eyes. holy shit!

controlling it with only your eyes. holy shit!

immediately makes me think of devices for people with cerebral palsy

I tried that just now and it took a few seconds before I realized that without a webcam on my end it didn't matter were I focused my eyes.

Kinect? Gives you colour image AND a depth map (per pixel distance measurement) and the software is capable of whizzy things like skeleton and face detection, can detect up to 6 bodies... Expensive though and I think the open source library has been bought by someone and and the new fork is lagging behind iirc.

oooo you could make the eyes widen in response to screams of terror

a kinect does seem like a good idea if you have the budget and can get it running fast enough on the pi

I'd never looked into how the kinect works. It seems that it has an illuminating IR LED and then an ASIC to quickly process what the camera is seeing reflected back using a trained machine learning algorithm. So it is like the deluxe version of my "stupid" idea for the ultrasound sensors.

those are all really good ideas! it looks like i'll have some options. tbh at the moment i am planning on it tracking only one (the closest) person. i'm working on some video cycling logic that lets it smoothly track from one target to another, so if one person is the target and then they suddenly turn around and leave, the eyes/face will smoothly work its way over to the next-closest person detected (or to it's default resting cycle if no one else is there).

i won't break any ground technically but i hope to make the videos interesting and transition from one another smoothly

have any of you used the Pine64 (https://www.pine64.org/)? or heard anything about it from someone who has?

i'm working on something involving speech recognition and synthesis and while i have the guts of it working on my raspberry pi 3, it's painfully slow. a 2 second speech recording takes about 6 seconds to convert to text, and that's before i even do text-to-speech. meanwhile, i can run the same python code on my mac and the conversion is near-instantaneous. that suggests to me that the slowdown is due to the lack of specs on the raspberry pi. fishing around for a way to improve the performance, i saw some chatter about the relatively new pine64, which apparently has more processing power and general capabilities while still in the $30-50 range.

or actually...maybe it would make more sense to look into inexpensive ways to augment/power up the RPi somehow?

How do you want to install the final project? Is it feasible to use the Pi as a client and offload the speech-to-text to a faster computer over the network? I can well imagine it being hard to get low-latency speech to text performance on any cheapo device.

really hoping to make it as modular and mobile as possible, like completely independent and offline units that could be arranged in different ways. i wanted offline because it has to run for a few hours straight and i didn't want the whole thing to break if the wi-fi went down or something.

Is it feasible to use the Pi as a client and offload the speech-to-text to a faster computer over the network?

maybe? i think it would work, but i don't know how much faster it would be. each Pi unit has its own USB microphone to pick up the last signal in the chain, and each unit also has it's own speaker output connected to it. So each pi unit would have to send the recorded audio file to the faster computer, which would convert it (very quickly), but then send the converted file back to the pi unit again before playing it through the speaker. it just seems like the wait times for sending it back and forth would negate the improvement in speech-to-text conversion speed.

Sensible of you. I think you might want to treat this as a “prototyping” stage and live with what $50 gets you. It looks like for $500 you can get a development board for the NVidia Tegra X1, which is the chip the Nintendo Switch uses and probably has several orders of magnitude better performance than a Pi. Expensive but still cheaper and smaller than duct-taping a MacBook to something.

Don’t overestimate the round trip time of the network though. Six seconds is a long time.

oh, i'll definitely give it a shot! i'm also going to look into adjusting the dictionary i'm using for the speech recognition (using Sphinx). i saw a video of some dude online who is getting a decent turnaround on his RPi3, so maybe he was using a limited dictionary?

this stuff is sometimes frustrating but really fun to look into (especially when it works in the end)

Are there any other speech recognition softwares you've tried?
If the idea is to just robotize people's speech, maybe have it recognize/generate phonemes instead of words?

all the other speech recognition i've found requires an API: a couple google versions limited to 50 API calls a day, IBM, Bing, Jasper, Houndify, others.

i've been thinking of a way to take advantage of the slow processing. i'm thinking a set of 4 units that each work on 12 second cycles: 2 for listening to speech, 6 for converting to text, 2 for text-to-speech playback, and some buffer time in between each of those steps. the timing of the units would overlap so there'd be playback every 2 seconds. like this:

https://i.imgur.com/6afC7mA.png

(sorry for ugly colors)

actually, the cycle length would vary, but each would loop every 32 seconds with the timeline above. i'm thinking i can use python's time.time() to keep everything on schedule, so even if individual components in the cycle (record, convert, playback) take longer or shorter than expected, each step should begin at a precise point in time. the biggest issue (i think) will be with calibrating the units at the beginning. i'm thinking i can start the first unit on it's eternal loop, then manually trigger the launch of the second unit, listen for a few cycles to make sure they're in alignment, continue to the 3rd, and then the 4th.

if all that works i hope to include to introduce human speech (whoever is in the room with this thing) into the mix as well

it's eternal loop

jfc, its

is the idea that these units get a lot of crosstalk from each other so it ends up as cacophonous siri feedback, or are they more orchestrated?
would it be bad if they get slightly out of phase?

the goal was to try to make it tightly coordinated, but i was i interested in what would happen when they inevitably went out of phase.

but shit, bad news. i thought the easy part would be the speech synthesis, because as I already had that working on my mac. turns out that audio and raspberry pi don't work well together at all - playback is wildly irregular, almost always cutting off pieces of sentences, and sometimes starting playback a second into the .wav file, or cutting off the last bit. it sounds the same whether i use aplay or omxplayer to play sound, or the native functions of TTS software like festival, festival lite, espeak.

i don't think it's me, either, i think it's the RPi. reading through this thread (https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=47942) the guy who nails it is the pair of posts from nPHYN1T3 on page 2. i'm feeling you nPHYN1T3, i'm right there too

sorry for drunk. but i'm kind of super bummed because i thought the RPi would be powerful enough to at least play sound files on command. if it can't do that i probably need to start over.

Seems like these weenies will sell you an outboard DAC board for your Pi?

https://www.hifiberry.com/products/dacplus/

Tho to be clear I have no idea if that would help at all lol