Raspberry Pi - A credit card sized, linux based mini-computer for $30

lol. another slightly "dumb" option could be to place a few sensors in places that are best for the look of the thing. grid out the room. get some volunteers and measure every possible permutation you'd like to account for. Then you'd just have to calculate a "distance" between every measurement and one of your pre-measured cases and select the closest one. A display response would be associated with each pre-measured permutation. This could get messy if you find that different people have widely varying scattering cross-sections. Would also be a disaster if the sensors were moved or varied with time in any way.

oh man, i just went through a cycle of despair and hope. up until a few minutes ago, i thought any multi-sensor solution was hopeless because of the crosstalk sufjan mentioned above. i started thinking about the waves bouncing at obliuque angles off of objects (and walls) and getting picked up by the second sensor long after they were sent out by the first.

but i think it's ok as long as the two sensor's measurement windows are spaced a few tenths of a second apart. each measurement takes place in a very short amount of time. the speed of sound is 343 m/s at sea level. so if an object is 10 m away, the ultrasonic wave will reflect off the object in ~0.0145 seconds, and return to the sensor at ~0.029 seconds. if i understand correctly, each measurement from the ultrasonic sensor takes place in a brief window of time - it sends a wave out and listens for a defined amount of time (say, 0.1 seconds), then it closes up. so even if a wave hit an object and bounced around the room several times, it would be undetectable to the second sensor because the second sensor's measurement window wouldn't be open yet.

so you'd have the two sensors collecting measurements staggered by 0.2 seconds from each other, and the signals wouldn't interfere with each other, and you'd still be able to get two different readings on the same object as about the same moment in time.

i'm still not sure exactly which method to use to translate the readings into different buckets of videos, but i think it might be something that could be done in a messy/dumb fashion through trial and error.

yes, that time division multiplexing approach should work so long as there isn't a strong resonance in the room at the frequency of the sensor. You can also maybe get sensors operating at different frequencies that filter for that frequency if TDM doesn't work. I would not despair. I think you'll get something good working. You seem to have a good intuition for this stuff.

Raspberry PewDiePieBeret

this is silly, but i'm thinking about other ways to track people near the display. vibration sensors are super cheap. i'm wondering if i could lay down a rug in front of the display and then install a grid of vibration sensors. no matter where someone stood on the rug, they'd be setting off one or more of the sensors on the grid, and it would be easy to track multiple people - almost like an overhead view version of the rug area. downside - no one would ever step on the rug. i think people would think they weren't supposed to touch the rug. bummer. oh well, this could at least be fun just to have at home

i'm still going to order the ultrasonic sensors too, though, since they're really cheap as well

I'm at a loss here since I was digging around to find which Japanese video artist I saw a work from at the NY MoMA a couple weeks ago, but all of this motion sensor talk is reminding me of it: a room with a series of moving projectors in the center, triggered by the presence and movement of viewers in the room, that projected a group of people walking and running on the walls of the room, overlapping but never touching

webcam solution possible (and free)!
https://webgazer.cs.brown.edu/#examples

that is amazing!!! i think you'd have to require a calibration for every person that used it, though, and it wouldn't make sense for more than one person at once. i bet there's some other really cool stuff you could do with that, though!

even the basic demo game is kind of otherworldly, controlling it with only your eyes. holy shit!

controlling it with only your eyes. holy shit!

immediately makes me think of devices for people with cerebral palsy

I tried that just now and it took a few seconds before I realized that without a webcam on my end it didn't matter were I focused my eyes.

Kinect? Gives you colour image AND a depth map (per pixel distance measurement) and the software is capable of whizzy things like skeleton and face detection, can detect up to 6 bodies... Expensive though and I think the open source library has been bought by someone and and the new fork is lagging behind iirc.

oooo you could make the eyes widen in response to screams of terror

a kinect does seem like a good idea if you have the budget and can get it running fast enough on the pi

I'd never looked into how the kinect works. It seems that it has an illuminating IR LED and then an ASIC to quickly process what the camera is seeing reflected back using a trained machine learning algorithm. So it is like the deluxe version of my "stupid" idea for the ultrasound sensors.

those are all really good ideas! it looks like i'll have some options. tbh at the moment i am planning on it tracking only one (the closest) person. i'm working on some video cycling logic that lets it smoothly track from one target to another, so if one person is the target and then they suddenly turn around and leave, the eyes/face will smoothly work its way over to the next-closest person detected (or to it's default resting cycle if no one else is there).

i won't break any ground technically but i hope to make the videos interesting and transition from one another smoothly

have any of you used the Pine64 (https://www.pine64.org/)? or heard anything about it from someone who has?

i'm working on something involving speech recognition and synthesis and while i have the guts of it working on my raspberry pi 3, it's painfully slow. a 2 second speech recording takes about 6 seconds to convert to text, and that's before i even do text-to-speech. meanwhile, i can run the same python code on my mac and the conversion is near-instantaneous. that suggests to me that the slowdown is due to the lack of specs on the raspberry pi. fishing around for a way to improve the performance, i saw some chatter about the relatively new pine64, which apparently has more processing power and general capabilities while still in the $30-50 range.

or actually...maybe it would make more sense to look into inexpensive ways to augment/power up the RPi somehow?

How do you want to install the final project? Is it feasible to use the Pi as a client and offload the speech-to-text to a faster computer over the network? I can well imagine it being hard to get low-latency speech to text performance on any cheapo device.

really hoping to make it as modular and mobile as possible, like completely independent and offline units that could be arranged in different ways. i wanted offline because it has to run for a few hours straight and i didn't want the whole thing to break if the wi-fi went down or something.

Is it feasible to use the Pi as a client and offload the speech-to-text to a faster computer over the network?

maybe? i think it would work, but i don't know how much faster it would be. each Pi unit has its own USB microphone to pick up the last signal in the chain, and each unit also has it's own speaker output connected to it. So each pi unit would have to send the recorded audio file to the faster computer, which would convert it (very quickly), but then send the converted file back to the pi unit again before playing it through the speaker. it just seems like the wait times for sending it back and forth would negate the improvement in speech-to-text conversion speed.

Sensible of you. I think you might want to treat this as a “prototyping” stage and live with what $50 gets you. It looks like for $500 you can get a development board for the NVidia Tegra X1, which is the chip the Nintendo Switch uses and probably has several orders of magnitude better performance than a Pi. Expensive but still cheaper and smaller than duct-taping a MacBook to something.

Don’t overestimate the round trip time of the network though. Six seconds is a long time.

oh, i'll definitely give it a shot! i'm also going to look into adjusting the dictionary i'm using for the speech recognition (using Sphinx). i saw a video of some dude online who is getting a decent turnaround on his RPi3, so maybe he was using a limited dictionary?

this stuff is sometimes frustrating but really fun to look into (especially when it works in the end)

Are there any other speech recognition softwares you've tried?
If the idea is to just robotize people's speech, maybe have it recognize/generate phonemes instead of words?

all the other speech recognition i've found requires an API: a couple google versions limited to 50 API calls a day, IBM, Bing, Jasper, Houndify, others.

i've been thinking of a way to take advantage of the slow processing. i'm thinking a set of 4 units that each work on 12 second cycles: 2 for listening to speech, 6 for converting to text, 2 for text-to-speech playback, and some buffer time in between each of those steps. the timing of the units would overlap so there'd be playback every 2 seconds. like this:

https://i.imgur.com/6afC7mA.png

(sorry for ugly colors)

actually, the cycle length would vary, but each would loop every 32 seconds with the timeline above. i'm thinking i can use python's time.time() to keep everything on schedule, so even if individual components in the cycle (record, convert, playback) take longer or shorter than expected, each step should begin at a precise point in time. the biggest issue (i think) will be with calibrating the units at the beginning. i'm thinking i can start the first unit on it's eternal loop, then manually trigger the launch of the second unit, listen for a few cycles to make sure they're in alignment, continue to the 3rd, and then the 4th.

if all that works i hope to include to introduce human speech (whoever is in the room with this thing) into the mix as well

it's eternal loop

jfc, its

is the idea that these units get a lot of crosstalk from each other so it ends up as cacophonous siri feedback, or are they more orchestrated?
would it be bad if they get slightly out of phase?

the goal was to try to make it tightly coordinated, but i was i interested in what would happen when they inevitably went out of phase.

but shit, bad news. i thought the easy part would be the speech synthesis, because as I already had that working on my mac. turns out that audio and raspberry pi don't work well together at all - playback is wildly irregular, almost always cutting off pieces of sentences, and sometimes starting playback a second into the .wav file, or cutting off the last bit. it sounds the same whether i use aplay or omxplayer to play sound, or the native functions of TTS software like festival, festival lite, espeak.

i don't think it's me, either, i think it's the RPi. reading through this thread (https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=47942) the guy who nails it is the pair of posts from nPHYN1T3 on page 2. i'm feeling you nPHYN1T3, i'm right there too

sorry for drunk. but i'm kind of super bummed because i thought the RPi would be powerful enough to at least play sound files on command. if it can't do that i probably need to start over.

Seems like these weenies will sell you an outboard DAC board for your Pi?

https://www.hifiberry.com/products/dacplus/

Tho to be clear I have no idea if that would help at all lol

hmm...it might! i was thinking about that earlier (actually, i was thinking of connecting to an arduino and using a wave shield, which would just be one more unnecessary step). i'm not sure if it would work either. it's not so much the sound quality, it's more just the weird thing about the beginning and ending of audio clips getting cut off. super weird. it would be one thing if the cutoff was predictable, because then you could just build in extra silence at the edges. but sometimes it cuts off half of the whole file, sometimes just the first second, sometimes only the first syllable. weird.
i might give the DAC a shot. thanks!

Isn’t there supposed to be like some media decoding in hardware on these chips? This is perverse but does an mp3 play back without an issue?

are you playing back audio through the HDMI or headphone jack? also, it might be a stretch but someone had figured out how to use raspberry pi to transmit FM radio (attach a bit of wire to one of the pins to act as an antenna) so that might be another way of getting generated audio off the pi.

Same problem with wav and mp3, yeah.

Same problem with 3.5 mm output. It’s really the craziest thing, seems like people have been complaining for years too.

I will troubleshoot some more tomorrow, but for tonight I declare temporary failure

Temporary failure is just the prelude to permanent success!

—Steve Jobs

I could track it down, but I had a cheap dac that was pretty darn good connected to a pi for quite a while. As in, purchased on alibaba and available in bulk cheap. I don’t remember any playback issues but I was playing music and not short clips

is it only happening on starting and finishing an audio clip? maybe keep the audio engaged by playing an infinite silent stream simultaneously so there's never a start/stop?

Philip i did run across a proposed solution along those lines, and i'm going to give that a shot. plan B (actually more like Plan Z-42 at this point) is going with an audio DAC. i'm also going to try to see if there are any maker-oriented physical stores in the Chicago area.

i also want to look into boosting the processing power - i'm kind of wary to overclock because my normally-clocked unit is having issues with power (if i have a USB microphone connected and then run chromium at the same time, the pi crashes). but there might also be ways to use all 4 cores of the pi rather than just one (this thread has several ideas along these lines: https://www.raspberrypi.org/forums/viewtopic.php?t=198425)

i was super bummed about this the other night but now i'm back in the saddle.

thanks for the ideas and help, by the way! really appreciate it.

are you playing back audio through the HDMI or headphone jack?

i tried the headphone jack again, this time by just right-clicking the desktop GUI sound icon and changing it to Analog. It worked! At the moment it's not cutting off any text, and isn't taking too long to process the TTS either.

i'm going back to the voice recognition (via eSpeak) part again tonight, only this time my plan is to use a pretty limited dictionary/language model (around 500 words) rather than trying to attempt to recognize everything in English. i think that might have been the source of the slow processing speed (it was taking about 6 seconds to process 2 seconds of audio).

Is it feasible to use the Pi as a client and offload the speech-to-text to a faster computer over the network? I can well imagine it being hard to get low-latency speech to text performance on any cheapo device.

― direct to consumer online mattress brand (silby),

silby, i'm a fool. i think this could work! i wanted to avoid dependence on the internet because i didn't want some sort of uncontrollable network connection issue to break it. but a friend just explained to me that it's possible to my computer as the server, the raspberry pi's as clients, and a local modem to transmit the wi-fi signal, without my computer needing to be connected to the internet itself. right? i'm a noob in pretty much everything i'm doing here so i might be way off, but it seems like the only possible connection issue would be the building's wi-fi interfering with the local wi-fi between my laptop and the RPIs, and even that would be avoidable by just using a unique port (or something) on my router?

with all that said, i'm viewing all of that as a backup option, and i'd still rather figure out a way to get these devices working together without any network, as fully independent links in the chain. but if i'm not able to do it, it'll be great to have the server/client processing as an option.

a minor victory today! i didn't realize that time.time() is so consistent and precise between UNIX devices. when i call it on either my laptop or pi, they return the same exact value (in seconds since jan 1, 1970..siiiick). i can run the following bit of code on two different devices, and they'll both play the sound at the exact same time, independently.

import time
import subprocess
def Launch():
	print ("unit initialization time: ", time.time())
	print("waiting for next 18-second cycle to begin...")
	while time.time() % 18 > abs(.005):
		time.sleep(.001)
def Cycle():
	while True:
		print("new cycle: ", time.time())
		subprocess.call('afplay test.wav', shell=True)
		while time.time() % 18 > abs(.005):
			time.sleep(.001)
Launch()
Cycle()

the way i'm making it wait for 18 second intervals is a little hacky but it works.

what's nice is that i can start up one device and then wait as long as i want to start the second, and they'll both sync up as soon as the start of the next 18-second cycle comes around. to me, this is exciting because it means i can define various events in an endless 18-second loop (play a message, record a message, choose a new phrase and play it back, etc), and even if the processing speeds for the various events vary from unit to unit, i can still specify exact times in the cycle for the events to happen so that they stay in sync. plus, it means that the entire chain of devices can be modular, constrained only by the cost of buying all the equipment and the size of the room.

while time.time() % 18 > abs(.005):

why abs() on a constant positive value?