They’re bad folks, everybody knows it, everybody says it. But they’re simultaneously better and worse than you think, because apparently nobody actually knows how these products actually work. (I’m mostly going to be discussing google home, because that’s what I’m familiar with; I assume echos use similar technology, but I can’t speak for them).
So to get started, no, your smart speaker isn’t always listening. At a hardware level, there’s two boards and a tiny bit of cache. Your speaker is constantly listening for the trigger words, and processing about 2 seconds of audio stored on the cache at a time. However, this is all done at a local level on the first board. Only once it recognizes the trigger words does it establish a connection to the cloud, and use that to process your request. Once your request is complete, it goes back into standby mode. You can look at the packets coming out of the device, and see that it only connects to the internet when it needs to. The onboard cache is small, and constantly being overwritten, so there’s literally no way for it to constantly be monitoring you, by design.
However, what IS nefarious is the amount of permissions you have to give google in regards to the data it does capture. Of course, they use the captured audio for expected things like training their voice recognition AI, but you also give them permission to store all that data indefinitely, with metadata tracing it back to you, AND it’s not off limits to engineers.
That’s right, there’s the possibility, however small, that real people will be listening whenever you ask google to play your erotic jazz playlist. Once that audio is on the cloud, you basically don’t own it anymore, and google can do whatever they want with it.
So should you be worried? If you want to be, I guess. I resigned to the fact that I lost all my digital privacy before I was even born, and will happily tell google to turn off my lights while laying in bed like a fat sack of shit, but it comes down to what you’re comfortable with. Either way, I just want people to actually understand what they are and how they work, because there is a lot to criticize, so it pays to be criticizing the right things.
The good news is that “smart homes” aren’t going to be a huge thing for many years if you are afraid. If you wanted to do a “smart house” then I’d look into systems used on the market by office buildings like wattstopper. These are usually wired locally and programmed locally so you don’t need to give google or Amazon more information than you need to. You could control some of your options remotely by using a land line which is way more secure in my opinion than going through a cloud with Amazon or google. This is the system that I see being installed in “smart” buildings that I wire up as an electrician.