Smart home privacy: What Amazon, Google and Apple do with your data

And what you can do about it

How tech companies use smart speaker data

Nearly 70 million Americans own a smart speaker – that's more than a fifth of the population. Add to that those who use Google Assistant, Siri, Alexa (and yes, Cortana and Bixby) on their smartphones, in their cars, and even on their computers, and it becomes clear that artificially intelligent voice assistants are now part of everyday life. But what are we giving up in exchange for the ability to ask questions, listen to music, hear the weather, and set timers all with just our voice?

News stories revealing yet another privacy violation by a tech company are becoming a regular occurrence, making it increasingly hard to brush off concerns as merely isolated incidents. When you put these always-listening-for-their-wake-word devices in your home the concept of surveillance is not just a worry for those with something to hide, it becomes about protecting a basic human right. This makes it even more important to understand what companies are and aren't doing with the information they collect in your home.

There's no denying that smart speakers and smart displays are incredibly useful for smart home owners. They've essentially removed the need for a smartphone to control your home and have made it simpler to use connected devices in multi-person households.

But now that they're in your home and potentially capable of listening to everything you say, you need to educate yourself about what the companies who control these devices are doing with your data. We've waded through the privacy policies, dissected all the news stories, and spoken to the big three smart speaker manufacturers – Amazon, Apple and Google – to bring you this guide to what smart speakers do with your data, and what you can do about it.


Alexa: Amazon Echo devices

Amazon's suite of smart speakers, smart displays, third-party devices with Alexa Voice Services built in, plus the Alexa smartphone and tablet apps, makes Alexa the most dominant AI out there. Here's a look at what Amazon does with your voice when you say one of Alexa's magic words:

What does the device record and when is it supposed to record?

When you speak to Alexa, a recording of what you said is sent to Amazon’s cloud, where it and other information is processed to formulate a response. For example, when you ask “Alexa, play top hits on Amazon Music,” Alexa uses the recordings of your request and information from Amazon Music to play top hits.

By default, Echo devices are designed to detect only your chosen wake word (Alexa, Amazon, Computer, or Echo). The device detects the wake word by identifying acoustic patterns that match it. No audio is stored or sent to the cloud unless the device detects the wake word (or Alexa is activated by pressing a button).

You will know when Alexa is recording and sending your request because a blue light indicator appears or an audio tone sounds.

If the device has a camera, are any video recordings stored?

Video calling sends video to the cloud, but Amazon says is only streamed and never stored.

Where are the recordings stored and how are they secured? Can I be identified by my recordings?

All voice recordings streamed to the cloud are encrypted and securely stored on Amazon’s servers. All voice requests are associated with your Amazon account. This allows you to review your voice recordings, access other Amazon services, and helps Alexa give you a more personalised experience.

Who else is listening?

Amazon has a voice review program – comprised of Amazon employees and contractors – that analyses a random sample of recordings captured by Echo and other Alexa devices in order to improve Alexa's intelligence. Amazon says it annotates a fraction of one percent of interactions from a random set of customers. Since the beginning of 2019, data analysts transcribed 0.2% of all requests to Alexa. A typical Alexa recording averages only 2 seconds in length. Contractors are subject to privacy agreements and Amazon says no one has direct access to information that can identify the person or account.

What happens to recordings made by mistake? Are they still analysed?

If an Echo device wakes up due to a word in background conversation sounding like Alexa or one of the other available wake words, it stops processing the audio and ends the audio stream to the cloud once it determines the audio is not intended for it. A short portion of the audio is stored in your Voice History in the Alexa app or on the website.

Can I opt-out of my voice recordings being analysed by humans?

Following recent news reports around the voice review program, Amazon now gives you the option to prevent your recordings being analysed. Go to the Alexa app. navigate to Settings > Alexa Privacy > Manage How Your Data Improves Alexa and turn off the tab under Help Improve Amazon Services and Develop New Features. Turning this off will potentially result in new features and voice recognition not working as well for you., but you now have the choice.

Can a user access recordings and can they delete them? Do they auto-delete at any point?

You can review and delete all voice recordings associated with your account in the Alexa App or on the web. You can also turn on the ability to delete recordings by voice, and delete the voice recording of your last request by saying “Alexa, delete what I just said,” or delete all the voice recordings from your account for the day by saying “Alexa, delete everything I said today.”

How long does Amazon keep customers’ voice recordings?

Voice recordings are kept until you delete them.

How does the voice data benefit Amazon?

As an artificial intelligence engine, Alexa is designed to learn. The more data Amazon can use to train these systems, the better Alexa works. Your data also has value when it comes to advertising and marketing. Amazon will serve up personalised ads to you on its various properties based on the data it has about you. You can opt out of these in your Amazon account preferences.

According to the company, the majority of Alexa interactions are not used for advertising. The experience on Alexa is similar to what you’d see on the Amazon website or Amazon app. For example, if you play a song on Alexa, you may see recommendations in the Amazon Music app for other artists you might enjoy. Or, if you order paper towels via Alexa, you may see recommendations for other similar products on the Amazon website.

Amazon says it doesn't use your other interactions with Alexa, like asking for a recipe or the weather, for product recommendations. Amazon does not allow advertising on Alexa outside of certain third-party skills such as streaming radio skills like Pandora or news skills like CNN.

How does the voice data benefit you?

The more you use Alexa, the more it will adapt to your speech patterns, vocabulary, and personal preferences. For example, keeping track of the songs you have listened to helps Alexa choose what songs to play when you say, “Alexa, play music.” Alexa can also learn what you like based on your requests and recommend skills you might want based on the skills you already use.

Do third party apps/other properties owned by Amazon have access to data from voice recordings?

Amazon says it does not share voice recordings with any third parties. When you use a third party service through Alexa, it exchanges related information with the third party so they can provide the service, but not actual voice recordings.

A restaurant booking skill might ask for your email address

No personally identifiable information is shared without your agreement. For example, a restaurant booking skill might ask for your email address to send you confirmation of your reservation, but Amazon only shares that with your permission.

You can control which Alexa Skills have requested permission to access data in the app – head to Settings > Alexa Privacy > Manage Skill Permission in the Alexa app or on the website.

Do third party speakers and devices (such as thermostats or smoke alarms with Alexa built in) have access to voice recordings?

No. Amazon says it doesn’t share voice recordings with any third parties.

What data does the Echo device collect, other than voice recordings?

In addition to voice recordings, Amazon says it collects other data necessary to provide and improve the service, such as information about the use of the device, network diagnostics and log files.


Google Assistant: Google Home speakers

With a small but growing cadre of smart speakers and smart displays, Google Assistant is fast finding a place in our homes, and if you're an Android user you've likely been chatting away to Google for a few years now. Here's what the company that once embraced "Don't be evil" as its motto does with your data:

What does the device record and when is it supposed to record?

As with Alexa, queries given to the Google Assistant are stored as recordings in Google's cloud, where they're processed in order to formulate a response. And like Alexa, the Google Assistant is always listening (unless you've muted it) but it will only start recording when it hears the wake words, such as "Hey Google" or "Okay Google".

There are different ways to know if the Google Assistant is listening, depending on which device you're using. On Google Home speakers you'll see the four coloured icons light up to indicate it's listening, while on Google Smart Displays you'll see an icon on the screen. If you're using a third-party speaker such as a Sonos One, you'll hear a bleep and see an indicator light flash. It depends on what you're using, but there should always be some audio or visual cue set by default.

Where are the recordings stored and how are they secured? Can I be identified?

All voice recordings are encrypted and stored on Google’s servers. Google says it strips personal identifiers from voice recordings and instead attaches a unique number, rather than a name. More concerning is the high number of "false accepts" that the Google Assistant is apparently processing, and which are landing on the desks of Google's AI reviewers.

While these recordings aren't directly linked to people's identities, unintended voice recordings are more likely to contain sensitive information that may give their identity away.

Okay, who else is listening?

Some recordings are shared with contractors who are tasked with analysing voice data to improve the Google Assistant. Google says that 0.2% of all audio recordings captured are listened to by reviewers.

If the device has a camera, are any video recordings stored?

Not with third-party cameras that work with Google Assistant, but for the company's own Nest devices, video footage is stored in your Google account. You can access, review, and delete this footage – head here for more information on how.

Google's upcoming Nest Hub Max smart display will have a built-in camera, which will sometimes be used for Face Match, a tool for determining who is using the device. Google says that video will be sent from the device to its own servers during the setup process, but not beyond that.

What happens to recordings made by mistake? Are they still analysed?

If any false accepts are taken, there's nothing to stop those recordings being processed and reaching the point where they're reviewed. Reviewers are directed not to transcribe anything that is deemed a background conversation, but there’s still a chance these clips will reach the ears of other humans.

A recent privacy blunder for Google saw more than 1,000 Assistant voice queries leaked by a contractor gone rogue. Among them were 153 snippets that were clearly recorded by mistake.

Can a user access recordings and can they delete them? Do they auto-delete at any point?

You can review voice recordings associated with your account and delete those voice recordings one by one or all at once. Here's how to delete your Google Assistant voice data.

How long does Google keep customers’ voice recordings?

Voice recordings are kept until you choose to delete them.

Is data used to give me ads?

Here's where the messaging is less clear. The official line from Google's Nest Help privacy page is as follows: “We commit to you that for all our connected home devices and services, we will keep your video footage, audio recordings, and home environment sensor readings separate from advertising, and we won’t use this type of data for ad personalization. When you interact with your Assistant, we may use those interactions to inform your interests for ad personalization.”

That last sentence sounds like a contradiction, and when we approached Google for clarification, it simply told us that it does not use audio for advertising purposes. However, the company says it does not sell personal information to any third parties, but may share some things like your phone number if you allow it. Which brings us to…

Do third-party apps/other properties owned by Google have access to data from voice recordings?

Google says it does not sell personal information to third parties, but it will share data when users interact with third-party services.

If you're using Google Assistant to interact with a third-party service, some information may be given over. Giving the example of Uber, Google says it will send information provided in order "to complete a booking or confirm a ride." However in these cases, you must have given Google permission to share that information with the service.

Google transcribes what you say and sends the text, but not the audio, to the third parties

Google elaborates a little more in its terms of service: "We don’t share information that personally identifies you with our advertising partners, such as your name or email, unless you ask us to share it. For example, if you see an ad for a nearby flower shop and select the “tap to call” button, we’ll connect your call and may share your phone number with the flower shop."

On the Google Home privacy terms page, the section titled 'Does the third-party service provider get an audio recording of what I said?' Google's response is, "Generally no. Google transcribes what you say and sends the text, but not the audio, to the third-party service provider."

With third-party devices on your network, such as smart security devices, Google says it will share data with those providers to provide a "helpful experience," but again you must give permission for it to do so.

In sum, voice recordings are never shared with third parties, transcriptions may be, and some personal details such as phone numbers and email addresses can be, but only if you give permission. But the fact we had to source this information from three different places goes to show that Google is doing a bad job of making this clear.

How does the voice data benefit Google and you?

Like Alexa, the more Google Assistant learns about you, the better it is at understanding your speech patterns, personal preferences etc. This goes for things like music preferences; if Google knows your music habits, asking the Assistant to “play music” will make it better at selecting songs you’re going to like.

The more information Google can process, the better the Assistant can get at parsing accents and dialects. For example, Google partners with language experts around the world who, in Google’s own words, process “a small set of queries” to help it better understand different languages.


Siri: Apple HomePod and iOS devices

Slower to seek out a space in our homes, Siri only comes in two flavours outside of iOS devices – HomePod and Apple TV (which doesn't really count as you need to use a remote to activate it). However, the power of the fruit shall not be underestimated and as HomeKit rapidly gains functionality the number of us using Siri on these devices to execute our every whim in our homes will grow. Here's what Apple does with your digital voice emissions:

What does the device record and when is it supposed to record?

Siri is triggered when it hears "Hey Siri." At this moment everything is still happening locally on the device, whether that's a HomePod, Apple TV, or iPhone. But once you've spoken to Siri, the audio is sent to Apple's servers for further processing.

However, some Siri activity does stay private on the device. For example, with the arrival of iOS 13, Siri can detect events in your apps and suggest adding reminders to the calendar, but all of this will take place without sending anything to the server.

If the device has a camera, are any video recordings stored?

The only case this might happen is with FaceTime. Here's Apple's line on the matter: "Apple may record and store information about FaceTime calls, such as who was invited to a call, and your device’s network configurations, and store this information for up to 30 days. Apple doesn’t log whether your call was answered, and can’t access the content of your calls."

Where are the recordings stored and how are they secured?

Queries sent to Apple's servers are accompanied by info such as your name, contacts and music preferences, in a package that's encrypted end-to-end. However, your Apple ID is stripped from this information, which is instead given a random identifier.

Apple also uses something known as differential privacy, whereby random but "carefully calibrated" noise is mixed in with the data it collects to obscure individual entries. (You can read Apple's whole spiel about it here).

However, it has been reported that some information such as location, contact and app data is associated with these recordings. Apple did not elaborate on this when we asked for further comment.

Who else is listening?

Like Amazon and Google, Apple has a fleet of contractors who review Siri voice recordings in order to improve the AI. Apple says less than 1% of audio recordings are put under review. All these contractors are subject to strict privacy agreements.

As of August 2019, Apple has suspended what it refers to as "Siri grading," and plans to roll out an option in a future software update that will let you choose to opt out of having humans potentially review your voice recordings.

What happens to recordings made by mistake? Are they still analysed?

In short, yes. Siri will start listening when it think it hears the wake words, and as reports have shown, many of those recordings still fall in the laps of reviewers. This means recordings containing sensitive information, including anything that identifies the speaker, could be heard by another person.

Can a user access recordings and can they delete them? Do they auto-delete at any point?

No, and this is somewhere Apple falls down. The company would likely justify not offering users the option by pointing at its systems for anonymising recordings – but we wish there was a way to review and delete individual recordings. For now, all you can do is delete your Siri voice history from your device by disabling Siri, but there's nothing to suggest this will wipe the recordings from Apple's servers.

How long does Apple keep customers’ voice recordings?

Apple told us it stores recordings along with their identifiers for six months. After this time, a copy of each recording is stored on Apple's servers for the purpose of improving Siri, and will stay there for up to two years. However, these copies are stripped of their identifiers at the point of copy.

After two years, Apple says a "small sub-set" of recordings and transcripts are saved for ongoing work to improve Siri, but they are stored without their identifiers.

Do third-parties have access to voice recordings?

Apple says it shares some data with third parties "to provide or improve our products and services, including to deliver products at your request, or to help Apple market to consumers." Which, yeah, is pretty vague. However, the company also says that "personal information will never be shared with third parties for their marketing purposes. In the past the company has said explicitly that it does not give third parties access to voice recordings.

Apple says it shares some data with third parties

"Apps supported by HomeKit are restricted by our developer guidelines to using data solely for home configuration or automation services. Apple does not know what devices you’re controlling or how and when you’re using them." Apple says that it associates your HomeKit devices with your Siri identifier, not your personal ID.

When you control HomeKit devices remotely, the information is encrypted. "Apple doesn’t know which devices you’re controlling or how you’re using them," the company says.

What data does the Apple device collect, other than voice recordings?

Location is the big one. If you have Location Services switched on in your settings, your device's location will be sent to Apple to help improve Siri's accuracy.

How does the voice data benefit Apple and you?

In short, it makes Siri smarter. And let's be honest, compared to Alexa and the Google Assistant, it needs it. But the fundamentals here aren't much different across the board.

Which smart speaker is the most private?

At first glance, Siri appears to be the most private smart speaker, as Apple anonymises your voice recordings before sending them to the cloud. Neither Google nor Amazon do this.

However, you can delete any and all recordings on those platforms, which you can't do with Siri. Siri is also a lot harder to turn off than Alexa or Google, both of which have quick access mute buttons on their speakers – not the case with HomePod. Also, and perhaps more importantly, Siri really isn't that capable as a voice assistant, and unless you have a HomeKit home, it's not a solution for controlling your lights, locks and other gadgets.

So, what's a smart home user to do if they're concerned about privacy but want the convenience of voice control? When it comes down to it, if you're going to use a smart speaker it's about which company you're more comfortable with, and that is always going to be a personal preference.

However there are some actions you can take – they require a bit more effort, but just like using a shredder, they'll be worth it in the long run. Delete your voice recordings every day; don't place smart speakers with cameras in bedrooms or bathrooms; mute the nearest speaker if you are having a sensitive conversation, and always remember that, however slim, there is a chance that what you say to or near one of these devices will be recorded and stored on a server somewhere for perpetuity.

TAGGED   amazon   google   apple   smart home   speakers

Recent stories

speakers China's Baidu overtakes Google in smart speaker sales
amazon Shhhh! How to turn on Alexa's Whisper Mode
amazon The best IFTTT Alexa and Echo Applets for your smart home
What do you think?
Reply to
Your comment