New study pans the intelligence of voice assistants

Natural language holding back true digital assistance

Study finds assistants are pretty dumb

A new study of voice assistants has tested the capacity of Amazon Alexa, Google Assistant and Siri to help satisfy people’s queries – and the results are not brilliant.

The Nielsen Norman Group (NNG) has performed an extensive – if less than scientific test – measuring not only the response, but expectations of users. It got together 17 participants from New York and San Francisco, all of which were experienced voice assistant users, and put them in a lab to carry out a number of smart speaker tasks, and interviewed them about their expectations and quality of results.

While it rated voice input as a good experience for users, and noted that support for accents was improving, it rated most other elements as “bad”. Those include using natural language, the voice output of voice assistants, and the ability to tap into users’ services (calendars, email, to-do lists) and pull out useful information.

The study found that natural language was one of the biggest barriers, with voice assistants regularly tripped up by sentences with multiple causes:

“The majority of the participants felt that complex, multiclause sentences (such as “What time should I leave for Moss Beach on Saturday if I want to avoid traffic?” or “Find the flight status of a flight from London to Vancouver that leaves at 4:55pm today”) were unlikely to be understood by the assistants,” the study reads.

Getting the language right was a theme of the study and Alexa’s skills came under fire from the participants. Requiring users to remember the exact name of a given skill was one major criticism, and the specific requirements of phrasing commands.

“The majority of the Alexa users did not know what skills were; some had encountered them before, installed one or two, and then completely forgotten about their existence.”

While the NNG survey is far from scientific, and doesn’t really generate any clear or meaningful conclusions, it’s refreshing for a study to put value on the expectations of users. Voice assistant tests usually mean parroting the same questions at speakers, but this is better designed to be a frank assessment of the challenges faced by the likes of Amazon, Google and Apple to produce a true digital assistant.

TAGGED   speakers

Recent stories

amazon ​Amazon Alexa ultimate guide: How to use and get more from your Amazon Echo
amazon Amazon Echo Show guide: Everything you need to know about the 2018 edition
google The 70 best Google Home Easter eggs to try right now

What do you think?

Reply to
Your comment