Augmented reality in the smart home, outside the bounds of smartphone screens and smartglasses, could make our interactions with technology more social, more playful and more useful. Add depth sensors, projectors and voice interaction into the mix and suddenly every object in our homes could be enhanced with virtual features.
That's what Swift Creatives, a design and innovation studio, has been up to with its recent batch of AR concepts. Matthew Cockerill, creative director and head of studio for Swift Creatives London, told us that the project came out of work they did for HTC last year and the limitations the team saw in Apple's ARKit and Google's ARCore. The concepts below, he says, are "one to two years away".
"We fill our houses with furniture thatās 100 years old and high tech TVs," he says. "People want a rich, warm environment that allows them to have rich experiences with the people in their homes, thatās why AR is exciting for us."
Cockerill and his team built prototypes for each of the four augmented reality concepts, but he notes that in the videos, some of the actions are based on real data from sensors and some are simulated, to save time and money. The next stage that Swift Creatives is moving onto this year is figuring out what can be achieved with existing platforms including ARKit 2.0, ARCore and Microsoft HoloLens.
We asked Cockerill to talk us through each of the AR smart home ideas.
The connected worktop
"There have been lots of these concepts where they will project the steps of how to make the bread onto the table," says Cockerill. "And in a way, consumers donāt need that because you can get that valuable experience with an iPad that youāve already got. So you donāt need to buy a new system. But actually what augmented reality means is that the system intimately knows the 3D environment.
"What weāve done with these prototypes is look at how the system intimately understands the topology. So when youāre pouring flour, itās actually understanding the profile of that flour and so it can calculate the volume of how much is there.
"So itās actually allowing you to free pour flour onto a table until the system tells you that thereās enough flour for your recipe. That gives you something that a smartphone canāt do.
"Iād imagine connecting it to Alexa. We didnāt do that with the prototypes but you could say, 'Alexa, I want to make some bread.' Alexa might say, 'OK letās measure out the flour.' You look at our visuals, a dot appears on the table and it says, āStart pouring nowā."
The flight tracking globe
"If I have a globe at home, I could ask Alexa 'Alexa, whereās Janeās flight right now?' says Cockerill. "Then an AR digital picture of an aeroplane flies around the globe and starts hovering where the plane actually is in relation to the world. Itās totally doable. The great thing is, that digital information is already there in APIs for flights. This is about having digital more connected to our physical objects.
"If you rotate the globe, it shows you the estimated time of arrival because spinning a globe around is obviously like fast-forwarding things, If you rotate it around, you can see a dotted line to where itās going to land and how many hours it is from that destination. Itās not only about showing data on an object thatās relevant so an aircraft on a globe, but the fact that you can interact with that physical object and the data can change. Thereās really interesting areas to explore around manipulating physical objects and having digital data update.
"Then thereās the idea of metadata. We think in a way this globe represents the idea that physical objects have this digital metadata associated with them. So the sell by date on food in the fridge, an eBay bid on objects I'm selling or my globe, all have digital data connected to them. And when you start looking at the world like that, you could think about ā when I look at my car, what digital overlay could I do on physical elements of my car? As designers it opens up a whole world of possibilities for us which weāre quite excited about."
The snowfall that sees your ornaments
"What weāre trying to do is use technology to assist in your daily life ā from that Hygge, very Scandinavian perspective, which is the origins of our consultancy," says Cockerill.
"What weāre trying to do is create a more delightful and useful life in the home. We donāt want to be focused on tech. The snowfall, for instance, itās a bit of a clichĆ© I guess but you could consider it almost a lava lamp, having low level interesting lighting in your home. But it recognises things like pictures and objects so it can exclude those from the snowfall to give you more of a sense that itās integrated with that environment.
"Right now I could create some snowfall and it would project onto a room but it would just project onto everything and youād lose that illusion. What our system can do is take the picture and understand that thereās a horizontal edge and start building up a snow pile onto it."
The ambient baby mobile
"The way we conceived these was to show how these experiences could be valuable through the day. So from the morning in the babyās room, through to cooking, through to socialising and just relaxing in the evening.
"What you see in the videos is one to two years out because of two issues," he explains. "Thereās the latency. If I have an object then I move it and Apple ARKit at the moment canāt update quickly enough to provide that experience.
"And the other thing is about trying to track lots of things so the mobile, for instance, where itās all moving and we map the physical mobile to those shadows. Thereās literally too many of those elements moving at once to do that. Itās about tracking and mapping and itās about latency."