91av

“I expect that after many months of using the VEST, people who are deaf will have a direct perceptual experience of hearing that is the equivalent of how you or I hear” (Image: Dan Winters)

You have described the brain as “locked in a vault of silence and darkness”, so how does it create such a rich reality for us to experience?
That’s one of the great mysteries of neuroscience: how do electrochemical messages in your brain get turned into your subjective experience of the world? What we know is that the brain is good at extracting patterns from our environment and assigning meaning to them. I’m interested in how we can plug alternative patterns into the brain and experience additional aspects of reality.

What new realities could we perceive?
We only pick up on a small fraction of signals that are going on in the world: those for which we have developed specialised sensors. There are many other signals out there – X-rays and gamma rays among others – but we’re completely blind to them. No matter how hard we try, we’ll never see that part of the spectrum naturally.

But the brain is really flexible about what it can incorporate into its reality. It receives information in the form of electrochemical signals from our eyes, our nose, our skin, and works out meaning from them. Crucially, it doesn’t care where these signals are coming from; it just figures out how to use them.

I think of the brain as a general-purpose computer. Our senses are just plug-and-play devices that we have inherited through evolution. And if that’s the case, we should be able to interface any data stream into the brain and it will figure out how to deal with it.

“Our senses are just plug-and-play devices inherited through evolution”

So how do you plan to patch a new data source into the brain?
We’re experimenting with what we call the versatile extrasensory transducer, or . It’s a wearable device covered with vibratory motors (pictured). When you wear the VEST, at first it just feels like strange patterns of vibrations on your torso, but the brain is really good at unlocking sensory patterns and figuring out what the input means.

(Image: James Duncan Davidson/TED)

How does a vibrating jacket allow us to experience a different reality?
Well, for example, we are trialling it with deaf participants at the moment. We capture sound from their environment and translate it into different patterns of vibrations. After a week or so our volunteers are able to figure out what’s being said using the vibrations alone. They can understand the auditory world through their skin.

Do these people actually start experiencing hearing in the same way non-deaf people do?
That’s a question we’re working on right now. My expectation is that after many months of using the VEST, people who are deaf will have a direct perceptual experience of hearing that is the equivalent of how you or I hear.

We don’t know if it will be exactly the same experience – it’s hard to communicate what a sense feels like, because our language is only capable of sharing things that we have in common. Nonetheless, the meaning of what someone is saying is directly evident to the deaf participant. It might be like learning a foreign language – at first you read it and translate it in your head to get the meaning, but with practice you just read it and understand the meaning directly.

But the brain is specialised to hear different frequencies. Can it really figure out speech from vibrations on the skin?
It seems crazy to hear via a moving pattern of touch on the skin, but ultimately, this is just translated into electrochemical signals coursing round the brain – which is all that regular hearing ever is. Traditionally, these signals come via the auditory nerve, but here they come via nerves in the skin.

We already know that the brain can figure out meaning from arbitrary signals. For example, when a blind person passes their fingers over Braille the meaning is directly evident to them. And when you read words in a 91av article, you don’t have to think about the details of the squiggles – the meaning simply flows off the page. In the same way, we’re demonstrating that a deaf person can extract the meaning of words coming to them by vibratory patterns.

What else might we sense using the VEST?
Here’s an example we’re working on now. We stream 5 seconds of real-time data from the internet to a person wearing the VEST. Then two buttons appear on a screen, and the person has to make choice. A second later they get a smiley face or frowny face telling them whether their choice was the right one. The person has no idea that what they’re feeling is real-time stock market data, and that the buttons represent buy or sell decisions.

What could this experiment demonstrate?
We’re seeing whether participants get better at making trade decisions without having any idea what they are doing. People often come with a lot of assumptions about the market, and we want to see whether we can circumvent that by having the brain decrypt patterns without any pollution from prior knowledge. Then it’s simply a pattern-recognition problem for the brain. Eventually we’ll tell the participants what is really going on, and we are interested to know what the experience will be like for a person who wears this stock market VEST for long time. Are they suddenly going to feel a tightening in their stomach and think, “Oh gosh, the oil price is about to crash”?

Would they be able to describe in words what the different patterns meant to them?
Historically, people can figure out meaning from quite subtle data without ever knowing exactly what they’re doing. Take British plane-spotters for example. During the second world war, some were really good at distinguishing between British and German planes from a distance. The British government tried to get them to teach others to do it, but they were unable to. It was ineffable knowledge – they were picking out very subtle patterns without knowing exactly how.

The vibrating jacket has so many potential applications. What else have you in mind?
Yes, it’s hard to decide which possibilities to test first. We are playing with feeding the jacket a real-time sentiment analysis on Twitter, as filtered by a hashtag. Let’s say you’re a presidential candidate giving a speech. You could wear the jacket and feel how the Twittersphere is reacting as you’re going along. One of our other experiments involves working with pilots and feeding them cockpit data or drone information via the VEST.

“A politician using the jacket could feel the Twittersphere reacting to a speech”

That’s intriguing. Do the pilots end up feeling like they’ve become the drone, say?
Yes, it’s like extending your skin to the plane or drone, so you feel the pitch, yaw and roll. It’s a new perceptual experience that we believe will allow someone to pilot better. We’re thinking about astronauts too. They spend a lot of time looking at hundreds of monitors, so wouldn’t it be great if they could directly feel the state of the space station and know when things were changing or shifting?

Our whole lives are spent looking at little screens. In my view it’s better to experience the data rather than simply look at it.

Are there limits to how many extra senses you might acquire using wearables?
You mean, could you have a Twitter jacket and stock-market jeans? I don’t see why not. We don’t know if there are any limits to how many different things you could sense. My intuition is that the limits are distant. We’ve got an enormous amount of real estate in the brain. If you lose one sense, the area of the brain responsible for it gets taken over by other senses. The brain is great at redistributing space for what’s needed and there’s plenty of room to share real estate without noticing any diminished effects elsewhere.

Does the VEST system have limitations?
The thing that defines what is possible is what we can build sensors for. If we have a good sensor, it’s trivial to convert the information it captures into vibrations in the jacket.

Could this technology could help us monitor our health?
That’s a good example of waiting for the right sensors to come along. If you want to measure blood glucose you currently do it with a spot of blood, but now people are starting to develop sensors that can image glucose through the skin. When we have this sensor, it’ll be easy for us to use it to feed data to the VEST. Then you’d have a direct way of experiencing your blood sugar level.

It’s like when you sit still for too long in one position and then shift your body to get blood flowing to where it needs to go: you do that subconsciously using a stream of information from your muscles. It would be the same thing with the VEST. You’re taking something that is normally invisible and making it visible. You would directly experience blood sugar readings and think, “I need to eat”.

You mentioned in a recent TED talk that you might be able to give us 360-degree vision. How would that work?
I’m keeping that one under wraps for now, but the basic idea is that you’d wear a 360-degree camera and translate the data from that into the VEST. My intuition is that it wouldn’t end up feeling like vision – it would be more like a sixth sense, like “I can feel someone coming up behind me”.

If you could choose only one extra sense to have, what would it be?
That’s an interesting question. Right now, everything about our society is engineered around the senses that we currently have. If I were suddenly able to have ultrasonic hearing, I would hear animal calls that no one else could hear. As a nature lover that would be amazing, but I don’t know if it would be lonely in that extrasensory space if no other human joins me there.

I’d also like to explore whether the VEST can allow us to better connect with other people. Perhaps if my wife and I both wore a VEST, and used it to somehow experience each other’s emotions, that might bring us to a new level of closeness. Or perhaps it would be detrimental [laughs] – we just don’t know until we try.

When can I buy one of these things?
In about nine months. We’re working with engineers and fashion designers, and making everything open source so that anyone can input whatever data stream they like. We anticipate that the world’s hive mind will come up with some great ways to use it.

This article appeared in print under the headline “Try my extrasensory jacket on for size”