Virtual Reality Keyboard for children with ASD
This story is a part of the Rewire.Education cycle. In order to understand what it’s all about better, you can read the introduction here.
Virtual Reality Keyboard
All teacher’s guides, exercises and games teachers requested us to develop are accessible to everyone at Rewire.Education.
What is it: A giant keyboard, hanging in a distraction-free virtual space. There is nothing else in the virtual space except for the player and the keyboard.
How does it work: A teacher is monitoring the process using a tablet or a smartphone. A player wearing an Oculus Go headset is typing a text using a pointer or the eye-tracking method by pressing big, realistic, and voluminous keyboard keys.
What we wanted to achieve: We wanted to find out whether the functionality of the subjects typing on the physical keyboard using the facilitation method would improve in a comfortable virtual environment.
How did it go: Project initiator — Olga Nikolayeva, Head of Development of voluntary activity department
When I stumbled upon your company, I felt that there was a potential that we can benefit from. In ‘Our Sunny World’ center, we are working on finding different ways to approach people with ASD to help them fix the difficulties in perception and in functionality. These people have two problems, to put it extremely plain and simple. Firstly, their perception is different, and secondly, there are things they can or cannot do. Unfortunately, there are no universal technologies for correcting these disorders yet. In our center, we are using an integrated approach: we use the most optimal combination of different methods to help people with ASD solve this issue.
ASD is a range of disorders that captures a vast spectrum of special developmental needs, and as a result, this means that individuals with functional, poorly distinguishable from neurotypical persons people can get included into the spectrum. There are many people with ASD who have not yet been diagnosed since they do not have serious problems: they can socialize, live a full life, but at the same time, they face difficulties, above all, in the social sense.
On the other side of the ASD spectrum, there are low-functioning people: both children and adults, who are usually not able to produce voluntary speech. They have special voluntary activity developmental needs. First, they cannot consciously and purposefully do anything, they have voluntary speech and attention disorders, and it is difficult for them to carry out targeted actions. Secondly, they cannot stop unconscious, most often repeated or even unpleasant actions or repeated words — so-called echolalia. This speech, however, does not carry communicative functions.
Low-functioning people special developmental needs are a topic I devoted the last decade of my life to. I grew very interested in this phenomenon because there is quite a lot of research material regarding the ASD spectrum, but it is usually descriptive. In fact, very little is known about the reasons for such behavior and such perception. And if there is not enough information about the reasons for this, conditionally speaking, impairment, then there is not enough knowledge about how it is possible to ‘fix’ it. To fix something, you need to understand how it works. And one of the ways to understand how it works is by trying to talk to the ‘impairment’ carrier.
You can speak verbally to high-functioning people with ASD. Speaking of low-functioning people, voluntary speech is not an option for them. The idea is that people with ASD will be able to talk in written form using a facilitator. The facilitated communication method is all about facilitator supporting the hand of a person who wants to communicate. The closer to the wrist, the higher the level of support.
This method follows two goals: first, communication itself, a possibility of speaking to other people. Second, in my case, it is about finding an opportunity to teach a person to write without any help from other people. It is known that the people with ASD do not have motor dysfunctions, they are able to use their hands, legs and etc. on their own. But it is hard for them to write on their own. Since I’ve been communicating to a decent number of teens and adults for 10 years, we tried to understand why is it all like that together. What is more, there are studies of various kinds written by different authors regarding the characteristics of their affective sphere and voluntary activity … And as a result, I came up with a hypothesis that one of the reasons why it is difficult for them to write independently is the deficiency of the affective sphere, which makes difficult for them to feel that they are a subject of activity.
In order to carry out some purposeful voluntary activity, we need a subject — that is, someone who actually does, perceives, speaks, types. And people with ASD often have such a thing as … Stanley Greenstone called it ‘the fear of manifestation.’ A person is emotionally intolerable to experience a sense of independence. It is difficult to experience the feeling that I am doing something myself. Therefore, the touch of the facilitator’s hand gives the feeling that ‘I am not doing this myself’ or ‘we are doing it together’.
That’s why it makes the same impression on the audience. There are a lot of articles about facilitated communication being quackery, when the facilitator writes using the hand of a person with ASD in order to achieve their own selfish goals or because of their strange fantasies. I myself was very skeptical about this before, but after trying it, I realized that it works, it is noticeable when a person types on their own when someone touches their hand. I sought for a way to withdraw support when carrying my activities in this field.
I had only one case when we saw that the support being completely withdrawn. The process took 6 years: the boy initially wrote with a high degree of hand support near the wrist, but in the end, we managed to withdraw the support and he continued to write on his own, by hand, and did not type on the computer. In addition, at that very moment, he gained an ability to speak voluntary — before that, he spoke with echolalia, and when he began to write independently, he began to speak independently, went to another functional level. Now we, unfortunately, have lost contact, because I understand that his family and he himself are not very pleased to recall this rather difficult period.
At the moment, I have about 6 adolescents and adults with whom I am working with permanently using this technology, everyone has different degrees of support withdrawal. Before the interaction with Luden.io, one young man, Ivan, wrote without tactile support, but support could not be considered removed completely, since the presence of a facilitator was necessary, meaning that I had to sit next to him and hold my hand over the keyboard, pretending that I am pointing the keys for him. In fact … my hand is at such a distance above the keyboard that you cannot point at a specific key. This is a kind of trick that allows the child to feel that he is not quite doing it himself. At the same time, he is a young man with no intellectual impairment, all these people are cognitively comparable to their age norm. That is, he is aware of what is happening, but nevertheless, he cannot get rid of this presence of the facilitator dependence.
The remaining five people wrote with different levels of tactile support: from
holding one’s wrist to holding one’s elbow. I thought: how can virtual reality help these people solve their problems? One of my thoughts was to create a virtual keyboard in virtual reality: a person puts on a headset, sees a keyboard and they have a pointer that helps them to choose the right key using a controller. The person presses the keys and text appears over the keyboard. So, you can call it an analog of a keyboard the person sees in reality, but with one important difference — the person does not see themselves.
According to my hypothesis, this, firstly, could reduce the fear of experiencing subjectness, of one’s own existence — since you cannot see yourself with your own eyes, it may be easier for you to deceive this little deficit mechanism of yours. Secondly, there is a feeling that this is all for fun, not real at all. There is still such a special quality in people with ASD that one of the mechanisms that give them the opportunity to behave functionally is a decrease in the significance of what is happening for them. When too much attention from others is directed towards what they are doing or not doing, it is more difficult for them to accomplish the task. And if everything happens as if not seriously, this task becomes a lot simpler for them. Reduced sense of significance. And the third factor is that there is no obvious feeling that the person is being watched because they do not see the observers.
During the study, we tried to use this keyboard with our teenagers and adults. It was easy for us because they gave us feedback: they were writing, they communicated with help from our side and they could be asked about their impressions, they could answer questions. They made several suggestions: for example, to put the keys a bit farther from other keys so that it was easier for them to type what they want and make input possible not only using a hand controller, but also with head tracking tools.
At the beginning we didn’t see any special functionality: these were only chaotic keystrokes, sticky key lines and other similar actions. Later, we got very inspired by one of the first experiences of a girl who dealt with another program [one of the Rewire.Education games created under the guidance of ABA specialists from the USA], where you need to insert figures into holes of appropriate shape and color (Seguin Form Board). She has demonstrated that she is able to bear with the tasks quite well, despite the fact that in reality, she is unable to do this — she is easily distracted by literally anything.
Later on, we concentrated on the keyboard — occasionally used it, because it was not easy for people with ASD to stay in this reality. Most of them stayed in it for a few minutes and said that that’s enough, they grew tired. But judging by the feedback, everyone was encouraged, no one had negative feelings. The reactions were different: surprise, joy, hope that it would really help… One young man wrote that his ideas about reality had somewhat changed after he tried to do something in virtual reality. And he said that not only his ideas about how things work have changed — he wrote that when he comes out of virtual reality, he sees reality in a different way. A very interesting story.
At the end of the experiment with the virtual keyboard, two people managed to write something: one subject wrote the word ‘Olya’ (Olya is the name of one of the teachers in our center), the second wrote ‘Hello’. In both cases, they controlled the pointer with head movements and pushed the controller button when they selected a key. In addition, almost all the subjects found and pressed the backspace key, in various ways — they erased the gibberish that had been written before.
It is the dynamics after interacting with virtual reality that brings the greatest joy and surprise. There was a feeling that the functionality of these people outside the program increased noticeably. That is they gained more resources and opportunities to type independently. Even in the process of how I was writing with them using facilitated communication I had a feeling, physically, that they were doing it more confidently. There was a feeling that I had an opportunity to pull my hand off for a while until the child who was writing noticed this. They inertially finished writing what they began to write — until they noticed. But previously [before the VR sessions], such episodes happened to them less often.
In the end, I decided to try to withdraw the tactile support in analogy with the way I took it withdrew it with Ivan. I withdrew tactile support with him using a method I came up with a few years ago. I stop touching his hand and first point my finger at the key that he needs to press. I physically touch it, and then I stop pressing certain keys. I do not tell him what to write — I just make a gesture pointing at the keyboard area at no key specifically. At the same time, he gets the impression that someone is showing him what to write.
We had an opportunity to communicate this way only with Ivan. I also tried to interact with virtual reality with at least a few of the subjects — it did not work. And now, by the end of our interaction, tactile support has been removed from almost all of the subjects. That is, the facilitator is there, holds its hand over the keyboard, but does not touch the person who is writing.
It’s hard to tell how closely this dynamic is tied to the use of virtual reality. The only thing that leads us to this idea is that all these people have gone through the same, non-altered special education program. Nothing in their program has changed… maybe they all had something at the same time happening in their personal life, we cannot take all the factors into account. But there were no significant changes in the special education program. The only thing that happened during this period was the appearance of the virtual keyboard. Therefore, this gives us reason to assume that probably this experience has given them an opportunity to feel more independent, more functional in reality. Moreover, they themselves wrote that they also had the feeling that their independence was increasing.
My area of professional interest lies in increasing the functionality of people with ASD. If I see a product that probably contributes to the enhancement of this functionality, I naturally grow interested in using it in exceptional education work. It would be nice to continue, firstly, the fragmentary (it’s simply easier) use of this product with people who have already used it before. And, secondly, when new ones appear — and a sufficient number of them (people capable of such kind of communication) will appear, — use this product to increase their functionality.
We would like to express our gratitude to Jonathan Ravasz for carrying research regarding the convenience of keyboards in virtual reality and his Punchkeyboard, project, which served as a reference for us.
Conclusion
Olga successfully continues to use the virtual keyboard during the lessons. We hope to interview one of the students who can now print independently in the nearest future.
In addition, Olga’s second idea has already been implemented into one of our projects, but tests have not yet been conducted. We ‘ve already added an ability to remove and bring objects of any size closer in the virtual space. According to Olga’s hypothesis, this can help with the identification of objects, depending on what part of the object and at what distance the student sees them. We will be happy to share the results with you as soon as they appear.
All teacher’s guides, exercises and games teachers requested us to develop are accessible to everyone at Rewire.Education.
