Magnetic joystick and VR
November 24, 2016, 4 min to read
But that’s enough introducing, what matters today are magnets.
Magnets and VR? What’s the link?
Not so obvious. One must be aware of all the sensors contained inside a smartphone.
Actually a smartphone is filled with sensors. Some are obvious such as the touch screen and the microphone, but there are more, even though the every day user may not be aware of it: an accelerometer coupled to a gyroscope to get the orientation of the device, a light sensor to adapt the luminosity of the screen… And also a magnetometer, which allows to orient the device relatively to the world, especially in order to make the compass work.
Most of these sensors are useful in order to make the Cardboard apps work. The accelerometer and the gyroscope allow to detect the user’s head orientation, hence move the image on the screen according to where the user looks.
The magnetometer also mattered in the first versions of the Cardboard. Indeed, in order to interact with a virtual world, the user needs to be able to tell to the app he wants to select something in a menu, in an environment… A button just does that. Google chose to make this button in an original manner. After several experiments including whistling or teeth-chattering, they chose to use magnets.
Two of these are used: one fixed inside the Cardboard, one vertically movable on the outside. When pulled down, it automatically comes back up to its initial position, and the variation in the magnetic field generated by the move of the magnet is detected thanks to the magnetometer by the app which understands the button has been used.
What could we do to enhance this?
One of the engineers that worked on the project said in a post explaining in more details how the magnets are used that further implementations could detect double-clicks, or could even modify the physical constraints to move the magnet in a more elaborate way than vertical, like a joystick.
It could be pretty useful indeed to be able to add a joystick to an object that allows to get into virtual worlds, even just in order to move inside of them! Furthermore, keeping the magnet to allow this interaction keeps the initial philosophy of the project that is to offer virtual reality with basic materials. Adding any peripheral device would be a nonsense.
However, regarding the implementation, it is quite more complex than detecting a significant variation of a magnetic field. One must find a way to transform a magnetic field into a position on the plan of the Cardboard.
From a magnetic field to a position on a plan
The solution used to make this transformation is for sure not the only one that works, but at least it is quite simple.
A prototype with two magnets has been recreated on a Cardboard that doesn’t use magnets at first (the most recent ones work this way, using buttons that push on the touch screen). One is fixed on the inside, the other completely free on the outside. The magnetometer can give us a raw magnetic field with three components (x,y,z). By observing the evolution of these fields while moving the magnet along the surface of the Cardboard, the magnetic field evolves (no surprise for those who did a bit of physics) in a non linear way. Hence, the transformation from the magnetic field to its position isn’t goint to be trivial. Nevertheless, the way the components are evolving can be interpolated in order to approximately represent the surface representing their evolution.
Bicubic interpolation was used to do the job. It is a surface interpolating method, sometimes used for image redimensioning, which requires at least an input set of 9 points to recreate a surface. The magnet is placed on 9 points of the plan, then the interpolation is triggered for each component of the magnet field. Once the 3 surfaces are obtained, we can look for the place these components could be on the surfaces (a nearest neighbor search), and then we have the position.
To show the potential of this method, a demo app has been created with the help of SceneKit, the high-level 3D scene editor from Apple. Some effects were added in order to have a proper distortion effect on the screen, but fortunately some projets on Github were already showing the way to do it (the author of the article didn’t have the chance before to deal with OpenGL).
The result is two simple applications: a universe in which you can fly, another one with an infinite slope on a bobsleigh.
These applications are only an example of what can be achieved with these magnets, some other applications could be fast forwarding/backwarding on 360 videos, changing apps as they are launched in virtual reality, and surely many others.
The source code is available here, so that you can reproduce the experience at home! You will need an iPhone, a Cardboard working without magnets, a piece of checkered paper, two small magnets and scotch tape #DIY.
Feel free to reach us on Twitter if you have any question on the implementation or any idea of improvement!