Augmented Reality software tools

Programming or pinning?

The best of both worlds – that’s what many advertising slogans say when it comes to combining two aspects: emotion and innovation, naturalness and modernity, pragmatism and perfectionism. But what if two worlds collide and open up endless possibilities? Augmented Reality (AR) combines reality and virtuality. The product is the best of both worlds.

AR is an interface of real objects and virtual content. What sounds simple is really a promising technology with immense advantages. This is visible for example when learning with AR. If content is prepared with virtual or spatial simulations, the learner doesn’t have to navigate in a User Interface or search for information. His or her cognitive burden is minimized and the complete focus on the content is therefore enabled. The quality of learning is enhanced.

Customer specific knowledge transfer needs good programmers

Software Development Kits (SDKs) and Software Tools that are necessary for implementing such interfaces are already widely spread, e.g. ArCore, Apple Arkit and HoloLens Toolkit. Guidelines based on experience values and extensive studies are also available. But teaching content and expectations of companies of knowledge transfer are mostly subject-specific and individual. This is why it takes programmers who develop tailor-made solutions for each individual customer. This process is cost- and time-consuming. The programmer has to deal with the tracking and the interaction with virtual and real objects, create user interfaces and pay attention for a good user experience and easy handling.

In spite of these individual demands parallels can be found in the use cases which are about learning and training workflows. In the industrial area, for example, the animation and description of certain worksteps with audiovisual and spatial instructions is reasonable. This is the reason why TeamViewer developed tools which enable the user to generate content and manuals himself easily. He doesn’t have to have knowledge about tracking, interaction or design of the user interface and there are no programming skills needed. With TeamViewer Pins virtual content can be ‘pinned’ onto real objects. With TeamViewer Steps, instructing, learning and training step-by-step on virtual objects are made possible.

3 components + a good idea = an interactive application

In order to build an interactive app, a good idea is indispensable. It helps to know the three main components of each AR app. These are:

1. Tracking
2. Content
3. Interaction

The tracking

For AR glasses to be more than just a screen, they need a tracking system. This anchors a virtual object in space so that it maintains its position even when the viewer moves around the object.

The proprietary tracking systems of glasses can be extended by tracking-SDKs. These can usually be integrated independently of special glasses. With these additional tracking-SDKs, not only the position of the glasses and the general environment can be detected, but also special objects in the environment, such as people, tools, machines or gestures.

The HoloLens uses inside-out tracking, which is typical for AR devices. It uses the sensors built directly into the glasses (cameras, infrared cameras, light sensors and inertial measurement units) to position itself and objects in space. The glasses look for so-called “features”, i.e. “interesting” points in the room, which they reliably recognize over several frames and from several angles. Thus the glasses can determine in which relation itself stands to the environment.

Additional algorithms can access the sensors. There are usually two different optical approaches: The first approach is based on recognizing natural features such as entire objects and is followed by VisionLib and OpenCV, for instance. The disadvantage is that the computation power needed is often comparatively high and the applications prove to be less reliable. On the positive side, the environment does not need to be specially prepared.

The second approach is to place markers or QR codes (so-called fiducial markers) at prominent positions or objects. The camera can easily distinguish these from other objects. The Marker-based SDKs include Wikitude and ARToolKit. An integration into Unity has already taken place.

Tracking can be improved with additional hardware. A current research approach deals with exo-suits, such as the HoloSuit, for body tracking or additional sensors such as the LeapMotion, which specializes in tracking individual fingers. There are currently no field-proven systems in conjunction with the HoloLens, even though the initial results of the research and development teams are very promising.

Which tracking system you choose depends strongly on the use case. Sometimes the standard functions of the glasses are sufficient, if the content can be viewed independently of the environment. In more specific cases, the advantages and disadvantages of additional SDKs must be weighed up in order to achieve the best possible result.

The Content

To view holograms, 3D models can simply be imported into Unity. Most of the time, however, this is not the end of the story, because you usually don’t just want to view a model, but want to add additional content such as animations, instructions, text, images, and videos. If there are only a few additions, entering them into Unity is not a problem. If, on the other hand, you want to display several or hundreds of work instructions, animations or media content, it becomes difficult to scale the number of contents.

With the help of an editor or content creation tool, it is possible to transfer the input of content to people who do not need a deeper understanding of 3D engines, instead of having to assign it to a programmer. This means a large initial development effort.

In addition to the number of contents, it must be considered that the size of the graphic objects must be adapted to the respective glasses and their abilities. The HoloLens has a display resolution of 1268×720 px and a custom-made processor. Nevertheless, you should not overcharge your glasses with the rendering of large files. Images and videos must therefore, be compressed and the polygons of the 3D objects reduced to guarantee a smooth 60 FPS (frames per second).