Apple’s recently published patent application talks about a way to control the device, where any surfaces visible through glasses can be used as interactive elements. A patent application entitled ” Method and device for detecting touch between two objects ” was filed back in 2016, but was published only this week. It argues that overlaying controls on objects in the real world will be necessary to interact with smart glasses.

The company focuses on the fact that as the displays of AR devices become smaller and closer to the user’s eyes, appearing in the form of glasses, the usual touch control becomes inconvenient. In some cases, the use of conventional capacitive touch screens is highly impractical or even impossible. For example, AR glasses, which you have to take off and touch the lens with your finger every time to interact, cannot be called a device that makes life easier.

Apple sees a way out of this situation as follows. The company plans to implement a function in which controls will be projected onto objects of the surrounding world, and special sensors built into the glasses will read the user’s touches to them. Thus, the user’s eyes will see a virtual object, in order to interact with which the user will need to touch what is in this place in the real world.

Unfortunately, the implementation of this function in practice is much more complicated than it might seem at first glance. At this stage, such interaction is possible only with a real device that resembles an iPad, the display of which does not display anything, but the sensor reads touches. Apple is working to provide users with the ability to interact with absolutely any surface.

The patent says that the most realistic way to achieve a goal is to equip an object or part of the human body (for example, a fingertip) with a sensor capable of sensing touch. However, Apple wants to implement technology that doesn’t require the human body to be embedded in a circuit. The company plans to provide a camera built into the glasses with the ability to track the position of the user’s fingers and record touches to the surface onto which the system interface is projected.

The problem is that when displaying an array of adjacent buttons, trying to reach one of them, the user will press all the keys located along the path. This can be solved by using very large buttons in the interface or a small number of them, but this can adversely affect ergonomics.

Apple plans to use heat to solve the problem. The patent says that if two objects with different temperatures come into contact, the area where the touch occurs changes its temperature and then slowly returns to its original value. Thus, Apple Glass will offer interaction with projected controls by capturing temperature changes. The appendix to the application lists various methods for detecting thermal changes, including thermal imaging, but mainly it is about the accuracy that can be achieved. The end device must be able to distinguish the user’s intentional touch from the touch of foreign objects and accurately localize them.

It is worth noting that this technology is not intended exclusively for Apple Glass, but is part of Apple’s wider AR work. This demonstrates the company’s far-sightedness since the patent application was filed four years ago when augmented reality technologies were just beginning.