WO2018046349A1

WO2018046349A1 - Head-mounted display for interaction with a user

Info

Publication number: WO2018046349A1
Application number: PCT/EP2017/071627
Authority: WO
Inventors: Frank Dietrich; Olaf Dressel; Frank Fritze; Andreas Daniel Hartl
Original assignee: Bundesdruckerei Gmbh
Priority date: 2016-09-07
Filing date: 2017-08-29
Publication date: 2018-03-15
Also published as: EP3510471A1; DE102016116774A1

Abstract

The invention relates to a head-mounted display (100) for interaction with a user, where the head-mounted display (100) can be worn by the user. The head-mounted display (100) comprises a camera (101) which is designed to record a video sequence with an object surface and an input element of the user. The head-mounted display (100) also comprises a display device (103) which is designed to display an operating field with a plurality of operating elements, the operating field being superimposed on the object surface. The head-mounted display (100) further comprises a processor (105) which is designed to record a position of the input element in relation to the object surface on the basis of the video sequence, and to associate an operating element of the displayed operating field with the recorded position, in order to detect an actuation of the operating element.

Description

Data glasses for interacting with a user

The present invention relates to the field of human-machine interaction, in particular the interaction between a user and data glasses. The interaction between a user and data glasses is becoming increasingly important for Augmented Reality (AR) applications.

An interaction of a user with a data glasses is currently usually by means of a control by predefined gestures or buttons on the data glasses and feedback to the user. Furthermore, integrated sensors, such as motion sensors, may be used to interact with the user.

The predefined gestures or the function of the keys usually have to be learned in advance by the user. Also, the interaction using integrated sensors, such as motion sensors, requires training of the user to perform some interaction with the data goggles.

Especially when entering strings by the user, these aspects prove to be a challenge for the interaction with the data glasses. It is therefore the object of the present invention to provide an efficient concept for

To create interaction of a user with a data glasses.

This object is solved by the features of the independent claims. Advantageous forms of further development are the subject of the dependent claims, the

Description as well as the figures.

The invention is based on the realization that the above object can be achieved by a data goggles, which can recognize an object surface and an input element of the user, and can superimpose a control panel with a plurality of control elements over the object surface. By means of its input element, the user can actuate individual control elements of the control panel on the object surface and interact intuitively with the data glasses. The object surface may be, for example, a body part of the user, in particular a palm of the user, or a surface of a machine or a tool. The input element of the user may be, for example, a user's finger or a stylus of the user. This ensures that the user can interact efficiently with the data glasses. In particular, the user can intuitively enter a string, thereby

For example, an efficient authentication of the user is possible. The

For example, authentication of the user may be via a server

Communication network are signaled.

According to a first aspect, the invention relates to data glasses for interacting with a user, wherein the data glasses are portable by the user. The data glasses comprise a camera, which is designed to capture a video sequence with an object surface and an input element of the user. The data glasses further comprise a display device, which is designed to display a control panel with a plurality of control elements, wherein the control panel is superimposed on the object surface. The data glasses also include a processor, which is designed to detect a position of the input element relative to the object surface on the basis of the video sequence, and to associate the detected position with an operating element of the displayed control panel in order to detect an actuation of the operating element. This provides the advantage that an efficient concept for the interaction of a user with data glasses can be realized. The object surface may be a surface of any object, wherein the

Object surface predetermined properties, such as geometric dimensions, contours or textures may have. The object surface may, for example, a body part of the user, in particular a palm of the user, or a

Surface of a machine or a tool. The object surface may be planar or curved. The input element of the user may be, for example, a user's finger or a stylus of the user. The object surface as well as the

Input element can be detected for example by means of pattern recognition.

The control panel may be, for example, a keypad, wherein the plurality of controls may be formed by a plurality of keys. The plurality of operating elements may further comprise slider elements or menu structures which can be superimposed on the object surface or projected onto the object surface. The overlay can be done in the form of augmentation. According to one embodiment, the processor is formed on the basis of

Video sequence to capture a further position of the input element relative to the object surface, the detected further position, the control of the displayed

Associate control panel, and to detect the operation of the control element on the basis of the detected position and the detected further position. The processor can therefore detect at least one position of the input element relative to the object surface on the basis of the video sequence. This achieves the advantage that the

Actuation of the control element can be detected efficiently.

According to one embodiment, the processor is configured to press the

Operating element to detect as actuation of the control element, or is the

Processor configured to detect a retention of the input element over the operating element during a predetermined period of time as actuation of the operating element. As a result, the advantage is achieved that the actuation of the operating element can be detected in different ways. The predetermined period of time may be, for example, 0.5 second, 1 second, 2 seconds or 5 seconds.

According to one embodiment, each control element of the control panel is assigned a character, in particular a digit, wherein the display device is designed to display a character, in particular a digit, which is assigned to the actuated operating element of the control panel. This achieves the advantage that the user can efficiently verify the operated control element.

According to one embodiment, the data goggle further comprises an actuator, wherein the actuator is configured to mechanically vibrate in response to detecting the operation of the control element. As a result, the advantage is achieved that the user a successful operation of the control element can be displayed efficiently.

According to one embodiment, the data goggles further comprise a loudspeaker, wherein the loudspeaker is configured to emit an acoustic signal in response to detecting the actuation of the control element. As a result, the advantage is achieved that the user a successful operation of the control element can be displayed efficiently.

According to one embodiment, the processor is configured to detect an actuation of a further control element of the control panel, wherein the actuated Operating element and the actuated further control element actuated

Form operating sequence, and wherein the processor is formed, the actuated

Compare control element sequence with a pre-stored control sequence.

This has the advantage that, for example, strings can be entered efficiently by the user. The detection of the operation of the other

Control element can be done in the same way as the described detection of the operation of the control element. The operated control sequence may be an actuated key sequence and the prestored control sequence may be a pre-stored key sequence. According to one embodiment, the display device is formed, a position of the operating element and a further position of the further operating element in the

Control panel randomly to determine, to display the control element at the random position, and to display the other control element at the other randomly determined position. It is thereby achieved that an arrangement and / or a layout of the operating elements can be varied. As a result, for example, measures can be taken to unwanted spying on entered

To impede control sequences by third parties. The position of the controls can be determined at random for each operation, i. For example, no standardized number pad is used as the control panel. Furthermore, the arrangement of the control panel can be varied within the object surface.

According to one embodiment, the processor is configured to generate an authentication signal if the actuated control sequence of the prestored

Control sequence corresponds, wherein the authentication signal indicates an authentication of the user. As a result, the advantage is achieved that an authentication of the user can be realized by means of the interaction of the user with the data glasses.

According to one embodiment, the data glasses further comprise a

Communication interface, wherein the communication interface is adapted to communicate with a server via a communication network, and wherein the

Communication interface is designed to send the authentication signal via the communication network to the server. This provides the advantage that the authentication of the user to the server can be signaled via the communication network. According to one embodiment, the processor is configured to capture the object area and the input element of the user in the video sequence, or the processor is configured to track the object area and the input element of the user in the video sequence. As a result, the advantage is achieved that the position and / or the further position of the input element relative to the object surface can be detected efficiently.

According to one embodiment, the processor is configured to generate a three-dimensional environment model based on the video sequence, to determine a plurality of reference object areas based on the environment model, and to select the object area from the plurality of reference object areas. As a result, the advantage is achieved that an automatic determination of a suitable object surface for interaction with the user can take place. The user can choose the

Change object surface, for example, the control panel arbitrarily in another

Place object surface or move the control panel during operation. The generation of the environment model can be done, for example, using a Simultaneous Localization and Mapping (SLAM) approach at runtime.

According to one embodiment, the display device is designed to display a pointer element, wherein the pointer element represents the position of the input element relative to the object surface. This achieves the advantage that the user can position the input element efficiently relative to the object surface. The

Pointer element can be realized for example in the form of a mouse pointer. The pointer element can be displayed within the control panel with the plurality of controls, which is superimposed on the object surface or which on the

Object surface is projected.

According to one embodiment, the control panel of the object surface is transparent, in particular semi-transparent, superimposed. As a result, the advantage is achieved that the object surface remains visible to the user and an actuation of the operating element can be easily realized by the user.

According to one embodiment, the object surface is a body part of the user, in particular a palm of the user or an arm surface of the user. This provides the advantage that the interaction of the user with the data glasses can be particularly intuitive. According to one embodiment, the input element is a user's finger or a stylus of the user. This provides the advantage that the interaction of the user with the data glasses can be particularly intuitive. The stylus can be held in one hand by the user. According to one embodiment, the object surface is a palm of the user, wherein the input element is a finger of the user, and wherein the palm and the finger are associated with different hands of the user. Thereby, the advantage is achieved that an operation of the operating element can be performed efficiently. In a second aspect, the invention relates to a method of interacting with a user using smart glasses. The data glasses comprise a camera, a display device and a processor, the data glasses being portable by the user. The method comprises capturing a video sequence with a

Object surface and an input element of the user through the camera, displaying a control panel with a plurality of controls through the

A display device, wherein the control panel is superimposed on the object surface, detecting based on the video sequence a position of the input member relative to the object surface by the processor, and assigning the detected position to a control of the displayed control panel by the processor to an actuation of the control element to detect. This provides the advantage that an efficient concept for the interaction of a user with data glasses can be realized.

The procedure can be carried out by the data glasses. Further features of the method result directly from the features and / or functionality of the data glasses.

According to a third aspect, the invention relates to a computer program with a program code for carrying out the method according to the second aspect of the invention

Invention. This provides the advantage that the process can be carried out efficiently.

The data glasses, in particular the camera, the display device and / or the

Processor may be programmable to execute the program code or parts of the program code. The invention can be implemented in hardware and / or software.

Further embodiments will be explained with reference to the accompanying figures. Show it:

Fig. 1 is a schematic diagram of a data glasses for interacting with a

users; and

Fig. 2 is a schematic diagram of a method for interacting with a

Users using data glasses.

1 shows a schematic diagram of data glasses 100 for interacting with a user, where the data glasses 100 are portable by the user. The data glasses 100 comprise a camera 101, which is designed to be a video sequence with a

Object surface and an input element of the user to capture. The data glasses 100 further comprise a display device 103, which is designed to display a control panel with a plurality of controls, wherein the control panel is superimposed on the object surface. The data glasses 100 also include a processor 105, which is designed to detect a position of the input element relative to the object surface on the basis of the video sequence, and to associate the detected position with an operating element of the displayed control panel in order to detect an actuation of the operating element.

FIG. 2 shows a schematic diagram of a method 200 for interacting with a user using smart glasses. The data glasses comprise a camera, a display device and a processor, the data glasses being portable by the user. The method 200 comprises detecting 201 a video sequence with an object surface and an input element of the user by the camera, a display 203 of a control panel with a plurality of control elements through the

A display device, wherein the control panel is superimposed on the object surface, detecting 205 based on the video sequence a position of the input member relative to the object surface by the processor, and assigning the detected position 207 to a control of the displayed control panel by the processor to an actuation of the Detecting control element. The data glasses 100 and the method 200 for interaction with the user extend the possibilities of interaction with the data glasses 100, since the interactions can be solved independently of the mechanical structure of the data glasses 100. In particular, touch-sensitive surfaces on the data glasses 100 can be dispensed with. The approach is based on tracking the object surface and the input element of the user as well as the use of augmented reality (AR).

The approach to user interaction with the user begins with a detection or recognition of the previously determined object area by the camera 101 of the data glasses 100, for example by means of a tracking of the object surface. The object area may be, for example, the right-hand user's left palm, or the left-hand user's right palm.

Subsequently, a display or superimposition of operating elements by the display device 103 of the data glasses 100 by means of augmentation. For example, a control panel with a plurality of controls, such as buttons, for displaying a control sequence, such as a PIN, are displayed.

The user then moves an input element, for example a finger of his other hand, into the initially recognized object surface. The movement of the input element is also detected by the camera 101 of the data glasses 100 or

tracked and given by the display device 103, such as a display, the data glasses 100 feedback on the current position. The feedback can be made similar to a mouse pointer on a computer. If a desired position has been reached by the input element, a pressure on the virtual control panel can be triggered to actuate a control element. This pressure can be triggered, for example, by staying at the position for a longer time or by detecting a movement of the input element into the depth.

The data glasses 100 and the method 200 can be used, for example, for non-contact operation of machines in production environments.

All of the features explained and shown in connection with individual embodiments of the invention may be provided in different combinations in the article according to the invention in order to simultaneously realize their advantageous effects. LIST OF REFERENCE NUMBERS

100 data glasses

101 camera

103 Display device

105 processor

200 methods for interacting with a user

201 Capture a video sequence

203 Showing a Control Panel

205 detecting a position of the input element with respect to the object surface

207 Assigning the detected position

Claims

1 . Data goggles (100) for interacting with a user, the goggles (100) being portable by the user, comprising: a camera (101) adapted to capture a video sequence including an object surface and an input member of the user; a display device (103) which is designed to display a control panel with a plurality of control elements, wherein the control panel is superimposed on the object surface; and a processor (105) configured to detect, based on the video sequence, a position of the input element with respect to the object surface, and to associate the detected position with an operating element of the displayed control field

To detect actuation of the control element.

2. data goggles (100) according to claim 1, wherein the processor (105) is adapted to detect based on the video sequence, a further position of the input element relative to the object surface to assign the detected further position the control element of the displayed panel, and the operation of the Operating element based on the detected position and the detected further position to detect.

Data goggles (100) according to claim 2, wherein the processor (105) is adapted to detect a pressing of the operating element as actuation of the operating element, or wherein the processor (105) is designed to maintain the input element over the operating element during a predetermined time Duration as actuation of the

Detecting control element.

4. Data glasses (100) according to any one of claims 2 or 3, wherein each control element of the control panel is associated with a character, in particular a digit, and wherein the

Display device (103) is designed to display a character, in particular a digit, which is associated with the actuated operating element of the control panel.

Data goggles (100) according to one of claims 2 to 4, wherein the data goggles (100) further comprise an actuator, and wherein the actuator is adapted to mechanically vibrate in response to detecting the operation of the control element.

The data goggles (100) of any one of claims 2 to 5, wherein the data goggles (100) further comprise a speaker, and wherein the speaker is configured to emit an audible signal in response to detecting actuation of the control.

7. Data goggles (100) according to any one of claims 2 to 6, wherein the processor (105) is adapted to detect an actuation of another control element of the control panel, wherein the operated control element and the actuated further control element form an actuated operating sequence, and wherein the Processor (105) is adapted to compare the operated control sequence with a pre-stored control element sequence.

The data goggles (100) of claim 7, wherein the processor (105) is configured to generate an authentication signal if the operated control sequence corresponds to the pre-stored control sequence, and wherein the authentication signal indicates authentication of the user.

The data glasses (100) of claim 8, wherein the data glasses (100) further comprise a communication interface, wherein the communication interface is configured to communicate with a server via a communication network, and wherein the communication interface is adapted to send the authentication signal to the computer via the communication network Send server.

Data goggles (100) according to any one of the preceding claims, wherein the processor (105) is adapted to detect the object surface and the input element of the user in the video sequence, or wherein the processor (105) is formed

Trace the object area and the input element of the user in the video sequence.

1 1. Data glasses (100) according to one of the preceding claims, wherein the

Display device (103) is adapted to display a pointer element, and wherein the pointer element, the position of the input element relative to the object surface

represents.

12. Data glasses (100) according to any one of the preceding claims, wherein the control panel of the object surface is transparent, in particular semi-transparent, superimposed.

The smart phone (100) of any one of the preceding claims, wherein the object surface is a palm of the user, the input element being a finger of the user, and wherein the palm and the finger are associated with different hands of the user.

14. Method (200) for interacting with a user using data glasses (100), wherein the data glasses (100) comprise a camera (101), a display device (103) and a processor (105), the data glasses (100) is portable by the user, with:

Capturing (201) a video sequence having an object surface and an input element of the user by the camera (101);

Displaying (203) a control panel having a plurality of controls by the display (103), wherein the control panel is superimposed on the object surface; Detecting (205) on the basis of the video sequence a position of the input element with respect to the object surface by the processor (105); and

Associating (207) the detected position with a control of the displayed one

Control panel by the processor (105) to detect an actuation of the control element.

15. Computer program with a program code for carrying out the method (200) according to claim 14.