JP2012003690A - User interface - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a user interface that enables three-dimensional gesture input and touch input on a display surface.SOLUTION: A user interface of this invention comprises: a transparent display; an image output means for outputting an image on the transparent display; a distance image sensor that is installed behind the transparent display and obtains a distance image through the transparent display; and an analysis means for analyzing the distance image obtained by the distance image sensor to detect input operation of a user. The analysis means detects the input operation by gesture recognition when an object in the distance image is away from the display, and detects pointing operation as the input operation when the object touches the display substantially.

Description

  The present invention relates to a user interface device using a distance image sensor.

  As a technique for receiving an input from a user without using an input device such as a keyboard or a mouse, a user interface device using a touch panel or a distance image sensor is known.

  In a user interface using a touch panel, an operation is performed by a user touching the display panel with a finger or a stylus. As a touch panel method, a resistive film method, a capacitance method, an optical sensor method, an infrared method, a surface acoustic wave method (ultrasonic method), an electromagnetic induction method, and the like have been proposed. The resistive film method is the most popular method, but it cannot detect input until it touches the panel surface. On the other hand, in the capacitance method and the optical sensor method, proximity detection can be performed before an object (finger or stylus) touches the panel surface. However, the detectable distance is about several centimeters from the panel surface, and is not suitable for gesture input with a depth (three-dimensional input).

  In a user interface that uses a distance image sensor (also called a three-dimensional camera, a depth sensor, or the like), a user moves a hand or a body in front of a display, and an operation is performed by a gesture. The distance image sensor can detect a distance from several tens of centimeters to several meters. An input operation can be performed by a user operation in the air. However, in a situation where an icon on the screen is selected, a physical feedback can be obtained by directly touching the display, so that a highly accurate input can be performed.

US Patent Application Publication No. 2006/0026521 Special table 2009-528514 JP 2010-015553 A

  It is an object of the present invention to provide a user interface device that combines the advantages of both three-dimensional gesture input and input in a two-dimensional plane (display plane).

  In order to achieve the above object, the present invention employs a transparent display as a display, and detects a user input operation from a distance image obtained from behind the transparent display through the transparent display. At this time, if an object such as a user's finger or stylus is far away from the transparent display, it is interpreted as a gesture input, and if it is close to the transparent display, it is interpreted as a touch-type input to the panel.

More specifically, the user interface device according to the present invention includes a transparent display,
Image output means for outputting an image to the transparent display;
A distance image sensor installed on the back surface of the transparent display to obtain a distance image through the transparent display;
Analyzing means for analyzing a distance image acquired by the distance image sensor to detect a user input operation;
With
The analysis means includes
If the object in the distance image is more than a predetermined distance from the transparent display, the input operation is detected by recognizing the gesture of the object in the distance image,
When the object in the distance image is within the predetermined distance from the transparent display, a pointing operation for the position of the transparent display corresponding to the object in the distance image is detected as a user input operation.

  Here, it is preferable that the predetermined distance is a distance that can be regarded as being substantially in contact with the transparent display. That is, it is preferable to regard the input operation in contact with the display not only when the object is completely touching the transparent display but also when being slightly separated. This predetermined distance may be determined according to the size of the transparent display, for example.

  By adopting such a configuration, the user interface device according to the present invention can realize a three-dimensional gesture input and a two-dimensional plane input with only a distance image sensor without using a touch panel.

  Further, in the present invention, when the object in the distance image is separated from the transparent display by a predetermined distance and the shape of the object is a predetermined shape, a user interface element for supporting a pointing operation Is preferably displayed on a transparent display.

  Here, the predetermined shape is a shape for performing a pointing operation, and may be a sharp shape such as a palm shape or a stylus shape with only one finger extended. In addition, the display of the user interface element for supporting the pointing operation is a display of an element that can be selected by the user such as a menu element, or an enlarged display near the pointing position.

  As described above, when the user performs the pointing operation, the user interface element for the pointing operation is displayed before the user touches the transparent display. Therefore, a highly convenient user interface can be provided.

  In the present invention, it is preferable that image output to the transparent display and distance image acquisition by the distance image sensor are performed alternately.

  In this way, the distance image can be acquired without being affected by the image display. Further, if the switching between image output and distance image acquisition is performed at a sufficiently high speed, the distance image can be acquired without impairing the display image quality.

  The present invention can be understood as a user interface device including at least a part of the above means. Moreover, it can also be grasped as an information input method for performing these processes, and a program for realizing these methods. Each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

  According to the present invention, three-dimensional gesture input and two-dimensional plane input can be realized without using a touch panel.

It is a figure which shows the outline | summary of the user interface apparatus which concerns on this embodiment. It is a figure which shows the functional block of the user interface apparatus which concerns on this embodiment. It is a flowchart which shows the process of the user interface which concerns on this embodiment. It is a flowchart which shows the detail of the distance image analysis process in the process of the user interface which concerns on this embodiment. It is a figure explaining a gesture input area and a contact input area. It is a figure explaining the operation | movement in the case of performing the gesture input corresponding to scroll operation in a gesture input area | region. It is a figure explaining operation | movement in the case of making a fingertip contact a transparent display for the gesture input of pointing operation.

  An overview of the user interface device according to the present embodiment will be described with reference to FIGS. In the present embodiment, the image output from the information processing apparatus 1 is displayed on the transparent display 2. The distance image sensor 3 is installed on the back side of the transparent display 2, and the distance image is acquired through the transparent display 2.

  The information processing apparatus 1 is a computer including a CPU, a main storage device, an auxiliary storage device, a network device, and the like. The information processing apparatus 1 outputs an image to the transparent display 2. The distance image sensor 3 is used as an input device. Of course, in addition to the distance image sensor, a keyboard, a mouse, a controller (wired / wireless), or the like may be provided as an input device. The information processing apparatus 1 implements each function described in this specification by causing the CPU to execute a program developed on a memory.

  Although the information processing apparatus 1, the transparent display 2, and the distance image sensor 3 have been described as separate bodies, the transparent display 2 and the distance image sensor 3 or the information processing apparatus 1, the transparent display 2, and the distance image sensor 3 are separated. It may be configured as a single unit.

  The transparent display 2 is a display device capable of switching between a transparent state and a digital image display state. As the transparent display, for example, an EL display using an EL element, a liquid crystal display, or the like can be used. In this embodiment, a transparent organic EL display is used. This organic EL display is self-luminous, has low power, high brightness, high contrast, and a high viewing angle, so that the displayed image is easy to see, and unlike the liquid crystal display, it does not require a backlight and can be reduced in thickness. is there. An image output from the information processing apparatus 1 is displayed on the transparent display 2.

  The distance image sensor 3 irradiates a light pulse such as infrared light, and detects the reflected light that hits the object and its flight time, thereby measuring the shape and distance of the object. The distance image acquired by the distance image sensor 3 is input to the information processing apparatus 1.

  The image output unit 12 of the information processing apparatus 1 outputs the image data generated by the control unit 11 to the transparent display 2. The distance image input unit 13 acquires a distance image from the distance image sensor 3. The distance image analysis unit 14 analyzes a distance image and detects a user input operation to the information processing apparatus 1. In accordance with the input operation via the distance image, the control unit 11 executes the program.

The control unit 11 also controls the timing of image display on the transparent display 2 and the distance image acquisition by the distance image sensor 3, and the light for image display and the light pulse of the distance image sensor alternate in time division. Flash on. At this time, by making the blinking time sufficiently short so as not to be perceived by human eyes, it is possible to irradiate a light pulse for distance sensing from the back of the display without impairing the display image.

  FIG. 3 shows a flowchart of processing performed by the user interface device according to the present embodiment. The user interface device repeatedly executes the process shown in FIG. Here, it is assumed that the transparent display 2 displays an image of 30 frames per second. First, output of an image for one frame is started (S1), and image output is stopped after 1/60 seconds (S2). In a state where no image is displayed on the transparent display 2, a distance measurement light pulse is emitted from the distance image sensor 3 to acquire a distance image (S3). And a distance image is analyzed and the input from a user is acquired (S4). By repeating these processes, it is possible to display an image and accept an input from the user at the same time. Here, the distance image analysis process (S4) has been described as being performed in a state where no image is displayed on the transparent display 2. However, it is sufficient that the distance image is acquired in an image non-display state. This analysis may be performed during image display.

  Since the distance image may be acquired when the display image is not displayed, it is not necessary to perform the distance image measurement every time one frame is displayed, and the display display and the distance image measurement time need not be the same. For example, an operation may be employed in which the distance image is acquired once after displaying several frames on the transparent display 2 continuously.

  Next, details of the distance image analysis processing in step S4 will be described with reference to FIGS. FIG. 4 is a flowchart showing detailed processing of the distance image analysis processing. In the distance image analysis process, first, an object (such as a finger or a stylus) approaching the display is extracted from the acquired distance image (S41). Then, based on the distance from the object, it is determined whether the object is located within the gesture input area or the contact input area (S42). As shown in FIG. 5, the contact input area is an area whose distance from the transparent display 2 is d1 or less. Here, d1 is a distance that can be considered that the object is substantially in contact with the transparent display 2. The gesture input area is an area whose distance from the transparent display 2 is greater than d1 and within d2. The distances d1 and d2 are determined according to user interface requirements, the size of the transparent display, and the like. When the distance between the distance image sensor 3 and the transparent display 2 is L, if there is an object between the distances L to L + d1 in the distance image, it is determined that the object exists in the contact input region, and between the distances L + d1 to L + d2 If there is an object, it is determined that it exists in the gesture input area.

  If the object detected from the distance image is within the gesture input area, the gesture is recognized from the shape or movement of the object (S43). The relationship between the shape and movement of the object and the input operation may be determined by design as appropriate. As one of such relationships, a sharp shape such as a fingertip or a stick is associated with the pointing operation. As a result of the gesture recognition in step S43, if it is determined that the shape or movement of the object is other than the pointing operation, the gesture of the object is processed as an input (S44). For example, a gesture for moving an open hand is processed as a scroll operation, or a zoom-in / zoom-out operation is performed by bringing both hands closer to or away from each other. Conversely, if it is determined as a result of gesture recognition in step S43 that the shape or motion of the object represents a pointing operation, a user interface element for supporting the pointing operation is displayed on the transparent display 2 (S45). . For example, it is conceivable to display selectable menu parts, display a cursor, or display an enlarged portion of the cursor.

  If it is determined in step S42 that the object is in the contact input area, the position (pointing position) in the transparent display 2 to which the object points is calculated (S46) and processed as a pointing operation for that position. (S47).

  A specific example of the user input operation will be described with reference to FIGS.

  As shown in FIG. 6, when the user moves the open hand at a position away from the transparent display (in the gesture input area), the gesture is determined to be a gesture other than the pointing operation in the gesture input area. Then, a process corresponding to the gesture of moving the open hand, here, a screen scroll process is executed.

  On the other hand, as shown in FIG. 7A, when the user brings his fingertip close to the transparent display at a position away from the transparent display (in the gesture input area), it is determined that the gesture is for a pointing operation in the gesture input area. The Then, a menu screen is displayed on the transparent display for pointing operation support. As shown in FIG. 7B, when the user touches the fingertip closer to the transparent display (more precisely, moves it into the contact area), the pointing operation for the fingertip position, that is, the menu of the position touched by the fingertip is displayed. The process of selecting is executed.

  According to the user interface device according to the present embodiment, both the three-dimensional gesture input and the touch input to the display can be realized. Conventionally, both a distance image sensor for 3D gesture input and a touch panel for contact input are required. However, according to the present embodiment, both types of input can be realized without using a touch panel. Furthermore, since the distance image is acquired from the back surface of the transparent display, there is less deviation between the position of the object in the distance image and the position in the display that the user intends to operate, which is highly convenient for the user. Input is possible.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Transparent display 3 Distance image sensor 11 Control part 12 Image output part 13 Distance image input part 14 Distance image analysis part

Claims (4)

A transparent display,
Image output means for outputting an image to the transparent display;
A distance image sensor installed on the back surface of the transparent display to obtain a distance image through the transparent display;
Analyzing means for analyzing a distance image acquired by the distance image sensor to detect a user input operation;
With
The analysis means includes
If the object in the distance image is more than a predetermined distance from the transparent display, the input operation is detected by recognizing the gesture of the object in the distance image,
When the object in the distance image is within the predetermined distance from the transparent display, a pointing operation for the position of the transparent display corresponding to the object in the distance image is detected as an input operation.
User interface device. The predetermined distance is a distance that can be regarded as being substantially in contact with the transparent display.
The user interface device according to claim 1. When the object in the distance image is farther than the predetermined distance from the transparent display and the shape of the object is a predetermined shape, the analyzing means is a user for assisting a pointing operation. Displaying interface elements on the transparent display;
The user interface device according to claim 1 or 2. The image output to the transparent display and the distance image acquisition by the distance image sensor are performed alternately.
The user interface device according to claim 1.

Images