JP2024005440A - Input output device, input output method, and program - Google Patents

Abstract

To provide an input output device which corrects misalignment between an operation position on an aerial operation surface and a response position of a display.SOLUTION: An input output device 1 includes: a display surface for displaying content; a detection unit 7 which detects motion of a user on the content displayed, in a position not on the display surface; an imaging unit (camera 5) which images the user; a correction unit which corrects the detection position detected by the detection unit 7 or the position of the content displayed on the display surface, on the basis of an image of the user captured by the imaging unit (camera 5); and an eye specifying unit which specifies the eyes of the user from the image captured by the imaging unit (camera 5). The correction unit corrects the detection position detected by the detection unit 7 on the basis of positions of the eyes of the user in the captured image specified by the eye specifying unit or a distance to the eyes of the user.SELECTED DRAWING: Figure 2

Description

The present invention relates to an input/output device, an input/output method, and a program.

For example, Patent Document 1 discloses a display means of a liquid crystal display panel, an input means of a transparent touch sensor panel, and a position information indicating means using a pen or the like, in which a transparent touch sensor panel is laminated on the display surface of the liquid crystal display panel. The information processing device inputs information by pointing at a displayed figure with a pen or the like, the information processing device comprising a plurality of microphones disposed near the touch sensor panel at predetermined intervals, and voice input means for receiving input from each microphone. It has a sound source position identifying circuit that identifies the position of the sound source based on the phase shift of the audio waveform input from the microphone, and a correction amount setting circuit, and detects the direction of the operator with respect to the touch sensor panel based on the audio from the sound source. A display device with an input function for an information processing device is disclosed, which is characterized by correcting parallax.

JP 8-101746

It is an object of the present invention to provide an input/output device that corrects a positional deviation between an operation position on an aerial operation surface and a reaction position of a display.

The input/output device 1 according to the present invention includes a display surface that displays content, a detection unit that detects a user's action on the displayed content at a position different from the display surface, and a photography unit that photographs the user. and a correction section that corrects the detection position detected by the detection section or the position of the content displayed on the display surface based on the image of the user photographed by the photographing section.

Preferably, the camera further includes an eye identifying section that identifies the user's eyes from the photographed image photographed by the photographing section, and the correction section determines the position of the user's eyes in the photographed image identified by the eye identifying section; Alternatively, the detection position detected by the detection unit is corrected based on the distance to the user's eyes.

Preferably, the correction unit determines the correction amount based on the position of the user's eye specified by the eye identification unit, the distance to the eye, and the area of the detection position on the display surface by the detection unit. do.

Preferably, the detection section detects a user's motion at a distance of 1 cm or more and 5 cm or less from the display surface, and the correction section determines the amount of correction at an upper, central, and lower portion of the display surface.

Preferably, the correction unit further includes a relative position calculation unit that calculates a distance from the display surface to the eyes based on the distance between the eyes specified by the eye identification unit, and the correction unit A correction amount is determined based on the calculated distance from the display surface to the eyes.

Preferably, the photographing section photographs the user during the operation, and the correction section corrects the operating position based on the image photographed by the photographing section.

The input/output method according to the present invention includes a detection step of detecting a user's action on the displayed content at a position different from a display surface on which the content is displayed, a photographing step of photographing the user, and the photographing step. and a correction step of correcting the detection position detected by the detection step or the position of the content displayed on the display surface based on the photographed image of the user.

The program according to the present invention includes a detection step of detecting a user's motion with respect to the displayed content at a position different from a display surface where the content is displayed, a photographing step of photographing the user, and a photographing step of the photographing step. The computer is caused to execute a correction step of correcting the detection position detected in the detection step or the position of the content displayed on the display surface based on the image of the user.

To provide an input/output device that corrects a positional deviation between an operation position on an aerial operation surface and a reaction position of a display.

FIG. 6 is an image diagram illustrating the deviation between the line of sight and the reaction position of the aerial operation surface 13 for users of various heights. (a) is a diagram illustrating the configuration of the input device 1 when viewed from the side, and (b) is a diagram illustrating the configuration of the input/output device 1 viewed from the user side. 1 is a diagram illustrating a hardware configuration of an input/output device 1. FIG. 3 is a diagram illustrating a functional configuration of a control unit 11. FIG. 3 is a diagram illustrating the amount of parallax shift between the aerial operation surface 13 and the four corners of the display 3. FIG. FIG. 3 is a diagram illustrating positional deviation correction processing (S10) by the input/output device 1. FIG.

[background]
The background on which the present invention was made will be explained.
In order to make KIOSK terminals and other devices contactless, an increasing number of devices are using fingertip detection in the air to operate panels without touching the screen. In order to detect whether or not the fingertip is in a predetermined position, there are methods of detecting with three-dimensional coordinates using an infrared sensor, a capacitive sensor, or a camera. When a sensor for non-contact operation in the air is retrofitted to an existing touch panel product, the operation surface is formed in the air parallel to the display surface.
Displays such as KIOSK terminals at convenience stores are installed diagonally when viewed from the front. In normal touch panel operation, even if you do not look directly at the screen, no problem occurs because the aerial operation surface responds at the same position in your line of sight.
On the other hand, if an operation surface is raised in the air for non-contact input, if any person attempts to operate the display, parallax will occur between the display and the reaction section. Therefore, when attempting to touch the selection button, the position of the line of sight differs from the height of the aerial operation, causing a deviation from the desired location.

Therefore, as illustrated in FIG. 2, the input/output device 1 uses a camera 5 installed on the display 3 to estimate the position of the user's eyes and the distance from the display 3, and based on the estimated distance, The positional deviation between the display 3 and the aerial operation surface 13 as seen from the line of sight is corrected, and the difficulty of use due to differences in height etc. is eliminated.

Embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows an example of which part of the aerial operation surface 13 is selected when a person from short to tall tries to touch the content displayed on the display 3 when operating a KIOSK terminal. This is an image diagram.
As illustrated in Figure 1, a short person selects the content at the top of the display 3 to be much lower, the content at the center of the display 3 to be slightly lower, and the content at the bottom of the display 3 to the lower part. , I end up choosing a little higher.
In addition, a person of medium height will select the content at the top of the display 3 slightly below, select the content at the center of the display 3 without shifting, and select the content at the bottom of the display 3 slightly above. I end up choosing.
Furthermore, tall people select content at the top of the display 3 without shifting, select content at the center of the display 3 slightly above, and select content at the bottom of the display 3 significantly above. I end up choosing.
Therefore, the amount of parallax shift changes depending on the position of the user's eyes and the positions of the top, center, and bottom of the display 3. That is, the input/output device 1 calculates the relative position between the midpoint of the user's eyes and the display 3, and calculates the error in the reaction position between the user's line of sight and the aerial operation surface 13 in real time based on the calculated relative position. By correcting this, the user can operate without feeling uncomfortable.

FIG. 2 is a diagram illustrating the configuration of the input/output device 1. As shown in FIG.
As illustrated in FIGS. 2(a) and 2(b), the input/output device 1 is a KIOSK terminal, and includes a display 3, a camera 5, a detection unit 7, a housing 9, and a housing 9. It has a control section 11 and an aerial operation surface 13.
The display 3 is attached and fixed so as to cover the opening on the upper side (front side) of the housing 9, and displays content. The display 3 is an example of a display surface according to the present invention.
The detection unit 7 detects a user's action on the content displayed on the display 3 at a position different from the display 3. Specifically, the detection unit 7 detects the user's motion at a distance of 1 cm or more and 5 cm or less from the display 3. More specifically, the detection unit 7 is a sensor and detects the user's motion at a distance of 1 cm or more and 3 cm or less in front of the display 3. Detects fingertip operations.
The camera 5 is a photographing device that photographs the user, and is disposed within the housing 9 to photograph the face of the user of the input/output device 1 . The camera 5 is an example of a photographing unit according to the present invention.
The housing 9 houses the control unit 11.
The control unit 11 has built-in software and is connected to the camera 5, the display 3, and the detection unit 7 by wire or wirelessly. 13 is corrected.
The aerial operation surface 13 is a range in which the detection unit 7 can detect the user's actions regarding the content displayed on the display 3.

FIG. 3 is a diagram illustrating the hardware configuration of the input/output device 1. As shown in FIG.
As illustrated in FIG. 3, the input/output device 1 includes a CPU 100, a memory 102, a nonvolatile memory 104, a network interface 106 (network IF 106), a display 3, a camera 5, and a detection unit 7, and these configurations are connected to each other via a bus 108.
CPU 100 is, for example, a central processing unit.
The memory 102 is, for example, a volatile memory and functions as a main storage device.
The nonvolatile memory 104 stores a computer program (for example, the positional deviation correction program 12 in FIG. 4) as a nonvolatile storage device, for example.
The network IF 106 is an interface for wired or wireless communication.
The display 3 is an example of a display surface, and is, for example, a liquid crystal touch panel.
The camera 5 is a photographing device that photographs the user.
The detection unit 7 is, for example, a capacitive touch panel.

FIG. 4 is a diagram illustrating the functional configuration of the control section 11. As shown in FIG.
As illustrated in FIG. 4, a positional deviation correction program 12 is installed in the control unit 11. The positional deviation correction program 12 includes a face detection section 120, an eye identification section 122, a map creation section 124, a relative position calculation section 126, a map switching section 128, and a correction section 130. 132 and a deviation correction section 134.
The face detection unit 120 determines whether the user's face is within the photographing range of the camera 5, and if the user's face is within the photographing range, the camera 5 photographs the user's face.

The eye identification unit 122 identifies the user's eyes based on the face image photographed by the face detection unit 120. Specifically, the eye identification unit 122 identifies the user's right eye and left eye. More specifically, the eye identifying unit 122 identifies an intermediate position between the right eye and the left eye.
The map creation unit 124 creates a map of the amount of deviation of positions within the display 3 (for example, 1000 positions for 100×100) with respect to a reference position that is an intermediate position of a certain user's eyes. Specifically, the map creation unit 124 calculates the amount of parallax shift between the reference position, the four corners of the display 3, and the aerial operation surface 13. As illustrated in FIG. 5, the amount of deviation of the four corners of the display 3 is determined by the point where it intersects with the aerial operation surface 13, and the four corners of the display 3 have the largest deviation in each direction. More specifically, the map creation unit 124 creates a map of the amount of deviation at each position within the display 3 by changing the distance and positional relationship with the reference position.

The relative position calculation unit 126 calculates the distance from the display 3 to the user's eyes based on the distance between the eyes.
The map switching unit 128 switches the map of the amount of deviation based on the relative position between the reference position of the eye (intermediate position of the eye) and the display 3.
The correction unit 130 corrects the detection position detected by the detection unit 7 or the position of the content displayed on the display 3 based on the image of the user captured by the camera 5. Specifically, the correction unit 130 corrects the operation position based on an image taken by the camera 5 during the user's operation.
The correction amount determining unit 132 determines the correction amount based on the position of the user's eyes, the distance to the user's eyes, and the area of the detected position on the display 3. Specifically, the correction amount determining unit 132 determines the correction amount based on the relative positions of the user's eyes and the top, center, and bottom of the display 3. More specifically, the correction amount determining unit 132 determines the correction amount based on the map of the amount of deviation switched by the map switching unit 128.
The deviation correction unit 134 performs deviation correction on the detected position of the finger based on the correction amount determined by the correction amount determination unit 132.
Note that part or all of the positional deviation correction program 12 may be realized by hardware such as an ASIC, or may be realized by borrowing some functions of an OS (Operating System).

FIG. 6 is a flowchart illustrating the positional deviation correction process (S10) by the input/output device 1.
As illustrated in FIG. 6, in step 100 (S100), the map creation unit 124 creates a map of the amount of deviation of each position in the display 3 with respect to the reference position of a certain user's eyes.
In step 105 (S105), the face detection unit 120 determines whether the user's face is within the shooting range of the camera 5. If it is within the photographing range, the process moves to S110 (S105: Yes), and if it is not within the photographing range, the process waits until the user's face falls within the photographing range (S105: No).
In step 110 (S110), the face detection section 120 photographs the user's face with the camera 5, and the eye identification section 122 identifies the intermediate position of the user's right eye and left eye in the photographed image as a reference position.

In step 115 (S115), the relative position calculation unit 126 calculates the relative distance between the reference position specified by the eye identification unit 122 and the display 3.
In step 120 (S120), the map switching unit 128 switches to a map with a shift amount according to the relative distance calculated by the relative position calculation unit 126.
In step 125 (S125), the positional deviation correction process (S10) moves to S130 when the detection unit 7 detects the user's operation on the aerial operation surface 13 (S125: Yes), and when it is not detected. (S125: No).
In step 130 (S130), the correction amount determination unit 132 determines the correction amount based on the switched map of the amount of deviation, and the deviation correction unit 134 uses the correction amount determined by the correction amount determination unit 132 to detect The position detected by the unit 7 is corrected.
In step 135 (S135), the input device 1 adjusts the corrected position to the display position on the display 3 side, and performs the same process as the touch operation on the display 3.

As explained above, the input/output device 1 of the present embodiment can correct the amount of parallax shift between the aerial operation surface 13 and the display 3 based on the distance between the user's eye position and the display 3. . As a result, even if users of different heights use the device, the parallax between the operating position on the aerial operating surface 13 and the reaction section of the display 3 is eliminated, and the operation can be performed without any discomfort.
Further, since the correction of the deviation is performed in real time based on the image photographed during the user's operation, it is possible to correct the deviation according to the user who is operating the input device 1.

1... Input/output device 3... Display 5... Camera 7... Detection unit 9... Housing 11... Control unit 12... Position shift correction program 13... Aerial operation surface 120... Face detection unit 122... Eye identification unit 124... Map creation unit 126 ...Relative position calculation section 128...Map switching section 130...Correction section 132...Correction amount determination section 134...Difference correction section

Claims (8)

a display surface for displaying content;
a detection unit that detects a user's action on the displayed content at a position different from the display surface;
a photography unit that photographs the user;
An input/output device comprising: a correction section that corrects a detection position detected by the detection section or a position of content displayed on the display surface based on an image of a user photographed by the photographing section. further comprising an eye identification unit that identifies the user's eyes from the captured image captured by the imaging unit;
The correction unit corrects the detection position detected by the detection unit based on the position of the user's eye in the captured image specified by the eye identification unit or the distance to the user's eye. input/output device. The correction unit determines the correction amount based on the position of the user's eye specified by the eye identification unit, the distance to the eye, and the area of the detection position on the display surface by the detection unit. Input/output devices described in . The detection unit detects a user's motion at a distance of 1 cm or more and 5 cm or less from the display surface,
The input/output device according to claim 1, wherein the correction unit determines the amount of correction at an upper portion, a center portion, and a lower portion of the display surface. further comprising: a relative position calculation unit that calculates a distance from the display surface to the eyes based on the distance between the eyes specified by the eye identification unit;
The input/output device according to claim 3, wherein the correction unit determines the correction amount based on the distance from the display surface to the eye calculated by the relative position calculation unit. The photographing unit photographs the user during operation;
The input/output device according to claim 1, wherein the correction section corrects the operation position based on an image photographed by the photographing section. a detection step of detecting a user's action on the displayed content at a position different from the display surface on which the content is displayed;
a photographing step of photographing the user;
An input/output method comprising: a correction step of correcting the detection position detected in the detection step or the position of the content displayed on the display surface based on the image of the user photographed in the photographing step. a detection step of detecting a user's action on the displayed content at a position different from the display surface on which the content is displayed;
a photographing step of photographing the user;
A program that causes a computer to execute a correction step of correcting the detection position detected in the detection step or the position of the content displayed on the display surface based on the image of the user photographed in the photographing step.

Images