WO2018154714A1 - Operation input system, operation input method, and operation input program - Google Patents

Abstract

[Problem] To reliably obtain an operation input made via fine quick movement of a finger tip or the like without the need for a direct manipulation device such as a mouse or a touch panel, thus imposing a reduced burden on the operator. [Solution] This operation input system is provided with: a transmission unit 5c which transmits electromagnetic waves or sound waves toward a monitored space region R3 including part or all of a viewing range R2 in which a display screen 5b is visible, said electronic waves and sound waves serving as transmission waves; reception units 6a0—2 which receive waves reflected from a target object irradiated with the transmission waves; a signal processing unit 112 which, on the basis of the transmission waves and the reflected waves, detects position information relating to a three-dimensional space position of the wave-reflecting target object and relating to changes in that position, generates an input signal associated with a coordinate position on the display screen 5b, and provides the generated input signal to an information processing terminal device; and an approach/departure detection unit 113 which compares the wavelengths of the reflected waves with those of the transmission waves and thereby detects the movement and the speed of the target object in the direction in which the target object approaches or departs from the display screen 5b.

Description

Operation input system, operation input method, and operation input program

The present invention relates to an operation input system, an operation input method, and an operation input program for inputting a coordinate position on a display screen of an information processing apparatus and an operation command related to the coordinate position.

Conventionally, as a device for inputting an operation command for an information processing apparatus such as a personal computer, in addition to a keyboard for inputting characters, a coordinate position on a display screen of a display such as a mouse, a touch pad, and a touch panel is designated. A method of inputting an operation command at the coordinate position, such as button click, tap, and swipe, has become widespread.

In addition, in order for the operator to improve the convenience of operation without using a special device such as a mouse, a touch pad, or a touch panel, the operator's hand is held by a camera provided in the information processing terminal. Take an image, set the operation area in the air near the position of the finger in correspondence with the screen of the information processing terminal, detect the movement of the finger in the operation area by image analysis, and from the movement of the finger, the screen There has been proposed an operation input device that moves the upper cursor or highlights and designates an icon (see Patent Document 1). According to the operation input device disclosed in Patent Document 1, it is possible to determine an operation input with less burden on the operator without requiring an operation of directly touching a finger such as a mouse, a touch pad, or a touch panel.

By the way, in the operation input device disclosed in Patent Document 1 described above, since the position of the finger is detected by analyzing the image taken by the camera, the movement of the finger forward and backward is due to a change in the size of the image of the fingertip. There is a problem that detailed movement cannot be detected.

On the other hand, for example, the minimum point in the distance image is detected using a distance image that is a distance distribution from the camera to the object surface as disclosed in Patent Document 2, and the operator's There has also been proposed an information input device that obtains the position of the fingertip and enables pointing on the screen by the movement.

JP 2013-171529 A JP 7-334299 A

However, since all of the devices disclosed in the above-mentioned patent documents detect the fingertip by image analysis and determine the pointing operation based on the movement, it is difficult to detect the fine and quick movement of the fingertip. It may cause malfunction.

Therefore, the present invention solves the above-described problems, and does not require a device that directly operates such as a mouse, a touch pad, or a touch panel, and has a small burden on the operator such as a thin and quick movement of a fingertip. An object of the present invention is to provide an operation input system, an operation input method, and an operation input program that can accurately acquire an operation input.

In order to solve the above problems, the present invention is an operation input system for inputting an operation command related to a coordinate position on the display screen of the information processing apparatus and the coordinate position,
A transmitter that transmits electromagnetic waves or sound waves as a transmitted wave to a monitoring space area including a part or all of the visual field range in which the display screen is visible;
A receiving unit that receives the transmitted wave reflected by the object as a reflected wave;
Compared with the wavelength of the transmitted wave and the wavelength of the reflected wave, a contact / separation detection unit that detects the movement of the object in the direction approaching or leaving the transmission unit or the reception unit and its speed,
Based on the transmitted wave transmitted and the received reflected wave, the position information on the position and the displacement of the object that generated the reflected wave in the three-dimensional space is detected, and the position information and the contact / separation detection unit And a signal processing unit that generates an input signal related to a coordinate position on the display screen based on a detection result and provides the input signal to the information processing apparatus.

The present invention is also an operation input method for inputting an operation command related to a coordinate position on the display screen of the information processing apparatus and the coordinate position,
While transmitting the electromagnetic wave or sound wave as a transmitted wave from the transmitter to a monitoring space area including part or all of the visual field range where the display screen is visible, the transmitted wave reflected by the object is used as a reflected wave. A step of receiving by the receiver;
Based on the transmitted wave and the received reflected wave, position information on the three-dimensional space of the object that has generated the reflected wave and position information regarding the displacement are detected, and signal processing is performed based on the position information. A unit that generates an input signal related to a coordinate position on the display screen and provides the input signal to the information processing apparatus;
By comparing the wavelength of the transmitted wave and the wavelength of the reflected wave, the contact / separation detection unit detects the movement and speed of the object in a direction approaching or leaving the transmission unit or the reception unit. And a step.

According to these inventions, the position of the object in the three-dimensional space and its displacement are detected based on the transmitted wave and the reflected wave, and input signals relating to cursor movement and basic command operations are provided to the information processing apparatus. In addition, a pointing operation by an operator can be input without requiring a device such as a mouse, a touch pad, or a touch panel that is directly operated. At this time, in order to monitor the so-called Doppler effect by comparing the change in wavelength of the transmitted wave and its reflected wave, the approach of the object to the transmitting unit or the receiving unit or without performing heavy processing such as image analysis The separation and the moving speed thereof can be detected, and an operation input with less burden on the operator such as a fine and quick movement of a fingertip or the like can be accurately acquired.

(Direction of transmitter / receiver)
In the above-described invention, a transmitter that emits a pulse wave from the display screen direction toward a space free of obstacles with directivity, and a mixture of re-reflection from the vicinity is avoided, and the fingertip in the air = from the monitored object It is preferable to further include a receiving unit that receives only the direct reflected wave with directivity, and to generate the input signal based on the reflected wave that returns directly from the monitoring target point.

(filter)
In the above invention, a filtering unit that extracts in advance only the minimum data necessary for the subsequent processing process of the received reflected wave, that is, only data from the initial wave of the reflected wave to a predetermined time, and sends it to the subsequent processing flow. In addition, it is preferable that the signal processing unit generates the input signal based on the reflected wave extracted by the filtering unit.

デ ー タ By selecting data to be processed at an early stage by this filtering, it is possible to reduce the amount of calculation processing, reduce the load, speed up the processing, and improve the responsiveness of the input operation.

(Pointing gesture recognition and fingertip detection)
In the above invention, the signal processing unit is characterized in that the shape of the amplitude-time profile of the reflected wave after the filter is such that the initial wave has the highest amplitude intensity and the subsequent wave has no to weak amplitude below a threshold value. Only when the pointing gesture is recognized, and when the pointing gesture is recognized, the initial wave of the amplitude-time profile, that is, the maximum amplitude intensity and the distance to the receiving unit is It is preferable to further include a module that detects the shortest point as a fingertip = monitoring target point, and detects the fingertip = monitoring target point in the operator's pointing gesture using a necessary and minimum device.

(Fingertip detection + 3-point ranging)
In the above invention, when the pointing gesture is recognized by the signal processing unit and the initial wave is detected as a fingertip = monitoring target point, the distance between the fingertip = monitoring target point and each of at least three receiving units is calculated. Obtaining the operator's pointing coordinates while omitting complicated image processing, further comprising a module that obtains the spatial coordinate position of the fingertip = monitoring target point from the at least three distances by the three-point ranging method Is preferred. Furthermore, it is preferable that the contact / separation detection unit detects a movement of the fingertip = a point to be monitored before and after and a speed thereof, and generates a command signal at the pointing coordinates.

(3D touch panel)
In the above invention, the signal processing unit is parallel to the display screen or has a predetermined angle in the monitoring space region, has the same shape or similar shape to the display screen, and is a distance from the display screen to the monitoring target point And an operation surface setting unit for setting a virtual continuous countless operation surface, that is, a 3D touch panel, whose size is changed in accordance with the coordinates of the monitoring target point on the virtual operation surface of the 3D touch panel It is preferable to generate the pointing input signal according to the position.

The 3D touch panel makes it easy to imagine a pointing operation such as moving a cursor by defining an infinite number of virtual operation surfaces that are continuous in the air between the operator and the display screen. Since the size of the virtual operation surface varies depending on the distance between the monitoring target point and the display screen (the farther it is, the larger the distance), the fingertip = when moving the monitoring target point parallel to the display screen by adjusting the distance The ratio between the speed of the fingertip = the point to be monitored and the speed of the pointer on the display screen changes (the farther it is, the slower the distance), and a more flexible pointing operation becomes possible.

Since the 3D touch panel is a so-called non-contact type device that performs a pointing operation using a space from a screen to an operator, it is difficult to perform a command operation by touching with a fingertip = a monitoring target point. In order to compensate for this weak point, it is preferable to improve the operability of the information processing terminal by adopting a button operation (Doppler command) using the Doppler effect.

(Subcommand key)
In the above invention, since the Doppler commands are limited to two types of button down and button up and combinations thereof, only basic commands that can be executed by the current mouse, touch pad, and touch panel can be realized. Therefore, it is preferable to further include one subcommand key in the pointing gesture and diversify input commands in addition to the Doppler command. This subcommand key can be replaced by other methods such as voice input.

(Distance image + 3D touch panel)
In the above invention, the module is changed to recognize the pointing gesture from the form by a distance image instead of the three-point ranging method, and directly obtain the spatial coordinates of the fingertip = monitoring target point, and more visually. The operator's pointing operation can also be acquired in an easy-to-understand manner. Even in this change, it is preferable that a pointing operation is performed in combination with the 3D touch panel, and that the contact / separation detection unit detects the movement of the fingertip = the front and back of the monitoring target point and the speed thereof by the Doppler effect. .

(Distance image + vector projection)
In the above invention, when the distance image is used, instead of a pointing operation by a combination of the fingertip = the position coordinates of the monitoring target point and the 3D touch panel, the shape of the finger at the time of the pointing gesture is schematically shown as a columnar shape by the distance image. It is also possible to adopt an operation method in which a pointer is displayed at a coordinate position where a vector directed to the axis of the columnar shape intersects with the display screen as a monitoring target point. Also in this case, it is preferable that the contact / separation detection unit monitors the Doppler effect using a reflected wave used for the distance image.

(Subcommand (second columnar) detection)
In the above invention, when using a distance image, a fingertip or a columnar shape having the second closest distance to the receiving unit in the monitoring space area is detected as a second monitoring target, and the signal processing unit It is preferable to change the content of the input signal according to the presence or absence of the second monitoring target point. In this case, it is possible to diversify the input commands by detecting an additional gesture such as raising a little finger in combination with a pointer operation with the index finger, for example.

(Operation input program)
The above-described apparatus and method according to the present invention can be realized by executing the operation input program of the present invention described in a predetermined language on a computer. That is, the program of the present invention is installed in a mobile terminal device, a smart phone, a wearable terminal, a mobile personal computer, other information processing terminals, an IC chip or a memory device of a general-purpose computer such as a personal computer or a server computer, and executed on the CPU. By doing so, the system of each function mentioned above can be constructed | assembled, and the method of this invention can be implemented.

That is, the program of the present invention is an operation input program for inputting a coordinate position on the display screen of the information processing apparatus and an operation command related to the coordinate position,
A transmitter that transmits electromagnetic waves or sound waves as a transmitted wave to a monitoring space area including a part or all of the visual field range in which the display screen is visible,
A receiving unit that receives the transmitted wave reflected by the object as a reflected wave;
Based on the transmitted wave and the received reflected wave, the position information on the three-dimensional space of the object that generated the reflected wave and position information regarding the displacement are detected, and the display is performed based on the position information. An input signal related to a coordinate position on the screen is generated and provided to the information processing device, and the wavelength of the transmitted wave and the wavelength of the reflected wave are compared, and the signal is transmitted to the transmitter or the receiver. On the other hand, it functions as a contact / separation detection unit that detects the movement and speed of the object in the direction of approaching or leaving.

In such an operation input program of the present invention, for example, as a package application that can be distributed through a communication line and that runs on a stand-alone computer by being recorded on a computer-readable recording medium. Can be transferred. Specifically, the recording medium can be recorded on various recording media such as a magnetic recording medium such as a flexible disk and a cassette tape, an optical disk such as a CD-ROM and a DVD-ROM, and a RAM card. According to the computer-readable recording medium on which the program is recorded, the above-described system and method can be easily implemented using a general-purpose computer or a dedicated computer, and the program can be stored, transported, and Easy installation.

According to these inventions, a mouse, a touch pad, a touch panel, or the like is operated by direct contact with a pointing device or a command input on a desktop or a display screen of an information processing terminal device such as a notebook personal computer, a tablet personal computer, or a smartphone. Without the need for a device, it is possible to accurately acquire an operation input that places little burden on the operator, such as a thin and quick fingertip movement.

It is a perspective view showing appearance composition of an information processing terminal unit for realizing an operation input system concerning a 1st embodiment. It is a block diagram which shows the hardware constitutions of the information processing terminal device which concerns on 1st Embodiment. It is a block diagram which shows the functional module constructed | assembled on CPU of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the detection process of the fingertip = monitoring target point of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the outline | summary of the three-point ranging method of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the Doppler effect detection process of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the 3D touch panel of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the 3D touch panel of the operation input system which concerns on 1st Embodiment. It is explanatory drawing which shows the filtering process and pointing gesture recognition of the operation input system which concern on 1st Embodiment. It is explanatory drawing which shows the example of the amplitude-time profile utilized for the pointing gesture recognition of the operation input system which concerns on 1st Embodiment, and a fingertip = monitoring target point detection process. It is a flowchart figure which shows the procedure of the operation input method which concerns on 1st Embodiment. It is a block diagram which shows the functional module constructed | assembled on CPU of the operation input system which concerns on 2nd Embodiment. It is explanatory drawing which shows the operation method of the operation input system which concerns on 2nd Embodiment. It is explanatory drawing which shows the vector projection process of the operation input system which concerns on 2nd Embodiment. It is explanatory drawing which shows the columnar shape detection process of the operation input system which concerns on 2nd Embodiment. It is explanatory drawing which shows the subcommand recognition process of the operation input system which concerns on 2nd Embodiment. It is a flowchart figure which shows the procedure of the operation input method which concerns on 2nd Embodiment. It is explanatory drawing which shows the 3D touchscreen which concerns on the example of a change. It is explanatory drawing which shows the outline | summary of the three-point distance measurement in the fingertip = monitoring target point detection process of the operation input system which concerns on the example of a change.

[First Embodiment]
(Overall configuration of operation input system)
Hereinafter, a first embodiment of an operation input system according to the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a conceptual diagram showing an external configuration of an information processing terminal device for realizing the operation input system according to the present embodiment, and FIG. 2 is an information process for realizing the operation input system according to the present embodiment. It is a block diagram which shows the hardware constitutions of a terminal device. The “module” used in the description refers to a functional unit that is configured by hardware such as an apparatus or a device, software having the function, or a combination thereof, and achieves a predetermined operation. .

The information processing terminal device according to the present embodiment can be realized by a general-purpose computer or a dedicated device. In the present embodiment, as shown in FIG. Install the operation input program. The notebook personal computer 1 is an information processing terminal device that includes a display screen 5b such as a monitor and is integrated so that a main body having a keyboard, a pointing device, and the like is folded in half, and includes a transmitter 5c and a receiver 6a0. 2 are arranged on the lower side of the display screen 5b toward the operator side. In the present embodiment, the case where the present invention is applied to a notebook personal computer will be described as an example. However, the present invention is not limited to this, and information processing that can be equipped or connected to the transmission unit 5c and the reception units 6a0 to 6a. If it is a terminal, it can be realized by a general-purpose computer such as a desktop personal computer or a dedicated device specialized in function, and a tablet personal computer, smartphone, mobile phone, wearable terminal device, etc. can also be adopted. .

As shown in FIG. 2, the hardware configuration of the notebook personal computer 1 includes a CPU 11, a memory 12, an input interface 16, a storage device 14, an output interface 15, and a communication interface 13. . In the present embodiment, these devices are connected via the CPU bus 10 and can exchange data with each other.

The input interface 16 is a module that receives an operation signal from an operation device such as a keyboard 6b, a pointing device, a touch panel, or a button. The received operation signal is transmitted to the CPU 11 and can perform operations on the OS and each application. . Further, input devices such as a CCD camera 6d and a microphone 6c can be connected to the input interface 16, and the receiving units 6a0 to 6a2 of the present invention are also connected to the input interface 16.

As shown in FIGS. 4 to 6, the receiving units 6a0 to 6a are receiving devices having directivity for blocking re-reflection from the vicinity and selectively receiving only the direct reflected wave from the monitoring target in the air. is there. More specifically, the receiving units 6a0 to 6a2 irradiate the space W that has nothing other than the monitoring target and the operator in the vicinity with a predetermined directivity, and the direction of the keyboard among the transmitted waves reflected by the target object. A direct reflected wave that reaches the receiving unit having directivity so as to prevent re-reflection from being received is received as a reflected wave W2, and the received reflected wave W2 is subjected to the synchronization processing shown in FIG. The signal is input to the signal processing unit 112 and the contact / separation detection unit 113 through the unit 111.

In this embodiment, the receiving units 6a0 to 6a2 are a plurality of infrared sensors that are physically separated and arranged adjacent to each other, and are arranged in a triangle on the lower right side of the display screen 5b. If there is sufficient resolution, various sensors corresponding to the types of electromagnetic waves and sound waves transmitted by the transmitter 5c, such as electromagnetic sensors other than infrared rays and ultrasonic sensors, can be employed.

The output interface 15 is a module that transmits video signals and audio signals in order to output video and audio from output devices such as the display screen 5b and the speaker 5a. A transmitter 5c that transmits electromagnetic waves or sound waves as a transmitted wave is also connected to the output interface 15.

As shown in FIG. 4, the transmitter 5 c has a predetermined directivity with respect to the monitoring space region R <b> 3 including part or all of the visual field range R <b> 2 that is visible on the display screen 5 b using electromagnetic waves or sound waves as a transmitted wave. This is a transmitting device, and transmits a transmitted wave in accordance with control by the synchronization processing unit 111 in FIG. In particular, in the present embodiment, the transmitter 5c employs an infrared light that irradiates an infrared laser pulse, and is arranged at the center of the receivers 6a0 to 6a2 arranged in a triangle at the lower right side of the display screen 5b. If the transmitter 5c has sufficient resolution to detect the fingertip = the monitoring target point, the transmitter 5c can detect electromagnetic waves and sound waves that can be detected by the receivers 6a0 to 2 such as an ultrasonic transmitter and a radio transmitter other than infrared rays. Various transmitters corresponding to such types can be employed.

The communication interface 13 is a module that transmits and receives data to and from other communication devices. As a communication method, for example, a public line such as a telephone line, an ISDN line, an ADSL line, and an optical line, a dedicated line, WCDMA (registered trademark) In addition to 3rd generation (3G) communication systems such as CDMA2000, 4th generation (4G) communication systems such as LTE, and 5th generation (5G) and later communication systems, WiFi (registered trademark), Bluetooth ( Wireless communication networks such as registered trademark).

The storage device 14 is a device that accumulates data in a recording medium and reads out the accumulated data in response to a request from each device. For example, a hard disk drive (HDD), a solid state drive (SSD), a memory card, and the like Can be configured.

The CPU 11 is a device that performs various arithmetic processes necessary for controlling each unit, and virtually constructs various modules on the CPU 11 by executing various programs. On the CPU 11, an OS (Operating System) is activated and executed, and the basic functions of the notebook personal computer 1 are managed and controlled by the OS. Furthermore, various applications can be executed on the OS, and the CPU 11 executes the OS program, thereby managing and controlling the basic functions of the notebook personal computer 1, and the CPU 11 executing the application program. By being executed, various functional modules are virtually constructed on the CPU.

(Configuration of various function modules)
FIG. 3 is a block diagram showing functional modules constructed on the CPU of the operation input system according to the first embodiment. In the present embodiment, by executing the operation input program of the present invention, three modules of the synchronization processing unit 111, the signal processing unit 112, and the contact / separation detection unit 113 are constructed on the CPU 11.

As shown in FIGS. 4 to 6, the synchronization processing unit 111 synchronizes the transmission process of the transmitted wave W1 in the transmission unit 5c and the reception process of the reflected wave W2 in the reception unit 6a, and transmits the transmitted wave W1 and the reflected wave W2. It is a module that associates with. In the present embodiment, the transmission unit 5c transmits intermittent pulsar waves as transmission waves W1, receives reflected waves corresponding to the transmitted pulsar waves, transmits and receives the order, time, transmission to reception. Based on the time length, each transmitted wave signal and each corresponding reflected wave signal are selected, the transmitted wave and the reflected wave signal are associated as a set, and sent to the signal processing unit 112.

The signal processing unit 112 detects the spatial position information of the object that generated the reflected wave W2 and its displacement based on the transmitted wave W1 transmitted by the transmitting unit 5c and the reflected wave W2 received by the receiving unit 6a. This is a module that generates an input signal related to a coordinate position on the display screen 5b and provides it to an OS or an application running on the information processing apparatus.
Specifically, the signal processing unit 112 according to the present embodiment includes a filtering unit 112a, a pointing gesture recognition unit 112b, a fingertip = monitor target point detection unit 112c, a position information calculation unit 112d, an operation surface setting unit 112e, and a pointing signal generation. 112f and an input signal generator 112g.

The filtering unit 112a is a module that excludes in advance signals other than the reflected waves that are required in subsequent processing operations when the monitoring space region R3 shown in FIG. 4 is monitored. Since processing operations scheduled to be performed in the future, that is, pointing gesture recognition and fingertip = monitoring target detection, only a reflected wave in a range (A1) corresponding to a depth of about 3 cm from the initial motion wave shown in FIG. 10 is required. The other reflected wave data is excluded at the previous stage, and only the excluded data is input to the pointing gesture recognition unit 112b.

The pointing gesture recognition unit 112b determines the presence or absence of the pointing gesture as shown in FIGS. 1, 4 to 9, 13, 13, 18, and 19. When the pointing shown in FIG. 9 is performed, the reflected wave has a strong amplitude at the fingertip B1 perpendicular to the traveling direction of the transmitted wave, and at the belly of the finger that is parallel to the traveling direction of the transmitted wave or is in contact with a predetermined angle (for example, an obtuse angle). Since there is no to weak amplitude, the reflected wave forms an amplitude-time profile as seen in the A1 region of FIG. Paradoxically speaking, if such a characteristic of the amplitude-time profile is observed in the reflected wave, it is presumed that a pointing gesture was present. In the present embodiment, it is determined that there is a pointing gesture when such a profile is observed simultaneously with all the predetermined number or more of the pulse waves of the three receiving units 6a0 to 6a2, and the reflected wave data is stored at the next fingertip. = Input to monitoring target point detection unit 112c.

The fingertip = monitoring target point detection unit 112c identifies the initial motion wave as the fingertip = monitoring target point shown in B1 of FIGS. 9 and 10 in the reflected wave data accompanying the pointing gesture shown in the range of A1 in FIG. The distance from the fingertip = the monitoring target point to each of the three receiving units is calculated by multiplying the round trip time of the transmitted wave pulse by the speed of light and dividing by 2. The three distance values calculated in this way are input to the position information calculation unit 112d.
Next, the positional information calculation unit 112d uses the three-point distance measurement method based on the three distances input from the fingertip = monitoring target point detection unit 112c to calculate the relative position information (x, y, z) is obtained and input to the pointing signal generator 112f.

In this embodiment, spatial position information is obtained by three-point ranging from three reception units arranged in a triangle. However, as the number of reception units increases, the arrangement is improved by devising the arrangement. The accuracy of the three-dimensional position information of the target point is improved. In the present embodiment, the number of transmitting units is one, but the number of transmitting units can be increased to the same number as the receiving units by synchronizing with each receiving unit. The proximity of the receiving unit and the transmitting unit increases the accuracy of distance measurement and increases the reliability of the spatial position information of the fingertip = monitoring target point. The combination of the number and arrangement of the transmitters and receivers can be optimized in accordance with the required accuracy of the monitoring target point position information. In the first embodiment, as shown in FIG. 1, the most basic 1 transmission unit + 3 reception unit is adopted, but as shown in FIG. 18, as a pair of three transmission units and three reception units corresponding to each of the three transmission units. Also good.

As shown in FIGS. 7 and 8, the operation surface setting unit 112e has the same or similar shape to the display screen 5b in parallel or obtuse with the display screen 5b in the monitoring space region R3. This is a module for setting a virtual continuous countless virtual operation screen VP set (3D touch panel) that changes its size in accordance with the distance to the target point.

In the 3D touch panel, the size of the VP changes according to the distance from the display screen 5b to the monitoring target point 3a. That is, the size of the virtual operation screen VP or VP2 changes according to the distance D1 or D2 between the operator's fingertip (3a or 3a ′) = the monitoring target point and the display screen 5b (larger as the distance increases), By adjusting the distance D1 (or D2), the ratio between the speed at which the fingertip = the monitoring target point (3a or 3a ′) is moved in parallel to the display screen 5b and the speed of the pointer on the display screen is changed. If you want to move the pointer quickly and greatly, move it near the screen. If you want to move it slowly and precisely, move your fingertip away from the screen. The virtual operation screen VP or VP2 is a bottom surface of a quadrangular pyramid, and is formed innumerably along the height of the quadrangular pyramid = the central axis direction of the transmitted wave (z in the figure). The virtual operation screen VP or VP2 as the bottom surface of the quadrangular pyramid may be set to be perpendicular to the normal line orthogonal to the display screen 5b, for example, and perpendicular to the extension line passing through the transmission unit or the reception unit Or you may set so that it may make a predetermined angle and you may set so that it may become parallel with respect to the display screen 5b.

That is, in the three-dimensional space in which the horizontal and vertical directions of the display screen 5b are the x axis and the y axis, respectively, and the direction perpendicular to the screen is the z axis, one VP having the same shape and different scale is assigned to an arbitrary z value. If the fingertip = monitoring target point 3a is moved forward and the z value becomes smaller, the virtual operation screen VP becomes smaller, and the virtual operation screen VP becomes larger by moving backward and the z value becomes larger. The framework = 3D grid of the 3D touch panel set by the operation surface setting unit 112e is input to the pointing signal generation unit 112f.

The pointing signal generation unit 112f is a module that generates a GUI (Graphical User Interface) such as a cursor, a pointer, or a scroll bar displayed on the screen as a pointing signal according to coordinates on the screen. Here, the spatial position information (x, y, z) of the fingertip = monitoring target point input from the position information calculation unit 112d is applied to the 3D touch panel framework = 3D grid set by the operation surface setting unit 112e. It is captured. One VP corresponding to the z value of the fingertip = monitoring target point is selected, and the position of the pointer on the display screen 5b is determined on the basis of the fingertip = position (x, y) of the monitoring target point on the VP. The signal is input to the signal generator 112g. This is the description of the signal processing unit.

Next, the contact / separation detector will be described. The contact / separation detector 113 is a module that detects the Doppler effect by comparing the wavelength of the transmitted wave and the wavelength of the reflected wave, and detects the movement of the object in the direction of approaching or leaving the object and the speed thereof. is there. The contact / separation detection unit 113 includes a Doppler effect detection unit 113a and a Doppler command signal generation unit 113b.

The Doppler effect detection unit 113a detects the Doppler effect by comparing the wavelength T1 of the transmitted wave W1, which is a transmitted pulse wave, with the wavelength T2 of the reflected wave W2, as shown in FIG. As the reflected wave W2, the initial wave at the fingertip B1 input from the fingertip = monitoring target point detection unit 112c shown in FIG. 10 is used. When the wavelength T2 of the reflected wave is shortened, that is, when a blue shift is detected, it is determined that the fingertip = monitor target point is approached. When the wavelength T2 is increased, that is, when a red shift is detected, the fingertip = monitor target point. Is determined to have left. When the pulse directions of the three receiving units are the same and all the receiving units recognize a pulse wave whose fingertip = monitoring point moving speed is equal to or greater than a threshold value more than a specific number of times, Of these, the average of the maximum speeds is input to the Doppler command signal generator 113b.

The Doppler command signal generation unit 113b is a module for converting the fingertip = the moving direction and the moving speed before and after the monitoring target point obtained by the Doppler effect detecting unit 113a into commands (Doppler commands). In the present embodiment, it is interpreted as a button down operation when it is a blue shift where the wavelength of the reflected wave is short, and as a button up operation when it is a red shift where the wavelength is long. For example, if there is a rapid button-down operation, it is determined that a click with the mouse is performed, and if the button-down operation continues twice rapidly, a double-click operation is determined. If there is a rapid button up, it is determined that the drag operation is started, and if the button is rapidly down after selecting the drag range, it is determined that the drag operation is completed. The Doppler command generated here is input to the input signal generation unit.

The input signal generation unit 112g integrates the pointer coordinates (x, y) on the screen obtained by the pointing signal generation unit 112f and the command signal obtained by the Doppler command signal generation unit 113b, and performs comprehensive operations on the GUI. An input signal translated as a command is generated. When the Doppler command signal is input to the input signal generation unit 112g, the input of the pointing signal to the input signal generation unit 112g is interrupted, and the pointer coordinates (x, y) remain at a position where the command is executed for a certain time. In addition, a time lag is provided in the input time of the pointing signal and the Doppler command signal so that the pointer coordinates do not fluctuate in the short time before the Doppler command is recognized, and the Doppler effect is detected after the fingertip = monitored point starts moving. It is preferable to exclude the movement of the finger up to the point from the pointing operation.

The input signal generated by the input signal generation unit 112g is provided to the notebook personal computer 1 through the OS executed on the CPU 11 and other applications. The input signal generation unit 112g may generate various commands by incorporating operation signals from other input devices in addition to the Doppler command. For example, with the pointer displayed on the display screen with a pointing gesture, the sub-command key 1a in the left front corner of the information processing terminal shown in FIG. 1 is briefly pressed once to display the menu screen, or the same key is pressed. You can zoom in and out by moving the pointer back and forth while holding down, and scroll the screen by moving it in parallel with the screen. In this way, by adding only one subcommand key, almost all current mouse and touch panel commands can be realized together with the Doppler command. Furthermore, it is more convenient if the GUI mode can be optimized to the operator's preference according to needs, such as adding assistance by voice input to the microphone 6c such as “click”, “drag”, “zoom in”, and the like.

Also, when generating the input signal, it is more convenient to include an image signal and character information for displaying the content of the command included in the input signal in the vicinity of the pointer on the screen. For example, a character string such as “click” or “double click” may be displayed as a message for about 1 second, or “drag”, “zoom”, and “scroll” may be continuously displayed during operation.

(Operation input procedure)
The operation input method of the present invention can be implemented by operating the operation input system described above. FIG. 11 is a flowchart showing the procedure of the operation input method according to the present embodiment.

First, in step S101, infrared rays or other electromagnetic waves or sound waves are transmitted as directional transmission waves from the transmission unit 5c to the monitoring space region R3 shown in FIG. 4 and reflected waves reflected by the object. Are received by the receiving units 6a0 to 6a so that re-reflection is not mixed.
In step S102, the transmission process of the transmission wave W1 in the transmission unit shown in FIG. 6 and the reception process of the reflected wave W2 in the reception unit are synchronized, and the transmission wave W1 and the reflection wave W2 are associated with each other.
In step S103, in order to remove unnecessary information in advance and reduce the processing load, the filtering unit 112a performs filtering to select only a reflected wave signal in a range corresponding to a depth of 3 centimeters from the initial motion wave. Send to step.
In step S104, the presence / absence of a pointing gesture is determined for the filtered signal. When a characteristic reflected wave pattern as shown in the area A1 in FIG. 10 is detected simultaneously with a predetermined number or more of pulse waves from the three receiving units, it is determined that there is a pointing gesture.

If it is determined in step S105 that there is a pointing gesture (“Y” in S105), the process proceeds to the next step S106, and the reflected wave data of the three receiving units shown in the A1 area of FIG. 10 is sent to the next step. . If the pointing gesture is not recognized (“N” in S105), the previous step is repeated until it is recognized.
In step S106, the initial motion wave of the reflected wave data in the A1 region in FIG. 10 is interpreted as the fingertip B1, and the distance from the time from when the transmitted wave is irradiated until it returns to each of the three receiving units to the fingertip is calculated. . Based on these three distances, the position information calculation unit 112d obtains the spatial position information (x, y, z) of the fingertip = monitoring target point based on the three-point distance measuring method.
In step S107, the fingertip = the monitoring target point detection unit 112c is compared with the wavelength of the initial reflected wave data of the three receiving units and the wavelength of the pulse transmission wave, and the presence or absence of the Doppler effect is checked. Detects the movement of a point back and forth.

If the deviation direction matches with a predetermined number or more of the pulse waves of each reception unit at the fingertip = monitoring target point and the speed exceeds the threshold value (“Y” in step S108), the Doppler command signal in step S111 Is generated.

When the Doppler command signal is input to the input signal generation unit 112g, the input of the pointing signal to the input signal generation unit 112g is interrupted, and the pointer coordinates (x, y) remain at a position where the command is executed for a certain time. In addition, a time lag is provided in the input time of the pointing signal and the Doppler command signal so that the pointer coordinates do not fluctuate in the short time before the Doppler command is recognized, and the Doppler effect is detected after the fingertip = monitored point starts moving. It is preferable to exclude the movement of the finger up to the point from the pointing operation. On the other hand, when the Doppler effect is not detected (“N” in step S108), the normal pointing operation is continued (S109).

In step S110, the pointing position information input from the pointing signal generation unit in step S109 and the Doppler command signal in step S111 are combined, and an input signal of a Doppler command at a specific pointer position (x, y) is output. Is provided to the notebook personal computer 1 through the OS and other applications running on the computer.

(Action / Effect)
According to the present embodiment, the spatial position of the fingertip = monitored point and its displacement can be detected based on the synchronized transmitted wave and reflected wave, and an input signal such as movement of the pointer or other operations can be output. In addition, an operator's pointing operation can be input without using a device such as a mouse, a touch pad, or a touch panel that is operated by direct contact. In this case, the so-called Doppler effect is monitored by comparing changes in the wavelength of the transmitted wave and its reflected wave, so that the approach of the target object can be performed easily and accurately without performing heavy processing such as image analysis. It is possible to detect the separation and the moving speed thereof, and it is possible to accurately detect a minute and quick movement of a fingertip or the like and convert it into a command.

Also, for other commands that are not covered only by Doppler commands, adding a single button operation and combining it with a pointing gesture will allow you to cover almost all commands on the current mouse, touchpad, and touch panel. . Furthermore, by adding voice commands, it is possible to meet various specifications that meet the needs of the operator.

In this embodiment, by obtaining the three-dimensional position of the fingertip = monitoring target point by the three-point distance measurement method, the spatial position information of the fingertip = monitoring target point that is not inferior to the distance image can be acquired by at least three receiving sensors. . Further, by increasing the number of receiving units, and by increasing the number of transmitting units linked to each receiving unit, the accuracy of fingertip = monitoring target point detection can be increased. In this embodiment, the most basic setting of one transmitting unit and three receiving units is adopted. However, in the modification example of the smartphone shown in FIG. 18, there are three pairs of receiving / transmitting units linked to the same position. Adopt more advanced settings.

In addition, instead of the three-point distance measurement method, it is also possible to recognize a pointing gesture using a distance image, acquire spatial position information of the fingertip = monitoring target point, and perform pointing and command operations in combination with a 3D touch panel and Doppler commands. If the distance image is used for other purposes, it may be diverted.

In this embodiment, in order to perform a statistical cross check that is adopted only when independent information provided from a plurality of pulse reflected waves of three different receiving units is matched in pointing gesture recognition and Doppler effect detection. The possibility of misidentification is kept low. As described above, the three reception points have the most basic specifications. If the reception unit and the pulse wave density increase, the number of cross checks increases, and the accuracy of determination further improves.

In the present embodiment, the transmitting unit irradiates a transmitted wave in a space where there are no objects to be monitored and an object other than the operator in the vicinity, and the receiving unit is configured to receive only reflected waves from the space from the beginning. Unnecessary signal = noise is suppressed. In addition, filtering and fingertip gesture recognition are performed, and the objectives of fingertip = monitoring target point detection and Doppler effect detection are achieved while selecting only necessary and minimum information. For this reason, unlike the conventional gesture GUI, a huge amount of data processing and complicated image processing by a high-spec device are not required, so that the load on the CPU can be greatly reduced.

In this embodiment, the combination of the 3D touch panel and the Doppler command allows the operator to freely perform a natural and free operation by freely using the space existing in front of the display screen of the information processing terminal. In the 3D touch panel, the size of the virtual operation screen VP changes depending on the distance from the display screen of the fingertip = monitoring target point, so if you want to move the pointer greatly, bring your finger to the screen, and if you want to perform more precise operations, You only need to move back backwards. The operability can be improved by combining with a Doppler command that produces an effect of actual touch by a touching gesture.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, a pointer is represented on a display screen on an extension line in the pointing direction based on the three-dimensional shape of the finger in a distance image that is a distance distribution to the object surface. FIG. 12 is a block diagram illustrating functional modules constructed on the CPU of the operation input system according to the second embodiment. In the present embodiment, the same components as those in the first embodiment described above are denoted by the same reference numerals, and the functions and the like are the same unless otherwise specified, and the description thereof is omitted.

(Operation input system configuration)
As shown in FIG. 12, the signal processing unit 112 according to the present embodiment includes a columnar detection unit 112i and a vector determination unit 112h as characteristic components.
In the present embodiment, one receiving unit 6a provided at the lower right of the display screen 5b is a so-called range image camera. A distance image camera is a semiconductor chip such as a CCD that captures an image, and measures and images the arrival distance of reflected waves received by a number of sensor elements to generate a so-called distance image.

The role of the pointing gesture recognition unit 112b is the same as that of the first embodiment, but in this embodiment, the pointing gesture is extracted based on the shape obtained from the image information. As shown in FIG. 14, since the finger directed to the display screen can be imitated in a long and narrow cylindrical shape, when a cylindrical shape having a predetermined diameter facing the display screen is recognized in the image data after filtering. Recognize that there is a fingertip gesture.

When looking at the pointing gesture in the distance image, a shadow is created on the side opposite to the direction of the transmitted wave irradiation, so the fingertip gesture is actually recognized on the basis of an elongated rectangle projected on the xz plane, that is, the keyboard and a horizontal plane. The distance image data that has been recognized with the pointing gesture in this manner is input to the columnar detection unit 112i.

In the columnar detection unit 112i, when a pointing gesture is recognized, based on the input distance image data, from the image of the portion that does not include a shadow (FIG. 15A), FIG. 14 and FIG. A cylindrical shape such as Ob is geometrically approximated and drawn three-dimensionally.

The vector determination unit 112h creates an approximate expression of the axis vector V1 (FIGS. 13 to 15) of the columnar shape Ob (FIGS. 14 and 15B) detected by the columnar detection unit 112i. The operation surface setting unit 112e sets a 3D grid with the display screen 5b as the XY axis and the direction orthogonal thereto as the Z axis in the monitoring space area. The 3D grid set by the operation surface setting unit 112e is input as a framework to the position information calculation unit 112d.

The position information calculation unit 112d calculates the coordinate position (x, y) where the approximate straight line of the vector V1 expressed in the 3D grid intersects the display screen 5b, and uses this as the pointer position to the pointing signal generation unit 112f. Deliver. Based on this, the pointing signal generator 112f generates an input signal that specifies the position and movement of the pointer.

In the present embodiment, if the second pointing gesture is recognized in the above process, it is captured as a subcommand signal and the content of the input signal is changed. For example, as shown in FIG. 16, if a gesture such as raising a little finger is performed in combination with an operation with a pointing finger with an index finger, this is determined as a subcommand signal, and a subcommand dialog is displayed on the screen. The range of selection operations by the operator can be expanded. This corresponds to the sub-command key (1a in FIG. 1) in the first embodiment. In this embodiment, this information is replaced with a gesture by making use of the abundant amount of information in the distance image.

(Operation input procedure)
The operation input method of the present invention can be implemented by operating the operation input system according to the present embodiment described above. FIG. 17 is a flowchart showing the procedure of the operation input method according to this embodiment. Although the basic process is the same as in the first embodiment, a pointing operation is performed based on a pointing gesture vector of distance image analysis instead of the position information of the fingertip = monitoring target point. In the following, the same contents as in the first embodiment are omitted, and only the steps unique to this embodiment are described.

In step S202, the object is monitored with a distance image camera, and in step S204, the presence or absence of a pointing gesture is determined based on the distance image data.
In step S205, if it is determined in the previous step that there is a pointing gesture, the columnar detection unit 112i geometrically analyzes the columnar shape Ob shown in FIGS. 14 and 15 based on the distance image data of the pointing gesture. Approximate.
In step S206, the vector determination unit 112h calculates the direction of the axis of the columnar shape Ob detected by the columnar detection unit 112i, that is, the approximate expression of the vector V1, as shown in FIGS.

In step S207, the position information calculation unit 112d calculates the coordinate position where the extended line of the vector V1 intersects the display screen 5b, and transfers this to the pointing signal generation unit 112f. Based on this information, the pointing signal generator 112f generates a pointing signal for designating the pointer position and movement (S209), and inputs the pointing signal to the input signal generator 112g.

The processes related to Doppler command generation in Step S208 and Step S211 are basically the same as those in the first embodiment. In this embodiment, the reflected wave sample for comparing the transmitted wave and the wavelength required for detecting the Doppler effect is directly extracted from the distance image data, not from the fingertip after fingertip gesture recognition = the initial motion wave corresponding to the monitoring target point. It is different in point to do.

In the first embodiment, the subcommand key is used. In this embodiment, the subcommand signal is generated by the second pointing gesture using the abundant amount of information in the distance image. When it is determined that the gesture of the subcommand is recognized, a process for displaying the subcommand menu is executed, and then a pointing signal for moving the pointer on the subcommand menu is generated. Thereafter, in step S210, based on the position information, the signal processing unit 112 generates and outputs an input signal related to the coordinate position on the display screen 5b.

[Example of change]
The above description of the embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made according to the design and the like as long as they do not depart from the technical idea of the present invention.

In the first and second embodiments described above, the notebook personal computer is used as the information processing terminal device, but the operation input program of the present invention is installed in the smartphone 1 '. In this modified example, as shown in FIG. 18, the transmitting / receiving unit is arranged in a triangle toward the operator side on the upper left of the display screen 5b. There are the same number of transmitters at the same place as the three receivers, and the pairs of transmitters and receivers work together to measure the distance from the fingertip to the monitoring target point at each location. Since the transmitter / receiver is located at the same location, the accuracy of distance measurement and the accuracy of the position information of the fingertip = monitored point are high. By adopting such a transmitter / receiver configuration, excellent operability can be realized even on a small screen such as a smartphone.

This change example assumes that you have a smartphone with your left hand and perform a pointing gesture with your right index finger, or a smartphone with your right hand and a pointing gesture with your right thumb. Unlike a notebook personal computer, smartphones have a limited number of keys that can be used freely, so a subcommand button 1b is newly provided at the lower left side. The function of this button 1b is the same as the subcommand key of the first embodiment shown in 1a of FIG. 1, and various commands can be executed in addition to the Doppler command by combining this button operation and the pointing gesture operation. .

VP ... Virtual operation screen W1 ... Transmitted wave W2 ... Reflected wave 1 ... Notebook personal computer 1 '... Smartphone 1a ... Subcommand key 3a ... Fingertip (monitoring point)
3b ... fingertip (second monitoring target point)
6a, 6a0 to 2 ... receiving unit 10 ... CPU bus 11 ... CPU
DESCRIPTION OF SYMBOLS 12 ... Memory 13 ... Communication interface 14 ... Memory | storage device 15 ... Output interface 16 ... Input interface 51 ... Pointer 111 ... Synchronization processing part 112 ... Signal processing part 112a ... Filtering part 112b ... Pointing gesture recognition part 112c ... Fingertip = inspection object inspection Output unit 112d ... Position information calculation unit 112e ... Operation surface setting unit 112f ... Pointing signal generation unit 112g ... Input signal generation unit 112h ... Vector determination unit 112i ... Columnar detection unit 113 ... Contact / separation detection unit 113a ... Doppler effect detection unit 113b ... Doppler command generator

Claims (12)

1. An operation input system for inputting a coordinate position on a display screen of an information processing device and an operation command related to the coordinate position, and a part or all of a visual field range visible on the display screen by using electromagnetic waves or sound waves as a transmission wave A transmitter for transmitting to a surveillance space area including
   A receiving unit that receives the transmitted wave reflected by the object as a reflected wave;
   Compared with the wavelength of the transmitted wave and the wavelength of the reflected wave, a contact / separation detection unit that detects the movement of the object in the direction approaching or leaving the transmission unit or the reception unit and its speed,
   Based on the transmitted wave transmitted and the received reflected wave, the position information on the three-dimensional space of the object that generated the reflected wave and position information regarding the displacement thereof are detected, and the position information and the contact / separation detection are detected. A signal processing unit that generates an input signal related to a coordinate position on the display screen based on a detection result by the unit and provides the input signal to the information processing device;
   An operation input system comprising:
2. At least one of the transmitters is spaced apart from each other, at least three of the receivers are provided, or at least three pairs of transmitters and receivers to be linked are provided,
   The signal processing unit grasps the spatial position of the fingertip = monitoring target point by a three-point ranging method based on the time length from the transmission of the transmitted wave to the reception of the reflected wave, and based on this, the input A position information calculation unit for generating a signal;
   The operation input system according to claim 1, wherein the contact / separation detection unit detects a movement and a speed of the monitoring target point as the object.
3. Based on a distance image that is a distribution of distances between the receiving unit and the object surface, a monitoring target inspection that detects a coordinate point having the shortest distance to the receiving unit as a monitoring target point in the monitoring space region It is further equipped with a branch
   The operation input system according to claim 1, wherein the contact / separation detection unit detects a movement and a speed of the monitoring target point as the object.
4. The signal processing unit
   In the monitoring space area, set innumerable continuous virtual operation surfaces parallel to each other and parallel to the display screen or having a predetermined angle and the same shape or similar shape to the display screen, and from the receiving unit An operation surface setting unit that changes the size of the virtual operation surface according to the distance to the monitoring target point;
   The operation input system according to claim 2, wherein the input signal is generated according to a coordinate position of the monitoring target point on the virtual operation surface.
5. Extracting only the reflected wave from the coordinate point where the distance to the receiving unit is the shortest to a predetermined distance, further comprising a filtering unit to send to the subsequent processing flow,
   The operation input system according to claim 2, wherein the signal processing unit generates the input signal based on the reflected wave extracted by the filtering unit.
6. In the three-point distance measurement method, the signal processing unit recognizes the operator's pointing gesture based on the morphological characteristics of the amplitude-time profile of the reflected wave and in the distance image based on the three-dimensional shape of the finger. 4. The operation input system according to claim 2, wherein an initial motion wave of the amplitude-time profile or a point closest to the receiving unit is detected as a fingertip = a monitoring target point. 5.
7. The operation input system according to claim 2 or 3, wherein the signal processing unit detects an arbitrary operation by an operator and changes the content of the input signal according to the presence or absence of the arbitrary operation. .
8. In the distance image, a columnar shape imitating the shape of a finger directed to the display screen in the pointing gesture is detected, and a coordinate position where an extension line of the detected axis of the columnar shape intersects the display screen is determined. It further includes a columnar detection unit and a vector determination unit that are generated as input signals to be specified,
   The operation input system, wherein the contact / separation detection unit detects the movement and speed of the monitoring target point.
9. In the pointing gesture recognition, based on the distance image, a second pointing gesture with a distance to each receiving unit in the monitoring space area is detected,
   The operation input system according to claim 3 or 8, wherein the signal processing unit changes the content of the input signal in accordance with the presence or absence of the second pointing gesture.
10. The transmitting unit irradiates the pulse wave transmitted from the receiving unit with directivity toward an unobstructed space,
    2. The operation input system according to claim 1, wherein the receiving unit has directivity for blocking re-reflection from the vicinity and selectively receiving only a direct reflected wave from a monitoring object in the air.
11. An operation input method for inputting an operation command related to a coordinate position on the display screen of the information processing apparatus and the coordinate position,
    While transmitting the electromagnetic wave or sound wave as a transmitted wave from the transmitter to a monitoring space area including part or all of the visual field range where the display screen is visible, the transmitted wave reflected by the object is used as a reflected wave. A step of receiving by the receiver;
    Based on the transmitted wave and the received reflected wave, the position of the object that generated the reflected wave in the three-dimensional space and position information regarding its displacement are detected, and signal processing is performed based on the position information. A unit that generates an input signal related to a coordinate position on the display screen and provides the input signal to the information processing apparatus;
    By comparing the wavelength of the transmitted wave and the wavelength of the reflected wave, the contact / separation detection unit detects the movement and speed of the object in a direction approaching or leaving the transmission unit or the reception unit. An operation input method comprising: steps.
12. An operation input program for inputting an operation command related to a coordinate position on the display screen of the information processing apparatus and the coordinate position, the computer,
    A transmitter that transmits electromagnetic waves or sound waves as a transmitted wave to a monitoring space area including a part or all of the visual field range in which the display screen is visible,
    A receiving unit that receives the transmitted wave reflected by the object as a reflected wave;
    Based on the transmitted wave and the received reflected wave, the position information on the three-dimensional space of the object that generated the reflected wave and position information regarding the displacement are detected, and the display is performed based on the position information. An input signal related to a coordinate position on the screen is generated and provided to the information processing device, and the wavelength of the transmitted wave and the wavelength of the reflected wave are compared, and the signal is transmitted to the transmitter or the receiver. An operation input program that functions as a contact / separation detection unit that detects movement and speed of the object in a direction of approaching or leaving.

Images