WO2022191276A1

WO2022191276A1 - Operation input device, operation input method, and program

Info

Publication number: WO2022191276A1
Application number: PCT/JP2022/010548
Authority: WO
Inventors: 堪亮坂本
Original assignee: 株式会社ネクステッジテクノロジー
Priority date: 2021-03-10
Filing date: 2022-03-10
Publication date: 2022-09-15
Also published as: JP7452917B2; JPWO2022191276A1

Abstract

In the present invention, a real camera 10 is fixed in place so that a sight line direction is inclined with respect to an indicated direction of an indicating body 321. A three-dimensional data generation unit 212 of an operation input device 20 generates three-dimensional data on the basis of output data of the real camera 10. A coordinate conversion unit 214: assumes a virtual camera 312 with a sight line direction rotated by an angle of θ from the sight line direction of the real camera 10, using a specific point 313 as the center; and converts the three-dimensional data of the real camera 10 to virtual three-dimensional data of the virtual camera 31. A contact determination unit 216 determines that the indicating body 321 has contacted a virtual touch panel 311 if the shortest distance from the virtual camera 321 to the indicating body 321 calculated on the basis of the virtual three-dimensional data is not greater than the distance from the virtual camera 312 to the virtual touch panel 311.

Description

Operation input device, operation input method and program

The present invention relates to an operation input device, an operation input method, and a program for inputting information related to operator's operations.

In recent years, many operation input devices have been developed that allow operators to input operations to information processing devices without contact. A non-contact operation input device is attracting attention as a means that can lighten the operator's operation burden and that an information processing device can be operated even during work at a surgical site, a cooking site, or the like.

Such an operation input device includes, for example, a camera that photographs an indicator such as an operator's finger, measures the three-dimensional position of the indicator, and performs an operation input to an information processing apparatus based on the three-dimensional position. (For example, Patent Document 1).

The interactive projector described in Patent Document 1 uses two cameras to detect the contact of a self-luminous indicator on the projection screen based on the light emission pattern, and to detect the contact of the non-luminous indicator on the projection screen by a position detection unit. It is explained that the detection accuracy of the contact of the pointer to the screen surface can be improved by detecting by .

JP 2016-186676 A

In the interactive projector described in Patent Document 1, in order to detect the three-dimensional position of the indicator, it is necessary to install two cameras above the head and direct the camera line-of-sight direction vertically downward. When this configuration is applied to a normal information processing device, it is necessary to install the camera above the front of the display. there were.

On the other hand, some conventional information processing devices such as personal computers and tablet terminals are equipped with a camera at the edge of the display that takes pictures in the direction perpendicular to the screen. When such a camera is used, there is a problem that the three-dimensional position of the tip of the pointer in the area near the display has a large error, and the position indicated by the pointer cannot be specified accurately.

The present invention has been made in view of the above circumstances, and aims to provide an operation input device, an operation input method, and a program capable of accurately detecting non-contact operation input with a simple configuration.

In order to achieve the above object, the operation input device of the present invention includes:
a three-dimensional data generation unit that generates three-dimensional data based on output data of a real camera that is fixed with respect to a display surface of a display means and whose line-of-sight direction is inclined with respect to the pointing direction of the indicator;
The first straight line is rotated by a predetermined angle about a point on the first straight line extending in the line-of-sight direction of the real camera and located at a predetermined first distance from the real camera. a coordinate transformation unit for transforming the three-dimensional data in the coordinate space of the real camera into virtual three-dimensional data in the coordinate space of the virtual camera, assuming a virtual camera whose viewing direction is the direction of the second straight line;
When a surface separated from the virtual camera by a predetermined second distance in a direction parallel to the second straight line is a virtual touch panel, the virtual touch panel is calculated based on the virtual three-dimensional data, and the When a third distance, which is the shortest distance from the virtual camera to the pointer, is compared with the second distance from the virtual camera to the virtual touch panel, and the third distance is equal to or less than the second distance a contact determination unit that determines that the pointer has come into contact with the virtual touch panel;
an operation input signal generation unit that generates an operation input signal based on the virtual three-dimensional data when the contact determination unit determines that the pointer has made contact.

According to the present invention, it is possible to accurately detect non-contact operation inputs with a simple configuration in which a camera for generating three-dimensional data is installed with the line-of-sight direction inclined.

2 is a block diagram showing the hardware configuration of the operation input system according to Embodiment 1; FIG. 2 is a functional block diagram showing the functional configuration of the operation input device according to Embodiment 1; FIG. FIG. 2 is a schematic side view of display means, a real camera, and a virtual touch panel according to Embodiment 1; FIG. 4 is a diagram showing the positional relationship between a real camera and a virtual camera; 4 is a flowchart of operation input processing according to Embodiment 1; FIG. 4 is a diagram showing installation positions of real cameras; FIG. 10 is a schematic side view of display means, a real camera, a virtual touch panel, and a hover surface according to Embodiment 2; FIG. 11 is a perspective view showing a schematic configuration of display means, a real camera, a virtual touch panel, and an instruction effective area according to Embodiment 3; 10 is a flowchart of operation input processing according to Embodiment 3; FIG. 14 is a diagram showing the arrangement of real cameras according to Embodiment 4; It is the schematic which looked at the display means, the transparent plate, and the virtual touch panel which concern on other embodiment from the side.

(Embodiment 1)
Embodiment 1 of the present invention will be described in detail with reference to the drawings.

The operation input system 1 according to the present embodiment is an information processing system that performs processing based on an operation input signal generated by determining an operation using an operator's finger or other indicator. FIG. 1 is a block diagram showing the hardware configuration of an operation input system 1 according to the first embodiment. The operation input system 1 comprises a real camera 10 and an operation input device 20, as shown in FIG.

The real camera 10 is a 3D camera (3 Dimensional Camera) that outputs data representing 3D information. Any conventional method may be used to obtain the three-dimensional information, and for example, a stereo method may be used. The real camera 10 is preferably capable of wide-angle photography, and is equipped with a wide-angle FOV 84° lens, for example. Although the actual camera 10 externally connected to the operation input device 20 will be described in the first embodiment, a dedicated camera built into the operation input device 20 may be used.

The operation input device 20 is an arbitrary information processing terminal, and may be a general-purpose information processing terminal such as a personal computer, a smartphone, or a tablet-type terminal in which a program for operation input processing is installed, or a dedicated terminal. As shown in FIG. 1, the operation input device 20 includes a CPU (Central Processing Unit) 21, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 23, a communication interface 24, a storage unit 25, a display Means 26 are provided.

The operation input system 1 detects the operator's virtual touch operation on a virtual touch panel imaginary in front of the display surface of the display means 26 of the operation input device 20 based on the output data of the real camera 10 .

The display means 26 of the operation input device 20 is an arbitrary display device that displays information such as images and characters. The display means 26 may be a tangible display including a liquid crystal display or an organic EL (Electro-Luminescence) display built in or external to a personal computer, a smartphone, a tablet terminal, etc. It may be an intangible or fluid type display device such as a display.

The indicator is an object that can be moved by the operator in order to specify the position on the display screen of the display means 26, and has an elongated shape extending in the pointing direction. The indicator is, for example, an operator's finger or a pointing stick. The arrangement and functional configuration of the real camera 10 and the operation input device 20 will be described in detail below.

The real camera 10 is fixed with respect to the display surface of the display means 26 of the operation input device 20 . Also, the line-of-sight direction of the real camera 10 (the direction toward the center of the field of view) is inclined with respect to the pointing direction of the indicator. For example, when the display means 26 is a liquid crystal display of a personal computer and the real camera 10 is fixed to the end of the liquid crystal display, the line-of-sight direction is the direction toward the area in front of the center of the display surface of the display means 26. Slanted with respect to the direction perpendicular to the plane. In the following description of the first embodiment, the case where the display means 26 is a liquid crystal display of a personal computer and the real camera 10 is fixed to the central upper end portion of the liquid crystal display will be described. In this case, the line-of-sight direction of the real camera 10 is obliquely downward with respect to the direction perpendicular to the display surface of the display means 26 .

The CPU 21 of the operation input device 20 controls each component of the operation input device 20, and executes each process including the operation input process by executing the programs stored in the ROM 23 and the storage unit 25. The RAM 22 is a memory in which data can be read and written at high speed, and temporarily stores data output from the real camera 10 and data read from the storage unit 25 for data processing executed by the CPU 21 . The ROM 23 is a read-only memory that stores programs for processing executed by the CPU 21, setting values, and the like.

The communication interface 24 is an interface through which data is transmitted and received with the real camera 10 . The storage unit 25 is a large-capacity storage device, and is composed of a flash memory or the like. The storage unit 25 stores the output data of the real camera 10 and the data generated by the processing of the CPU 21 . In addition, the storage unit 25 further stores programs executed by the CPU 21 . The display means 26 displays information such as images and characters generated by the CPU 21 .

By executing the operation input processing program stored in the storage unit 25, the CPU 21 and the RAM 22, as shown in FIG. It functions as a unit 214 , a pointer detection unit 215 , a contact determination unit 216 , and an operation input signal generation unit 217 .

The data acquisition unit 211 acquires output data of the real camera 10 . The three-dimensional data generation unit 212 develops the output data acquired by the data acquisition unit 211 in a three-dimensional coordinate space to generate three-dimensional data. The output data format of the real camera 10 may be any format, and the 3D data generation unit 212 executes data processing according to the output data format of the real camera 10 in order to generate 3D data in a predetermined format. do. The output data of the real camera 10 may be directly used as three-dimensional data.

The parameter acquisition unit 213 acquires parameters related to the position and orientation of the real camera 10 and the position and size of the virtual touch panel. The parameters are determined in advance by an initial setting input by the administrator of the operation input device 20 .

The coordinate conversion unit 214 converts the three-dimensional data generated based on the output data of the real camera 10 into virtual three-dimensional data corresponding to the virtual touch panel. A virtual camera is assumed in which the real camera 10 is virtually moved for conversion into virtual three-dimensional data. That is, the coordinate transformation unit 214 transforms the three-dimensional data in the coordinate space of the real camera 10 into virtual three-dimensional data in the coordinate space of the virtual camera.

The pointer detection unit 215 detects the point closest to the virtual camera as the tip of the pointer based on the virtual three-dimensional data. The contact determination unit 216 determines whether or not the tip of the pointer detected by the pointer detection unit 215 has come closer to the display unit 26 than the virtual touch panel.

Details of the three-dimensional data generation unit 212, the parameter acquisition unit 213, the coordinate conversion unit 214, the pointer detection unit 215, and the contact determination unit 216 will now be described with reference to FIGS. FIG. 3 is a schematic diagram of the real camera 10, the display means 26, and the virtual touch panel 311 viewed from the side. FIG. 4 is a diagram showing the positional relationship between the real camera 10 and the virtual camera 312. As shown in FIG.

The 3D data generation unit 212 generates 3D data based on the output data of the real camera 10 acquired by the data acquisition unit 211 . When the pointer 321 exists in the three-dimensional coordinate space of the real camera 10, the three-dimensional data generator 212 can generate three-dimensional data including the pointer 321. FIG.

In FIG. 3, when the plane on which the virtual touch panel 311 extends is defined as an extension plane P, and the plane parallel to the extension plane P that passes through a reference point that is the reference of the three-dimensional coordinates of the real camera 10 is defined as a reference plane B. , the distance between the extension plane P and the reference plane B is Pz. The distance Pz is a value within a range that is determined according to specifications such as the viewing angle of the real camera 10 or the three-dimensional coordinate measurable range, and is determined by the administrator's input of initial settings.

In Embodiment 1, the real camera 10 is fixed to the central upper end portion of the display means 26, and is oriented obliquely downward. When the line-of-sight direction of the real camera 10 is E, and the direction passing through the center of the virtual touch panel 311 and perpendicular to the extension plane P is V, the angle formed by the direction E and the direction V is θ. θ is a parameter input by the administrator according to the orientation of the real camera 10. The administrator assumes the position and size of the virtual touch panel 311, and calculates the distance Pz from the reference plane B to the virtual touch panel 311 along with the angle θ. to decide.

　Transformation from the three-dimensional data of the real camera 10 to virtual three-dimensional data is performed assuming a virtual camera 312 in which the real camera 10 is rotated and translated, as shown in FIG. The line-of-sight direction of the virtual camera 312 is defined as a direction V that passes through the center of the virtual touch panel 311 and is perpendicular to the virtual touch panel 311 . Also, it is defined that the reference point for determining the virtual three-dimensional data of the virtual camera 312 is on the reference plane B. FIG. Let S be the distance between the reference point of the real camera 10 and the specific point 313 when the intersection of the directions E and V is the specific point 313 .

The parameters that the parameter acquisition unit 213 acquires by the administrator's input are the angle θ for defining the line-of-sight direction of the real camera 10 and the distance Pz for defining the position of the virtual touch panel 311 shown in FIGS. and a distance S for defining a particular point 313 about which the rotation from the real camera 10 to the virtual camera 312 is centered. The administrator also considers the performance of the real camera 10 and determines the parameters. By setting these parameters, it is possible to specify a conversion method from three-dimensional data to virtual three-dimensional data.

In other words, a point on the first straight line extending in the line-of-sight direction E of the real camera 10, and centering on the specific point 313 located at a distance S (first distance) from the reference point of the real camera 10, the first straight line A virtual camera 312 is assumed to have a line-of-sight direction V of a second straight line rotated by an angle θ. Then, the three-dimensional data in the coordinate space of the real camera 10 is coordinate-transformed into virtual three-dimensional data in the coordinate space of the virtual camera 312 .

The virtual touch panel 311 is separated from the reference point of the virtual camera 312 by Pz (second distance) in a direction parallel to the second straight line. The shape of the virtual touch panel 311 may be any shape as long as it can correspond to the display surface of the display means 26 . In the following description, for simplification of description, a case where the display surface of the display unit 26 is rectangular and the virtual touch panel 311 is also rectangular will be described. In this case, the second straight line passes through the rectangular center point of the virtual touch panel 311, and the virtual touch panel 311 extends in a direction perpendicular to the second straight line. The second straight line passes through the reference point of virtual camera 312 and the center of virtual touch panel 311 .

The coordinate conversion performed by the coordinate conversion unit 214 will be described. The three-dimensional data generated by the three-dimensional data generation unit 212 based on the output data of the real camera 10 is, as shown in FIG. It is represented by coordinates (x, y, z) with the direction as the x-axis. When the coordinate space of the real camera 10 is rotated around a straight line passing through the specific point 313 and parallel to the x-axis, the line-of-sight direction V of the virtual camera 312 is taken as the z'-axis, and the horizontal direction of the virtual touch panel is taken as the x-axis. A coordinate space is obtained, represented by the coordinates (x, y', z').

Here, when the real camera 10 is rotated, the camera position is located at B' behind the reference plane B, so the camera position is translated along the direction V to the reference plane B. The translation distance Dz is S-Scos θ. In other words, the virtual three-dimensional data in the coordinate space by the virtual camera 312 is the coordinates (x, y', z'') shifted from (x, y', z') by Dz (Dz=S-Scos θ) in the z'-axis direction. ) In this way, the three-dimensional data in the coordinate space represented by (x, y, z) based on the output data of the real camera 10 is rotated and translated by the virtual camera 312 ( x, y', z'') can be converted into virtual three-dimensional data in a coordinate space.

The pointer detection unit 215 detects the point closest to the virtual camera 312 as the tip of the pointer 321 based on the virtual three-dimensional data converted by the coordinate conversion unit 214 . The tip of the indicator 321 is the pointing position 322 of the operator. The pointer detection unit 215 outputs virtual three-dimensional coordinates of the pointing position 322 based on the virtual three-dimensional data.

The contact determination unit 216 determines the z″ direction (the direction V ), and the contact determination unit 216 compares the distance Vz with the distance Pz (second distance) in the z″ direction from the virtual camera 312 to the virtual touch panel.

In other words, the contact determination unit 216 determines the distance Vz, which is the shortest distance from the reference plane B passing through the reference point of the virtual camera 312 to the pointing position 322 of the indicator 321, the distance Pz from the reference plane B to the virtual touch panel, compare. As shown in FIG. 4 , when Vz≦Pz, the indicated position 322 of the pointer 321 can be said to be at the same position as the virtual touch panel 311 or closer to the virtual camera 312 than the virtual touch panel 311 . At this time, it is determined that the pointer 321 touches the virtual touch panel 311 . If Vz>Pz, it is determined that there is no input from the pointer 321 to the virtual touch panel 311 .

When the contact determination unit 216 determines that the tip of the pointer 321 that is the pointing position 322 has touched, the operation input signal generating unit 217 detects the operator's operation based on the virtual three-dimensional coordinates of the pointing position 322 at that time. generates an operation input signal indicated by .

Specifically, the operation input signal generation unit 217 generates an operation input signal for displaying a cursor at a position corresponding to the virtual three-dimensional coordinates of the pointing position 322 on the display screen of the display means 26 .

After that, the operation input signal generation unit 217 generates an operation input signal for moving the cursor, selecting, moving, etc. according to the information on the temporal change of the pointing position 322 output by the contact determination unit 216 . Further, in an information processing terminal including the operation input device 20 in which an arbitrary application is installed in advance, an operation input signal for instructing execution of an application indicated by an icon displayed on the display means 26 is generated.

The operation input processing of the operation input device 20 configured in this way will be described along the flowchart shown in FIG. FIG. 5 is a flowchart showing operation input processing. The operation input process starts when the administrator of the operation input system 1 executes the operation input program.

First, the parameter acquisition unit 213 displays a parameter input screen on the display means 26. The administrator hypothesizes the position and orientation of the virtual touch panel 311 and inputs parameters for specifying the virtual touch panel 311 . The parameter acquisition unit 213 acquires parameters input in advance by the administrator (step S101).

The first parameter acquired by the parameter acquisition unit 213 is the angle θ of the line-of-sight direction E of the real camera 10 with respect to the direction V perpendicular to the extension plane P of the virtual touch panel 311 . The second parameter is a value obtained when a specific point 313 is an intersection of a second straight line passing through the center of the virtual touch panel 311 and extending in the direction V and a first straight line extending in the line-of-sight direction E of the real camera 10 . It is the distance S from the reference point of the camera 10 to the specific point 313 . The third parameter is the distance Pz from the reference plane B to the virtual touch panel 311 when the plane passing through the reference point of the real camera 10 and parallel to the virtual touch panel 311 is taken as the reference plane B. FIG. The parameters acquired by parameter acquisition unit 213 may include other parameters for specifying virtual touch panel 311 .

Based on the parameters acquired by the parameter acquisition unit 213, the coordinate transformation unit 214 determines a method of coordinate transformation from the three-dimensional data of the coordinate space of the real camera 10 to the virtual three-dimensional data of the coordinate space of the virtual camera 312 (step S102). ). The virtual camera 312 is a camera virtualized at a position where the real camera 10 is rotated by an angle θ around the specific point 313 and translated along the direction V to the reference plane B. FIG.

In the example shown in FIG. 4, the coordinate transformation unit 214 rotates the coordinate space of the real camera 10 by an angle θ around the specific point 313 and transforms the coordinate space along the direction V by a length Dz (Dz=S−Scos θ). Assuming a parallel coordinate space of the virtual camera 312, a coordinate transformation method for transforming the three-dimensional data of the coordinate space of the real camera 10 into virtual three-dimensional data of the coordinate space of the virtual camera 312 is determined (step S102).

The data acquisition unit 211 acquires the output data of the real camera 10 (step S103). The three-dimensional data generator 212 develops the output data of the real camera 10 in a three-dimensional coordinate space to generate three-dimensional data. (Step S104: 3D data generation step).

Thereafter, the coordinate transformation unit 214 transforms the three-dimensional data in the coordinate space of the real camera 10 into virtual three-dimensional data in the coordinate space of the virtual camera 312 using the coordinate transformation method determined in step S102 (step S105: coordinates conversion step).

Next, the pointer detection unit 215 detects the point closest to the virtual camera 312 as the pointed position 322 of the pointer 321 based on the virtual three-dimensional data, and calculates the virtual three-dimensional coordinates of the pointed position 322 . Then, the contact determination unit 216 calculates the shortest distance Vz from the reference plane B to the indicated position 322 based on the virtual three-dimensional coordinates of the indicated position 322 (step S106). The contact determination unit 216 compares the distance Pz from the reference plane B to the virtual touch panel 311 with the distance Vz to the indicated position 322 (step S107), and determines whether or not the distance Vz is equal to or less than the distance Pz (step S108: contact determination step).

If the distance Vz is equal to or less than the distance Pz (step S108: Yes), the coordinates on the virtual touch panel 311 are calculated based on the virtual three-dimensional coordinates of the pointing position 322 obtained in step S106, and the operation input signal generation unit 217 (step S109). If the distance Vz is greater than the distance Pz (step S108: No), the process returns to step S103.

After outputting the coordinates on the virtual touch panel 311 obtained in step S109, if the administrator issues an instruction to end the operation input process (step S110: Yes), the process ends. If there is no end command (step S110: No), the process returns to step S103.

The operation input signal generation unit 217 generates an operation input signal based on the time change of the coordinates on the virtual touch panel 311 obtained in step S109 (operation input signal generation step) and outputs it. The operation input signal is a signal for instructing selection, movement, or the like by moving a cursor, or a signal for instructing execution of an application installed in advance.

In this way, the operation input device 20 converts the three-dimensional data based on the output data of the real camera 10 into virtual three-dimensional data by the virtual camera 312, and compares the pointing position 322 with the position of the virtual touch panel 311. , the touch operation of the pointer 321 on the virtual touch panel 311 is detected.

As described above, in Embodiment 1, the real camera 10 is fixed with respect to the display means 26 with the line-of-sight direction directed in a direction inclined with respect to the pointing direction of the indicator 321 . A three-dimensional data generation unit 212 of the operation input device 20 generates three-dimensional data based on the output data of the real camera 10 . The coordinate transformation unit 214 assumes a virtual camera 312 whose line-of-sight direction is a direction V obtained by rotating the line-of-sight direction E of the real camera 10 by an angle θ about a specific point 313, and converts three-dimensional data from the real camera 10 into virtual camera 312. 312 into virtual three-dimensional data. Then, based on the virtual three-dimensional data, the pointer detection unit 215 detects the point closest to the virtual camera 312 as the pointing position 322, and the contact determination unit 216 moves away from the virtual camera 312 in the direction V by the distance Pz. Assuming that the surface is the virtual touch panel 311, the distance Vz from the virtual camera 312 to the pointing position 322 in the direction V is compared with the distance Pz from the virtual camera 312 to the virtual touch panel 311, and if the distance Vz is the distance Pz or less, It is determined that the pointer 321 touches the virtual touch panel 311 at some point. This makes it possible to accurately detect non-contact operation input with a simple configuration in which the real camera 10 is installed with the line-of-sight direction inclined.

In the first embodiment, the case where the real camera 10 is installed at the central upper end portion of the display means 26 has been described, but as shown in FIG. may When the coordinate transformation unit 214 determines the coordinate transformation method, by changing the direction of rotation from the real camera 10 to the virtual camera 312 , the 3D data from the real camera 10 at another location can also be converted to the virtual 3D data from the virtual camera 312 . It can be transformed into dimensional data.

Even if the real camera 10 is installed at a position other than the central upper end, the line-of-sight direction of the real camera 10 is inclined with respect to the pointing direction of the indicator 321 . In the first embodiment described above, the rotation angle θ is set from the real camera 10 to the virtual camera 312 with a straight line parallel to the x-axis as the rotation axis, and the coordinate transformation represented by this rotation is performed. On the other hand, for example, when the position of the real camera 11 in FIG. , and coordinate transformation represented by this rotation is performed. In addition, when it is at the position of the real camera 12, a rotation angle with a straight line parallel to the x-axis as a rotation axis and a rotation angle with a straight line parallel to the y-axis as a rotation axis are set, and a straight line parallel to the x-axis is set. Coordinate transformation represented by rotation about a rotation axis and rotation about a straight line parallel to the y-axis may be performed.

That is, the rotation from the real camera 10 to the virtual camera 312 is rotation about a straight line passing through the specific point 313 and parallel to the extending direction of the virtual touch panel 311. Performs coordinate transformation represented by a rotation of . Thereby, a virtual touch operation on the virtual touch panel 311 can be detected regardless of the installation position of the real camera 10 .

In addition, if the position of the real camera 10 can capture the pointer 321 from a direction that is inclined with respect to the pointing direction of the pointer 321, the operator operating toward the virtual touch panel 311 as well as the outer periphery of the virtual touch panel 311 It may be up, down, left, right, or behind. In this case, by setting the angle θ to a value of 90° or more, coordinate conversion can be performed as appropriate, and a virtual touch operation on the virtual touch panel 311 can be detected.

(Embodiment 2)
Embodiment 2 of the present invention will be described in detail with reference to FIG. FIG. 7 is a schematic side view of the display unit 26, the real camera 10, the virtual touch panel 311, and the hover surface 315 according to the second embodiment.

The operation input system 1 according to the second embodiment has the same configuration as that of the first embodiment, and executes the same operation input process. The difference is that the hover plane 315 is assumed.

The hover surface 315 is a surface parallel to the virtual touch panel 311 and is located at a predetermined distance Hz from the virtual touch panel 311 . The distance Hz between the virtual touch panel 311 and the hover surface 315 can be set by the administrator, and is a value between 5 cm and 10 cm, for example.

The functions and operations of the data acquisition unit 211, the three-dimensional data generation unit 212, the coordinate conversion unit 214, and the pointer detection unit 215 are the same as in the first embodiment. The parameter acquisition unit 213 acquires the distance Hz from the virtual touch panel 311 to the hover plane 315 in addition to the parameter angle θ, the distance S, and the distance Pz in the first embodiment.

Before comparing the distance Vz from the reference plane B to the indicated position 322 with the distance Pz to the virtual touch panel 311, the contact determination unit 216 determines the distance from the reference plane B to the hover plane 315 (fourth distance: (Pz+Hz )). When the indicated position 322 approaches and passes through the hover surface 315 and the distance Vz becomes equal to or less than the distance (Pz+Hz), the operation input signal generation unit 217 displays a display indicating that the virtual touch panel 311 is approaching. Generates an operation signal to perform.

For example, the operation input signal generator 217 may cause the display means 26 to display a cursor when the distance Vz becomes equal to or less than the distance (Pz+Hz). This allows the operator to recognize the current pointing position. Further, the size or shape of the cursor may be changed to display the distance to the virtual touch panel 311 recognizable. For example, as the pointing position 322 crosses over the hover surface 315 and approaches the virtual touch panel 311, the cursor may be made smaller step by step, or the color may be made darker step by step.

With a conventional touch panel, the operator can visually recognize the distance to the touch panel, but in the case of the virtual touch panel 311 in space, the distance to the virtual touch panel 311 cannot be visually recognized. In contrast, in the configuration in which the hover surface 315 is provided in front of the virtual touch panel 311 according to the second embodiment, the instruction position 322 approaches the virtual touch panel 311 and the hover surface 315 is displayed before the operator touches the virtual touch panel 311 . , the area where the cursor is located and the distance to the virtual touch panel can be recognized. Also, the positional error when the virtual touch panel 311 is touched can be reduced. Thereby, operability in space can be improved.

As described above, in the second embodiment, the hover surface 315 is assumed to be spaced apart from the virtual touch panel 311 in the direction opposite to the display surface of the display means 26, and the contact determination unit 216 instructs Passage of hover surface 315 is determined prior to touching virtual touch panel 311 at location 322 . Then, the operation input signal generation unit 217 generates an operation signal for displaying that the pointed position 322 is approaching the virtual touch panel 311 . This makes it possible for the operator to recognize the area where the pointed position 322 exists and the distance to the virtual touch panel 311 .

(Embodiment 3)
Embodiment 3 of the present invention will be described in detail with reference to FIGS. FIG. 8 is a perspective view showing a schematic configuration of display means 26, real camera 10, virtual touch panel 311, and instruction effective area 330 according to the third embodiment. FIG. 9 is a flowchart of operation input processing according to the third embodiment.

The operation input system 1 according to Embodiment 3 has the same configuration as Embodiments 1 and 2, but the functions of the parameter acquisition section 213 and the pointer detection section 215 are partially different. In the operation input system 1 according to the third embodiment, the operation input device 20 assumes an instruction effective area 330 in which an instruction by the indicator 321 is valid, and detects only the indicator 321 inside the instruction effective area 330. It is valid, and the indicator 321 is not detected outside the indicator effective area 330 .

The instruction effective area 330 is an area limited from the space area in front of the display means 26, as shown in FIG. The shape of the boundary of the instruction effective area 330 is arbitrary, and, for example, as shown in FIG. In addition, it may have a cylindrical shape or an elliptical cylindrical shape whose side surface is perpendicular to the virtual touch panel 311 , or a shape in which the area expands or narrows as the distance from the virtual touch panel 311 increases or decreases. In the present embodiment, a case where the boundary of instruction effective area 330 is a rectangular parallelepiped will be described.

Note that a hover surface 315 similar to that in the second embodiment may be further included in the instruction effective area 330, and it is determined whether or not the pointer 321 has passed through the hover surface 315 before the pointing position 322 of the pointer 321 touches the virtual touch panel 311. You may make it

The parameter acquisition unit 213 acquires parameters for specifying the instruction effective area 330 in addition to parameters for specifying the virtual touch panel 311 . The parameters for specifying the virtual touch panel 311 are the angle θ of the line-of-sight direction E of the real camera 10 as in Embodiment 1, the distance S from the reference point of the real camera 10 to the specific point 313, and the distance S from the reference plane B It is the distance Pz to the virtual touch panel 311 .

The parameter for specifying the instruction effective area 330 is an arbitrary parameter indicating the boundary of the instruction effective area 330 in the coordinate space of the virtual camera 312. For example, the virtual three-dimensional coordinate value indicating the boundary of the instruction effective area 330 is The upper and lower bounds or coefficients of a three-dimensional function representing the bounding surface of the indication effective area 330 . The parameters acquired by parameter acquisition section 213 may include other parameters for specifying virtual touch panel 311 or instruction effective area 330 .

The pointer detection unit 215 detects the closest object from the virtual camera 312 based on the virtual three-dimensional data within the instruction effective area 330 specified by the parameter, among the virtual three-dimensional data from the virtual camera 312 output by the coordinate transformation unit 214 . A point is detected as the indicated position 322 of the pointer 321 .

Other configurations of the operation input device 20 are the same as in the first and second embodiments. Operation input processing of the operation input device 20 configured in this manner will be described with reference to the flowchart shown in FIG. In FIG. 9, the processes assigned the same reference numerals as those in FIG. 5 are the same as those in the first embodiment.

First, the parameter acquisition unit 213 displays a parameter input screen on the display means 26. The administrator hypothesizes the position and orientation of the virtual touch panel 311 and inputs parameters for specifying the virtual touch panel 311 and parameters for specifying the instruction effective area 330 . The parameter acquisition unit 213 acquires parameters input in advance by the administrator (step S201).

The parameters for specifying the virtual touch panel 311 acquired by the parameter acquisition unit 213 are the angle θ of the line-of-sight direction E of the real camera 10 as in Embodiment 1, and the distance from the reference point of the real camera 10 to the specific point 313. S and a distance Pz from the reference plane B to the virtual touch panel 311 . A parameter for specifying the instruction effective area 330 is a parameter indicating the boundary of the instruction effective area 330 in the coordinate space of the virtual camera 312 . The parameters acquired by parameter acquisition section 213 may include other parameters for specifying virtual touch panel 311 or instruction effective area 330 .

Next, using the parameters acquired by the parameter acquisition unit 213, the coordinate transformation unit 214 determines a method of coordinate transformation from the three-dimensional data in the coordinate space of the real camera 10 to the virtual three-dimensional data in the coordinate space of the virtual camera 312. Determine (step S102). The data acquisition unit 211 acquires the output data of the real camera 10 (step S103). The three-dimensional data generator 212 develops the output data of the real camera 10 in a three-dimensional coordinate space to generate three-dimensional data. (Step S104: 3D data generation step).

Thereafter, the coordinate transformation unit 214 transforms the three-dimensional data in the coordinate space of the real camera 10 into virtual three-dimensional data in the coordinate space of the virtual camera 312 using the coordinate transformation method determined in step S102 (step S105: coordinates conversion step). The detailed processing of steps S102 to S105 is the same as that of the first embodiment.

Next, the pointer detection unit 215 excludes the data outside the instruction effective area 330 specified based on the parameters acquired in step S201 from the virtual three-dimensional data coordinate-transformed in step S105 (step S202). .

After that, based on the virtual three-dimensional data excluding data outside the instruction effective area 330, the pointer detection unit 215 detects the point closest to the virtual camera 312 as the designated position 322 of the pointer 321, Calculate virtual three-dimensional coordinates. Then, the contact determination unit 216 calculates the shortest distance Vz from the reference plane B to the indicated position 322 based on the virtual three-dimensional coordinates of the indicated position 322 (step S106).

The contact determination unit 216 compares the distance Pz from the reference plane B to the virtual touch panel 311 with the distance Vz to the indicated position 322 (step S107), and determines whether or not the distance Vz is equal to or less than the distance Pz (step S108: contact determination step).

After outputting the coordinates on the virtual touch panel 311 obtained in step S109, if the administrator issues an instruction to end the operation input process (step S110: Yes), the process ends. If there is no end command (step S110: No), the process returns to step S103. The detailed processing of steps S106 to S110 is the same as that of the first embodiment.

In this way, the operation input device 20 detects data in the instruction effective area 330 from among the data obtained by converting the three-dimensional data based on the output data of the real camera 10 into the virtual three-dimensional data by the virtual camera 312. By comparing the indicated position 322 and the position of the virtual touch panel 311, the touch operation of the pointer 321 on the virtual touch panel 311 is detected.

The effects of the third embodiment will be explained. In some display means 26, only a part of the display means 26 is targeted for instruction operation, and other operation units 331 such as card insertion and bar code reader are provided. In such a case, the operation input device 20 must avoid erroneously detecting an operation to the other operation unit 331 as an instruction operation. According to the third embodiment, the instruction effective area 330 is set, and only the instruction position 322 of the indicator 321 existing inside the instruction effective area 330 is detected. It is possible to accurately detect a non-contact operation input without erroneously detecting as.

As described above, in the third embodiment, the pointer detection unit 215 of the operation input device 20 detects the virtual camera 312 based on the data within the instruction effective area 330 of the virtual three-dimensional data obtained by the virtual camera 312 . The point closest to 312 is detected as designated position 322, and contact determination unit 216 compares distance Vz from virtual camera 312 to designated position 322 with distance Pz from virtual camera 312 to virtual touch panel 311, It is determined that the pointer 321 has touched the virtual touch panel 311 when the distance Vz is equal to or less than the distance Pz. This makes it possible to accurately detect a non-contact operation input without erroneously detecting another operation by the operator outside the instruction effective area 330 .

(Embodiment 4)
Embodiment 4 of the present invention will be described in detail with reference to FIG. FIG. 10 is a diagram showing the arrangement of real cameras according to the fourth embodiment.

The operation input system 1 according to Embodiment 4 executes the same operation input processing as in Embodiment 1, but differs in that two or more real cameras including

real cameras

12 and 13 are used. Although the number of real cameras may be any number of two or more, in the following description, a configuration including

real cameras

12 and 13 will be described.

The operator may perform operation input using two or more indicators. For example, by touching the touch panel with two fingers at the same time, various operation inputs are possible depending on the distance between the two fingers or the moving direction and moving distance of the two fingers. The operation input device 20 according to the fourth embodiment can reliably detect an operation input to the virtual touch panel 311 even when there are two or more pointers 321 .

The first real camera 12 and the second real camera 13 are fixed at locations separated from each other. For example, as shown in FIG. 10 , the first real camera 12 is installed at the upper right end of the display means 26 and the second real camera 13 is installed at the upper left end of the display means 26 . That is, the first real camera 12 is positioned at the upper right side of the virtual touch panel 311 , and the second real camera 13 is positioned at the upper left side of the virtual touch panel 311 . The line-of-sight directions of the

real cameras

12 and 13 are directions inclined with respect to the pointing direction of the indicator 321 and different from each other. The configuration of the operation input device 20 is the same as that of the first embodiment.

An administrator of the operation input device 20 selects a master camera from among the two or more

real cameras

12 and 13, and sets parameters for the master camera. Here, a case where the real camera 12 is the master camera will be described. The parameter obtaining unit 213 obtains an angle θ _x and a specific The angle θy about a straight line passing through the point 313 and parallel to the _y -axis, the distance S from the real camera 12 to the specific point 313, and the distance Pz from the reference plane B to the virtual touch panel 311 are obtained.

The 3D data generation unit 212 generates 3D data based on the output data of the two or more

real cameras

12 and 13 and the position information of the two or more

real cameras

12 and 13 . Note that the three-dimensional data generation unit 212 performs calibration for generating a single three-dimensional data for two or more

real cameras

12 and 13 in advance.

Specifically, the three-dimensional data generation unit 212 expands the output data of the real camera 12, which is the master camera, into a three-dimensional coordinate space to generate three-dimensional data. Similarly, the three-dimensional data generator 212 generates three-dimensional data based on the output data of the real camera 13, which is a slave camera. This is three-dimensional data that has been calibrated and corrected based on the relationship and output data. The three-dimensional data generator 212 supplements the three-dimensional data based on the output data of the real camera 12 with calibrated three-dimensional data based on the output data of the real camera 13 to generate a single three-dimensional data. .

The coordinate transformation unit 214 rotates the line-of-sight direction E of the real camera 12 around the specific point 313 by an angle θ _x about a straight line parallel to the x-axis and an angle θ _y about a straight line parallel to the y-axis. is the line-of-sight direction V, and a virtual camera 312 having a reference point on the reference plane B is assumed. Then, the coordinate transformation unit 214 determines a coordinate transformation method for transforming the three-dimensional data based on the output data of the

real cameras

12 and 13 generated by the three-dimensional data generation unit 212 into virtual three-dimensional data by the virtual camera 312 .

The coordinate transformation unit 214 transforms the three-dimensional data generated by the three-dimensional data generation unit 212 into virtual three-dimensional data using the determined coordinate transformation method.

The contact determination unit 216 hypothesizes a surface that is separated from the virtual camera 312 by a distance Pz in the direction V and that is perpendicular to the direction V as the virtual touch panel 311 . Then, the distance Vz from the virtual camera 312 to the

pointers

323 and 324 in the direction V is calculated based on the virtual three-dimensional data coordinate-transformed from the three-dimensional data from the

real cameras

12 and 13 .

Here, in the fourth embodiment, in order to detect operations by two or

more pointers

323 and 324, the contact determination unit 216 uses the virtual camera 312 in the direction V calculated based on the virtual three-dimensional data. to the

pointers

323 and 324 is detected as one or more distances Vz (third distance). In other words, the contact determination unit 216 plots the distances from the virtual camera 312 to points in the three-dimensional space based on the virtual three-dimensional data, and detects points indicating the minimum distance as indicated positions of the

indicators

323 and 324 .

The contact determination unit 216 compares one or more distances Vz with the distance Pz from the virtual camera 312 to the virtual touch panel 311, and when the distance Vz is equal to or less than the distance Pz, the

pointer

323, 324 determines that it has touched.

Here, the effect of using two or more

real cameras

12 and 13 will be described. In the example of FIG. 10, from the three-dimensional data based only on the output data of the real camera 12, the pointer 324 is behind the pointer 323 and cannot be detected. In the fourth embodiment, the three-dimensional data is supplemented by the output data of the real camera 13 with different installation positions and line-of-sight directions. The virtual three-dimensional coordinates of the indicated position on the body 324 can be output.

In other words, since the virtual three-dimensional coordinates of the pointed position are output based on the output data of two or more

real cameras

12 and 13, the problem of failing to detect the pointed position can be avoided.

The operation input signal generation unit 217 generates an operation input signal based on the time change of the coordinates on the virtual touch panel of the pointing position determined by the contact determination unit 216 to be in contact. At this time, when contact at two or more different designated positions is determined, operation input signals corresponding to two or more preset contacts are generated.

As described above, in the fourth embodiment, the operation input system 1 is fixed apart from each other, and the line-of-sight directions are tilted with respect to the pointing directions of the

indicators

323 and 324 and are different from each other. Two or more

real cameras

12 and 13 are provided. A three-dimensional data generation unit 212 generates three-dimensional data based on the output data of two or more

real cameras

12 and 13, and a coordinate transformation unit 214 performs a coordinate transformation method based on parameters related to the real camera 12, which is a master camera. is used to convert to virtual three-dimensional data. Then, the contact determination unit 216 acquires the distance Vz, which is one or more minimum values of the distances from the virtual camera 312 to the

pointers

323 and 324 in the direction V calculated based on the virtual three-dimensional data, and obtains one or more instruction points. The indicated positions of the

bodies

323 and 324 are to be detected. As a result, the pointer 324 that has not been detected by one of the real cameras 12 can also be detected by the other real camera 13, and reliable operation input by a plurality of touches is possible.

In Embodiments 1-4 above, the real camera 10-13 is installed at the end of the display means 26, and the administrator sets parameters related to coordinate transformation. It may be built in the input device 20 . In this case, when the operation input device 20 is assembled, the line-of-sight direction of the real camera 10-13 is tilted with respect to the direction perpendicular to the display surface of the display means 26, and the coordinate transformation is determined according to the tilt angle. method may be used.

Further, in addition to the configuration of Embodiments 1 to 4 above, a transparent plate 316 may be further installed in front of the display means 26 and the real camera 10, as shown in FIG. The transparent plate 316 is an arbitrary transparent plate that transmits the light output by the display means 26, such as a glass plate or an acrylic plate. The display means 26 and the real camera 10 can be protected, and the operator can be urged to operate in a space a certain distance away.

Further, in Embodiments 1 to 4 above, it is assumed that one rectangular and flat virtual touch panel 311 corresponding to one rectangular display surface of the display means 26 is used to detect an operation input. And the size is arbitrary, and it does not have to be flat. Assuming two or more virtual touch panels 311 on the display surface of one display means 26, the three-dimensional data obtained by the real cameras 10-13 are subjected to coordinate transformation by two or more virtual cameras 312, and each virtual An operation input to the touch panel 311 may be detected. In these cases, the parameter acquisition unit 213 acquires parameters including the shape and size of the virtual touch panel 311, and the coordinate conversion unit 214 converts three-dimensional data according to the parameters representing the shape and size of the virtual touch panel 311. Coordinate transformation including enlargement, reduction or transformation of the coordinate values of

Further, in Embodiment 4 above, two or more

real cameras

12 and 13 are used to detect one or more indicated positions. A distance Vz that is one or more minimum values of the distance from the camera 312 to the indicator 321 may be acquired, and the indicated position of one or more indicators 321 may be detected.

Also, the hardware configuration and flowcharts shown in the above embodiments are examples, and can be arbitrarily changed and applied. Each function realized by the CPU 21 can be realized using a normal computer system without depending on a dedicated system.

For example, a computer-readable CD-ROM (Compact Disc Read-Only Memory), DVD (Digital Versatile Disc), MO (Magneto Optical Disc), memory card, etc. A computer capable of realizing each function may be configured by storing and distributing the program in a recording medium of the above and installing the program in the computer. When each function is shared between an OS (Operating System) and an application, or by cooperation between the OS and an application, only portions other than the OS may be stored in the recording medium.

Various embodiments and modifications of the present invention are possible without departing from the broad spirit and scope of the present invention. Moreover, the embodiment described above is for explaining the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims rather than the embodiments. Various modifications made within the scope of the claims and within the meaning of the invention equivalent thereto are considered to be within the scope of the present invention.

This application is based on Japanese Patent Application No. 2021-038462 filed on March 10, 2021. The entire specification, claims, and drawings of Japanese Patent Application No. 2021-038462 are incorporated herein by reference.

DESCRIPTION OF SYMBOLS 1...

Operation input system

10, 11, 12, 13... Actual camera 20... Operation input device 21... CPU
22 RAM
23 ROM
24 Communication interface 25 Storage unit 26 Display unit 211 Data acquisition unit 212 Three-dimensional data generation unit 213 Parameter acquisition unit 214 Coordinate conversion unit 215 Pointer detection unit 216 Contact determination unit 217 Operation input signal Generation part 311... Virtual touch panel 312... Virtual camera 313... Specific point 315... Hover surface 316...

Transparent plate

321, 323, 324... Indicator 322... Indication position 330... Indication effective area 331... Other operation part

Claims

a three-dimensional data generation unit that generates three-dimensional data based on output data of a real camera that is fixed with respect to a display surface of a display means and whose line-of-sight direction is inclined with respect to the pointing direction of the indicator;
The first straight line is rotated by a predetermined angle about a point on the first straight line extending in the line-of-sight direction of the real camera and located at a predetermined first distance from the real camera. a coordinate transformation unit for transforming the three-dimensional data in the coordinate space of the real camera into virtual three-dimensional data in the coordinate space of the virtual camera, assuming a virtual camera whose viewing direction is the direction of the second straight line;
When a surface separated from the virtual camera by a predetermined second distance in a direction parallel to the second straight line is a virtual touch panel, the virtual touch panel is calculated based on the virtual three-dimensional data, and the A third distance, which is the shortest distance from the virtual camera to the pointer, is compared with the second distance from the virtual camera to the virtual touch panel, and when the third distance is equal to or less than the second distance a contact determination unit that determines that the pointer has come into contact with the virtual touch panel;
an operation input signal generation unit that generates an operation input signal based on the virtual three-dimensional data when the contact determination unit determines that the pointer has made contact;
Operation input device.
a parameter including a rotation angle from the first straight line to the second straight line, the first distance from the real camera to the specific point, and the second distance from the virtual camera to the virtual touch panel; further comprising a parameter acquisition unit that acquires
The coordinate transformation unit uses the parameters to perform coordinate transformation represented by rotation about the specific point from the real camera to the virtual camera.
The operation input device according to claim 1.
When a plane passing through the reference point of the real camera and parallel to the extending direction of the virtual touch panel is taken as a reference plane, the reference point of the virtual camera is on the reference plane,
The first distance is the distance from the reference point of the real camera to the specific point, the second distance is the distance from the reference plane to the virtual touch panel, and the third distance is the distance from the reference plane to the instruction. is the shortest distance to the body,
The coordinate transformation unit is represented by rotation from the real camera to the virtual camera about the specific point, and translation along the second straight line from the post-rotation position to the reference point. perform a coordinate transformation that
The operation input device according to claim 1 or 2.
The rotation from the real camera to the virtual camera about the specific point is rotation about a straight line that passes through the specific point and is parallel to the extending direction of the virtual touch panel.
The operation input device according to claim 2 or 3.
The virtual touch panel has a shape extending in a direction perpendicular to the second straight line, and the second straight line passes through the center of the virtual touch panel.
The operation input device according to any one of claims 1 to 4.
The contact determination unit assumes a hover plane parallel to the virtual touch panel, which is spaced apart from the virtual touch panel in a direction opposite to the display means, and the hover plane in a direction parallel to the second straight line. to the virtual camera is a fourth distance, and if the third distance is equal to or less than the fourth distance, it is determined that the pointer is at a position closer to the virtual touch panel than the hover surface. death,
The operation input signal generation unit generates an operation input signal for displaying on the display means that the pointer is approaching the virtual touch panel.
The operation input device according to any one of claims 1 to 5.
The contact determination unit calculates the third distance from the virtual camera to the pointer, which is the shortest distance from the virtual camera to the pointer, which is calculated from data within a predetermined instruction effective area among the virtual three-dimensional data output from the coordinate conversion unit. determine contact based on distance;
The operation input device according to any one of claims 1 to 6.
The three-dimensional data generation unit includes output data of two or more of the real cameras fixed apart from each other and having line-of-sight directions that are inclined with respect to the pointing direction of the pointer and in different directions, and two or more generating the three-dimensional data based on the position information of the real camera of
The operation input device according to any one of claims 1 to 7.
The coordinate transformation unit performs calibration for generating the single three-dimensional data for the two or more real cameras in advance,
The operation input device according to claim 8.
The contact determination unit calculates one or more third distances, which are one or more minimum values of a distance from the virtual camera to the pointer in a direction parallel to the second straight line, calculated based on the virtual three-dimensional data. The distance is compared with the second distance from the virtual camera to the virtual touch panel in the direction parallel to the second straight line, and when one or more of the third distances is less than or equal to the second distance, the determining that one or more of the indicators have come into contact with the virtual touch panel;
The operation input signal generation unit generates the operation input signal according to the number of the pointers determined to be in contact by the contact determination unit.
The operation input device according to any one of claims 1 to 9.
a three-dimensional data generation step of generating three-dimensional data based on output data of a real camera fixed with respect to a display surface of a display means and having a line-of-sight direction inclined with respect to a pointing direction of an indicator;
The first straight line is rotated by a predetermined angle about a point on the first straight line extending in the line-of-sight direction of the real camera and located at a predetermined first distance from the real camera. a coordinate transformation step of transforming the three-dimensional data in the coordinate space of the real camera into virtual three-dimensional data in the coordinate space of the virtual camera, assuming a virtual camera whose viewing direction is the direction of the second straight line;
When a surface separated from the virtual camera by a predetermined second distance in a direction parallel to the second straight line is a virtual touch panel, the virtual touch panel is calculated based on the virtual three-dimensional data, and the When a third distance, which is the shortest distance from the virtual camera to the pointer, is compared with the second distance from the virtual camera to the virtual touch panel, and the third distance is equal to or less than the second distance a contact determination step of determining that the pointer has come into contact with the virtual touch panel;
an operation input signal generation step of generating an operation input signal based on the virtual three-dimensional data when it is determined in the contact determination step that the pointer has made contact;
Manipulation input method.
the computer,
a three-dimensional data generation unit that generates three-dimensional data based on output data of a real camera that is fixed with respect to the display surface of the display means and whose line-of-sight direction is inclined with respect to the pointing direction of the indicator;
The first straight line is rotated by a predetermined angle about a point on the first straight line extending in the line-of-sight direction of the real camera and located at a predetermined first distance from the real camera. A coordinate transformation unit that transforms the three-dimensional data in the coordinate space of the real camera into virtual three-dimensional data in the coordinate space of the virtual camera, assuming a virtual camera whose viewing direction is the direction of the second straight line;
When a surface separated from the virtual camera by a predetermined second distance in a direction parallel to the second straight line is a virtual touch panel, the virtual touch panel is calculated based on the virtual three-dimensional data, and the A third distance, which is the shortest distance from the virtual camera to the pointer, is compared with the second distance from the virtual camera to the virtual touch panel, and when the third distance is equal to or less than the second distance a contact determination unit that determines that the pointer has come into contact with the virtual touch panel;
an operation input signal generation unit that generates an operation input signal based on the virtual three-dimensional data when the contact determination unit determines that the pointer has made contact;
A program to function as
a three-dimensional data generation unit that generates three-dimensional data based on output data of a real camera that is fixed with respect to a display surface of a display means and whose line-of-sight direction is inclined with respect to the pointing direction of the indicator;
Assuming a virtual camera whose line-of-sight direction is the direction of a second line obtained by rotating the first line by a predetermined angle about a specific point on the first line extending in the line-of-sight direction of the real camera, a coordinate transformation unit that transforms the three-dimensional data in the coordinate space of the real camera into virtual three-dimensional data in the coordinate space of the virtual camera;
Assuming a virtual touch panel on a surface away from the virtual camera in a direction parallel to the second straight line, the instruction from the virtual camera in a direction parallel to the second straight line calculated based on the virtual three-dimensional data a contact determination unit that determines that the pointer has come into contact with the virtual touch panel when the shortest distance to the body is equal to or less than the distance from the virtual camera to the virtual touch panel;
an operation input signal generation unit that generates an operation input signal based on the virtual three-dimensional data when the contact determination unit determines that the pointer has made contact;
Operation input device.