JP6873326B2 - Eye 3D coordinate acquisition device and gesture operation device - Google Patents

Description

The present invention is an eye three-dimensional coordinate acquisition device for acquiring eye three-dimensional coordinates indicating the position of the eye included in the two-dimensional image based on a two-dimensional image captured by the two-dimensional imaging unit, and a gesture operation device having the same. Regarding.

In recent years, in various information devices, a gesture operation device for inputting information by a gesture operation such as a gesture of an operator has been used. In the gesture operation device, three-dimensional coordinates indicating the position of a part of the operator's body are acquired from a three-dimensional image (that is, a stereoscopic image) acquired by using a device such as a stereo camera. There is something that improves operability for the operator (see, for example, Patent Documents 1 and 2).

Japanese Unexamined Patent Publication No. 2011-175617 (for example, paragraphs 0010 to 0013, FIG. 2). Japanese Unexamined Patent Publication No. 2011-54118 (for example, paragraph 0010, FIG. 2)

However, a device having a function of acquiring three-dimensional coordinates indicating the position of a body part using a three-dimensional imaging unit has a problem that the number of parts is large and the structure and control are complicated.

The present invention has been made to solve the above-mentioned problems of the prior art, and is based on a two-dimensional image captured by a two-dimensional imaging unit, and has three-dimensional eye coordinates indicating the position of the eye included in the two-dimensional image. It is an object of the present invention to provide an eye three-dimensional coordinate acquisition device capable of acquiring the above and a gesture operation device having the same.

The eye three-dimensional coordinate acquisition device according to one embodiment of the present invention indicates the position of the eye viewing the gazing point from the two-dimensional image captured by the two-dimensional imaging unit and the gazing point position information indicating the gazing point position. A device that acquires three-dimensional eye coordinates, the two-dimensional eye coordinate calculation unit that calculates the two-dimensional eye coordinates indicating the two-dimensional position of the eye in the two-dimensional image, and the maximum image captured by the two-dimensional imaging unit. From the angle and the two-dimensional coordinates of the eye, the first three-dimensional vector from the first starting point, which is the imaging position of the two-dimensional imaging unit, to the position of the eye is used as the direction vector, and the first starting point is used as the starting point. From the first straight line calculation unit that calculates one straight line, the two-dimensional image, and the two-dimensional coordinates of the eye, a second three-dimensional vector from the second starting point, which is the position of the eye, to the position of the gazing point. And the 3D vector calculation unit that calculates the above, and the 3D vector that is the inverse vector of the 2D 2D vector and that goes from the 3rd starting point, which is the position of the gazing point, to the position of the eye, is used as the direction vector. Moreover, the triangular survey method is used from the second straight line calculation unit that calculates the second straight line starting from the third starting point, the first straight line, the second straight line, and the gazing point position information. It is characterized by having an eye three-dimensional coordinate calculation unit for calculating eye three-dimensional coordinates.

The gesture operation device according to another embodiment of the present invention includes the eye three-dimensional coordinate acquisition device, an imaging device that captures a two-dimensional image including the gesture operation, and a virtual surface which is a virtual operation surface on which the gesture operation is performed. It is characterized by having a gesture coordinate acquisition unit for acquiring a virtual in-plane gesture coordinate indicating the position of the gesture operation in the above.

The eye three-dimensional coordinate acquisition device according to the present invention can acquire eye three-dimensional coordinates indicating the position of the eye included in the two-dimensional image based on the two-dimensional image captured by the two-dimensional imaging unit. Further, according to the eye three-dimensional coordinate acquisition device according to the present invention, the structure and control of the device can be simplified.

Since the gesture operation device according to the present invention calculates the position of the virtual surface on which the gesture operation is performed based on the eye three-dimensional coordinates acquired by the eye three-dimensional coordinate acquisition device, the gesture operation of the operator is accurately detected. can do. Further, according to the gesture operation device according to the present invention, the structure and control of the device can be simplified.

It is a block diagram which shows the eye 3D coordinate acquisition apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the image pickup position, the eye position, and the gaze point position of the 2D image pickup unit in Embodiment 1 on the zx plane. It is a figure which shows the example of the image pickup position, the eye position, and the gaze point position of the 2D image pickup unit in Embodiment 1 on the xy plane. It is a figure which shows the example of the light emitting element as a gaze point in Embodiment 1. FIG. It is a figure which shows the example of the 2D image received by the eye 2D coordinate calculation unit of the eye 3D coordinate acquisition apparatus which concerns on Embodiment 1, and the 2D coordinate of the eye position output by the eye 2D coordinate calculation unit. It is a figure which shows the calculation method of the yaw component of the 3D vector in the direction from the imaging position of the 2D imaging unit to the position of an eye in Embodiment 1. FIG. It is a figure which shows the calculation method of the pitch component of the 3D vector in the direction from the imaging position of the 2D imaging unit to the position of an eye in Embodiment 1. FIG. The first three-dimensional vector in the direction from the first starting point, which is the imaging position of the two-dimensional imaging unit in the first embodiment, to the position of the eye, the first three-dimensional vector is used as the direction vector, and the first starting point is used as the starting point. It is a figure which shows the example of the 1st straight line. It is a figure which shows the example of the 2nd three-dimensional vector in the direction from the 2nd origin which is the position of an eye in Embodiment 1 toward the position of a gaze point. The third three-dimensional vector (that is, the inverse vector of the second three-dimensional vector) and the third three-dimensional vector in the direction from the third starting point, which is the position of the gazing point in the first embodiment, to the position of the eye, are used. It is a figure which shows the example of the 2nd straight line which is a direction vector and starts from a 3rd origin. It is a figure which shows the 1st projective line and the 2nd projective line obtained by projecting the 1st straight line shown in FIG. 8 and the 2nd straight line shown in FIG. 10 on the triangulation execution plane. It is a figure which shows the intersection of the normal line passing through the intersection point on the triangulation execution plane shown in FIG. 11, and the first straight line and the second straight line. It is a figure which shows the intersection of the normal and the 1st straight line which pass through the intersection on the triangular survey execution plane shown in FIG. 11 and the intersection of a normal and a 2nd straight line. It is a figure which shows the example of the presentation method of the gaze point in Embodiment 2 of this invention. It is a figure which shows another example of the gaze point presentation method in Embodiment 2. FIG. It is a figure which shows still another example of the method of presenting the gaze point in Embodiment 2. FIG. It is a block diagram which shows the gesture operation apparatus which concerns on Embodiment 3 of this invention. It is the schematic which shows the example of the 3D virtual surface in Embodiment 3. FIG. It is the schematic which shows the example of the 2D virtual surface in Embodiment 3. FIG. It is a schematic diagram which shows the example of the gesture image in the 2D virtual plane in Embodiment 3. FIG. It is the schematic which shows the example of the coordinate which shows the gesture position in the 2D virtual plane in Embodiment 3. FIG. It is a block diagram which shows the gesture operation apparatus which concerns on Embodiment 4 of this invention. It is a block diagram which shows the gesture operation apparatus which concerns on Embodiment 5 of this invention. It is a figure which shows the operation of the gesture operation apparatus which concerns on Embodiment 5. It is a block diagram which shows the example of the hardware composition of the modification of Embodiments 1-5.

The eye three-dimensional coordinate acquisition device and the gesture operation device having the eye three-dimensional coordinate acquisition device according to the embodiment of the present invention will be described below with reference to the attached drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

In the following description, a world coordinate system, which is an xyz Cartesian coordinate system, is used to indicate a three-dimensional position. In the world coordinate system, the x-axis is a lateral coordinate axis orthogonal to the optical axis of the two-dimensional imaging unit. The y-axis is a vertical coordinate axis orthogonal to the optical axis of the two-dimensional imaging unit. The z-axis is a coordinate axis that coincides with the optical axis that passes through the imaging position of the two-dimensional image pickup unit (that is, the center position of the image pickup element of the two-dimensional image pickup unit). In the following embodiments, the imaging position is the origin of the world coordinate system.

Further, in the following description, an XY Cartesian coordinate system indicating a two-dimensional position in a two-dimensional image captured by the two-dimensional imaging unit is used. In the XY Cartesian coordinate system, the X-axis is the coordinate axis corresponding to the horizontal direction of the two-dimensional image, and the Y-axis is the coordinate axis corresponding to the vertical direction of the two-dimensional image.

When the optical distortion in the two-dimensional imaging unit is sufficiently small, the x-axis in the world coordinate system and the X-axis in the XY Cartesian coordinate system can be considered to be parallel to each other. When the optical distortion in the two-dimensional imaging unit is sufficiently small, the y-axis in the world coordinate system and the Y-axis in the XY Cartesian coordinate system can be considered to be parallel to each other. However, usually, the unit length in the world coordinate system and the unit length in the XY orthogonal coordinate system do not match. For example, the unit length in the XY Cartesian coordinate system, which corresponds to the unit length of 1 mm in the world coordinate system, is the length of one pixel (that is, the distance between the center positions of adjacent pixels).

Embodiment 1.
FIG. 1 is a block diagram showing an eye three-dimensional coordinate acquisition device 100 according to the first embodiment. The eye three-dimensional coordinate acquisition device 100 captures the imaging region with the two-dimensional imaging unit 101 to obtain a two-dimensional image (that is, two-dimensional image data) D11 and a position of a gazing point (also referred to as a “gazing target point”). It is a device that acquires the eye three-dimensional coordinate D18 indicating the position of the eye viewing the gazing point in the three-dimensional coordinates of the world coordinate system from the gazing point position information D12 indicating the gazing point 31. The eye is, for example, the eye of an operator (for example, a user) of the device having the eye three-dimensional coordinate acquisition device 100. The gazing point is an object or a displayed image portion that the operator can easily pay attention to (that is, the operator's line of sight is directed).

As shown in FIG. 1, the eye 3D coordinate acquisition device 100 has an eye 2D coordinate calculation unit 103 that calculates the eye 2D coordinates D14 in the 2D image D11, and an imaging position as a starting point (that is, a first starting point). ), The first straight line calculation unit 104 that calculates the first straight line LV1 (that is, D15), and the second three-dimensional vector V2 (that is, D16) that starts from the position of the eye (that is, the second starting point). ), The second straight line calculation unit 106 that calculates the second straight line LV3 (that is, D17) starting from the position of the gazing point (that is, the third starting point), and the eye. It has an eye three-dimensional coordinate calculation unit 107 that calculates the three-dimensional coordinate D18. The eye three-dimensional coordinate acquisition device 100 may include a two-dimensional imaging unit 101. Further, the eye three-dimensional coordinate acquisition device 100 may include a gazing point position information providing unit 102.

FIG. 2 is a diagram showing an example of the imaging position 11 of the two-dimensional imaging unit 101, the eye position 21 of the operator 20, and the gazing point position 31 in the first embodiment on the zx plane. FIG. 3 is a diagram showing an example of the imaging position 11 of the two-dimensional imaging unit 101, the eye position 21 of the operator 20, and the gazing point position 31 on the xy plane. The two-dimensional imaging unit 101 acquires a two-dimensional image (that is, two-dimensional image data) D11 of the photographing region. The two-dimensional image pickup unit 101 is, for example, a camera provided with an image pickup element that acquires an image of a shooting area. In FIGS. 2 and 3, the operator 20 directs the line of sight 22 to the position 31 of the gazing point.

FIG. 4 is a diagram showing an example of a light emitting element 30 as a gazing point in the first embodiment. In FIG. 4, the operator 20 directs the line of sight 22 to the position 31 of the gazing point, which is the center position of the lit light emitting element 30. The light emitting element 30 is, for example, an LED (Light Emitting Diode) arranged at a predetermined position. The position 31 of the gazing point can be changed by providing a plurality of light emitting elements 30 and lighting any one of them. The gazing point position information providing unit 102 is a device that provides gazing point position information D12 (for example, three-dimensional coordinates of the gazing point position 31) indicating the gazing point position 31. The gazing point position information providing unit 102 is a storage unit that stores the gazing point position information D12, a receiving unit that receives the gazing point position information D12, and the like. The gazing point position information providing unit 102 provides a device that provides the gazing point position information D12 to the eye three-dimensional coordinate acquisition device 100, or provides information that enables the gazing point position information D12 to be calculated by the eye three-dimensional coordinate acquiring device 100. Any device may be used as long as it is a device. The gazing point may be the center point of the lamp, the center point of the rotating light, the light irradiation point by the laser pointer, or a predetermined position that is not provided with any special means but is expected to be gazed by the operator. .. The gaze point position information providing unit 102 may be a device provided with a light emitting element 30 as a gaze point or a device connected to the light emitting element 30 as a gaze point. Further, in FIG. 4, the light emitting element 30 as the gazing point exists on the x-axis, but the position where the light emitting element 30 exists is not limited to the x-axis and may exist at any point in the space.

FIG. 5 shows the two-dimensional image D11 received by the eye two-dimensional coordinate calculation unit 103 of the eye three-dimensional coordinate acquisition device 100 according to the first embodiment and the eye two-dimensional output of the eye two-dimensional coordinate calculation unit 103. It is a figure which shows the example of the coordinate D14. FIG. 6 is a diagram showing a method of calculating the yaw component of the three-dimensional vector in the direction from the imaging position 11 of the two-dimensional imaging unit 101 toward the eye position 21. FIG. 7 is a diagram showing a method of calculating the pitch component of the three-dimensional vector in the direction from the imaging position 11 of the two-dimensional imaging unit 101 toward the position 21 of the eye. Note that FIG. 6 also shows _{angles γ X} and γ _Y determined by a straight line connecting the gazing point position 31 and the eye position 21 and a straight line passing through the eye position 21 and perpendicular to the x-axis. These will be described in Embodiment 2.

The eye two-dimensional coordinate calculation unit 103 detects (that is, obtains by calculation) the eye two-dimensional coordinate D14 in the two-dimensional image D11 acquired by the two-dimensional image pickup unit 101. When detecting the two-dimensional eye coordinates D14 from the two-dimensional image D11, for example, an extraction method of a target object using ASM (Active Shape Model) or AAM (Active Appearance Models) can be used.

As shown in FIG. 6, the X coordinate (for example, the number of pixels in the X direction) of the eye two-dimensional coordinate D14 in the _{two-dimensional image D11 is represented by X e} , and is 1 of the total number of pixels in the horizontal direction of the two-dimensional image D11. The number of pixels of / 2 _{is represented by X max} , and the yaw angle of the 3D vector, which is the angle formed by the 3D vector and the z-axis, which is the optical axis of the imaging system, is _{represented by α X} in the horizontal direction of the 2D image D11. The maximum shooting angle is represented _{by β X.} At this time, the following relational expression holds.
X _e : X _max = tanα _X : tanβ _X
Therefore, the yaw angle α _X of the three-dimensional vector can be calculated by the following equation (1).

As shown in FIG. 7, Y coordinates of the eye two-dimensional coordinate D14 in the 2-dimensional image D11 (e.g., Y direction number of pixels) and expressed in Y _e, 1 of the total number of pixels in the vertical direction of the two-dimensional image D11 The number of pixels of / 2 _{is represented by Y max} , and the pitch angle of the 3D vector, which is the angle formed by the 3D vector and the z-axis, which is the optical axis of the imaging system, is _{represented by α Y} in the vertical direction of the 2D image D11. The maximum shooting angle is expressed _{in β Y.} At this time, the following relational expression holds.
Y _e : Y _max = tanα _Y : tanβ _Y
Therefore, the pitch angle α _Y of the three-dimensional vector can be calculated by the following equation (2).

The eye two-dimensional coordinate calculation unit 103 may use the eye two-dimensional coordinate D14 indicating the eye position 21, that is, the representative position of the eye based on one or both eyes as the _{coordinates (X e} , _Ye). it can. For example, the eye two-dimensional coordinate calculation unit 103 can use the position of the right eye, the position of the left eye, the position of both eyes, the average position of the positions of both eyes, or the like as the representative position of the eye. When only the position of one eye is used as the position 21 of the eye, the position of the eye can be calculated even when the two-dimensional coordinate calculation unit 103 of the eye can detect only the position of one eye. This has the advantage of being able to reduce the frequency of undetectable eye positions. When both eye positions are used as the eye positions 21, there is an advantage that the eye position can be calculated with higher accuracy than when only one eye position is used.

In FIG. 8, the first three-dimensional vector V1 in the direction from the first starting point, which is the imaging position 11 of the two-dimensional imaging unit 101 in the first embodiment, to the position 21 of the eye is used as the direction vector, and the first starting point is used as the starting point. It is a figure which shows the example of the 1st straight line LV1 (that is, D15). _{The first linear calculation unit 104 includes the maximum imaging angles β X} and β _Y (that is, D13) of the two-dimensional image D11 captured by the two-dimensional imaging unit 101 and the eye acquired by the two-dimensional coordinate calculation unit 103. From the two-dimensional coordinate D14 of the eye corresponding to the position 21 of, the imaging position / eye which is a unit vector having information in the three-dimensional direction from the first starting point which is the imaging position 11 of the two-dimensional imaging unit 101 toward the position 21 of the eye. The direction three-dimensional vector, that is, the first three-dimensional vector V1 is calculated, and the first straight line LV1 having the first three-dimensional vector V1 as the direction vector and the first starting point as the starting point is calculated.

FIG. 9 is a diagram showing an example of a second three-dimensional vector V2 (that is, D16) in the direction from the second starting point, which is the position 21 of the eye, to the position 31 of the gazing point in the first embodiment. The three-dimensional vector calculation unit 105 is a unit vector having information in the three-dimensional direction from the two-dimensional image D11 and the two-dimensional coordinate D14 of the eye from the second starting point, which is the position 21 of the eye, to the position 31 of the gazing point. -Calculate a three-dimensional vector in the gazing point direction, that is, a second three-dimensional vector V2. In order to calculate the direction of the second three-dimensional vector V2, the three-dimensional vector calculation unit 105 uses a known method such as the outer corner of the eye method, the corneal reflex method, the bright pupil method, or the dark pupil method. Can be done. Further, since the direction of the face has a strong correlation with the direction of the line of sight, the direction of the face may be used as the direction of the line of sight. In this case, the three-dimensional vector calculation unit 105 detects the entire range of the operator 20's face in the two-dimensional image by a method such as a cascade detector, SVM (Support Vector Machine), or deep learning, and the detected face. The positions of the eyes, nose, or mouth, which are the necessary facial parts within the entire range, are calculated by the ASM method or the AAM method, and the position of the facial parts is biased within the detected range of the entire face. The orientation can be calculated.

FIG. 10 shows a second configuration in which the third three-dimensional vector V3 in the direction from the third starting point, which is the position 31 of the gazing point in the first embodiment, to the position 21 of the eye is used as the direction vector and the third starting point is used as the starting point. It is a figure which shows the example of the straight line LV3 (that is, D17). The third three-dimensional vector V3 is an inverse vector of the second three-dimensional vector V2. The second linear calculation unit 106 is a gaze point / eye direction three-dimensional vector, that is, a third gaze point / eye direction three-dimensional vector, which is a unit vector having information in the three-dimensional direction from the third starting point which is the gaze point position 31 toward the eye position 21. 3D vector V3 is calculated, and a second straight line LV3 is calculated with the third three-dimensional vector V3 as the direction vector and the third starting point as the starting point.

FIG. 11 shows a first projective line PV1a and a second projective line obtained by projecting the first straight line LV1 shown in FIG. 8 and the second straight line LV3 shown in FIG. 10 onto the triangular survey execution plane 40. It is a figure which shows the straight line PV3a and the intersection 23a of the 1st projective line PV1a and the 2nd projective line PV3a. FIG. 12 is a diagram showing an intersection 23 of the eye existence range straight line 41, which is a normal line passing through the intersection 23a on the triangulation execution plane 40 shown in FIG. 11, and the first straight line LV1 and the second straight line LV3. ..

The eye three-dimensional coordinate calculation unit 107 calculates the eye three-dimensional coordinate D18 from the first straight line LV1, the second straight line LV3, and the gazing point position information D12 by using the triangulation method. An example of the calculation method of the three-dimensional eye coordinates D18 will be described below.

In the triangulation method, a triangulation plane 40, which is an arbitrary plane that projects a straight line in the three-dimensional space and a point in the three-dimensional space, is used. In FIGS. 11 and 12, the triangulation execution plane 40 is an uv plane of an uv Cartesian coordinate system having a u-axis and a v-axis as coordinate axes. The u-axis and v-axis do not have to be parallel to the x-axis, y-axis, and z-axis of the world coordinate system. However, the u-axis or v-axis may be parallel to any of the x-axis, y-axis, and z-axis.

The triangulation execution plane 40 is, for example, a horizontal plane orthogonal to the direction of gravity (that is, the vertical direction). The triangulation execution plane 40 may be a plane parallel to a plane including a straight line orthogonal to the optical axis of the two-dimensional imaging unit 101 and parallel to the horizontal plane and the optical axis of the two-dimensional imaging unit 101. Further, another plane may be used as the triangulation execution plane 40.

After defining the triangular measurement execution plane 40, the eye three-dimensional coordinate calculation unit 107 projects the first straight line LV1 from the imaging position 11 toward the eye position 21 onto the triangular measurement execution plane 40 as shown in FIG. By doing so, the first projective line PV1a is obtained.

Similarly, as shown in FIG. 11, the eye three-dimensional coordinate calculation unit 107 projects a second straight line LV3 from the gazing point position 31 toward the eye position 21 onto the triangulation execution plane 40 to obtain a second. The projective line PV3a of is obtained.

Here, as shown in FIG. 11, the intersection 23a of the first projective line PV1a and the second projective line PV3a is the projection point on the triangulation execution plane 40 at the eye position 21. Therefore, as shown in FIG. 12, the three-dimensional eye coordinate D18 passes through the projection point (intersection point 23a) of the eye position 21 on the triangular survey execution plane 40 and is a straight line perpendicular to the triangular survey execution plane 40 (that is, that is). Assuming that the normal line) is the eye existence range straight line 41, it is determined at any position on the eye existence range straight line 41.

Further, a first straight line LV1 (that is, a straight line that passes through the imaging position 11 and has the first three-dimensional vector V1 as a direction vector) and a second straight line LV3 (that is, a straight line that passes through the position 31 of the gazing point) and the third The straight line with the three-dimensional vector V3 as the direction vector) theoretically intersects at the intersection 23, which is one point on the eye existence range straight line 41. Therefore, the eye three-dimensional coordinate calculation unit 107 outputs the coordinates of the intersection 23 as the eye three-dimensional coordinates D18.

FIG. 13 shows the intersection point 24 between the eye existence range straight line 41 and the first straight line LV1, which is a normal line passing through the intersection 23 on the triangular survey execution plane 40 shown in FIG. 11, and the eye existence range straight line 41 and the second straight line. It is a figure which shows the intersection 25 with LV3.

In the actual calculation, due to the calculation error of the two-dimensional eye coordinate D14, the calculation error of the second three-dimensional vector V2, the quantization error or the rounding error in the calculation of the computer, and the like, as shown in FIG. The intersection 24 of the straight line LV1 and the eye existence range straight line 41 may not match the intersection 25 of the second straight line LV3 and the eye existence range straight line 41. In other words, this is a case where the first straight line LV1 and the second straight line LV3 are in a twisted position relationship with each other. In this case, the eye three-dimensional coordinate calculation unit 107 connects the intersection 24 of the first straight line LV1 and the eye existence range straight line 41 and the intersection 25 of the second straight line LV3 and the eye existence range straight line 41, and both ends ( That is, the line segment including the intersections 24 and 25), that is, the three-dimensional eye coordinate D18 which is the coordinates of any point on the line segment connecting the intersections 24 and 25 is output. For example, the following method can be used to determine where the three-dimensional eye coordinate D18 is determined on the line segment connecting the intersections 24 and 25.

The eye three-dimensional coordinate calculation unit 107 sets either the intersection 24 of the first straight line LV1 and the eye existence range straight line 41 or the intersection 25 of the second straight line LV3 and the eye existence range straight line 41 as the eye three-dimensional coordinate D18. Can be decided. Further, the eye three-dimensional coordinate calculation unit 107 may determine the midpoint between the intersection 24 and the intersection 25 as the eye three-dimensional coordinate D18. Furthermore, the eye three-dimensional coordinate calculation unit 107 also calculates the reliability of the positions of both eyes when the eye two-dimensional coordinate calculation unit 103 calculates the positions of both eyes, and calculates the positions of both eyes. The internal division point internally divided according to the reliability of the eye may be determined as the three-dimensional eye coordinate D18. When the first straight line LV1 and the second straight line LV3 are in a twisted position with each other in this way, both the first straight line LV1 and the second straight line LV3 are used as the triangular survey execution plane 40. By using a plane perpendicular to a straight line (that is, a common perpendicular line) that is orthogonal to each other, the line segment connecting the intersections 24 and 25 can be set to be the shortest, and the range of the eye three-dimensional coordinate D18 can be narrowed to the shortest.

As described above, the eye three-dimensional coordinate acquisition device 100 according to the first embodiment is composed of the two-dimensional image D11 captured by the two-dimensional imaging unit 101 and the gazing point position information D12 indicating the gazing point position 31. , The eye three-dimensional coordinate D18 indicating the position of the eye looking at the gazing point can be acquired. Further, according to the eye three-dimensional coordinate acquisition device 100 according to the first embodiment, since it is not necessary to use the three-dimensional imaging unit, the structure and control of the device can be simplified.

Embodiment 2.
The eye three-dimensional coordinate acquisition device according to the second embodiment is the same as the eye three-dimensional coordinate acquisition device 100 according to the first embodiment, except for the method of presenting the gazing point. In the second embodiment, the gazing point is presented to the operator by an image portion that is a part of the displayed image. In the second embodiment, FIGS. 1 to 13 used in the description of the first embodiment are referred to.

FIG. 14 is a diagram showing an example of a method of presenting a gazing point in the second embodiment. In the example of FIG. 14, the gazing point is displayed as a mark (for example, an arrow-shaped pointer) 32 on the display screen of the VDT (Visual Display Thermals) device 32a, which is a display device. The mark 32 may be blinked so as to alternately turn on and off. In addition, a message prompting the user to pay attention to the mark 32 may be displayed. In this case, the gazing point position information providing unit 102 provides the eye three-dimensional coordinate acquisition device with the position information indicating the position of the mark 32.

FIG. 15 is a diagram showing another example of the method of presenting the gazing point in the second embodiment. In the example of FIG. 15, the gazing point is projected by the projector device 33a as a mark (for example, an arrow-shaped pointer) 33. The mark 33 may be blinked so as to alternately turn on and off. In addition, a message prompting the user to pay attention to the mark 33 may be displayed. In this case, the gazing point position information providing unit 102 provides the eye three-dimensional coordinate acquisition device with the position information indicating the position of the mark 33.

FIG. 16 is a diagram showing still another example of the method of presenting the gazing point in the second embodiment. In the example of FIG. 16, the gazing point is displayed as a mark (for example, an arrow-shaped pointer) 34 on the display screen of the VDT device 34a, which is a display device. In the example of FIG. 16, the two-dimensional imaging unit 101 is arranged near the upper center of the VDT device 34a. In this case, the mark 34 as the gazing point is displayed at the position farthest from the imaging position 11 of the two-dimensional imaging unit 101. In this way, by moving the position of the mark 34 indicating the gazing point far from the imaging position 11 (moving to the lower left corner in FIG. 16), the expected angle in triangulation (for example, π / 4- in FIG. 6). α _X + γ _X , π / 4-α _Y + γ _Y in FIG. 7) can be increased, and as a result, the measurement accuracy of the three-dimensional eye coordinate D18 can be improved.

As described above, the eye three-dimensional coordinate acquisition device according to the second embodiment is based on the two-dimensional image D11 captured by the two-dimensional imaging unit 101 and the gazing point position information D12 indicating the gazing point position 31. It is possible to acquire the eye three-dimensional coordinate D18 indicating the position of the eye looking at the gazing point. Further, according to the eye three-dimensional coordinate acquisition device 100 according to the first embodiment, since it is not necessary to use the three-dimensional imaging unit, the structure and control of the device can be simplified.

Further, the eye three-dimensional coordinate acquisition device according to the second embodiment may include the VDT device 32a of FIG. 14, the projector device 33a of FIG. 15, or the VDT device 34a of FIG. 16 as a part of the device.

Embodiment 3.
FIG. 17 is a block diagram showing the gesture operation device 300 according to the third embodiment. As shown in FIG. 17, the gesture operation device 300 according to the third embodiment captures a two-dimensional image including the gesture operation of the operator and the eye three-dimensional coordinate acquisition device 100 according to the first or second embodiment. It has an imaging device and a gesture coordinate acquisition unit 310 that acquires gesture coordinates D22 in a virtual plane indicating the position of the gesture operation on the virtual plane which is a virtual operation plane on which the gesture operation is performed. In FIG. 17, the imaging device is the two-dimensional imaging unit 101, but it may be another two-dimensional imaging unit different from the two-dimensional imaging unit 101.

The gesture operation device 300 according to the third embodiment is a gesture operation UI (User Interface). The gesture operation device 300 transmits the virtual in-plane gesture coordinates D22 corresponding to the position of the operator's hand in space to the operation target device (not shown). The pointer on the screen of the display device (not shown) of the operation target device moves to a position corresponding to the gesture coordinate D22 in the virtual plane, and is one of the operation parts such as an icon displayed at a specific position on the screen. Is selected. By using the gesture operation device 300 instead of a physical input device such as a remote controller or a mouse, it is possible to easily operate the operation target device. Further, in the gesture operation device 300, the operator can perform the operation by moving a body part (for example, a hand) in the space, and it is not necessary to touch the input operation unit with the hand, which is excellent in terms of hygiene. There is.

In the gesture operation device 300 according to the third embodiment, a configuration for displaying and moving a pointer on the screen of the display device of the operation target device that has received the virtual in-plane gesture coordinates D22 according to the position and movement of the operator's hand will be described. To do. Gesture operations may be performed on body parts other than the hands, such as the face, arms, whole body, or legs. In addition, the gesture operation device 300 does not move the pointer on the screen of the display device of the operation target device in response to the gesture operation, but in the coordinates at which the gesture operation is started, the type of the body part on which the gesture operation is performed, and the gesture operation. The operation target device may be configured to perform an operation of the content according to the direction of movement of the body part. The screen of the display device is, for example, a VDT device, a projector device, a head-mounted display, or the like. In addition, the gesture operation device 300 can also be used to operate an operation target device that does not have a display device.

As shown in FIG. 17, the gesture coordinate acquisition unit 310 calculates the virtual surface 3D coordinate D19 based on the eye 3D coordinate D18 provided by the eye 3D coordinate acquisition device 100. From 301, the virtual surface 2D coordinate calculation unit 302 that calculates the virtual surface 2D coordinate D20 based on the virtual surface 3D coordinate D19, and the 2D image D11, the 2D coordinates of the position where the gesture operation is performed are obtained. From the gesture coordinate calculation unit 303 that calculates the gesture coordinate D21 in the two-dimensional image shown, the virtual surface two-dimensional coordinate D20, and the gesture coordinate D21 in the two-dimensional image, the gesture coordinate D22 in the virtual plane in the virtual surface two-dimensional coordinate D20 is calculated. It has a virtual in-plane gesture coordinate calculation unit 304. In the third embodiment, the gesture coordinate D21 in the two-dimensional image is a position within the movement range of the position of the hand in the two-dimensional image. For the calculation of the gesture coordinates D21 in the two-dimensional image, a rule-based method, a template matching method, a method using deep learning, or the like can be used.

FIG. 18 is a schematic view showing an example of a virtual surface 371 (also referred to as a “virtual operation surface”) in the third embodiment. The virtual surface three-dimensional coordinate calculation unit 301 calculates the virtual surface three-dimensional coordinate D19, which is the coordinate indicating the virtual surface 371, from the eye three-dimensional coordinate D18. It is desirable from the viewpoint of operability that the virtual surface 371 exists in the direction from the operator 20 toward the screen of the display device or the device to be operated. Therefore, the virtual surface three-dimensional coordinate calculation unit 301 is on a straight line extending from the three-dimensional vector V30 (for example, the second three-dimensional vector V2) in a predetermined direction from the starting point which is the eye three-dimensional coordinate D18. A point within the movable range of the operator's arm is set as a virtual surface reference point 372, and a surface including the virtual surface reference point 372 is set as a virtual surface 371.

The virtual surface reference point 372 is, for example, the center of the virtual surface 371. However, the virtual surface reference point 372 may be a point other than the center of the virtual surface 371. The virtual plane reference point 372 is, for example, a point on a straight line connecting the position 21 of the eye and the position 31 of the gazing point, and the movable range of the hand by the operator is offset in any direction from the position of this straight line. Because it is. For example, since the movable range of the hand is below the position 21 of the eye in the vertical direction, the operability can be set by offsetting the virtual surface 371 slightly below the second three-dimensional vector V2 in the vertical direction. Will increase. Further, when the position of one eye is used as the position 21 of the eyes, the operability is improved by offsetting the virtual surface 371 in the direction of the position of the one eye. Further, when it is known in advance whether the hand performing the gesture operation is the right hand or the left hand, the operability is improved by offsetting the virtual surface 371 in the direction of the hand performing the gesture operation. In this way, by setting the virtual surface reference point 372 such that the virtual surface 371 is offset according to the specifications of the gesture operation, the operability of the device can be improved.

The movable range of the operator's arm is within a predetermined value range. When the average value of the arm length is about 70 cm (the average value of Japanese people), it is appropriate that the distance from the three-dimensional eye coordinate D18 to the virtual plane reference point 372 is within the range of 0 to 70 cm. .. However, if this distance is too short or too long, the operability will deteriorate. Therefore, in the third embodiment, the case where the distance from the three-dimensional eye coordinate D18 to the virtual plane reference point 372 is 35 cm, which is 1/2 of the average value of the arm lengths, will be described. The distance between the three-dimensional eye coordinate D18 and the virtual plane reference point 372 can be selected within the range equal to or less than the length of the operator's arm.

The virtual surface 371 may be a curved surface as shown in FIG. 18, or may be a flat surface. Since the movable range of the hand is circular or spherical around the base of the arm (position of the shoulder joint), the operability is improved when the virtual surface 371 is a curved surface close to the movable range of the hand. However, for the sake of simplification of calculation, the virtual surface 371 may be a plane.

FIG. 19 is a schematic view showing an example of the virtual surface 381 according to the third embodiment. FIG. 20 is a schematic view showing an example of the two-dimensional image D11 including the gesture operation in the third embodiment. As shown in FIGS. 19 and 20, the virtual surface two-dimensional coordinate calculation unit 302 is a virtual surface which is the coordinates of the virtual surface 381 which is the projection surface of the virtual surface 371 shown in FIG. 18 on the two-dimensional image D11. The two-dimensional coordinates D20 are calculated using the virtual surface three-dimensional coordinates D19 on the virtual surface 371.

FIG. 21 is a schematic view showing an example of coordinates indicating the gesture position 390 in the virtual surface 381 according to the third embodiment. As shown in FIG. 21, the virtual surface in-plane gesture coordinate calculation unit 304 calculates the virtual in-plane gesture coordinate D22 from the virtual surface two-dimensional coordinate D20 and the two-dimensional image in-plane gesture coordinate D21. In FIG. 21, the gesture position 390, which is the center of the gesture region where the gesture operation was performed, and the four corners 391-394 of the gesture region are shown as the positions indicated by the gesture coordinates D22 in the virtual plane. The position indicated by the gesture coordinate D22 in the virtual plane is not limited to the example of FIG.

Further, as shown in FIG. 20, the virtual surface 381 is generally set in a narrow range as compared with the two-dimensional image D11. Therefore, the gesture coordinate D22 in the virtual plane may be outside the virtual plane 381. It is assumed that such a situation is a situation in which the gesture coordinate calculation unit 303 erroneously detects the coordinates of the region where the gesture operation is not performed as the coordinates of the gesture operation. Therefore, the virtual in-plane gesture coordinate calculation unit 304 may determine that the gesture operation is invalid when the in-plane gesture coordinate D22 is outside the virtual surface 381.

As described above, the gesture operation device 300 according to the third embodiment determines the position of the virtual surface 371 on which the gesture operation is performed based on the eye three-dimensional coordinate D18 acquired by the eye three-dimensional coordinate acquisition device 100. The virtual surface 381 is set by calculating and projecting the virtual surface 371 onto the two-dimensional image D11, and the virtual in-plane gesture coordinates D22, which are the gesture coordinates in the virtual surface 381, are output. Since the virtual surface 371 is set to an appropriate position from the position 21 of the eyes of the operator 20, the virtual surface 381 is also set to an appropriate position. Therefore, the gesture operation can be detected accurately. Further, according to the gesture operation device 300, since the three-dimensional imaging unit is not provided, the structure and control of the device can be simplified.

Embodiment 4.
FIG. 22 is a block diagram showing the gesture operation device 400 according to the fourth embodiment. In FIG. 22, components that are the same as or correspond to the components shown in FIG. 17 are designated by the same reference numerals as those shown in FIG. The gesture operation device 400 according to the fourth embodiment is different from the gesture operation device 300 according to the third embodiment in the processing method performed by the gesture coordinate calculation unit 403 of the gesture coordinate acquisition unit 410.

The gesture coordinate calculation unit 303 in the third embodiment extracts a portion corresponding to the gesture operation, for example, a range of the hand performing the gesture operation from the entire two-dimensional image D11. However, such processing requires a very large amount of information processing, and if the amount of information processing is reduced, the extraction accuracy of the hand range is lowered. That is, there is a trade-off relationship between the amount of information processing and the extraction accuracy. For example, when determining whether the image in the sliding window corresponds to the range of the gesture operation while gradually moving the sliding window which is the cutout range in the two-dimensional image D11, the gesture can be made by increasing the moving step of the sliding window. If the extraction accuracy of the image is lowered and the moving step of the sliding window is made smaller, the number of times the process is repeated increases, so that the amount of information processing increases.

Therefore, the gesture coordinate calculation unit 403 in the fourth embodiment detects the gesture operation only from the range indicated by the virtual surface two-dimensional coordinate D20. As a result, the amount of information processing can be reduced, and the gesture operation is not erroneously detected in a region outside the range indicated by the virtual surface two-dimensional coordinate D20.

As described above, the gesture operation device 400 according to the fourth embodiment can obtain the effect described in the third embodiment. Further, the gesture operation device 400 according to the fourth embodiment can suppress false detection and reduce the amount of information processing at the same time.

Except for the above points, the gesture operating device 400 according to the fourth embodiment is the same as the gesture operating device 300 according to the third embodiment.

Embodiment 5.
FIG. 23 is a block diagram showing the gesture operation device 500 according to the fifth embodiment. In FIG. 23, components that are the same as or correspond to the components shown in FIG. 17 are designated by the same reference numerals as those shown in FIG. The gesture operating device 500 according to the fifth embodiment is different from the gesture operating device 300 according to the third embodiment in that the gesture coordinate acquisition unit 510 is configured. The gesture operation device 500 according to the fifth embodiment replaces the virtual surface two-dimensional coordinate calculation unit 302, the gesture coordinate calculation unit 303, and the virtual in-plane gesture coordinate calculation unit 304 in the gesture operation device 300 according to the third embodiment. A two-dimensional image virtual surface projection unit 502 and an in-plane gesture coordinate calculation unit 503 are provided.

In the gesture operation device 300 according to the third embodiment, the gesture coordinate calculation unit 303 extracts the gesture coordinates D21 in the two-dimensional image from the two-dimensional image D11. However, when the operator performing the gesture operation is at a position away from the front of the two-dimensional imaging unit 101, which is an imaging device, that is, at a position away from the optical axis, or when the operator is in the normal direction of the two-dimensional imaging unit 101. When the direction deviates from the optical axis, that is, the direction deviates from the optical axis, the gesture operation in the two-dimensional image D11 is detected as a deformed gesture operation, and the gesture in the two-dimensional image is detected. A situation may occur in which the coordinate D21 cannot be accurately detected.

Therefore, in the gesture operation device 500 according to the fifth embodiment, as shown in FIG. 23, the two-dimensional image virtual surface projection unit 502 uses the two-dimensional image D11 and the virtual surface three-dimensional coordinates D19 to form a virtual surface. The deformation of the above is compensated, and the two-dimensional image virtual surface projection image D51 which is the compensation result is output.

FIG. 24 is a diagram showing the operation of the gesture operation device 500 according to the fifth embodiment. Let the coordinates of the pixel P in the two-dimensional image D11 be (X _p , Y _p ). As shown in FIG. 24, the X coordinate of the pixel P in the two-dimensional image D11 (for example, the number of pixels in the X direction) is _{represented by X e,} which is 1/2 of the total number of pixels in the horizontal direction of the two-dimensional image D11. The number of pixels _{is represented by X max} , and the yaw angle of the 3D vector, which is the angle formed by the 3D vector V51 and the z-axis, which is the optical axis of the imaging system, is represented _{by α P.} The angle of view is represented _{by β P.} At this time, the following relational expression holds.
_{X P: X max = tanα P} : tanβ P
Therefore, the yaw angle α _P of the three-dimensional vector can be calculated by the following equation (3).

The pitch angle of the three-dimensional vector V51 is calculated in the same manner as in the equation (3).

By obtaining the direction vector passing through the pixel P in this way, a straight line (that is, passing through the imaging position) obtained by extending the three-dimensional vector V51 from the imaging position to the pixel P as the starting point. The intersection of the three-dimensional vector V51 as a direction vector) and the virtual surface 571 is obtained as the projection point 572 of the pixel P. The virtual surface 571 is acquired in the same manner as the virtual surface 371 in the third embodiment.

By performing this projection processing on all the pixels of the pixels on the two-dimensional image D11, or by performing the projection processing on the image within the range necessary for obtaining the projected image on the virtual surface 571 in order to reduce the processing amount. A two-dimensional image virtual surface projection image D51 is obtained. At this time, since the pixel size of the two-dimensional image D11 and the virtual surface 571 are generally different, some kind of interpolation is required. For interpolation, an image processing method such as the nearest neighbor method, the bilinear method, the bicubic method, or the Lanczos method can be used.

Further, the corresponding coordinates on the virtual surface 571 are obtained starting from the pixel P of the two-dimensional image D11, but conversely, the corresponding coordinates on the two-dimensional image D11 are obtained starting from the pixel in the virtual surface 571. Coordinates may be calculated.

The virtual in-plane gesture coordinate calculation unit 503 calculates the in-plane gesture coordinates D52 from the two-dimensional image virtual surface projection image D51 thus obtained.

As described above, the gesture operation device 500 according to the fifth embodiment can improve the gesture recognition accuracy. Further, the gesture operation device 500 according to the fifth embodiment can suppress false detection and reduce the amount of information processing at the same time.

Except for the above points, the gesture operating device 500 according to the fifth embodiment is the same as the gesture operating device 300 according to the third embodiment.

Modification example.
FIG. 25 is a block diagram showing an example of the hardware configuration of the modified examples of the first to fifth embodiments. The eye three-dimensional coordinate acquisition device 100 shown in FIG. 1 uses a memory 91 as a storage device for storing a program as software and a processor 92 as an information processing unit for executing a program stored in the memory 91. It can be realized (for example, by a computer). In this case, a part of each component in FIG. 1 can be realized by the memory 91 shown in FIG. 25 and the processor 92 that executes the program.

Further, the gesture coordinate acquisition units 310, 410, and 510 shown in FIGS. 17, 22, and 23 execute a memory 91 as a storage device for storing a program as software and a program stored in the memory 91. It can also be realized by using a processor 92 as an information processing unit (for example, by a computer). In this case, all or part of the gesture coordinate acquisition units 310, 410, and 510 can be realized by the memory 91 shown in FIG. 25 and the processor 92 that executes the program.

11 Imaging position, 20 people (operator), 21 eye position, 22 line of sight, 23 intersection of first straight line and second straight line, 23a intersection of first projection straight line and second projection straight line, 24 Intersection point, 25 intersection point, 30 light emitting element, 31 gazing point position, 31a gazing point position projection point, 32, 33, 34 pointer, 32a, 34a VDT device (display device), 33a projector device, 40 triangular survey execution plane , 41 Eye existence range straight line (normal line), 100 eye 3D coordinate acquisition device, 101 2D imaging unit, 102 gazing point position information providing unit, 103 eye 2D coordinate calculation unit, 104 1st linear calculation unit, 105 3D vector calculation unit, 106 2nd straight line calculation unit, 107 eye 3D coordinate calculation unit, 301 virtual surface 3D coordinate calculation unit, 302 virtual surface 2D coordinate calculation unit, 303 gesture coordinate calculation unit, 304 in virtual plane Gesture coordinate calculation unit, 310,410,510 Gesture coordinate acquisition unit, 371 virtual plane, 372 virtual plane reference point, 381 virtual plane, 571 virtual plane, 572 projection point of 2D image on virtual plane, D11 2D image, D12 gazing point position information, D13, β _X , β _Y maximum imaging angle, D14 eye two-dimensional coordinates, D15, LV1 first straight line, D16, V2 second three-dimensional vector, D17, LV3 second straight line, D18 eye 3D coordinates, D19 virtual surface 3D coordinates, D20 virtual surface 2D coordinates, D21 2D image in-plane gesture coordinates, D22, D42, D52 virtual in-plane gesture coordinates, PV1a first projection line, PV3a second Projection straight line.

Claims (15)

It is an eye 3D coordinate acquisition device that acquires eye 3D coordinates indicating the position of the eye viewing the gazing point from the 2D image captured by the 2D imaging unit and the gazing point position information indicating the position of the gazing point. hand,
An eye two-dimensional coordinate calculation unit that calculates eye two-dimensional coordinates indicating the two-dimensional position of the eye in the two-dimensional image, and an eye two-dimensional coordinate calculation unit.
From the maximum imaging angle of the two-dimensional imaging unit and the two-dimensional coordinates of the eye, a first three-dimensional vector from the first starting point, which is the imaging position of the two-dimensional imaging unit, to the position of the eye is used as a direction vector. A first straight line calculation unit that calculates a first straight line starting from the first starting point, and
A three-dimensional vector calculation unit that calculates a second three-dimensional vector from the two-dimensional image and the two-dimensional coordinates of the eye from the second starting point, which is the position of the eye, to the position of the gazing point.
A third vector that is the inverse vector of the second three-dimensional vector, has a third three-dimensional vector from the third starting point, which is the position of the gazing point, toward the position of the eye as a direction vector, and has the third starting point as the starting point. A second straight line calculation unit that calculates two straight lines, and
An eye three-dimensional coordinate calculation unit that calculates the eye three-dimensional coordinates using the triangulation method from the first straight line, the second straight line, and the gazing point position information.
An eye three-dimensional coordinate acquisition device characterized by having. The eye three-dimensional coordinate calculation unit
The first projective straight line is obtained by projecting the first straight line onto an arbitrary triangulation plane.
A second projective straight line is obtained by projecting the second straight line onto the triangulation execution plane.
The three-dimensional eye coordinates based on the normal line of the triangular survey execution plane at the intersection of the first projective line and the second projective line, and the intersection of the first straight line and the second straight line. The eye three-dimensional coordinate acquisition device according to claim 1, wherein the device is calculated. The eye three-dimensional coordinate calculation unit
The first projective straight line is obtained by projecting the first straight line onto an arbitrary triangulation plane.
A second projective straight line is obtained by projecting the second straight line onto the triangulation execution plane.
The above is based on the intersection of the normal plane of the triangular survey execution plane at the intersection of the first projection straight line and the second projection straight line with the first straight line, or the intersection of the second straight line. The eye three-dimensional coordinate acquisition device according to claim 1, wherein the eye three-dimensional coordinates are calculated. The eye three-dimensional coordinate acquisition device according to claim 2 or 3, wherein the triangulation execution plane is a plane parallel to a horizontal plane orthogonal to the direction of gravity. The second or third aspect of claim 2 or 3, wherein the triangular survey execution plane is a plane including an optical axis of the two-dimensional imaging unit and a straight line orthogonal to the direction of gravity and intersecting the optical axis. Eye 3D coordinate acquisition device. The eye three-dimensional coordinate acquisition device according to claim 2 or 3, wherein the triangulation execution plane is a plane perpendicular to the first straight line and the straight line orthogonal to the second straight line. From claim 1, the position of the gazing point is the position of the light emitting point of the light emitting element, the position of the light irradiation point by the laser pointer, or a predetermined position where the line of sight is expected to be directed. The eye three-dimensional coordinate acquisition device according to any one of 6. The eye according to any one of claims 1 to 6, wherein the position of the gazing point is the position of the image portion as the gazing point displayed on the display device or projected by the projector device. 3D coordinate acquisition device. The eye three-dimensional coordinate acquisition device according to any one of claims 1 to 8.
An imaging device that captures a two-dimensional image including gesture operations,
A gesture coordinate acquisition unit that acquires gesture coordinates in a virtual plane indicating the position of the gesture operation on a virtual surface that is a virtual operation surface on which the gesture operation is performed, and a gesture coordinate acquisition unit.
A gesture operating device characterized by having. The gesture operation device according to claim 9, wherein the image pickup device is the two-dimensional image pickup unit. The gesture coordinate acquisition unit
A virtual surface 3D coordinate calculation unit that calculates the virtual surface 3D coordinates included in the virtual surface based on the eye 3D coordinates provided by the eye 3D coordinate acquisition device.
A virtual surface 2D coordinate calculation unit that calculates virtual surface 2D coordinates from the virtual surface 3D coordinates,
The gesture operating device according to claim 9 or 10. The virtual surface three-dimensional coordinate calculation unit starts from the eye three-dimensional coordinate provided by the eye three-dimensional coordinate acquisition device, and extends from the eye three-dimensional coordinate in the direction of the second three-dimensional vector. The gesture operation device according to claim 11, wherein a surface on the obtained straight line including a point at a predetermined distance from the three-dimensional coordinates of the eye is set as the three-dimensional coordinates of the virtual surface. A gesture coordinate calculation unit that calculates gesture coordinates in a two-dimensional image indicating the range of gesture operations in the two-dimensional image captured by the imaging device.
The 11 or 12 according to claim 11, further comprising a virtual in-plane gesture coordinate calculation unit that calculates the in-virtual gesture coordinates from the two-dimensional virtual surface coordinates and the in-two-dimensional image gesture coordinates. Gesture control device. Among the two-dimensional images, a gesture coordinate calculation unit that calculates the gesture coordinates in the two-dimensional image of the range in which the gesture operation has occurred from the range indicated by the virtual surface two-dimensional coordinates.
Any of claims 11 to 13, further comprising a virtual in-plane gesture coordinate calculation unit that calculates the in-virtual gesture coordinates from the two-dimensional virtual surface coordinates and the in-two-dimensional image gesture coordinates. The gesture operating device according to item 1. A two-dimensional image virtual surface projection unit that outputs an image obtained by projecting the two-dimensional image onto the virtual surface as a two-dimensional image virtual surface projection image using the virtual surface three-dimensional coordinates.
The gesture operation according to claim 11, further comprising a virtual in-plane gesture coordinate calculation unit that calculates a range in which a gesture operation has occurred as the in-virtual gesture coordinates from the two-dimensional image virtual surface projection image. apparatus.

Images