JP2017191426A - Input device, input control method, computer program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input device which suppresses false recognition of gesture operation.SOLUTION: A HMD (Head Mount Display) 100 includes a camera 105 for acquiring a picked-up image including a user hand and a reference point fixedly set in a real space. The HMD 100 includes: a camera coordinate system conversion unit 209 which detects a position of the reference point in the picked-up image, from a motion of the camera 105; a hand gesture recognition unit 205 which recognizes gesture operation of the hand in the picked-up image, according to the change of position of the hand in the picked-up image within a predetermined time; a reference point gesture recognition unit 211 which recognizes a gesture of the reference point in the picked-up image, according to the change of a position of the reference point in the picked-up image within a predetermined time; and an output determination unit 212 which compares the gesture operation of the hand in the picked-up image with the gesture of the reference-point in the picked-up image, and determines whether or not to output a result corresponding to the gesture operation by the hand, on the basis of a comparison result.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an input device that recognizes a user's gesture as an input operation.

  There is an input device that accepts an input operation by continuously capturing a user with an imaging device and recognizing the user's gesture as an input operation from the captured image. An input operation using a gesture is referred to as a “gesture operation”. A gesture is a movement of an indicator such as a user's hand, a part of the body such as a finger, or an indicator. It is important for an input device using a gesture operation to accurately recognize a user's gesture. When the imaging device can be worn on the user's body, the imaging device moves according to the user's movement. For example, in an apparatus such as an HMD (Head Mount Display) in which an imaging device and a display device are integrated and worn on the user's head, the imaging range of the imaging device moves according to the movement of the user's neck. . Even when the user does not perform the gesture operation, the input device moves the position of the indicator in the captured image due to the movement of the imaging device according to the movement of the user, and this is used as the gesture operation. May be misrecognized. This is a gesture operation unintended by the user, leading to a decrease in operability.

  The gesture input device disclosed in Patent Literature 1 discloses a technique for recognizing a gesture from a captured image, obtaining a variation amount of the posture of the imaging device according to a user's body movement, and outputting a command corresponding to the gesture based on the variation amount. To do. This gesture input device suppresses erroneous recognition of a gesture operation by restricting the output of a command according to the gesture operation when the fluctuation amount of the imaging device is large.

JP 2012-146220 A

  The essential factor of misrecognition of gesture operation due to the movement of the imaging device is not the magnitude of the movement amount of the imaging device, but the indicator in the captured image moves like a gesture operation according to the movement of the imaging device. is there. It is difficult to determine whether or not the indicator in the captured image is moving like a gesture operation based on the amount of motion of the imaging device.

  In order to solve the above problems, it is a main object of the present invention to provide an input device that suppresses erroneous recognition of a gesture operation caused by movement of an imaging device.

  An input device according to the present invention includes an imaging unit that continuously captures a range including a predetermined reference point that is fixedly set in a real space and an indicator that performs an input operation, and an image that is captured by the imaging unit. First position detecting means for detecting the position of the indicator in the position, attitude detecting means for detecting the position and attitude of the imaging means, and the position of the reference point in the captured image from the position and attitude of the imaging means. An input operation based on a movement of the indicator within the captured image is detected by a change within a predetermined time of the position of the indicator detected from the second captured position detection unit and the captured image continuously detected by the first position detection unit. First recognizing means for recognizing and second recognizing means for recognizing movement of the reference point in the captured image based on a change in the position of the reference point detected by the second position detecting means within the predetermined time. Whether the movement of the indicator in the captured image is compared with the movement of the reference point in the captured image, and whether to output a result corresponding to the input operation by the indicator based on the comparison result Output judging means for judging whether or not.

  ADVANTAGE OF THE INVENTION According to this invention, the movement of the reference point in a captured image and the movement of a pointer can be compared, and the misrecognition of gesture operation can be suppressed according to a comparison result.

(A), (b), (c) is explanatory drawing of a structure of HMD. (A), (b) is explanatory drawing of gesture operation. (A), (b) is explanatory drawing of gesture operation. (A), (b) is explanatory drawing of a reference point. (A), (b) is explanatory drawing of a reference point. The flowchart showing an input control process. (A), (b) is explanatory drawing of a gesture recognition process. (A), (b) is explanatory drawing of a gesture recognition process. The functional block diagram of HMD. The flowchart showing an input control process. The functional block diagram of HMD. The flowchart showing an input control process.

  Hereinafter, embodiments will be described in detail with reference to the drawings. However, the components described in the present embodiment are merely examples, and are not intended to limit the scope of the present invention only to them.

[First Embodiment]
In this embodiment, an example will be described in which an HMD attached to the user's head is used as the input device. The HMD includes an imaging device and a display device, and displays a captured image captured by the imaging device on the display device. The input device accepts an input operation (gesture operation) by a movement (gesture) of a part of the user's hand, a part of the body such as a finger, or an indicator such as a pointer, and displays the result of the input operation on the display device. Alternatively, the input device transmits a command according to the accepted input operation to a predetermined information processing device, and causes the information processing device to execute processing according to the command. In addition, if the input device of this embodiment is a structure provided with the imaging device which can be mounted | worn with a user's body, it is not necessary to provide a display apparatus. In this case, the display device is provided separately from the imaging device. A display device provided separately from the imaging device is, for example, a projector.

(Constitution)
FIG. 1 is an explanatory diagram of the configuration of the HMD. FIG. 1A is an external view of the HMD 100. The HMD 100 is a spectacle type and is worn on the user's head to recognize a gesture operation by an indicator. In this embodiment, an example in which a user's hand is used as an indicator will be described. In order to recognize a gesture operation, the HMD 100 includes a camera 105 that is an imaging device at a position where the field of view of the user is included in the imaging range when worn. The camera 105 is a stereo camera, but may be an infrared distance camera. The HMD 100 also includes a display 106 that displays an image captured by the camera 105. For example, the HMD 100 displays an operation result corresponding to the gesture operation on the display 106.

  FIG. 1B is a hardware configuration diagram of the HMD 100. The HMD 100 is an information processing apparatus including a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and a storage 104. The CPU 101, ROM 102, RAM 103, storage 104, camera 105, and display 106 are connected via a system bus 107 so that they can communicate with each other. The CPU 101 controls the operation of the HMD 100 by reading a computer program stored in the ROM 102 and the storage 104 and executing it using the RAM 103 as a work area. The ROM 102 stores, for example, a startup program for performing startup processing of each hardware of the HMD 100 at startup. The storage 104 stores a control program for performing various processes by the HMD 100, for example. The storage 104 is a hard disk or an external storage medium. The CPU 101 controls the operations of the camera 105 and the display 106 by executing a computer program. The camera 105 continuously captures images and transmits the captured images that have been captured, for example, moving images, to the CPU 101.

  The case where the HMD 100 of the present embodiment displays a recognition result according to a gesture operation based on a captured image captured by the camera 105 on the display 106 will be described. FIG. 1C is a functional configuration diagram of the HMD 100 for performing such processing. The HMD 100 realizes each function by the CPU 101 executing a computer program, but at least a part may be realized by hardware. The HMD 100 functions as a captured image acquisition unit 201, an output determination unit 212, and a display control unit 213. The HMD 100 functions as a hand detection unit 202, a hand position detection unit 203, a hand position storage unit 204, and a hand gesture recognition unit 205 for recognizing a gesture operation. The HMD 100 functions as a reference point setting unit 206 and a first reference point storage unit 207 for determining a reference point that is a reference for determining the movement of the camera 105 in the world coordinate system representing the real space. The HMD 100 functions as a camera posture detection unit 208, a camera coordinate system conversion unit 209, a second reference point storage unit 210, and a reference point gesture recognition unit 211 for detecting the movement of the camera 105 in the real space.

The captured image acquisition unit 201 acquires a captured image in a range including the user's hand as a pointer and a reference point captured by the camera 105. Since the camera 105 continuously captures images, the captured image acquisition unit 201 continuously acquires captured images.
The hand detection unit 202 detects the user's hand region from the captured image acquired by the captured image acquisition unit 201. The hand detection unit 202 detects a hand region by detecting a skin color region in the captured image. The hand detection unit 202 may detect only a skin color region and a region whose distance from the HMD 100 (camera 105) is within a threshold as a hand region. The threshold value is determined according to the length of a human hand. This suppresses erroneous detection of a skin color area other than the user's hand as a hand area.

  The hand position detection unit 203 detects the position of the hand region detected by the hand detection unit 202 in the captured image. The position in the captured image is represented by a coordinate system based on the captured image (hereinafter referred to as “camera coordinate system”). The position of the hand region is generally represented by the barycentric coordinates of each pixel constituting the hand region. In addition, for example, a pixel having the longest shortest distance to the contour of the hand region among the pixels constituting the hand region may be set as the position of the hand region. In this case, the position near the center of the palm is the hand position. The hand position storage unit 204 stores the coordinates (position) of the hand region detected by the hand position detection unit 203 in the camera coordinate system. In order to continuously acquire captured images, the hand position storage unit 204 stores the position of the hand area in association with the detection time of the hand area.

  The hand gesture recognition unit 205 recognizes whether the hand movement is a gesture operation based on time-series data within a predetermined time of the position of the hand region stored in the hand position storage unit 204. The hand gesture recognizing unit 205 recognizes the movement of the hand based on a change in the position of the hand in consecutive captured images. The hand gesture recognition unit 205 calculates a feature amount such as the speed and acceleration of the hand from the position of the hand region and the change in the position of the hand region over time. The hand gesture recognition unit 205 performs matching between the calculated feature amount and a recognition target gesture model learned in advance, and recognizes the gesture operation when the likelihood of matching is large. The hand gesture recognition unit 205 models a recognition target gesture using an HMM (Hidden Markov Model), and uses a Viterbi algorithm for maximum likelihood path search at the time of matching. In addition, the hand gesture recognition unit 205 may perform matching using a rule-based recognition algorithm such as a decision tree.

  The reference point setting unit 206 sets a predetermined position in the world coordinate system as a reference point. The reference point is fixed in the world coordinate system and does not move. The first reference point storage unit 207 stores the coordinates (position) of the reference point set by the reference point setting unit 206 in the world coordinate system. Details of the “reference point” will be described later.

  A camera posture detection unit 208 detects the position and posture of the camera 105. The camera posture detection unit 208 uses, for example, a motion vector in continuous captured images acquired from the captured image acquisition unit 201 or a position / orientation detection marker provided in the world coordinate system around the user. The position and orientation of the camera 105 can be detected without using it. The camera posture detection unit 208 can also detect the position and posture of the camera 105 with high accuracy by separately adding sensors such as an infrared beacon, an acceleration sensor, and a gyro sensor to the HMD 100.

  The camera coordinate system conversion unit 209 converts the coordinates of the reference point stored in the first reference point storage unit 207 in the world coordinate system into a camera coordinate system. That is, the camera coordinate system conversion unit 209 detects the position of the reference point in the captured image by converting the position of the reference point set in the world coordinate system into a position in the camera coordinate system. The camera coordinate system is a coordinate system in a captured image captured by the camera 105, and is determined based on the position and orientation of the camera 105 detected by the camera orientation detection unit 208. The camera coordinate system conversion unit 209, for example, based on the parallel movement parameter and the rotation parameter based on the movement of the camera 105 in addition to the focal length, principal point coordinates, and distortion coefficient of the camera 105, Performs coordinate conversion to the camera coordinate system. The second reference point storage unit 210 stores the coordinates (position) of the reference point converted into the camera coordinate system. Since the captured images are continuously acquired, the camera posture detection unit 208 continuously detects the position and posture of the camera 105. For this purpose, the camera coordinate system conversion unit 209 continuously determines the camera coordinate system, and converts the coordinates of the reference point in the world coordinate system to the camera coordinate system. The second reference point storage unit 210 stores the position of the reference point coordinate-converted into the camera coordinate system in association with the coordinate determination time.

  The reference point gesture recognition unit 211 recognizes the movement of the reference point as a gesture operation based on time series data within a predetermined time of the position of the reference point in the camera coordinate system stored in the second reference point storage unit 210. The reference point gesture recognition unit 211 recognizes the movement of the reference point based on a change in the position of the reference point in consecutive captured images. The reference point gesture recognition unit 211 detects the position of the reference point in the same time interval as the time interval (hereinafter referred to as “gesture execution interval”) that is considered to be a gesture execution in the gesture recognition process of the hand gesture recognition unit 205. Processing is performed using time series data. Further, the reference point gesture recognition unit 211 performs matching on the assumption that the time series data of the position of the reference point in the camera coordinate system is the time series data of the hand position, and recognizes the gesture of the reference point. This is to determine whether or not the movement is recognized as a gesture when the same movement as the reference point is performed by hand. However, the movement of the reference point is a movement in the captured image corresponding to a change in the movement of the camera 105. The recognition process of the reference point gesture recognition unit 211 can be performed by the same algorithm as the recognition process of the hand gesture recognition unit 205. When the reference point gesture recognition unit 211 requires the position of the reference point in the camera coordinate system and the feature amount other than the speed and acceleration required for the recognition process, the hand feature amount used by the hand gesture recognition unit 205 in the recognition process Is used. Alternatively, the reference point gesture recognition unit 211 performs recognition processing using an algorithm that excludes a likelihood calculation portion based on a feature amount that cannot be acquired from the reference point.

  The output determination unit 212 compares the recognition result of the hand gesture operation by the hand gesture recognition unit 205 with the recognition result of the reference point gesture by the reference point gesture recognition unit 211, and determines the gesture operation of the hand based on the comparison result. It is determined whether to output a recognition result. The display control unit 213 displays the recognition result of the hand gesture operation on the display 106.

(Gesture operation)
2 and 3 are explanatory diagrams of a gesture operation by a user wearing the HMD 100. FIG. In FIG. 2, the user performs a gesture operation to move the right hand from right to left. In FIG. 3, the user is not performing a gesture operation, and the head is swung from left to right. 2A and 3A are overhead views of the user wearing the HMD 100, and FIGS. 2B and 3B are illustrations of images captured by the camera 105. FIG.

  As shown in FIG. 2, when the user moves the right hand from right to left, the HMD 100 performs a gesture operation in which the user moves the hand from right to left by hand movement (position time-series data) in the captured image. Recognize what has been done. As shown in FIG. 3, when the user swings his / her neck from left to right without moving his / her hand, the hand in the captured image moves from right to left (FIG. 3B). This is the same as the hand movement in FIG. For this reason, the HMD 100 erroneously recognizes that the user has performed a gesture operation of moving his hand from right to left even though the user has not performed a gesture operation. The HMD 100 of the present embodiment suppresses such erroneous recognition.

(Reference point)
4 and 5 are explanatory diagrams of reference points used to suppress erroneous recognition of gesture operations. In the present embodiment, it is determined whether or not to output a recognition result of the gesture operation by hand based on the movement of the reference point in the camera coordinate system. Therefore, proper setting of the reference point is important for accurate gesture operation recognition. In FIG. 4, as in FIG. 2, the user performs a gesture operation to move the right hand from right to left. In FIG. 5, as in FIG. 3, the user does not perform a gesture operation, and swings his neck from left to right. 4A and 5A are overhead views of the user wearing the HMD 100, and FIGS. 4B and 5B are examples of images captured by the camera 105. FIG.

  In FIG. 4A, a reference point 401 is set in front of the user in the world coordinate system. When the user shakes his / her hand from right to left without moving his / her neck, the camera 105 captures an image in which the hand moves from right to left (FIG. 4B). The movement of the reference point 401 in the captured image (camera coordinate system) due to the movement of the HMD 100 is significantly different from the movement of the hand because the camera 105 does not move. The movement of the reference point 401 for that purpose is not recognized as a gesture of shaking from right to left. This indicates that the hand movement in the captured image is not caused by the movement of the HMD 100. Therefore, in this case, the HMD 100 permits the output of the recognition result of the gesture operation that the hand is swung from the right to the left.

  In FIG. 5A, the reference point 401 is set at the same position as in FIG. When the user swings his / her neck from left to right without moving his / her hand, the camera 105 captures an image in which the hand moves from right to left (FIG. 5B). The movement of the reference point 401 in the captured image (camera coordinate system) due to the movement of the HMD 100 is similar to the movement of the hand, and is recognized as a gesture of shaking from right to left in the captured image. This indicates that the hand gesture in the captured image is caused by the movement of the HMD 100. Therefore, in this case, the HMD 100 stops outputting the recognition result of the gesture operation that the hand is swung from the right to the left.

  The reference point 401 is set in the vicinity of a trajectory where the hand moves when the user performs a gesture operation, thereby improving the accuracy of gesture recognition determination. This is because the closer the position of the user's hand in the world coordinate system and the position of the reference point 401 are, the more similar the movement of the hand caused by the movement of the camera 105 and the movement of the reference point 401 are. For example, the reference point 401 is preferably set at a distance of 50 [cm] from the HMD 100 (camera 105) within the imaging range of the HMD 100 (camera 105) at the start of the gesture recognition process. Conversely, when the reference point 401 is set at a position far away from the HMD 100 (camera 105) or behind the user, the movement of the reference point 401 caused by the movement of the camera 105 and the movement of the hand are deviated, and the gesture operation is performed. The accuracy of determining the output of the recognition result will be lowered. The reference point 401 is a virtual point in the world coordinate system that is not actually displayed in the image captured by the camera 105. A virtual object having a volume may be set as the reference point 401.

(Gesture recognition processing)
FIG. 6 is a flowchart showing an input control process by such a user's gesture operation by the HMD 100. This process is performed when the user wears the HMD 100 on the head and performs a gesture operation. At the start of processing, the camera 105 has started imaging, and continuously inputs captured images to the CPU 101.

  In the HMD 100, the reference point setting unit 206 sets the position of the reference point 401 in the world coordinate system, and the coordinates of this position in the world coordinate system are stored in the first reference point storage unit 207 (S301). The camera orientation detection unit 208 detects the position and orientation of the camera 105 in the world coordinate system based on the captured image acquired by the captured image acquisition unit 201 (S302). The camera coordinate system conversion unit 209 converts the coordinates in the world coordinate system of the reference point 401 stored in the first reference point storage unit 207 into a camera coordinate system based on the position and orientation of the camera 105, and converts the second reference point It memorize | stores in the memory | storage part 210 (S303). Since the camera 105 continuously captures images, the second reference point storage unit 210 stores time-series data representing the position of the reference point 401 in the camera coordinate system according to the time series.

  The hand detection unit 202 detects the user's hand region based on the captured image acquired by the captured image acquisition unit 201 (S304). The hand position detection unit 203 detects the position of the hand region detected by the hand detection unit 202 in the camera coordinate system, and stores it in the hand position storage unit 204 (S305). Since the camera 105 continuously captures images, the hand position storage unit 204 stores time series data representing the position of the hand region in the camera coordinate system according to the time series. The hand gesture recognition unit 205 recognizes whether or not the user's hand movement is performing a gesture operation based on the time-series data of the position of the hand region stored in the hand position storage unit 204 (S306). When the hand gesture recognition unit 205 determines that the gesture operation has not been performed (S307: N), the HMD 100 determines whether or not the next captured image has been acquired (S312).

  When it is recognized that a gesture operation has been performed (S307: Y), the reference point gesture recognition unit 211 is based on time-series data representing the position of the reference point of the camera coordinate system stored in the second reference point storage unit 210. Then, a gesture due to the movement of the reference point is recognized (S308). The reference point gesture recognition unit 211 recognizes a gesture based on the reference point using time series data of the position of the reference point in the camera coordinate system in the same time interval as the gesture execution interval in the process of S306. When a volume object is set as the reference point 401, the reference point gesture recognition unit 211 may use the size of the object as a feature amount for the gesture recognition process.

  When the recognition result of the gesture by the reference point and the gesture operation by the hand match (S309: Y), the output determination unit 212 stops outputting the recognition result of the gesture operation by the hand (S310). That the gesture by the reference point matches the gesture operation by the hand means that the movement of the reference point 401 in the camera coordinate system due to the movement of the camera 105 is similar to the hand movement at the time of the gesture operation ( (Refer FIG.5 (b)). That is, it can be interpreted that a captured image that is recognized as a gesture operation without moving the hand is captured by the movement of the camera 105. Therefore, the HMD 100 determines that the gesture operation is recognized incorrectly, and stops outputting the gesture operation recognition result to the display control unit 213.

  When the recognition result of the gesture by the reference point and the gesture operation by the hand do not match (S309: N), the output determination unit 212 permits the output of the recognition result of the gesture operation by the hand (S311). The display control unit 213 displays an image corresponding to the recognition result of the gesture operation. The fact that the gesture based on the reference point and the gesture operation of the hand do not coincide with each other indicates that the movement of the reference point in the camera coordinate system caused by the movement of the camera 105 is greatly different from the movement of the hand during the gesture operation (see FIG. 4 (b)). That is, it can be interpreted that the movement of the camera 105 is not recognized as a gesture operation unless the user moves his / her hand. Therefore, the HMD 100 determines that the recognition result of the gesture operation is as intended by the user, and permits the output of the recognition result of the gesture operation to the display unit 123.

  The CPU 101 of the HMD 100 determines whether or not the next captured image has been acquired from the camera 105 (S312). When the next captured image is acquired (S312: Y), the CPU 101 repeats the processing from S302 onward. When the next captured image is not acquired (S312: N), the CPU 101 ends the process.

  The HMD 100 of the present embodiment as described above suppresses misrecognition of the gesture operation by the hand due to the movement of the camera 105 by using the reference point set in the world coordinate system. Therefore, the HMD 100 can prevent a gesture operation not intended by the user when the camera 105 attached to the user's body moves. In particular, the HMD 100 recognizes how the hand movement in the captured image caused by the movement of the camera 105, which is an essential factor of misrecognition of the gesture operation, is similar to the gesture operation. It is possible to make a determination based on

[Second Embodiment]
The reference point is preferably set appropriately according to the imaging range of the camera 105, but the reference point once set may deviate from an appropriate position depending on the movement of the user wearing the camera 105. 7 and 8 are explanatory diagrams of gesture recognition processing when the reference point 401 deviates from an appropriate position. In FIG. 7, the reference point 401 is set at a position far away from the HMD 100 (camera 105). In FIG. 8, the reference point 401 is located behind the user due to the movement of the user and is out of the imaging range of the camera 105. FIGS. 7A and 8A are overhead views of the user wearing the HMD 100, and FIGS. 7B and 8B are illustrations of images captured by the camera 105. FIG. In any case, since the reference point 401 is out of an appropriate position, the movement of the reference point 401 corresponding to the movement of the camera 105 is different from the movement of the user's hand, and it is difficult to accurately recognize the gesture operation. It is.

  For this purpose, the HMD 100 according to the second embodiment resets the reference point in accordance with the movement of the user wearing the imaging device (camera 105), such as the position and posture. Specifically, the HMD 100 resets the reference point at the timing when the camera 105 is stationary. Since the appearance and hardware configuration of the HMD 100 are the same as those of the HMD 100 of the first embodiment, description thereof is omitted.

  FIG. 9 is a functional configuration diagram of the HMD 100 according to the second embodiment. The functional configuration of the HMD 100 of the second embodiment is a configuration in which a camera still detection unit 501 is added to the functional configuration of the HMD 100 of the first embodiment shown in FIG. Explanation of similar functions is omitted. The camera still detection unit 501 detects whether the camera 105 is stationary based on changes in the position and orientation of the camera 105 detected by the camera orientation detection unit 208.

  FIG. 10 is a flowchart illustrating an input control process by a user's gesture operation by the HMD 100 according to the second embodiment. The same steps as those in the flowchart of the first embodiment shown in FIG.

  The HMD 100 that has detected the movement of the camera 105 after setting the reference point determines whether or not the detected movement of the camera 105 is stationary by the camera stationary detection unit 501 (S301, S302, S1001). The stillness determination is performed by comparing the amount of movement of the camera 105 in the world coordinate system, for example, the amount of change in the position and angle of the camera 105 with a predetermined threshold. The camera still detection unit 501 determines that the camera 105 is still when the amount of motion is equal to or less than the threshold. When not standing still (S1001: N), HMD100 performs processing after S303. When it is stationary (S1001: Y), the reference point setting unit 206 sets a new reference point in the world coordinate system and stores it in the first reference point storage unit 207 (S1002). When the setting of a new reference point is completed, the HMD 100 performs the processes after S303.

  As described above, the HMD 100 according to the second embodiment sets a new reference point each time the movement of the camera 105 stops, and prevents the reference point from deviating from the optimum position. Thereby, the reference point is always set in the vicinity of the position where the user performs the gesture. Therefore, in addition to the effects of the HMD 100 according to the first embodiment, the HMD 100 according to the second embodiment can detect an error in gesture operation by hand due to the movement of the camera 105 even when the user's movement is large. It has the effect of suppressing recognition.

[Third Embodiment]
In the first and second embodiments, the reference point is set before recognizing the gesture operation by hand, but the reference point may be set after recognizing the gesture operation. The HMD 100 of the third embodiment sets a reference point during execution of a hand gesture operation. Since the gesture operation is being performed, the reference point can be set to an appropriate position based on the position of the hand.

  Since the appearance and hardware configuration of the HMD 100 are the same as those of the HMD 100 of the first embodiment, description thereof is omitted. FIG. 11 is a functional configuration diagram of the HMD 100 according to the third embodiment. The functional configuration of the HMD 100 of the third embodiment is a configuration in which a world coordinate system conversion unit 601 and a camera posture storage unit 602 are added to the functional configuration of the HMD 100 of the first embodiment shown in FIG. Explanation of similar functions is omitted.

  The world coordinate system conversion unit 601 coordinates the position of the hand in the camera coordinate system detected by the hand position detection unit 203 in the world coordinate system based on the position and orientation of the camera 105 detected by the camera orientation detection unit 208. Convert. The camera posture storage unit 602 stores movements such as the position and posture of the camera 105 detected by the camera posture detection unit 208.

  FIG. 12 is a flowchart illustrating an input control process by a user's gesture operation by the HMD 100 according to the third embodiment. The same steps as those in the flowchart of the first embodiment shown in FIG. This process is performed when the user wears the HMD 100 on the head and performs a gesture operation. At the start of processing, the camera 105 has started imaging, and continuously inputs captured images to the CPU 101.

  Based on the captured image acquired by the captured image acquisition unit 201, the camera orientation detection unit 208 detects movements such as the position and orientation of the camera 105 and stores them in the camera orientation storage unit 602 (S1201). The hand detection unit 202 detects the user's hand region based on the captured image acquired by the captured image acquisition unit 201 (S304). The hand position detection unit 203 detects the position of the hand region detected by the hand detection unit 202 in the camera coordinate system, and stores it in the hand position storage unit 204 (S305). Since the camera 105 continuously captures images, the hand position storage unit 204 stores time series data representing the position of the hand region in the camera coordinate system according to the time series.

  The world coordinate system conversion unit 601 converts the position of the hand region stored in the hand position storage unit 204 in the camera coordinate system based on the position and posture (motion) of the camera 105 stored in the camera posture storage unit 602. The coordinates are converted into a system and stored in the hand position storage unit 204 (S1202).

  The hand gesture recognition unit 205 recognizes whether or not the user's hand movement is performing a gesture operation based on the time-series data of the position of the hand region in the camera coordinate system stored in the hand position storage unit 204 (S306). ). When the hand gesture recognition unit 205 determines that the gesture operation has not been performed (S307: N), the HMD 100 determines whether or not the next captured image has been acquired (S312).

  When it is recognized that a gesture operation has been performed (S307: Y), the reference point setting unit 206 uses the world coordinate system of the reference point based on the position of the hand region stored in the hand position storage unit 204 in the world coordinate system. Set the position at. The reference point setting unit 206 stores the position of the set reference point in the world coordinate system in the second reference point storage unit 210 (S1203). Here, the reference point setting unit 206 sets the average position of the hand region in the gesture execution section recognized in the gesture operation recognition process of S306 performed immediately before as the reference point position. By setting the position of the reference point after recognizing the gesture operation, the reference point can be arranged near the hand during the gesture operation.

  Based on the position and orientation of the camera 105 stored in the camera orientation storage unit 602, the camera coordinate system conversion unit 209 determines the coordinates (position) in the world coordinate system of the reference point set in S1203 as the camera coordinates in the gesture execution section. The system coordinates are converted (S1204). The camera coordinate system conversion unit 209 causes the second reference point storage unit 210 to store the coordinates (position) of the reference point converted into the coordinates of the camera coordinate system. After storing the coordinates (positions) of the reference points in the camera coordinate system in the second reference point storage unit 210, the HMD 100 performs the processing from S308 onward.

  As described above, the HMD 100 according to the third embodiment sets the reference point after recognizing the gesture operation by the hand, and retrospectively calculates how the position of the reference point in the camera coordinate system has changed in the gesture execution section. To do. Thereby, the HMD 100 of the third embodiment suppresses the difference between the reference point and the position of the hand in addition to the effects of the HMD 100 of the first embodiment, and misrecognizes the gesture operation by the hand due to the movement of the camera 105. It has the effect of suppressing.

  In the first to third embodiments, the case where one reference point is set has been described, but a plurality of reference points may be provided. When a plurality of reference points are provided, it is possible to suppress a difference between the reference point and the hand position by performing a gesture operation determination according to the movement of one or more reference points closest to the hand position in the camera coordinate system. it can. Further, the imaging device (camera 105) can be applied to a handy camera, an in-vehicle camera, or the like in addition to the one attached to the user's body.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

Imaging means for continuously imaging a range including a predetermined reference point fixedly set in the real space and an indicator for performing an input operation;
First position detection means for detecting the position of the indicator in the captured image captured by the imaging means;
Posture detection means for detecting the position and posture of the imaging means;
Second position detection means for detecting the position of the reference point in the captured image from the position and orientation of the imaging means;
First recognition means for recognizing an input operation due to movement of the indicator in the captured image by a change within a predetermined time of the position of the indicator detected from consecutive captured images by the first position detection means;
Second recognition means for recognizing movement of the reference point in the captured image by a change in the position of the reference point detected by the second position detection means within the predetermined time;
Whether the movement of the indicator in the captured image is compared with the movement of the reference point in the captured image, and whether to output a result corresponding to the input operation by the indicator based on the comparison result. An output determining means for determining,
Input device. The output determination unit stops outputting the result according to the input operation by the indicator when the movement of the indicator in the captured image matches the movement of the reference point in the captured image. When the movement of the indicator in the captured image and the movement of the reference point in the captured image do not match, output of a result according to the input operation by the indicator is permitted. And
The input device according to claim 1. It further comprises a reference point setting means for setting the reference point in the vicinity of a trajectory where the indicator moves.
The input device according to claim 1 or 2. The reference point setting means sets the reference point at a position of 50 [cm] from the imaging means.
The input device according to claim 3. Further comprising stationary detection means for detecting whether or not the imaging means is stationary based on a change in position and orientation of the imaging means detected by the orientation detection means;
The reference point setting means resets the reference point when the imaging means is detected stationary.
The input device according to claim 3 or 4. Further comprising conversion means for converting the position of the indicator in the captured image into a position in the real space based on the position and attitude of the imaging means detected by the attitude detection means;
The reference point setting means sets the reference point based on the position of the indicator converted to the position in the real space.
The input device according to claim 3 or 4. The second position detecting means performs coordinate conversion of the position of the reference point set by the first coordinate system representing the real space to a position in the second coordinate system based on the captured image, thereby Detecting a position in the captured image;
The first position detecting means detects a position of the indicator in the second coordinate system based on the captured image.
The input device according to claim 1. The second recognizing unit recognizes a movement of the reference point in the captured image by a process similar to that of the first recognizing unit.
The input device according to claim 1. The first recognizing unit calculates a feature amount from a change within a predetermined time of the position of the indicator detected from consecutive captured images by the first position detecting unit, and a recognition target gesture model learned in advance. Recognizing an input operation caused by a movement of the indicator in the captured image.
The input device according to claim 8. The imaging means is mounted on a user's body,
The input device according to claim 1. A method executed by an input device that acquires a captured image from an imaging unit that continuously captures a range including a predetermined reference point that is fixedly set in a real space and an indicator that performs an input operation,
Detecting the position and orientation of the imaging means;
Detecting the position of the reference point in the captured image according to the detected position and orientation of the imaging means;
Detecting the position of the indicator in the captured image captured by the imaging means;
A step of recognizing an input operation due to a movement of the indicator in the captured image by a change within a predetermined time of the position of the indicator detected from consecutive captured images;
Recognizing movement of the reference point in the captured image by a change in the position of the reference point in the captured image within the predetermined time;
Whether the movement of the indicator in the captured image is compared with the movement of the reference point in the captured image, and whether to output a result corresponding to the input operation by the indicator based on the comparison result. A step of determining,
Input control method. A computer that acquires a captured image from an imaging unit that continuously captures a range including a predetermined reference point that is fixedly set in a real space and an indicator that performs an input operation;
First position detection means for detecting the position of the indicator in the captured image captured by the imaging means;
Posture detection means for detecting the position and posture of the imaging means;
Second position detection means for detecting the position of the reference point in the captured image from the position and orientation of the imaging means;
First recognition means for recognizing an input operation due to movement of the indicator in the captured image based on a change within a predetermined time of the position of the indicator detected from consecutive captured images by the first position detection means;
Second recognition means for recognizing movement of the reference point in the captured image based on a change in the position of the reference point detected by the second position detection means within the predetermined time;
Whether the movement of the indicator in the captured image is compared with the movement of the reference point in the captured image, and whether to output a result corresponding to the input operation by the indicator based on the comparison result. Output judging means for judging,
Computer program to function as.   A computer-readable storage medium storing the computer program according to claim 12.