JP2024032409A - Information processing equipment and HMD - Google Patents

Abstract

The present invention provides a technology that enables highly accurate recognition of a user's own hand from among the hands included within the field of view of an HMD through simple processing. [Solution] An information processing device that recognizes the hand of a user wearing the HMD from an image photographed by the HMD; acquisition means for acquiring sensing data regarding the position of the device from a sensor mounted on the device worn on the hand; and a movement of the hand detected from the photographed image and the movement of the hand detected from the photographed image. A comparison process is performed to compare the movement of the device recognized from the sensing data, and if the movement of the hand and the movement of the device match, the hand detected from the captured image is determined to be the user's hand. and processing calculation means for determining that there is. [Selection diagram] Figure 1

Description

The present invention relates to a technique for recognizing a user's hand included within the field of view of an HMD.

There is a mixed reality (MR) technology that fuses real space and virtual space and allows users to interact with virtual objects. MR technology realizes interaction by combining and presenting computer graphics (CG) representing virtual objects with real scenery, and by expressing contact between real objects and virtual objects.

In MR technology, it is assumed that users perform gesture operations with their own hands to move virtual objects in relation to real scenery. Gesture operations allow CG objects to be moved and manipulated without a controller. However, if there are multiple people other than the person experiencing the experience in the same space, they may not be able to distinguish between their own hands and those of others, and their own HMD (Head Mount Display) may be unintentionally damaged by the gestures of the other person's hands. There is a possibility that it may be manipulated.

Technologies related to object recognition in three-dimensional space are disclosed in Patent Documents 1 and 2. Patent Document 1 discloses a method of recognizing a person's motion using an image or a motion sensor and identifying a target person based on the characteristics of the motion. Patent Document 2 discloses a method of accurately expressing the front-back relationship between a real object and a virtual object by measuring the depth position of a real object such as a hand using a stereo camera.

Japanese Patent Application Publication No. 2015-61577 Japanese Patent Application Publication No. 2012-13514

The conventional technology disclosed in Patent Document 1 learns the movements of a target person to be identified in advance, and determines the current movement by determining the learned features and the features extracted from the person moving within a predetermined area. Determine whether the person doing so is the person you want to identify. However, such learning-based methods require advance preparation such as registering the target person's movements and are complicated. There is also the inconvenience that those who have not registered their operation cannot use the system.

The present invention was made in view of the above-mentioned circumstances, and its purpose is to provide a technology that enables highly accurate recognition of the user's own hand from among the hands included in the field of view of an HMD through simple processing. Our goal is to provide the following.

The present disclosure is an information processing device that recognizes the hand of a user wearing an HMD from an image taken by an HMD (Head Mount Display), and detects the hand from the photographed image taken by the HMD. a detection means; an acquisition means for acquiring sensing data regarding the position of the device from a sensor mounted on the device worn on the hand of the user; A comparison process is performed to compare the movement of the hand detected from the sensing data with the movement of the device recognized from the sensing data, and if the movement of the hand and the movement of the device match, the hand detected from the captured image is and processing calculation means for determining that the hand is the user's hand.

According to the present invention, it is possible to recognize the user's own hand from among the hands included in the field of view of the HMD with simple processing and with high precision.

Block diagram showing the configuration of the mixed reality system Block diagram showing the hardware configuration of the HMD Block diagram showing the configuration of a small operating device Diagram showing how the mixed reality system is used Flowchart of operation target recognition processing in the first embodiment Diagram showing an example of movement of a small operating device Diagram showing an example of hand positions detected by a small operating device and HMD Diagram showing an example of hand movement in an HMD image Flowchart of operation target recognition processing in the second embodiment Flowchart of operation target recognition processing in the third embodiment Flowchart of operation target recognition processing in the fourth embodiment Flowchart of operation target recognition processing in the fifth embodiment Flowchart of operation target recognition processing in the sixth embodiment Flowchart of operation target recognition processing in the seventh embodiment

Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. Note that the embodiment described below is an example of means for implementing the present invention, and may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. It is also possible to combine the embodiments as appropriate.

(First embodiment)
The configuration of the mixed reality system according to the first embodiment will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing the overall configuration of a mixed reality system according to a first embodiment. FIG. 2 is a block diagram showing the hardware configuration of the HMD. FIG. 3 is a block diagram showing the configuration of the small operating device. FIG. 4 is a diagram showing how the mixed reality system is used.

The mixed reality system 1 includes an HMD (Head Mount Display) 100 and a small operating device 300. The HMD 100 is a device worn on the head of a user, and includes a goggle device 100A and an information processing device 103. Furthermore, the small operating device 300 is a device worn on the user's hand. FIG. 4 schematically shows an example of how a user wears the HMD 100 and the small operating device 300.

The CPU 200 is a processor that controls the entire system by executing programs for operating the system. The processing units 104 to 109 shown in FIG. 1 are realized in software by the CPU 200 executing a program read from the ROM 202. However, some of these processing units may be implemented using hardware (eg, ASIC or FPGA) independent of the CPU 200. A processor (GPU, DSP, etc.) may be provided to assist the arithmetic processing of the CPU 200.

The goggle device 100A of the HMD 100 is a head-mounted display device that includes an image capturing section 101 and an image display section 102. As shown in the schematic diagram of FIG. 4, in the HMD 100 of this embodiment, an image capturing unit 101 is arranged at the user's viewpoint position, and a live-action video captured by the image capturing unit 101 is displayed on the image display unit 102. The video is displayed in a see-through format. In the case of a video see-through type HMD 100, a method is generally used in which the image capturing unit 101 is treated as the user's viewpoint position and orientation. Note that although the HMD 100 of this embodiment is of a video see-through type, it should be understood that this is merely an example. For example, it is also applicable to a virtual reality HMD that does not display live-action video on the image display unit 102. In that case, the image photographing unit 101 may be used not as a see-through camera but as a camera for photographing the hand, which is the object to be measured.

The information processing device 103 of the HMD 100 uses the image photographed by the image photographing section 101 to create data to be displayed on the image display section 102. The information processing device 103 is a small computer including a processor (CPU 200) and memory (RAM 201, ROM 202) shown in FIG. The information processing device 103 may be built in the same housing as the goggle device 100A, or may be configured in a housing independent from the goggle device 100A. When the information processing device 103 and the goggle device 100A are configured in independent housings, the information processing device 103 and the goggle device 100A are connected to each other by wire or wirelessly so that they can communicate with each other.

The image capturing unit 101 is fixed to the casing of the goggle device 100A, and captures an image of real space at a predetermined frame rate (for example, 30 frames/second). The image photographed by the image photographing section 101 is stored in the photographed image storage section 104. The photographed image storage section 104 stores the image acquired from the image photographing section 101 in the RAM 201. This image capturing unit 101 is composed of two real cameras, and a real camera for the left eye and a real camera for the right eye are arranged at positions close to both eyes of the HMD wearer. Hereinafter, the pair of left and right images obtained by the image capturing unit 101 will be referred to as a stereo camera image.

The image display unit 102 is fixed to the casing of the goggle device 100A, and displays images generated by the information processing device 103. This image is, for example, an MR image in which 3DCG digital content (virtual object) is synthesized with a live-action video shot by the image shooting unit 101. The image display section 102 has a structure that covers the field of view of the user wearing the goggle device 100A, and can provide a highly immersive MR experience to the user viewing the images. This image display section 102 is configured with, for example, an organic EL display, a liquid crystal display, or the like.

A hand detection unit 105 detects a hand from each of the captured images stored in the captured image storage unit 104 by image recognition, and extracts contour points of the hand and fingers. The three-dimensional position calculation unit 106 calculates the three-dimensional position of the contour point extracted by the hand detection unit 105. The x, y coordinates of the contour point are acquired with the lower left of the VRAM area of the image photographed by the image photographing unit 101 of the HMD 100 as (0, 0). The z-coordinate in the depth direction is obtained from the stereo camera image. This coordinate system is called a camera coordinate system or a local coordinate system. The origin of the camera coordinate system does not have to be at the lower left of the VRAM area, and may be determined freely. In this embodiment, the hand detection unit 105 (or the hand detection unit 105 and the three-dimensional position calculation unit 106) constitutes a detection unit that detects a hand from a captured image captured by the HMD 100.

As an algorithm for detecting hands from images, classical machine learning represented by support vector machine may be used, or deep learning-based algorithms such as R-CNN, YOLO, SSD, and DCN may be used. Good too. Also, a rule-based detection algorithm may be used.

In order to correctly display the depth of the detected hand, there is a method of extracting the hand region from an image of the hand taken with a stereo camera and calculating the depth value of the hand with respect to the stereo camera. This method uses a method of determining depth values by triangulation for all corresponding points in a stereo image of the extracted hand outline (Patent Document 2). In Patent Document 2, a stereo camera installed in a video see-through type HMD, which is a display device for presenting mixed reality, is used.

LeapMotion, manufactured by LeapMotion, can measure the position and orientation of the fingers of the hand as well. In LeapMotion, the hand area can be detected using a stereo camera (image capturing unit 101). Since the position and orientation of the fingers of the hand can be estimated by using a separate depth sensor, the depth sensor may be installed in the HMD 100.

In this embodiment, an arbitrary point (referred to as a reference point or representative point) is determined from the contour points extracted by the hand detection unit 105, and a three-dimensional position with respect to the reference point is calculated. The reference point determined here is stored as a reference position during this process, and reference three-dimensional position information is periodically stored in the RAM 201 in accordance with the movement of the fingers.

The relative position/orientation calculation unit 107 calculates the position/orientation of each of the captured images stored by the captured image storage unit 104 . There are two ways to calculate the position and orientation: one is to create it using deep learning, and the other is to use an existing public library. The position and orientation of the photographed image corresponds to the position and orientation of the HMD 100 in real space, that is, the viewpoint position and line-of-sight direction of the user wearing the HMD 100. The position and orientation of the captured image are expressed in a global coordinate system.

The processing calculation unit 109 performs hand movements recognized based on three-dimensional position information calculated from the photographed image and a small hand movement recognized based on sensing data acquired from a sensor 303 installed in a small operating device 300 (described later). A comparison calculation with the movement of the operating device 300 is performed. Specifically, the processing calculation unit 109 evaluates whether the relative movement amounts of the coordinates of the hand detected by the HMD 100 and the small operating device 300 match. If they match, the processing calculation unit 109 determines that the hand detected by the HMD 100 and the hand on which the small operating device 300 is attached are the same hand, that is, the hand of the user of the HMD 100. Generally, coordinates obtained by a sensor have an error, so a certain error rate will be accepted, for example, the coordinates will be determined to be the same up to an error rate of 5%. The error rate may be determined freely.

When sensor raw data (unprocessed data) is sent from the small operation device 300 to the information processing device 103, the processing calculation unit 109 sends the raw data so that it can be compared with the value calculated by the three-dimensional position calculation unit 106. All you have to do is convert it to coordinate values. Alternatively, after converting the raw data into coordinate values on the small operation device 300 side, the coordinate values may be transmitted to the information processing apparatus 103.

The image drawing unit 108 draws (generates) an image to be displayed on the image display unit 102 of the HMD 100 based on the processing result of the processing calculation unit 109.

The close proximity wireless communication unit 110 includes, for example, an antenna for wireless communication, a modulation/demodulation circuit for processing wireless signals, and a communication controller. The close-range wireless communication unit 110 outputs a modulated wireless signal from an antenna and demodulates the wireless signal received by the antenna, thereby performing short-range wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). Realize. In this embodiment, Bluetooth (registered trademark) communication employs version 5.1 of Bluetooth (registered trademark) Low Energy, which has low power consumption. The HMD 100 can exchange data with an external device such as the small operating device 300 via the close proximity wireless communication unit 110.

The RAM 201 is a volatile memory used as a working area for temporarily storing data and programs. The ROM 202 is a non-volatile memory that non-temporarily stores data such as programs executed by the CPU 200, setting information for the HMD 100, and setting information required from next time onwards.

As shown in FIG. 3, the small operation device 300 includes a control section 301, an operation section 302, a sensor 303, a working memory 304, a power supply control section 305, and a communication section 306. As shown in FIG. 4, the small operating device 300 is a wearable device that can be worn on the user's hand or finger, and in this embodiment, a ring-shaped device is used. However, the form of the small operating device 300 is not limited to a hand-worn type, and any form of device may be used as long as it is capable of detecting the position and posture of the user's own hand. For example, a wristwatch-type device, a hand-held device, a finger-sticking device, or the like may be used.

The control unit 301 controls the small operating device 300 according to input signals and a program described below. Note that instead of the control unit 301 controlling the entire device, the entire device may be controlled by having multiple pieces of hardware share the processing.

The work memory 304 is used as a buffer memory for temporarily storing data acquired by the sensor 303, a work area for the control unit 301, and the like.

The sensor 303 is a sensor for measuring the three-dimensional position, movement, posture (orientation), etc. of the hand of the user wearing the small operating device 300, and is also called a position sensor, movement sensor, posture sensor, etc. As the sensor 303, for example, an IMU (internal measurement unit) sensor can be used. The IMU sensor can calculate posture changes, relative directions, and positions based on the angular velocity detected by the gyro and the acceleration detected by the accelerometer.

The operation unit 302 is used to receive instructions for the small operation device 300 from the user. The operation unit 302 includes, for example, a power button for instructing ON/OFF of the main power supply including power supply to the control unit 301 of the small operation device 300, a mode switching button for switching the function mode, and the like. Further, the operation unit 302 includes a button for starting communication with an external device such as the HMD 100 via the communication unit 306.

The power supply control section 305 is a unit for supplying power for the operation of the small operating device 300.

The communication unit 306, like the close proximity wireless communication unit 110 of the HMD 100, includes, for example, an antenna for wireless communication, a modulation/demodulation circuit for processing wireless signals, and a communication controller. The communication unit 306 realizes short-range wireless communication according to the IEEE802.15 standard (so-called Bluetooth (registered trademark)). The small operation device 300 can exchange data with an external device such as the HMD 100 via the communication unit 306.

As shown in FIG. 1, the close proximity wireless communication unit 110 of the HMD 100 and the small operation device 300 are connected by Bluetooth (registered trademark) communication. For example, various sensing data acquired by the sensor 303 can be transmitted to the close proximity wireless communication unit 110 of the HMD 100 via the communication unit 306 of the small operation device 300. In this embodiment, the close proximity wireless communication unit 110 constitutes an acquisition unit that acquires sensing data regarding the position and orientation of the small operating device 300 from the sensor 303 mounted on the small operating device 300.
Note that data communication between the HMD 100 and the small operating device 300 is not limited to Bluetooth (registered trademark) communication, and may be other communication methods.

FIG. 5 is a flowchart of operation target recognition processing executed in the HMD 100 according to the first embodiment. The operation target recognition process identifies the hand of the user wearing the HMD 100 (i.e., the person performing the gesture operation) from one or more hands (objects) included in the image taken by the HMD 100 (i.e., the field of view of the HMD 100). This is a process that recognizes the user's hand (operation target). While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 5 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 (processing calculation unit 109) of the HMD 100 determines whether the position of the small operating device 300 has changed based on the sensing data acquired from the small operating device 300. The presence or absence of a change in position can be determined, for example, by comparing multiple sensing data (that is, time-series position data). Note that since there is a certain error in the position coordinates measured by the sensor 303, if the change in position is slight, it may be regarded as "no change in position". If a change in the position of the small operating device 300 is detected in step S500, the CPU 200 assumes that the user of the HMD 100 is performing a gesture operation using the hand on which the small operating device 300 is attached, and proceeds to the process of step S501. move on. On the other hand, if no change in the position of the small operation device 300 is detected, the CPU 200 assumes that no gesture operation is being performed, and performs subsequent processing (recognition of the user's hand based on the image of the HMD 100, gesture recognition, etc.) Skip. In this way, by not executing the subsequent processing when the user's hand movement is not detected by the small operation device 300, it is possible to reduce wasteful calculation processing, reduce power consumption, and prevent other people from using the device. The possibility of erroneously recognizing the hand of the user can be eliminated as much as possible.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set to be determined as whether or not they are the user's own hands. Subsequently, the CPU 200 (three-dimensional position calculation unit 106) calculates the three-dimensional position coordinates of the contour points of each hand to be determined. Note that the photographed image used here is an image photographed at the same time as the sensing data referenced in step S500 (if there is no data at exactly the same time, then the closest time).

If one or more judgment target hands are detected, the judgment target hands are sequentially selected in step S502, and the process of step S503 is executed for the selected judgment targets. Since it is unclear which hand in the image is the user's own hand, the determination is made one by one. If no hand to be determined is detected, or if the process of step S503 is performed for all the determination targets, a NO determination is made in step S502, and the process ends.

In step S503, the CPU 200 (processing calculation unit 109) of the HMD 100 selects one outline point (reference point) to serve as a reference from among the hand outline points to be determined. Next, the processing calculation unit 109 takes into account the position and orientation of the photographed image calculated by the relative position and orientation calculation unit 107 and converts the coordinate values of the reference point from the camera coordinate system to the global coordinate system. Then, the processing calculation unit 109 determines whether the movement of the hand to be determined (that is, the change in the position of the reference point) matches the movement of the small operating device 300 (that is, the change in the position of the small operating device 300). evaluate. If it is determined that the movement of the hand to be determined and the movement of the small operating device 300 match (that is, they are movements of the same object), the CPU 200 advances the process to step S504. If it is determined that they do not match, the CPU 200 returns to step S502 and evaluates the next determination target.

In step S504, the processing calculation unit 109 recognizes that the hand to be determined is the hand (operation target) of the user wearing the HMD 100. Thereafter, this motion information of the operation target is subjected to gesture operation recognition processing (not shown).

FIG. 6 is an example of movement of the small operating device 300 for each frame. The position of the small operating device 300 changes from P1 (X1, Y1, Z1) to P2 (X2, Y2, Z2) to P3 (X3, Y3, Z3) as the hand moves. The position of the small operating device 300 for each frame is measured by the sensor 303. Since the small operating device 300 is attached (fixed) to the user's hand, the position of the small operating device 300 measured by the sensor 303 can be regarded as the position of the user's hand. The table on the left side of FIG. 7 is an example of time series data of the hand position P (that is, temporal changes in the hand position) measured and recorded by the sensor 303 of the small operating device 300.

FIG. 8 is an example of movement of a hand included in an image photographed by the image photographing unit 101 of the HMD 100 frame by frame. The position of the hand changes from Q1 (x1, y1, z1) to Q2 (x2, y2, z2) to Q3 (x3, y3, z3). In the example of FIG. 8, the contour point at the tip of the left index finger is selected as the reference point indicating the hand position. The table on the right side of FIG. 7 is an example of time-series data of the hand position Q (that is, temporal changes in the hand position) detected and recorded from the image by the HMD 100. The time series data of the hand position Q detected from the image is also called hand tracking information.

As shown in FIG. 7, the information processing device 103 of the HMD 100 stores, at each corresponding time t1, t2,... Information on the position Q is acquired. In the process of step S503 in FIG. 5, the processing calculation unit 109 calculates the relative movement amount of the hand position P detected by the small operating device 300 and the relative movement amount of the hand position Q detected from the image for a predetermined period. compare. The relative movement amount is the amount of relative change in the position of the hand in the current frame when the position of the hand in the previous frame is taken as a reference. For example, the relative movement amount between position P1 (X1, Y1, Z1) at time t1 and position P2 (X2, Y2, Z2) at time t2 is expressed as (X2-X1, Y2-Y1, Z2-Z1) . If the relative movement amount in the predetermined period is the same or within a predetermined error range, the processing calculation unit 109 determines that the hand to be determined is the hand of the user wearing the HMD 100. Here, the "predetermined period" used for comparison may be freely set. In this embodiment, the number of frames is set to about 3 to 10 frames. Further, the "predetermined error range" may be set in consideration of the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image.

By using the relative movement amount for comparison, there is no need to match the position detected by the small operating device 300 (the mounting position of the sensor 303) with the position of the reference point detected from the image, and the processing is extremely simplified. There are advantages. For example, as shown in the examples of FIGS. 6 and 8, even if the mounting position of the sensor 303 is the base of the index finger and the reference point is the tip of the index finger, the amount of relative movement between the two (see the dashed arrow) is approximately This is because they are the same and a simple comparison is possible. If absolute positions were to be compared, measurement preparation would be complicated, such as requiring calibration to precisely align the reference point detected from the image with the mounting position of the sensor 303 of the small operating device 300. .

As explained above, in this embodiment, IMU information acquired from the sensor 303 of the small operating device 300 and hand tracking information acquired from the image photographed by the image photographing unit 101 of the HMD 100 are used. In the hand included in the image, a small operating device 3
An object that moves in the same way as the hand movement detected in 00 is recognized as the hand of the user wearing the HMD 100. As a result, even if someone else's hand appears within the field of view of the HMD 100, it is possible to identify the user's own hand with simple processing and high accuracy, reducing malfunctions of gesture operations in MR. can be done. The method of this embodiment does not require learning the user's skeleton and movements unlike conventional methods, and also does not require advance preparation such as alignment (calibration) between the HMD 100 and the small operating device 300. Therefore, it is highly convenient.

(Second embodiment)
In the first embodiment, all the hands detected by the HMD 100 are compared with the detection information of the small operating device 300. However, in order to determine that it is the user's hand, it is necessary to continue comparing it with each determination target for a predetermined period of time, so if there are many hands within the field of view of the HMD 100, it will take a lot of time. There is a possibility that it will be stored away. In the second embodiment, when one or more hands are detected from a captured image, the hand is selected as a target of comparison with the detection information of the small operation device 300 based on the distance from the HMD 100 to each hand. Refine. As a result, the number of hand candidates to be compared with the detection information of the small operating device 300 can be reduced, and the comparison processing time can be expected to be shortened. Hereinafter, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the second embodiment will be mainly described.

FIG. 9 is a flowchart of operation target recognition processing executed in the HMD 100 according to the second embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 9 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether the position of the small operating device 300 has changed based on the sensing data acquired from the small operating device 300. The processing from step S501 onward is performed only when the movement of the small operating device 300 is detected.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are sequentially selected, and the process in step S900 is executed for the selected determination targets.

In step S900, the CPU 200 (processing calculation unit 109) of the HMD 100 calculates the distance D from the HMD 100 to the hand to be determined, and compares the distance D with a threshold ThD. The distance D can be obtained from the depth information obtained from the stereo images described in the first embodiment. Alternatively, if the HMD 100 is equipped with a depth sensor, information measured by the depth sensor may be used. The threshold ThD may be set to a value that is slightly larger than the reach (reachable distance) of the user of the HMD 100, for example, a value of about 0.8 m to 1.0 m. The threshold ThD is set in the HMD 100 in advance.

When the distance D to the hand to be determined is equal to or greater than the threshold ThD (NO determination in step S900), the CPU 200 determines that the hand to be determined is not the hand of the user of the HMD 100 but another person's hand. In this case, the comparison process (step S503) is not performed, and the process moves to the next determination target process (step S502).

On the other hand, if the distance D to the hand to be determined is closer than the threshold ThD (YES determination in step S900), the CPU 200 determines that the hand to be determined may be the hand of the user of the HMD 100, and performs the comparison. The process advances to step S503. The subsequent processing is similar to that described in the first embodiment.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. In addition, in this embodiment, candidates to be compared are narrowed down based on the distance from the HMD 100 to the hand, so even if many hands are captured within the field of view of the HMD 100, processing costs can be reduced and processing time can be reduced. The effect of shortening time and suppressing power consumption can be obtained.

(Third embodiment)
In the first embodiment, all the hands detected by the HMD 100 are compared with the detection information of the small operating device 300. However, in order to determine that it is the user's hand, it is necessary to continue comparing it with each determination target for a predetermined period of time, so if there are many hands within the field of view of the HMD 100, it will take a lot of time. There is a possibility that it will be stored away. In the third embodiment, when one or more hands are detected from a captured image, the hands to be compared with the detection information of the small operating device 300 are narrowed down based on the movement direction of each hand. As a result, the number of hand candidates to be compared with the detection information of the small operating device 300 can be reduced, and the comparison processing time can be expected to be shortened. Hereinafter, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the third embodiment will be mainly described.

FIG. 10 is a flowchart of operation target recognition processing executed in the HMD 100 according to the third embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 10 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether there has been a change in the position of the small operating device 300 based on the sensing data acquired from the small operating device 300. The processing from step S501 onward is performed only when the movement of the small operating device 300 is detected.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are sequentially selected, and the process in step S1000 is executed for the selected determination targets.

In step S1000, the CPU 200 (processing calculation unit 109) of the HMD 100 determines whether the moving directions of the hand to be determined and the small operating device 300 match. The moving direction may be calculated from position information of two arbitrary frames. For example, the movement direction may be the direction of a movement vector whose starting point is the three-dimensional position of the previous frame and whose end point is the three-dimensional position of the current frame. A specific example will be explained with reference to FIG. Assuming that the current frame is at time t2 and the previous frame is at time t1, the movement vector V1 of the hand to be determined is {x2-x1, y2-y1, z2-z1}, and the movement vector V2 of the small operating device 300 is is found as {X2-X1, Y2-Y1, Z2-Z1}. If the angle θ formed by the vectors V1 and V2 is smaller than a predetermined threshold Thθ, the processing calculation unit 109 determines that the moving directions of the hand to be determined and the small operating device 300 match. The threshold value Thθ may be set in consideration of the measurement error of the sensor 303 and the error of the three-dimensional position calculated from the image. The threshold Thθ is set to HMD100 in advance.
Set it to .

If the moving directions of the hand to be determined and the small operating device 300 are different (NO determination in step S1000), the CPU 200 determines that the hand to be determined is not the hand of the user of the HMD 100 but the hand of another person. In this case, the comparison process (step S503) is not performed, and the process moves to the next determination target process (step S502).

On the other hand, if the moving direction of the hand to be determined and the moving direction of the small operating device 300 match (YES determination in step S1000), the CPU 200 determines that the hand to be determined may be the hand of the user of the HMD 100. , the process proceeds to comparison processing (step S503). The subsequent processing is similar to that described in the first embodiment.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. In addition, the candidates to be compared are narrowed down based on the hand to be determined and the moving direction of the small operating device 300, so even if many hands are captured within the field of view of the HMD 100, processing costs can be reduced and the processing speed can be reduced. The effect of shortening time and suppressing power consumption can be obtained. In particular, in this embodiment, since a method of simply calculating the moving direction from two frames of position information is adopted, there is an advantage that candidates to be compared can be narrowed down immediately.

(Fourth embodiment)
In the first embodiment, all the hands detected by the HMD 100 are compared with the detection information of the small operating device 300. However, in order to determine that it is the user's hand, it is necessary to continue comparing it with each determination target for a predetermined period of time, so if there are many hands within the field of view of the HMD 100, it will take a lot of time. There is a possibility that it will be stored away. In the fourth embodiment, when one or more hands are detected from a captured image, the hands to be compared with the detection information of the small operating device 300 are narrowed down based on the extending direction of each hand. As a result, the number of hand candidates to be compared with the detection information of the small operating device 300 can be reduced, and the comparison processing time can be expected to be shortened. Hereinafter, detailed descriptions of the same parts as those in the first embodiment will be omitted, and the characteristic parts of the fourth embodiment will be mainly described.

FIG. 11 is a flowchart of operation target recognition processing executed in the HMD 100 according to the fourth embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 11 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether the position of the small operating device 300 has changed based on the sensing data acquired from the small operating device 300. The processing from step S501 onward is performed only when the movement of the small operating device 300 is detected.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are sequentially selected, and the process in step S1100 is executed for the selected determination targets.

In step S1100, the CPU 200 (processing calculation unit 109) of the HMD 100 determines whether the stretching direction of the hand to be determined is within a predetermined range. The extending direction of the hand is the direction from the arm to the tips of the fingers, and is expressed, for example, by the direction of a vector with the wrist as the starting point and the tips of the fingers as the ending point. When the user of the HMD 100 performs a gesture operation with the right hand, the user's right hand is usually seen extending from bottom to top or from right to left within the field of view of the HMD 100. Furthermore, when the user of the HMD 100 performs a gesture operation with his or her left hand, the user's left hand is usually seen extending from bottom to top or from left to right within the field of view of the HMD 100. Therefore, if the hand detected in the captured image of HMD 100 extends from the top of the field of view to the bottom, or if the hand extends from the side opposite to the hand performing the gesture operation, that hand is It is highly unlikely that this was done by someone else. The possible angles of the extending direction of the user's hand are set in advance in the HMD 100 as a "predetermined range." For example, if the upward direction in the photographed image (12 o'clock direction on the clock) is 0 degrees, and if we consider 0 to +180 degrees clockwise and 0 to -180 degrees counterclockwise, the hand extending from the bottom to the top is It generally falls within the range of -45 degrees to 45 degrees. Furthermore, the hand extending from right to left generally falls within the range of -90 degrees to -45 degrees, and the hand extending from left to right generally falls within the range of 45 degrees to 90 degrees. Therefore, in this embodiment, a predetermined range is set, for example, from −90 degrees to 90 degrees, and a hand in a stretching direction that falls outside of this range is determined to be another person's hand. Note that the method of setting the predetermined range is not limited to this, and can be set arbitrarily. For example, if it is known in advance whether the hand performing the gesture operation is the right hand or the left hand, the predetermined range may be set even narrower. Alternatively, if the user tilts his or her head to the side, the field of view (captured image) of the HMD 100 itself will rotate, which will affect the relative angle between the hand extension direction and the captured image. A margin may be provided.

If the stretching direction of the hand to be determined is outside the predetermined range (NO determination in step S1100), CPU 200 determines that the hand to be determined is not the hand of the user of HMD 100 but someone else's hand. In this case, the comparison process (step S503) is not performed, and the process moves to the next determination target process (step S502).

On the other hand, if the stretching direction of the hand to be determined is within the predetermined range (YES determination in step S1100), the CPU 200 determines that the hand to be determined may be the hand of the user of the HMD 100, The process advances to comparison processing (step S503). The subsequent processing is similar to that described in the first embodiment.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. In addition, in this embodiment, the candidates to be compared are narrowed down based on the stretching direction of the hand to be determined, so even if a large number of hands are captured within the field of view of the HMD 100, processing costs can be reduced and the processing cost can be reduced. The effect of shortening time and suppressing power consumption can be obtained.

(Fifth embodiment)
In the first embodiment, whether or not the hand is the user's hand is determined based on whether the hand movement detected by the HMD 100 matches the movement of the small operating device 300. On the other hand, in the fifth embodiment, the user's hand is determined by image recognition using an image photographed by the image photographing unit 101 of the HMD 100. Hereinafter, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the fifth embodiment will be mainly described.

FIG. 12 is a flowchart of operation target recognition processing executed in the HMD 100 according to the fifth embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation target recognition process shown in FIG. 12 is performed in the HMD 100 at a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether the position of the small operating device 300 has changed based on the sensing data acquired from the small operating device 300. The processing from step S501 onward is performed only when the movement of the small operating device 300 is detected.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are sequentially selected, and the process in step S1200 is executed for the selected determination targets.

In step S1200, the CPU 200 (processing calculation unit 109) of the HMD 100 determines whether the small operating device 300 is attached to the hand of the determination target by image recognition. As the image recognition algorithm, classical machine learning represented by support vector machine may be used, or deep learning-based algorithms such as R-CNN, YOLO, SSD, DCN, etc. may be used. Rule-based detection algorithms may also be used.

If the small operation device 300 is not attached to the hand to be determined (NO determination in step S1200), the CPU 200 determines that the hand to be determined is not the hand of the user of the HMD 100 but someone else's hand. In this case, the process transitions to the next determination target process (step S502).

On the other hand, if the small operation device 300 is attached to the hand of the determination target (YES determination in step S1200), the CPU 200 determines that the hand of the determination target is the hand of the user of the HMD 100, and proceeds to step S504. . The subsequent processing is similar to that described in the first embodiment.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. In addition, in this embodiment, it is possible to determine whether or not the small operating device 300 is attached to the hand of the person to be determined based on one frame of the photographed image, so the advantage is that the user's hand can be immediately identified. There is also.

(Sixth embodiment)
In the first embodiment, whether or not the hand is the user's hand is determined based on whether the hand movement detected by the HMD 100 matches the movement of the small operating device 300. On the other hand, in the sixth embodiment, the user's hand is identified based on whether the hand posture detected by the HMD 100 and the posture of the small operating device 300 match. The attitude of the small operation device 300 can be estimated from the angular velocity of IMU information acquired by the sensor 303. Hereinafter, detailed descriptions of the same parts as those in the first embodiment will be omitted, and the characteristic parts of the sixth embodiment will be mainly described.

FIG. 13 is a flowchart of operation target recognition processing executed in the HMD 100 according to the sixth embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 13 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether there has been a change in the position of the small operating device 300 based on the sensing data acquired from the small operating device 300. Step S50 only when the movement of the small operating device 300 is detected.
1 and subsequent processes are performed.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are sequentially selected, and the process in step S1300 is executed for the selected determination targets.

In step S1300, the CPU 200 (processing calculation unit 109) of the HMD 100 evaluates whether the posture of the hand to be determined matches the posture of the small operating device 300. Specifically, the processing calculation unit 109 calculates the attitude angle of the small operating device 300 from the angular velocity of the IMU information acquired from the sensor 303. Further, the processing calculation unit 109 calculates the posture angle of the hand to be determined by estimating the posture of the hand to be determined detected from the photographed image. Any posture estimation algorithm may be used. For example, the three-dimensional coordinates of a plurality of feature points (joints, contour points, etc.) on the hand to be determined may be calculated, and the posture of the hand may be estimated based on the relative positional relationship of the plurality of feature points. Alternatively, an algorithm based on deep learning may be used to estimate the pose of a hand image. Note that, taking into account errors in calculating the posture angle and errors in posture estimation, if the difference between the posture angle of the hand to be determined and the posture angle of the small operating device 300 is within a predetermined error range, the postures of the two will match. It may be determined that the

If the posture of the hand to be determined is different from the posture of small operation device 300 (NO determination in step S1300), CPU 200 determines that the hand to be determined is not the hand of the user of HMD 100 but someone else's hand. In this case, the process transitions to the next determination target process (step S502).

On the other hand, if the posture of the hand to be determined matches the posture of the small operating device 300 (YES determination in step S1300), the CPU 200 determines that the hand to be determined is the hand of the user of the HMD 100, and the CPU 200 determines that the hand to be determined is the hand of the user of the HMD 100, and performs the determination in step S504. Proceed to. The subsequent processing is similar to that described in the first embodiment.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. Note that if both the "comparison of movements" in the first embodiment and the "comparison of postures" in this embodiment are performed, and the movements and postures match, it is determined that the hand is the user's own hand. Good too. This can be expected to further reduce misjudgments (mistaking someone else's hand to be the user's hand).

(Seventh embodiment)
In the first embodiment, all the hands detected by the HMD 100 were compared with the movements of the small operating device 300. However, there may be cases where the hand tracking information of the user's hand cannot be appropriately acquired because the user's hand is out of the field of view of the HMD 100 or in the blind spot of the image capturing section 101. Therefore, in the seventh embodiment, when the user's hand cannot be recognized from the photographed image, the user is notified that the hand position is inappropriate so that the user moves the hand to a position where it can be detected by the HMD 100. Hereinafter, detailed descriptions of the same parts as in the first embodiment will be omitted, and the characteristic parts of the seventh embodiment will be mainly described.

FIG. 14 is a flowchart of operation target recognition processing executed in the HMD 100 according to the seventh embodiment. The same steps as in the first embodiment are given the same step numbers as in the flowchart of FIG. While the mixed reality system 1 is in operation, sensing data (IMU information) acquired by the sensor 303 of the small operation device 300 is periodically transmitted to the HMD 100, and the operation object recognition process shown in FIG. 14 is performed in the HMD 100 in a predetermined cycle. executed repeatedly.

In step S500, the CPU 200 of the HMD 100 determines whether the position of the small operating device 300 has changed based on the sensing data acquired from the small operating device 300. The processing from step S501 onward is performed only when the movement of the small operating device 300 is detected.

In step S501, the CPU 200 (hand detection section 105) of the HMD 100 acquires an image photographed by the image photographing section 101 from the photographed image storage section 104, and detects a hand from the photographed image. If a plurality of hands are included in the photographed image, all of the hands are set as determination targets. In step S502, hands to be determined are selected in order, and the process in step S503 is executed for the selected determination targets.

If no hand to be determined is detected from the photographed image, or if the hand of the user is not recognized even though the process in step S503 has been performed for all the determination targets, step S502 A NO determination is made in step S1400, and the process proceeds to step S1400. In step S1400, the CPU 200 (processing calculation unit 109) of the HMD 100 displays guidance warning that the user's hand is not displayed on the HMD 100 on the image display unit 102, so that the user of the HMD 100 can check the position of the hand. induce them to move. Note that the processing calculation unit 109 may notify that the position of the hand is not appropriate through audio guidance.

The configuration of this embodiment described above can also provide the same effects as the first embodiment. In addition, in this embodiment, if the position of the hand when the user performs a gesture operation is inappropriate, it can be detected and the user can be prompted to correct the position of the hand. Therefore, failures in gesture operations due to inappropriate hand positions can be reduced, and convenience can be improved.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention. The configurations described in the first to seventh embodiments may be combined with each other (as long as there is no technical contradiction).

Further, a case where a software program that implements the functions of the above-described embodiments is supplied directly from a recording medium or using wired/wireless communication to a system or device having a computer capable of executing the program, and the program is executed. Also included in the present invention. Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself that is supplied and installed in the computer also realizes the present invention. In other words, the present invention also includes a computer program itself for realizing the functional processing of the present invention. In this case, the form of the program does not matter, such as an object code, a program executed by an interpreter, or script data supplied to the OS, as long as it has the function of a program. The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magnetic optical storage medium, or a nonvolatile semiconductor memory. Further, as a method of supplying the program, a method may be considered in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and executes the computer program. The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure of this specification includes the following configurations, methods, and programs.
(Configuration 1)
An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the position of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement of the hand detected from the photographed image with the movement of the device recognized from the sensing data, and the movement of the hand and the movement of the device are performed. processing calculation means that determines that the hand detected from the photographed image is the user's hand when the two match;
An information processing device having:
(Configuration 2)
An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the posture of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the posture of the hand detected from the photographed image with the posture of the device recognized from the sensing data, and the posture of the hand and the posture of the device are compared. processing calculation means for determining that the hand detected from the photographed image is the user's hand when the postures match;
An information processing device having:
(Configuration 3)
An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the position and orientation of the device from a sensor installed in the device worn by the user;
A comparison process is performed to compare the movement and posture of the hand detected from the photographed image with the movement and posture of the device recognized from the sensing data, and the movement and posture of the hand detected from the photographed image are compared. and processing calculation means for determining that the hand detected from the photographed image is the user's hand when the posture matches the movement and posture of the device;
An information processing device having:
(Configuration 4)
The processing calculation means narrows down the hands to be subjected to the comparison processing based on the distance from the HMD to each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of configurations 1 to 3.
(Configuration 5)
The processing calculation means narrows down the hands to be subjected to the comparison processing based on the movement direction of each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of configurations 1 to 4.
(Configuration 6)
The processing calculation means narrows down the hands to be subjected to the comparison processing based on the stretching direction of each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of configurations 1 to 5.
(Configuration 7)
An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
Determining whether or not the device worn by the user is worn on the hand detected from the photographed image by image recognition, and when the device is worn on the hand detected from the photographed image; , processing calculation means for determining that the hand detected from the photographed image is the hand of the user;
An information processing device having:
(Configuration 8)
The processing calculation means notifies the user that the hand position is not appropriate when the user's hand is not recognized from the photographed image.
The information processing device according to any one of configurations 1 to 7.
(Configuration 9)
When a change in the position of the device is detected based on sensing data acquired from a sensor installed in the device, processing by the detection means and the processing calculation means is executed.
The information processing device according to any one of configurations 1 to 8.
(Configuration 10)
In the comparison process, the processing calculation means compares the relative movement amount of the hand position and the relative movement amount of the device in a predetermined period, and determines that the relative movement amount of the hand position and the relative movement amount of the device are determining that the hand movement and the device movement match if they are the same or within a predetermined error range;
The information processing device according to configuration 1 or 3.
(Configuration 11)
the sensor is an IMU sensor;
The information processing device according to any one of Configurations 1 to 10.
(Configuration 12)
The device is a wearable device worn on the user's hand or finger.
The information processing device according to any one of configurations 1 to 11.
(Configuration 13)
A goggle device that has an image capturing section and an image display section and is worn on the head by a user;
The information processing device according to any one of configurations 1 to 12;
An HMD with
(Configuration 14)
A goggle device that has an image capturing section and an image display section and is worn on the head by a user;
The information processing device according to any one of configurations 1 to 12;
A device that is attached to the user's hand and has a sensor that measures the position and posture;
A system with
(Method 15)
A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the position of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement of the hand detected from the photographed image with the movement of the device recognized from the sensing data, and the movement of the hand and the movement of the device are performed. a step of determining that the hand detected from the photographed image is the user's hand if the two match;
Recognition method to perform.
(Method 16)
The above-mentioned HM from among the images taken by HMD (Head Mount Display)
A recognition method for recognizing the hand of a user wearing D,
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the posture of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the posture of the hand detected from the photographed image with the posture of the device recognized from the sensing data, and the posture of the hand and the posture of the device are compared. If the postures match, determining that the hand detected from the captured image is the hand of the user;
Recognition method to perform.
(Method 17)
A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the position and orientation of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement and posture of the hand detected from the photographed image with the movement and posture of the device recognized from the sensing data, and the movement and posture of the hand detected from the photographed image are compared. and determining that the hand detected from the captured image is the user's hand if the posture matches the movement and posture of the device;
Recognition method to perform.
(Method 18)
A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
Determining whether or not the device worn by the user is worn on the hand detected from the photographed image by image recognition, and when the device is worn on the hand detected from the photographed image; , determining that the hand detected from the photographed image is the hand of the user;
Recognition method to perform.
(Program 19)
A program for causing an information processing device to execute each step of the recognition method described in any one of Methods 15 to 18.

100 HMD
103 Information processing device 105 Hand detection unit 109 Processing calculation unit 110 Proximity wireless communication unit 300 Small operation device 303 Sensor

Claims (19)

An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the position of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement of the hand detected from the photographed image with the movement of the device recognized from the sensing data, and the movement of the hand and the movement of the device are performed. processing calculation means that determines that the hand detected from the photographed image is the user's hand when the two match;
An information processing device having: An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the posture of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the posture of the hand detected from the photographed image with the posture of the device recognized from the sensing data, and the posture of the hand and the posture of the device are compared. processing calculation means for determining that the hand detected from the photographed image is the user's hand when the postures match;
An information processing device having: An information processing device that recognizes the hand of a user wearing an HMD from images taken by an HMD (Head Mount Display),
Detection means for detecting a hand from a captured image captured by the HMD;
acquisition means for acquiring sensing data regarding the position and orientation of the device from a sensor installed in the device worn by the user;
A comparison process is performed to compare the movement and posture of the hand detected from the photographed image with the movement and posture of the device recognized from the sensing data, and the movement and posture of the hand detected from the photographed image are compared. and processing calculation means for determining that the hand detected from the photographed image is the user's hand when the posture matches the movement and posture of the device;
An information processing device having: The processing calculation means narrows down the hands to be subjected to the comparison processing based on the distance from the HMD to each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of claims 1 to 3. The processing calculation means narrows down the hands to be subjected to the comparison processing based on the movement direction of each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of claims 1 to 3. The processing calculation means narrows down the hands to be subjected to the comparison processing based on the stretching direction of each hand when one or more hands are detected from the photographed image.
The information processing device according to any one of claims 1 to 3. An information processing device that recognizes the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD,
Detection means for detecting a hand from a captured image captured by the HMD;
It is determined by image recognition whether or not the device worn by the user is worn on the hand detected from the captured image, and if the device is worn on the hand detected from the captured image, , processing calculation means for determining that the hand detected from the photographed image is the hand of the user;
An information processing device having: The processing calculation means notifies the user that the hand position is not appropriate when the user's hand is not recognized from the photographed image.
The information processing device according to any one of claims 1 to 3 and 7. When a change in the position of the device is detected based on sensing data acquired from a sensor installed in the device, processing by the detection means and the processing calculation means is executed.
The information processing device according to any one of claims 1 to 3 and 7. In the comparison process, the processing calculation means compares the relative movement amount of the hand position and the relative movement amount of the device in a predetermined period, and determines that the relative movement amount of the hand position and the relative movement amount of the device are determining that the hand movement and the device movement match if they are the same or within a predetermined error range;
The information processing device according to claim 1 or 3. the sensor is an IMU sensor;
The information processing device according to any one of claims 1 to 3. The device is a wearable device worn on the user's hand or finger.
The information processing device according to any one of claims 1 to 3. A goggle device that has an image capturing section and an image display section and is worn on the head by a user;
The information processing device according to any one of claims 1 to 3 and 7,
An HMD with A goggle device that has an image capturing section and an image display section and is worn on the head by a user;
The information processing device according to any one of claims 1 to 3 and 7,
A device that is attached to the user's hand and has a sensor that measures the position and posture;
A system with A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the position of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement of the hand detected from the photographed image with the movement of the device recognized from the sensing data, and the movement of the hand and the movement of the device are performed. a step of determining that the hand detected from the photographed image is the user's hand if the two match;
Recognition method to perform. The above-mentioned HM from among the images taken by HMD (Head Mount Display)
A recognition method for recognizing the hand of a user wearing D,
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the posture of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the posture of the hand detected from the photographed image with the posture of the device recognized from the sensing data, and the posture of the hand and the posture of the device are compared. If the postures match, determining that the hand detected from the captured image is the hand of the user;
Recognition method to perform. A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
acquiring sensing data regarding the position and orientation of the device from a sensor mounted on the device worn by the user;
A comparison process is performed to compare the movement and posture of the hand detected from the photographed image with the movement and posture of the device recognized from the sensing data, and the movement and posture of the hand detected from the photographed image are compared. and determining that the hand detected from the captured image is the user's hand if the posture matches the movement and posture of the device;
Recognition method to perform. A recognition method for recognizing the hand of a user wearing an HMD (Head Mount Display) from images taken by the HMD, the method comprising:
The information processing device
Detecting a hand from a captured image captured by the HMD;
It is determined by image recognition whether or not the device worn by the user is worn on the hand detected from the captured image, and if the device is worn on the hand detected from the captured image, , determining that the hand detected from the photographed image is the hand of the user;
Recognition method to perform. A program for causing an information processing device to execute each step of the recognition method according to any one of claims 15 to 18.

Images