WO2023209818A1 - 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム - Google Patents
操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム Download PDFInfo
- Publication number
- WO2023209818A1 WO2023209818A1 PCT/JP2022/018937 JP2022018937W WO2023209818A1 WO 2023209818 A1 WO2023209818 A1 WO 2023209818A1 JP 2022018937 W JP2022018937 W JP 2022018937W WO 2023209818 A1 WO2023209818 A1 WO 2023209818A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information
- image
- information processing
- unit
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
- G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2021—Shape modification
Definitions
- the present invention relates to an operating device, an information processing device, an information processing system, an information processing method, and an information processing program.
- Patent Document 1 a technique for tracking the movements of a user's fingers is known for use in VR (Virtual Reality), etc. (see, for example, Patent Document 1).
- the position of a user's fingertip is tracked using a glove equipped with a sensor at a position corresponding to the user's fingertip.
- Information indicating the position tracked by the glove is converted into various control signals and is used for various purposes, such as generating and displaying an image that reproduces the user's fingers and fingers in a virtual space.
- An operating device is an operating device that receives an operation by a user, and includes a detection unit that detects the operation, an acquisition unit that acquires a first image of the user's hand, and a first
- the present invention includes a feature information generation section that generates feature information regarding the hand and fingers based on the image, and a feature information output section that outputs the detection result by the detection section and the feature information.
- An information processing device includes: an acquisition unit that acquires a detection result by an operating device including a detection unit that detects an operation by a user; and a first image of a user's hand; a feature information generation unit that generates feature information regarding the hand and fingers based on the image; and an image generation unit that generates a second image that reproduces the shape of the hand and fingers based on the detection result by the detection unit and the feature information. , and a display section that displays the second image.
- An information processing device includes an acquisition unit that acquires a detection result by an operating device including a detection unit that detects an operation by a user, and a first image of the user's finger; a feature information generation section that generates feature information regarding the hand and fingers based on the image of the hand; an operation information generation section that generates operation information for moving the operation target based on the detection result by the detection section; , an operation information adjustment section that adjusts the operation information, and an operation information output section that outputs the adjusted operation information to the operation target.
- An information processing method is an information processing method implemented by an operating device that includes a detection unit and receives an operation by a user, and the method includes acquiring a first image of a user's finger.
- the method includes an acquisition procedure, a feature information generation procedure for generating feature information regarding the hand and fingers based on the first image, and a feature information output procedure for outputting the detection result by the detection unit and the feature information.
- An information processing program is an information processing program that is executed by an information terminal that includes a detection unit and is connected to an operating device that accepts operations by a user, and that is executed by an information terminal that is connected to an operating device that accepts operations by a user. an acquisition step of acquiring a captured first image; a feature information generation step of generating feature information regarding the hand and fingers based on the first image; and feature information that outputs a detection result by the detection unit and the feature information. Execute the output step.
- FIG. 1 is a schematic diagram showing a schematic configuration of an information processing system according to a first embodiment.
- FIG. 1 is a perspective view schematically showing an operating device according to a first embodiment. The figure which shows the example of a reference image. A diagram explaining feature information. The figure which shows the example of a model image.
- FIG. 3 is a diagram illustrating adjustment of a 3D model based on feature information. 3 is a flowchart illustrating an example of a processing method according to the first embodiment.
- FIG. 2 is a schematic diagram showing a schematic configuration of an information processing system according to a second embodiment.
- 7 is a flowchart illustrating an example of a processing method according to a second embodiment.
- FIG. 1 is a schematic diagram showing a schematic configuration of an information processing system according to a first embodiment.
- the information processing system 1 according to the first embodiment includes an operating device 10, an information processing device 20, and an HMD unit 30, as shown in FIG.
- the information processing system 1 uses the information processing device 20 to generate a model image (second image) that reproduces the user's fingers in the virtual space in response to an operation on the operating device 10 , and generates a model image (second image) in the HMD unit 30 . ) is displayed.
- the operating device 10 includes a detection section 11 and an output section 12, and receives operations by a user.
- the detection unit 11 detects the position of the user's fingers (hands and fingers).
- the output unit 12 outputs the detection result by the detection unit 11 to the outside of the operating device 10 via wireless or wired communication.
- FIG. 2 is a perspective view schematically showing the operating device 10.
- the operating device 10 is attached to a user's finger using a glove-shaped attachment member.
- the mounting member is made of a stretchable material that can fit tightly onto the user's hand.
- the operating device 10 includes, as a detection unit 11, a plurality of sensors 111a, 111b, 111c, 111d, and 111e provided according to each finger, as shown in FIG. A sensor 112 is provided.
- the plurality of sensors 111a, 111b, 111c, 111d, and 111e are provided at positions corresponding to the fingertips of each finger and at positions on the nail side of each finger. More specifically, for example, as shown in FIG. 2, the sensor 111a is provided corresponding to the first finger (thumb), and detects the position of the sensor 111a as the position of the fingertip of the thumb. Similarly, the sensors 111b, 111c, 111d, and 111e are provided corresponding to the second finger (index finger), the third finger (middle finger), the fourth finger (ring finger), and the fifth finger (little finger), respectively. The position of is detected as the position of the fingertip of each finger. The sensor 112 is provided at a position corresponding to the back of the hand, and detects the position of the sensor 112 as the position of the back of the hand.
- each of the sensors 111a, 111b, 111c, 111d, 111e, and the sensor 112 may be any known sensor. Further, the arrangement positions of the sensors 111a, 111b, 111c, 111d, 111e, and the sensor 112 are not limited to the example in FIG. 2.
- the sensors 111a, 111b, 111c, 111d, and 111e may be provided at the ventral side of each finger, at the tip of each finger, or in part or all of each finger. It may be provided so as to cover the In this embodiment, as shown in FIG. 2, the output unit 12 is provided within a casing 121 provided at a position corresponding to the user's wrist.
- the control section may also be provided within the casing 121.
- the housing 121 may be provided integrally or independently at the same position as the sensor 112, which is provided at a position corresponding to the back of the hand, or may be provided at another position.
- the information processing device 20 includes a communication section 21 and a control section 22, as shown in FIG.
- the communication unit 21 receives the detection result by the detection unit 11 output from the output unit 12 of the operating device 10, and is mutually connected to the communication unit 32 of the HMD unit 30 via wireless or wire.
- the control unit 22 includes the functions of a feature information generation unit 221 and an image generation unit 222, which are realized by a processor operating according to a program stored in a memory or received via a communication interface. The functions of each part will be described later.
- the HMD unit 30 can be used by being attached to the user's head using a mounting portion (not shown). Furthermore, the HMD unit 30 has the function of a display device and the function of an imaging device that captures an image from a user's first-person viewpoint, so-called inside-out.
- the HMD unit 30 may be configured to be attached to the user's head with a mounting portion such as a band, or may have any configuration such as a helmet type or glasses type. Further, when wearing the HMD unit 30, it is preferable to guide the user to properly wear the HMD unit 30 by displaying a tutorial or the like on the display device of the HMD unit 30, for example.
- the HMD unit 30 includes an imaging section 31, a communication section 32, a control section 33, and a display section 34.
- the imaging unit 31 includes an image sensor (not shown), and generates an image by capturing an image from the user's first-person viewpoint.
- the image sensor may be an image sensor that generates an image that includes color information, or a depth sensor that generates an image that includes depth information.
- the imaging section 31 may include a plurality of imaging elements.
- the imaging unit 31 may include both an image sensor and a depth sensor to generate an image including color information and an image including depth information, respectively. Further, for example, it may include a plurality of image sensors to generate an image including color information and generate depth information from the plurality of images.
- the imaging unit 31 captures a reference image (first image) obtained by capturing a user's finger.
- the reference image is information used to generate a model image that reproduces the user's fingers in the virtual space.
- the communication unit 32 is mutually connected to the communication unit 21 of the information processing device 20 via wireless or wire.
- the control section 33 controls each section within the HMD unit 30.
- the display unit 34 includes a display element such as an LCD (Liquid Crystal Display) or an organic EL, and an optical device such as a lens, and can present a display image to the user by displaying it.
- the display element of the display section 34 may be a transmissive display element or a non-transmissive display element.
- a terminal device such as a smartphone that can be attached to and detached from the housing of the HMD unit 30 may be used as the display device.
- a wearable device such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses may be used as the HMD unit 30.
- the feature information generation unit 221 generates feature information regarding the hand and fingers based on the reference image. Similar to the reference image described above, the feature information is information used to generate a model image, and includes at least one of the following two types.
- Position information of feature points on fingers The feature information generation unit 221 calculates position information of feature points, such as joint information on the user's fingers, from the reference image using, for example, a machine learning method. Examples of the feature points include the end points of each finger, the joints of each finger, and the joints of the wrist.
- the trained model used for machine learning performs supervised learning using, for example, an image of a person's hand and finger that has multiple feature points as input data, and coordinate information indicating the position of the feature point on the person's finger as correct answer data. It can be pre-built by doing this. Note that a detailed description of the specific method of machine learning will be omitted since various known techniques can be used.
- the feature information generation unit 221 includes a relationship learning unit, and each time a reference image is generated, the relationship between an image based on the input reference image and coordinate information indicating the position of a feature point on a person's finger is calculated. It may also be configured to learn and update the learned model.
- FIG. 3B illustrates feature points based on the reference image. In the example of FIG.
- the end points of each finger, the joints of each finger, and the joints of the wrist are detected as feature points, and the position information of each feature point is calculated as feature information.
- the feature information generation unit 221 extracts the silhouette of the user's hand and fingers using a method such as skin color detection on a reference image including color information, and generates position information of the feature point on the hand and finger based on the extracted silhouette. May be generated. Further, the feature information generation unit 221 may generate position information of feature points on the fingers by estimating the thickness around the joints of the fingers, etc., based on the depth information, for example, with respect to the reference image including depth information. good.
- Texture information on the surface of the user's fingers calculates texture information indicating the surface state of the user's fingers from the reference image using, for example, a method such as skin color detection. Texture information includes the color and texture of the user's skin, as well as surface irregularities such as scratches, moles, and wrinkles.
- the image generation unit 222 generates a model image (second image) that reproduces the user's hand and fingers in the virtual space based on the detection results received from the operating device 10 via the communication unit 21.
- An example of a model image is shown in FIG. 4A.
- the image generation unit 222 controls the 3D model of the hand and fingers based on the detection results received from the operating device 10 described above, adjusts the 3D model of the hand and fingers based on the feature information generated by the feature information generation unit 221, and adjusts the 3D model of the hand and fingers based on the feature information generated by the feature information generation unit 221.
- a model image is generated based on the 3D model.
- the image generation unit 222 uses, for example, the well-known Inverse Kinetics (hereinafter referred to as "IK") technology to determine the starting point (IK route) and end point (IK end) for the joint. ) and a target point (IK goal), the 3D model of the finger skeleton (bone) is controlled by performing posture correction such as bending and stretching movements.
- IK Inverse Kinetics
- the image generation unit 222 when the tip of a joint structure is moved, correction calculations are performed by tracing the rotation of the joint in the opposite direction (parent direction) during that time.
- the image generation unit 222 then generates a model image by performing skinning processing on the 3D model to set the influence range of the skeleton.
- the skeleton of the 3D model from which the model image is generated matches the skeleton of the finger based on the detection results received from the operating device 10. This is desirable. For example, if the hand of the 3D model is larger than the hand of the user operating the operating device 10, the sensors (sensors 111a, 111b, 111c, 111d, and 111e) will become embedded in the fingertips of the 3D model. Furthermore, if the hand of the 3D model is smaller than the hand of the user operating the operating device 10, the sensors (sensors 111a, 111b, 111c, 111d, 111e) will be located at a distance from the fingertips. .
- the skeleton of the 3D model from which the model image is generated does not match the skeleton of the hand and fingers based on the detection results received from the operating device 10, so the model image generated based on the 3D model does not match.
- the bending of the intermediate joints is not correctly reproduced, resulting in unnatural movements and giving the user a sense of discomfort.
- the image generation unit 222 adjusts the 3D model as necessary based on the feature information generated by the feature information generation unit 221, and then generates a model image. .
- Generation of model images is performed according to the type of feature information.
- the image generation unit 222 calculates the length of the fingers in the 3D model and the ratio of each part of the fingers based on the feature information. Adjust. More specifically, the image generation unit 222 enlarges or reduces the size of the fingers in the 3D model, expands or contracts the length of the fingers in the 3D model based on feature information, or changes the ratio of each part of the fingers in the 3D model.
- the 3D model is adjusted by making changes based on the feature information.
- the image generation unit 222 then generates a model image by performing skinning processing on the adjusted 3D model.
- the image generation unit 222 uses the texture information for the hand and finger in the model image as the texture of the image of the finger in the reference image. Based on the feature information, a model image is generated that reflects the user's skin tone, texture, and surface irregularities such as scratches, moles, and wrinkles. Note that when images of both the palm side and the back side of the user's fingers are captured as reference images, the images of both sides are adjusted in the model image as well.
- FIG. 4B illustrates a model image generated based on the feature information.
- the image generation unit 222 may perform the image generation processing described in (1) and (2) in combination, or may perform either of the image generation processes described in (1) and (2). Only one of the image generation processes may be performed. Further, the content of the image generation process may be settable by the user.
- FIG. 5 is a flowchart showing the processing executed by the control unit 22.
- the control unit 22 determines whether or not the reference image has been acquired by the imaging unit 31 (step S101), and when determining that the reference image has been acquired (step S101 YES), the feature information generation unit 221 generates information based on the reference image. feature information is generated (step S102). Then, the control unit 22 determines whether the detection result by the operating device 10 has been acquired via the communication unit 21 (step S103). If it is determined that the detection result has been obtained (step S103 YES), the image generation unit 232 adjusts the 3D model image according to the feature information generated in step S102 (step S104), and generates a model image (step S105). .
- step S106 when the communication unit 21 outputs the generated model image to the HMD unit 30 (step S106), the process returns to step S103. That is, when the feature information is generated once through the processing in steps S101 and S102, thereafter, the processing in steps S103 to S106 is repeatedly executed every time a detection result by the operating device 10 is obtained. Therefore, since a model image is output to the HMD unit 30 every time a detection result is obtained by the operating device 10, the model image displayed on the display unit 24 is sequentially updated, and at that time, the model image is always modeled based on the reference image. An image will be generated.
- the information processing system 1 includes an acquisition unit (communication unit 21) that acquires a detection result by the operating device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image of the user's hand and fingers. , a feature information generation unit 221 that generates feature information regarding the hand and fingers based on the reference image, and a model that is a second image that reproduces the shape of the hand and fingers based on the detection result by the detection unit 11 and the feature information. and an image generation unit 232 that generates an image.
- acquisition unit communication unit 21
- the information processing device 20 includes an acquisition unit (communication unit 21) that acquires a detection result by the operating device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image of the user's hand and fingers.
- a feature information generation unit 221 that generates feature information regarding the hand and fingers based on the reference image
- a model that is a second image that reproduces the shape of
- the image generation unit 222 controls the 3D model of the hand and fingers based on the detection result, adjusts the 3D model of the hand and fingers based on the feature information, and generates a model image based on the adjusted 3D model. According to such a configuration, by matching the skeleton of the 3D model from which the model image is generated and the skeleton of the hand based on the detection result received from the operating device 10, the movement of the hand and fingers can be correctly performed in the model image. Can be reproduced.
- the feature information generated by the feature information generation unit 221 includes position information of feature points on the fingers. According to such a configuration, by adjusting the model image based on the positional information of the feature points on the fingers, it is possible to accurately reproduce the shape of the actual user's fingers. Therefore, defects occurring in the model image can be eliminated or reduced, and the actual user's fingers and their movements can be accurately reproduced in the model image.
- the feature information generated by the feature information generation unit 221 includes texture information on the surface of the fingers. According to such a configuration, by adjusting the model image based on the texture information of the surface of the finger, it is possible to preferably reproduce the atmosphere and appearance of the actual user's finger. Therefore, it can be expected to improve the sense of reality and immersion.
- the reference image captured by the imaging unit 31 is an image containing at least one of color information and depth information. According to such a configuration, feature information necessary for adjusting a model image can be easily generated from a reference image that can be easily obtained.
- FIG. 6 is a schematic diagram showing the entire information processing system according to the second embodiment of the present invention.
- the information processing system 2 according to the second embodiment includes an imaging device 40, an information processing device 50, and an operation target device 60.
- the information processing system 2 generates operation information in the information processing device 50 in response to an operation on the operation device 10, and controls the operation target device 60 based on the operation information.
- the imaging device 40 includes an imaging section 41 and an output section 42, as shown in FIG.
- the imaging unit 41 includes an image sensor (not shown), and generates an image by capturing an image from a user's third-person viewpoint, so-called outside-in.
- the details of the image pickup device and the number of image pickup devices are the same as those of the image pickup section 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, so a description thereof will be omitted.
- the imaging unit 41 like the imaging unit 31 of the HMD unit 30 in the information processing system 1 of the first embodiment, captures a reference image (first image) that captures the user's finger.
- the output unit 42 outputs the reference image captured by the imaging unit 41 to the outside of the imaging device 40 via wireless or wire.
- the information processing device 50 includes a communication section 51 and a control section 52, as shown in FIG.
- the communication unit 51 receives the detection result by the detection unit 11 output from the output unit 12 of the operating device 10 and the reference image output from the output unit 42 of the imaging device 40. Furthermore, the communication unit 51 outputs the operation information generated by the control unit 52 to the outside of the information processing device 50 via wireless or wired communication. Details of the operation information will be described later.
- the control unit 52 includes a feature information generation unit 521, an operation information generation unit 522, and an operation information adjustment unit 523, which are realized by a processor operating according to a program stored in a memory or received via a communication interface. Includes each function. The functions of each part will be further explained below.
- the feature information generation unit 521 generates feature information regarding the hand and fingers based on the reference image.
- the feature information includes position information of feature points on the fingers. Similar to the feature information generation unit 221 in the information processing device 20 of the first embodiment, the feature information generation unit 521 calculates position information of feature points on the user's fingers from the reference image.
- the operation information generation unit 522 generates operation information for operating the operation target device 60 based on the detection result received from the operation device 10 via the communication unit 51.
- the operation information may be the detection result itself, or may be control information indicating the drive location and drive amount of the operation target device 60 calculated based on the detection result.
- a problem may occur in the operation of the operation target device 60.
- the size of the corresponding location on the operation target device 60 is significantly different from the size of the actual user's finger, the motion at the corresponding location on the operation target device 60 may not be correctly reproduced.
- the movement of the actual user's finger will be correctly reproduced on the operation target device 60.
- the operation information adjustment section 523 adjusts the operation information generated by the operation information generation section 522 based on the characteristic information generated by the characteristic information generation section 521 in order to eliminate or reduce the above-mentioned problems. A specific method of adjustment will be described later.
- the operation target device 60 is a device that is operated in response to an operation on the operating device 10. Examples of the device 60 to be operated include a robot hand, a manipulator, and the like.
- the operation target device 60 includes a communication section 61 and a drive section 62, as shown in FIG.
- the communication unit 61 receives operation information output from the communication unit 51 of the information processing device 50.
- the drive unit 62 includes a plurality of motors and actuators such as servo motors depending on the shape of the robot hand and the manipulator, and drives at least one of them. At this time, the drive unit 62 is driven based on operation information (for example, a control signal for operating a servo motor) received via the communication unit 61.
- the installation positions of the motor and actuator may be fixed or may be movable.
- the motor and actuator of the drive unit 62 installed at the joints of the robot hand may be fixed or movable.
- information indicating the installation positions of the motors and actuators in the drive section 62 of the operation target device 60 is stored in advance in the control section 52 or is sent to the operation target device 60 via the communication section 51.
- the configuration is such that it can be obtained from The same applies to the case where the operation target device 60 is a manipulator.
- the operation information adjustment unit 523 adjusts the operation information based on the characteristic information. More specifically, the operation information adjustment unit 523 first finds a correlation between the feature points in the feature information and the installation position of the drive unit 62. The operation information adjustment unit 523 then adjusts the operation information according to the correlation. That is, by matching the operation information to the installation position of the drive unit 62 in the operation target device 60, the difference between the actual user's fingers and the installation position of the motor and actuator in the drive unit 62 of the operation target device 60 is reduced. .
- the operation information adjustment section 523 adjusts the motors and actuators in the same manner as the model image adjustment described in the first embodiment. Adjustment information for enlarging or reducing the installation position based on the characteristic information, adjustment information for expanding or contracting the length between the installation positions of the motor and actuator based on the characteristic information, adjustment information for adjusting the installation position of the motor and actuator based on the characteristic information.
- the operation information is adjusted by adding adjustment information, etc. for changing the operation information to the operation information.
- the motor and actuator in the drive unit 62 of the operation target device 60 can be adjusted to match the actual user's hand and fingers. Reduce the difference with the installation location.
- the information processing device 50 receives the detection result from the operating device 10 and also receives the reference image from the imaging device 40. The information processing device 50 then calculates feature information based on the reference image. Then, operation information is generated based on the detection result, and the operation information is adjusted based on the feature information.
- FIG. 7 is a flowchart showing the processing executed by the control unit 52 of the information processing device 50.
- the control unit 52 determines whether or not a reference image has been acquired via the communication unit 51 (step S201), and when determining that the reference image has been acquired (step S201 YES), the feature information generation unit 521 Feature information is generated based on (step S202). Then, the control unit 52 determines whether the detection result by the operating device 10 has been obtained via the communication unit 51 (step S203). If it is determined that the detection result has been obtained (step S203 YES), the operation information generation unit 522 generates operation information (step S204), and the operation information adjustment unit 523 performs the operation based on the feature information generated in step S202.
- step S205 The information is adjusted (step S205). Further, when the communication unit 51 outputs the adjusted operation information (step S206), the process returns to step S203. That is, when the feature information is generated once through the processing in steps S201 and S202, thereafter, the processing in steps S203 to S206 is repeatedly executed every time a detection result by the operating device 10 is obtained. Therefore, the operation information is output every time a detection result by the operation device 10 is obtained, and at that time, the operation information is always adjusted based on the reference image.
- the information processing system 2 includes an acquisition unit (communication unit 51) that acquires a detection result by the operating device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image of the user's hand and fingers. , a feature information generation unit 521 that generates feature information regarding the hand and fingers based on the reference image, and operation information that generates operation information for operating the operation target device 60, which is the operation target, based on the detection result by the detection unit 11. It includes a generation unit 522 and an operation information adjustment unit 523 that adjusts operation information based on feature information. The adjusted operation information is output to the operation target device 60 via the communication unit 51.
- acquisition unit communication unit 51
- the information processing device 50 includes an acquisition unit (communication unit 51) that acquires a detection result by the operating device 10, which includes a detection unit 11 that detects an operation by a user, and a reference image that is a first image of the user's hand and fingers.
- a feature information generation unit 521 that generates feature
- the operating device 10 is attached to the user's finger using a glove-shaped attachment member, and the detection unit 11 detects the position of the user's finger (hand and fingers).
- the present invention is not limited to this, and the present invention can be similarly applied to an operating device 10 in the shape of a stick or the like that is held by a user's fingers.
- the information processing device 20 and the HMD unit 30 were described as examples of the information processing device.
- the present invention is not limited to this, and similarly to the second embodiment, the present invention can be similarly applied to other information processing apparatuses that include an imaging device inside or outside.
- a configuration may also be adopted in which part of the processing performed by the HMD unit 30, which is the information processing device, in the first embodiment is performed by the operating device 10.
- a configuration may be adopted in which part or all of the functions of the feature information generation section 221 and the image generation section 222 in the HMD unit 30 of the first embodiment are performed by the operating device 10.
- the control section 22 of the information processing device 20 may be provided in the HMD unit 30.
- a part of the processing performed by the information processing device 50 in the second embodiment may be performed by the operating device 10 or may be performed by the operation target device 60.
- the configuration may be such that the operation device 10 performs some or all of the functions of the feature information generation section 521, the operation information generation section 522, and the operation information adjustment section 523 in the information processing device 50 of the second embodiment.
- a configuration may be adopted in which a part or all of the functions of the feature information generation unit 521, the operation information generation unit 522, and the operation information adjustment unit 523 in the information processing device 50 of the second embodiment are performed by the operation target device 60. good.
- the operating device 10 is attached to the user's finger using a glove-shaped attachment member.
- the attachment member is not limited to this, and may have a band shape.
- the operating device 10 may be a so-called exoskeleton controller.
- An operating device that accepts an operation by a user, which includes a detection unit that detects the operation, an acquisition unit that acquires a first image of the user's hand, and a feature related to the hand and finger based on the first image. It includes a feature information generation section that generates information, and a feature information output section that outputs the detection result by the detection section and the feature information.
- the feature information includes position information of feature points on the fingers.
- the feature information includes texture information on the surface of the finger.
- the detection unit detects the position of the finger.
- the first image is an image including at least one of color information and depth information.
- An acquisition unit that acquires a detection result by an operating device including a detection unit that detects an operation by a user and a first image of a user's hand; and feature information regarding the hand and fingers based on the first image.
- an image generation section that generates a second image that reproduces the shape of the hand and fingers based on the detection result by the detection section and the characteristic information; and a display section that displays the second image.
- An information processing device comprising: [7] In the information processing device according to [6], the image generation unit controls the 3D model of the hand and fingers based on the detection result, adjusts the 3D model of the hand and fingers based on the feature information, and generates the adjusted 3D model. A second image is generated based on the model. [8] An acquisition unit that acquires a detection result by the operating device including a detection unit that detects an operation by the user and a first image of the user's hand and fingers, and a feature related to the hand and finger based on the first image.
- a feature information generation unit that generates information, an operation information generation unit that generates operation information for operating an operation target based on the detection result by the detection unit, and an operation information adjustment that adjusts the operation information based on the feature information. and an operation information output unit that outputs adjusted operation information to an operation target.
- An information processing system including an operating device including a detection unit that detects an operation by a user, and the information processing device according to [6] or [8].
- the information processing system according to [9] wherein the information processing device has an image sensor and further includes an image capture unit that captures the first image.
- the information processing device further includes a mounting section capable of mounting at least an image sensor on a user's body.
- An information processing system including an operating device including a detection unit that detects an operation by a user, the information processing device according to [8], and an operation target that operates based on operation information based on a detection result.
- An information processing method implemented in an operating device that includes a detection unit and accepts operations by a user the method comprising: an acquisition procedure for acquiring a first image of a user's hand; and a method based on the first image.
- An information processing method comprising: a feature information generation procedure for generating feature information regarding a hand; and a feature information output procedure for outputting a detection result by a detection unit and the feature information.
- An information processing program that includes a detection unit and is executed by an information terminal connected to an operating device that accepts operations by a user, the information processing program acquiring a first image of a user's hand and fingers on the information terminal.
- An information processing program that executes an acquisition step, a feature information generation step of generating feature information regarding a finger based on a first image, and a feature information output step of outputting a detection result by a detection unit and feature information.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Computer Graphics (AREA)
- Architecture (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2022/018937 WO2023209818A1 (ja) | 2022-04-26 | 2022-04-26 | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム |
| US18/855,377 US20250252695A1 (en) | 2022-04-26 | 2022-04-26 | Operating apparatus, information processing apparatus, information processing system, information processing method, and information processing program |
| JP2024517664A JP7743616B2 (ja) | 2022-04-26 | 2022-04-26 | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2022/018937 WO2023209818A1 (ja) | 2022-04-26 | 2022-04-26 | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2023209818A1 true WO2023209818A1 (ja) | 2023-11-02 |
Family
ID=88518234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2022/018937 Ceased WO2023209818A1 (ja) | 2022-04-26 | 2022-04-26 | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US20250252695A1 (cg-RX-API-DMAC7.html) |
| JP (1) | JP7743616B2 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2023209818A1 (cg-RX-API-DMAC7.html) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20240404171A1 (en) * | 2023-06-01 | 2024-12-05 | Apple Inc. | Synthesized hand texture |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013218535A (ja) * | 2012-04-09 | 2013-10-24 | Crescent Inc | 三次元モデリングされたcg画像内にcg画像化された手指を表示する方法及び装置、並びに三次元モデリングされたcg画像を表示する広視野角ヘッドマウントディスプレイ装置 |
| JP2018516399A (ja) * | 2015-04-15 | 2018-06-21 | 株式会社ソニー・インタラクティブエンタテインメント | ヘッドマウントディスプレイでのピンチアンドホールドジェスチャのナビゲーション |
| JP2021514516A (ja) * | 2018-02-19 | 2021-06-10 | バルキリー・インダストリーズ・リミテッドValkyrie Industries Limited | バーチャルリアリティに対する触覚フィードバック |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7308112B2 (en) | 2004-05-14 | 2007-12-11 | Honda Motor Co., Ltd. | Sign based human-machine interaction |
| JP2017142856A (ja) * | 2017-05-02 | 2017-08-17 | アキュウェザー, インク.Accuweather, Inc. | カスタマイズ可能な気象分析システム |
| JP6719418B2 (ja) | 2017-05-08 | 2020-07-08 | 株式会社ニコン | 電子機器 |
-
2022
- 2022-04-26 WO PCT/JP2022/018937 patent/WO2023209818A1/ja not_active Ceased
- 2022-04-26 US US18/855,377 patent/US20250252695A1/en active Pending
- 2022-04-26 JP JP2024517664A patent/JP7743616B2/ja active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013218535A (ja) * | 2012-04-09 | 2013-10-24 | Crescent Inc | 三次元モデリングされたcg画像内にcg画像化された手指を表示する方法及び装置、並びに三次元モデリングされたcg画像を表示する広視野角ヘッドマウントディスプレイ装置 |
| JP2018516399A (ja) * | 2015-04-15 | 2018-06-21 | 株式会社ソニー・インタラクティブエンタテインメント | ヘッドマウントディスプレイでのピンチアンドホールドジェスチャのナビゲーション |
| JP2021514516A (ja) * | 2018-02-19 | 2021-06-10 | バルキリー・インダストリーズ・リミテッドValkyrie Industries Limited | バーチャルリアリティに対する触覚フィードバック |
Also Published As
| Publication number | Publication date |
|---|---|
| JP7743616B2 (ja) | 2025-09-24 |
| US20250252695A1 (en) | 2025-08-07 |
| JPWO2023209818A1 (cg-RX-API-DMAC7.html) | 2023-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7283506B2 (ja) | 情報処理装置、情報処理方法、及び情報処理プログラム | |
| CN113632176B (zh) | 用于基于神经肌肉数据的低等待时间身体状态预测的方法和装置 | |
| EP3743892B1 (en) | Visualization of reconstructed handstate information | |
| US6862006B2 (en) | Image processing apparatus and image processing method, and image processing program and recording medium of the same | |
| US20190228330A1 (en) | Handstate reconstruction based on multiple inputs | |
| CN110549353B (zh) | 力觉视觉化装置、机器人以及存储力觉视觉化程序的计算机可读介质 | |
| US20170136621A1 (en) | Dynamic Multi-Sensor and Multi-Robot Interface System | |
| JP2009276996A (ja) | 情報処理装置、情報処理方法 | |
| CN102470530A (zh) | 生成机器人的教导数据的方法以及机器人教导系统 | |
| US11422625B2 (en) | Proxy controller suit with optional dual range kinematics | |
| JP2019012965A (ja) | 映像制御方法、映像制御装置、及び映像制御プログラム | |
| WO2019181263A1 (ja) | 情報処理装置、情報処理方法およびプログラム | |
| CN111902859B (zh) | 信息处理装置、信息处理方法和程序 | |
| JP2015503393A (ja) | 運動を行っているユーザーの手及び/又は手首の回転を追跡するための方法及び装置 | |
| JP4048999B2 (ja) | 画像処理装置及び画像処理方法 | |
| JP7743616B2 (ja) | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム | |
| JP7300436B2 (ja) | 情報処理装置、システム、情報処理方法および情報処理プログラム | |
| JP7262763B2 (ja) | リハビリ支援装置、及びプログラム | |
| CN115066668A (zh) | 显示控制装置、显示控制方法和记录介质 | |
| WO2024147195A1 (ja) | 操作装置、情報処理装置、情報処理システム、情報処理方法、および情報処理プログラム | |
| US12401883B2 (en) | System, imaging apparatus, information processing apparatus, information processing method, and information processing program | |
| JP7535968B2 (ja) | ロボット遠隔操作制御装置、ロボット遠隔操作制御システム、ロボット遠隔操作制御方法、およびプログラム | |
| JP7536312B2 (ja) | 画像インタフェース装置、画像操作装置、操作対象物操作装置、操作対象物操作システム、操作対象物提示方法および操作対象物提示プログラム | |
| JP2024049842A (ja) | ロボット遠隔操作制御装置、ロボット遠隔操作制御方法、およびプログラム | |
| JP7770728B2 (ja) | 支援システム、支援方法及びプログラム |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22940098 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 18855377 Country of ref document: US |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2024517664 Country of ref document: JP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 22940098 Country of ref document: EP Kind code of ref document: A1 |
|
| WWP | Wipo information: published in national office |
Ref document number: 18855377 Country of ref document: US |