WO2021145068A1

WO2021145068A1 - Information processing device and information processing method, computer program, and augmented reality system

Info

Publication number: WO2021145068A1
Application number: PCT/JP2020/043524
Authority: WO
Inventors: 石川　毅; 木村　淳; 山野　郁男; 真一河野; 壮一郎稲谷
Original assignee: ソニーグループ株式会社
Priority date: 2020-01-17
Filing date: 2020-11-20
Publication date: 2021-07-22

Abstract

Provided is an information processing device for processing information on augmented reality.　The information processing device is equipped with an acquisition unit for acquiring the position and posture of a user's hand, and a control unit for controlling display operation of a display device for displaying a virtual object superimposed on a real space, wherein the control unit controlsthe display device so as to display information on how to hold the virtual object when the hand approaches the virtual object. The control unit controls the display device so as to display the information near the hand or on the virtual object.

Description

Information processing equipment and information processing methods, computer programs, and augmented reality systems

The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device and an information processing method for processing information related to augmented reality, a computer program, and an augmented reality feeling system.

Virtual reality (VR), augmented reality (AR), and MR (Mixed Reality) are becoming widespread as technologies for realizing realistic experiences. VR is a technology that allows virtual space to be perceived as reality. In addition, AR is a technology that expands the real space seen by the user by adding, emphasizing, attenuating, or deleting information to the real environment surrounding the user. Further, MR is a technology for displaying a virtual object (hereinafter, also referred to as “virtual object”) that replaces an object in real space and interlacing the real and the virtual. AR and MR are realized by using, for example, a see-through type head-mounted display (hereinafter, also referred to as “AR glass”). According to AR technology, virtual objects are superimposed and displayed on the real space landscape that the user observes through AR glasses, specific real objects are emphasized or attenuated, and specific real objects are deleted as if they do not exist. You can make it look like. Further, a proposal has been made for an information processing device that presents a contact between a real object (such as a user's finger) and a virtual object to the user (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2019-40226

An object of the present disclosure is to provide an information processing device and an information processing method for processing information related to augmented reality, a computer program, and an augmented reality feeling system.

The first aspect of the disclosure is
An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
It is an information processing device.

The control unit is at least one of a gripping method in which the information is held in the vicinity of the virtual object or the hand, and the virtual object is gripped by the thumb and one other finger, or the gripping method in which the virtual object is gripped by the entire hand. The display device is controlled so as to display the information including.

The control unit is at least one of a state in which the hand is holding the virtual object, a position where the hand is holding the virtual object, and a movement in which the virtual hand is holding the virtual object at the position of the hand. The display device is controlled so as to display the information indicating the above.

The second aspect of the present disclosure is
The acquisition step to acquire the position and posture of the user's hand,
A control step that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Have,
In the control step, the display device is controlled so as to display information on a method of grasping the virtual object when the hand approaches the virtual object.
It is an information processing method.

In addition, the third aspect of the present disclosure is
Acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space.
Written in computer readable format so that the computer works as
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
It is a computer program.

The computer program according to the third aspect of the present disclosure defines a computer program written in a computer-readable format so as to realize a predetermined process on the computer. In other words, by installing the computer program according to the third aspect of the present disclosure on the computer, a collaborative action is exhibited on the computer, and the same action and effect as the information processing device according to the first aspect of the present disclosure can be obtained. Obtainable.

In addition, the fourth aspect of the present disclosure is
A display device that superimposes and displays virtual objects in real space,
An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of the display device,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
It is an augmented reality system.

However, the "system" here means a logical assembly of a plurality of devices (or functional modules that realize a specific function), and each device or functional module is in a single housing. It does not matter whether or not it is.

According to the present disclosure, it is possible to provide an information processing device and an information processing method, a computer program, and an augmented reality system that realize realistic interaction with a virtual object by a user's hand or finger.

It should be noted that the effects described in the present specification are merely examples, and the effects brought about by the present disclosure are not limited thereto. In addition to the above effects, the present disclosure may have additional effects.

Still other objectives, features and advantages of the present disclosure will be clarified by more detailed description based on embodiments and accompanying drawings described below.

FIG. 1 is a diagram showing a functional configuration example of the AR system 100. FIG. 2 is a diagram showing a state in which AR glasses are attached to the user's head. FIG. 3 is a diagram showing a configuration example of the AR system 300. FIG. 4 is a diagram showing a configuration example of the AR system 400. FIG. 5 is a diagram showing an example in which the controller 500 is attached to the user's hand. FIG. 6 is a diagram showing an example of a functional configuration included in the control unit 140. FIG. 7 is a diagram showing how a virtual object is arranged around a user wearing AR glasses on his / her head. FIG. 8 is a diagram for explaining a mechanism for displaying a virtual object so that the AR glass follows the movement of the user's head. FIG. 9 is a diagram showing a state according to the distance between the user's hand and the virtual object. FIG. 10 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 11 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 12 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 13 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 14 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 15 is a diagram showing a display example of a UI that guides a method of grasping a virtual object. FIG. 16 is a diagram showing an example in which a user's hand approaches a virtual object from various directions. FIG. 17 is a diagram showing an example in which the UI that guides the method of grasping the virtual object is switched according to the contact state between the finger and the virtual object. FIG. 18 is a diagram showing an example in which the UI that guides the method of grasping the virtual object is switched according to the contact state between the finger and the virtual object. FIG. 19 is a flowchart showing a processing procedure for presenting a UI that guides a user how to grasp a virtual object. FIG. 20 is a diagram showing a display example of a virtual finger when the user's finger touches the virtual object. FIG. 21 is a diagram showing a display example of a virtual finger when the user's finger approaches the virtual object. FIG. 22 is a diagram showing another display example of the virtual finger when the user's finger approaches the virtual object. FIG. 23 is a flowchart showing a processing procedure for displaying a virtual hand to the user. FIG. 24 is a diagram showing a configuration example of the remote control system 2400. FIG. 25 is a diagram showing an operator approaching a virtual object with his / her hand on the master device 2410 side. FIG. 26 is a diagram showing a state in which the robot 2421 is approaching an object so as to follow the movement of the operator's hand on the slave device 2420 side. FIG. 27 is a flowchart showing a processing procedure for presenting a UI that guides the operator how to grasp the virtual object.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

In the real world, an object can be held by a method of picking or grasping, and the shape of the object is changed by the force applied from the hand to pinch or grasp. On the other hand, in the virtual world, since the object does not actually exist, the hand slips through the object, and it is not possible to hold the object in the same manner as in the real world. For example, an augmented reality system that provides a user interface (UI) in which a finger is thrust into an object in a virtual world and pinched with a fingertip, or a frame provided on the outer circumference of the object is pinched is also conceivable. However, the method of holding an object through the UI in the virtual world has a large divergence from the method of holding an object in the real world, and the reality is greatly impaired.

Also, since the object in the virtual world does not actually exist, when the object is held by the method of picking or grasping, the hand slips through the object, and the user cannot obtain a realistic tactile sensation. For example, if you attach an exoskeleton type force sense presentation device to your hand and hold an object in the virtual world, you can lock the movement of your hand so that the hand does not slip through the object, and you can hold the object in the real world. A method of realizing a similar virtual world method is also conceivable. However, it can be used only by a limited number of users and environments because the purchase cost of the force sense presentation device is high and a place where the force sense presentation device is installed is required.

Therefore, in this disclosure, a method of holding an object in the virtual world is realized without using an external device such as a force sense presentation device by a method that does not deviate from the method of holding an object in the real world.

A. System Configuration FIG. 1 shows an example of a functional configuration of the AR system 100 to which the present disclosure is applied. The illustrated AR system 100 includes a first sensor unit 110 that detects the position of the user's hand wearing the AR glass and the shape of the user's finger, a second sensor unit 120 mounted on the AR glass, and AR. It includes a display unit 131 that displays virtual objects on the glass, and a control unit 140 that comprehensively controls the operation of the entire AR system 100. The first sensor unit 110 includes a gyro sensor 111, an acceleration sensor 112, and a directional sensor 113. The second sensor unit 120, which is mounted on the AR glass, includes an outward camera 121, an inward camera 122, a microphone 123, a gyro sensor 124, an acceleration sensor 125, and a directional sensor 126.

Further, the AR system 100 includes a speaker 132 that outputs an audio signal such as a voice related to a virtual object, a vibration presentation unit 133 that provides feedback by vibration presentation to the back of the user's hand or other body parts, and an AR system 100 externally. A communication unit 134 for communicating with the communication unit 134 may be further provided. Further, the control unit 140 may be equipped with a large-scale storage unit 150 including an SSD (Solid State Drive) or the like.

The AR glass body is generally a spectacle-type or goggle-type device, which is used by the user by wearing it on the head, superimposing digital information on the visual field of the user's eyes or one eye, or emphasizing a specific real object. It can be degraded or attenuated, or a particular real object can be deleted to make it appear as if it does not exist. FIG. 2 shows a state in which AR glasses are attached to the user's head. In the illustrated AR glass, a display unit 131 for the left eye and a display unit 131 for the right eye are arranged in front of the left and right eyes of the user, respectively. The display unit 131 is transparent or translucent, and displays a virtual object superimposed on a landscape in real space, emphasizes or attenuates a specific real object, or deletes a specific real object to make it appear as if it does not exist. Or something. The left and right display units 131 may be independently displayed and driven, for example, to display a parallax image, that is, a virtual object in 3D. Further, an outward camera 121 directed in the user's line-of-sight direction is arranged substantially in the center of the AR glass.

The AR system 100 can be composed of two devices, for example, an AR glass worn on the user's head and a controller worn on the user's hand. FIG. 3 shows a configuration example of an AR system 300 including an AR glass 301 and a controller 302. The AR glass 301 includes a control unit 140, a storage unit 150, a second sensor unit 120, a display unit 131, a speaker 132, and a communication unit 134. Further, the controller 302 includes a first sensor unit 110 and a vibration presenting unit 133.

As another configuration example, the AR system 100 is composed of three devices: an AR glass worn by the user on the head, a controller worn on the user's hand, and an information terminal such as a smartphone or tablet. FIG. 4 shows a configuration example of an AR system 400 including an AR glass 401, a controller 402, and an information terminal 403. The AR glass 401 includes a display unit 131, a speaker 132, and a second sensor unit 120. The controller 402 includes a first sensor unit 110 and a vibration presenting unit 133. Further, the information terminal 403 includes a control unit 140, a storage unit 150, and a communication unit 134.

The specific device configuration of the AR system 100 is not limited to FIGS. 3 and 4. Further, the AR system 100 may further include components other than those shown in FIG.

Each component of the AR system 100 will be described with reference to FIG.

As shown in FIGS. 3 and 4, the first sensor unit 110 and the vibration presentation unit 133 are configured as a controller to be worn on the user's hand. The first sensor unit 110 includes a gyro sensor 111, an acceleration sensor 112, and a directional sensor 113. The first sensor unit 110 may be an IMU (Inertial Measurement Unit) including a gyro sensor, an acceleration sensor, and a directional sensor. Further, the vibration presenting unit 133 is configured by arranging electromagnetic type or piezoelectric type vibrators in an array. The sensor signal of the first sensor unit 110 is transferred to the control unit 140.

FIG. 5 shows an example in which the controller 500 including the first sensor unit 110 and the vibration presentation unit 133 is attached to the user's hand. In the example shown in FIG. 5,

IMUs

501, 502, and 503 are attached to the thumb and the proximal phalanx and the middle phalanx of the index finger by

bands

511, 512, and 513, respectively. Thereby, the posture of the thumb, the posture of the proximal phalanx and the middle phalanx of the index finger (or the angle of the second joint of the index finger) can be measured. Further, the vibration presenting unit 133 is attached to the back of the hand. The vibration presenting unit 133 may be fixed to the back of the hand with a band (not shown), an adhesive pad, or the like.

However, FIG. 5 shows an example of the first sensor unit 110, and another IMU may be attached to another place of the thumb and the index finger, or the IMU may be attached to a finger other than the thumb and the index finger. May be good. Further, the method of fixing the IMU to each finger is not limited to the band. Further, although FIG. 5 shows an example in which the first sensor unit 110 and the vibration presenting unit 133 are attached to the right hand, they may be attached to the left hand instead of the right hand, or may be attached to both hands.

Further, the sensor signal from the first sensor unit 110 (IMU501, 502, 503 in the example shown in FIG. 5) is transmitted to the control unit 140, and the drive signal of the vibration presentation unit 133 is received from the control unit 140. It is assumed that there is a wired or wireless transmission line. The control unit 140 can detect the position and posture of the fingers based on the sensor signal of the first sensor unit 110. As shown in FIG. 5, when the

IMUs

501, 502, and 503 are attached to the base and middle nodes of the thumb and index finger, the control unit 140 is based on the detection signals of the

IMUs

501, 502, and 503, respectively. Therefore, it is possible to recognize the position and orientation of the fingers (or the shape of the fingers) and the gestures of the fingers, such as the opening angle between the thumb and the index finger, the angle of the second joint of the index finger, and the presence or absence of contact between the thumb and the tip of the index finger.

Refer to FIG. 1 again to continue the explanation of each component of the AR system 100.

The second sensor unit 120 is mounted on the AR glass, and includes an outward camera 121, an inward camera 122, a microphone 123, a gyro sensor 124, an acceleration sensor 125, and a directional sensor 126.

The outward-facing camera 121 is composed of, for example, an RGB camera, and is installed so as to photograph the outside of the AR glass, that is, the front direction of the user wearing the AR glass. The outward-facing camera 121 can capture the operation of the user's fingers, but if the user's fingers are hidden behind an obstacle, or if the fingertips are hidden by the back of the hand, the user is behind the body. It is not possible to capture the operation of the user's fingers when turning. Further, the outward-facing camera 121 may further include any one of an IR camera including an IR light emitting unit and an IR light receiving unit, and a TOF (Time Of Flight) camera. When an IR camera is used for the outward camera 121, a retroreflective material is attached to an object to be captured such as the back of the hand, and the IR camera emits infrared light and emits infrared light reflected from the retroreflective material. Receive light. The image signal captured by the outward camera 121 is transferred to the control unit 140.

The inward camera 122 is composed of, for example, an RGB camera, and is installed so as to photograph the inside of the AR glass, specifically, the eyes of a user wearing the AR glass. The line-of-sight direction of the user can be detected based on the captured image of the inward-facing camera 122. The image signal captured by the inward camera 122 is transferred to the control unit 140.

The microphone 123 may be a single sound collecting element or a microphone array including a plurality of sound collecting elements. The microphone 123 collects the voice of the user wearing the AR glass and the ambient sound of the user. The audio signal picked up by the microphone 123 is transferred to the control unit 140.

The gyro sensor 124, the acceleration sensor 125, and the azimuth sensor 126 may be composed of an IMU. The sensor signals of the gyro sensor 124, the acceleration sensor 125, and the directional sensor 126 are transferred to the control unit 140. The control unit 140 can detect the position and posture of the head of the user wearing the AR glasses based on these sensor signals.

The display unit 131 is composed of a transmissive display (glasses lens, etc.) installed in front of both eyes or one eye of the user wearing AR glasses, and is used for displaying a virtual world. Specifically, the display unit 131 expands the real space seen by the user by displaying information (virtual objects) and emphasizing, attenuating, or deleting real objects. The display unit 131 performs a display operation based on a control signal from the control unit 140. Further, the mechanism for see-through display of virtual objects on the display unit 131 is not particularly limited.

The speaker 132 is composed of a single sounding element or an array of a plurality of sounding elements, and is installed in, for example, an AR glass. For example, the speaker 132 outputs the sound related to the virtual object displayed on the display unit 131, but other audio signals may be output.

The communication unit 134 has a wireless communication function such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The communication unit 134 mainly performs a communication operation for realizing data exchange between the control unit 140 and an external system (not shown).

The control unit 140 is installed in the AR glass or is arranged in a device (smartphone or the like) separated from the AR glass together with a drive power source such as a storage unit 150 or a battery. The control unit 140 executes various programs read from the storage unit 150 to perform various processes.

FIG. 6 schematically shows an example of a functional configuration included in the control unit 140. In the illustrated example, the control unit 140 includes an application execution unit 601, a head position / posture detection unit 602, an output control unit 603, a finger position / posture detection unit 604, and a finger gesture detection unit 605. These functional modules are realized by executing various programs read from the storage unit 150 by the control unit 140. However, FIG. 6 shows only the minimum necessary functional modules for realizing the present disclosure, and the control unit 140 may further include other functional modules.

The application execution unit 601 executes the application program including the AR application under the execution environment provided by the OS. The application execution unit 601 may execute a plurality of application programs in parallel at the same time. AR apps are applications such as video playback and 3D object viewers, but virtual objects can be superimposed or specified in the view of a user wearing AR glasses (see Fig. 2) on their heads. Emphasizes or attenuates a real object in, or deletes a particular real object to make it appear as if it doesn't exist. The application execution unit 601 also controls the display operation of the AR application (virtual object) by using the display unit 131. Virtual objects generated by the AR application are arranged all around the user. FIG. 7 schematically shows how a plurality of

virtual objects

701, 702, 703, ... Are arranged around 700 around the user wearing the AR glass on the head. The application execution unit 601 has each

virtual object

701, 702, 703 around the user with reference to the position of the user's head or the position of the center of gravity of the body estimated based on the sensor information from the second sensor unit 120. , ... are placed.

The head position / orientation detection unit 602 is based on the sensor signals of the gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126 included in the second sensor unit 120 mounted on the AR glass, and is based on the sensor signals of the user's head. The position and orientation are detected, and the user's line-of-sight direction or visual field range is recognized.

The output control unit 603 controls the output of the display unit 131, the speaker 132, and the vibration presentation unit 133 based on the execution result of an application program such as an AR application. For example, the output control unit 603 specifies the user's visual field range based on the detection result of the head position / posture detection unit 602 so that the virtual object arranged in the visual field range can be observed by the user through the AR glass. That is, the display operation of the virtual object is controlled by the display unit 131 so as to follow the movement of the user's head.

A mechanism for displaying a virtual object so that the AR glass follows the movement of the user's head will be described with reference to FIG. In FIG. 8, the depth direction of the user's line of sight is the z _w axis, the horizontal direction is the y _w axis, and the vertical direction is the x _w axis, and the origin position of the user's _{reference axis x w} y _w z _{w is the user's viewpoint position.} do. Roll θ _z corresponds to the movement of the user's head _{around the z w} axis, tilt θ _y corresponds to the movement of the user's head around the y _w axis, and pan θ _z _{corresponds to the movement of the user's head around the x w axis.} .. The head position / orientation detection unit 602 moves the user's head in each of the roll, tilt, and pan directions (θ _z , θ _{y) based on the sensor signals of the gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126.} , Θ _z ) and the posture information consisting of the translation of the head. Then, the output control unit 603 moves the display angle of view of the display unit 131 in the real space (for example, see FIG. 7) in which the virtual object is arranged so as to follow the posture of the user's head. , The image of the virtual object existing at the display angle of view is displayed on the display unit 131. Specifically, the region 802-1 is rotated according to the roll component of the user's head movement, the region 802-2 is moved according to the tilt component of the user's head movement, and the user's head movement is performed. The display angle of view is moved so as to cancel the movement of the user's head by moving the area 802-3 according to the pan component of. Therefore, since the virtual object arranged at the display angle of view moved according to the position and orientation of the user's head is displayed on the display unit 131, the user can see the virtual object superimposed on the AR glass. You can observe the space.

The functional configuration of the control unit 140 will be described with reference to FIG. 6 again.

The finger position / posture detection unit 604 detects the position / posture of the user's hand and fingers wearing the AR glass based on the recognition result of the image taken by the outward camera 121 or the detection signal of the first sensor unit 110. .. Further, the finger gesture detection unit 605 detects the gesture of the user's finger wearing the AR glass based on the recognition result of the image taken by the outward camera 121 or the detection signal of the first sensor unit 110. The gesture of the finger referred to here includes the shape of the finger, specifically the angle of the third joint and the second joint of the index finger, and the presence or absence of contact between the thumb and the fingertip of the index finger.

In the present embodiment, the finger position / posture detection unit 604 and the finger gesture detection unit 605 are mainly from the first sensor unit 110 (gyro sensor 111, acceleration sensor 112, and orientation sensor 113) attached to the user's hand. The posture of the finger and the gesture of the finger are detected with higher accuracy by using the information of the position and the posture of the finger and the restraint condition of the position and the posture that the finger can take. For example, when the user turns his / her hand toward the back side of the body such as the back or hips, the fingers cannot be detected by image recognition from the head, but high accuracy can be obtained by using the sensor signal of the first sensor unit 110 attached to the hand. The position and orientation of the fingers can be detected with. On the other hand, in the case of the method of detecting the position and orientation of the fingers and the gestures of the fingers using the outward-facing camera 121, it may not be possible to detect with high accuracy due to occlusion or the like.

B. Presenting a guide on how to hold a virtual object The AR system 100 according to the present disclosure presents a guide on how to hold a virtual object to a user so as to hold a virtual object so as not to deviate from the method of holding an object in the real world. .. Specifically, the finger position / posture detection unit 604 detects the position / posture of the user's hand trying to grasp the virtual object. The application execution unit 601 determines the distance between the user's hand and the virtual object based on the positional relationship between the position and orientation of the user's hand detected by the finger position / posture detection unit 604 and the virtual object arranged in the real space. Correspondingly, a process of presenting a guide of the gripping method is performed around the hand. Then, the output control unit 603 outputs a display to the display unit 131 (or AR glass) of a virtual object that guides the gripping method around the hand.

Here, as the distance between the user's hand and the virtual object, three states of "approach", "contact", and "immersion" are defined. FIG. 9 shows three states of “approach”, “contact”, and “entry”. "Approach" is a state in which the shortest distance between the user's hand and the virtual object is equal to or less than a predetermined value. "Contact" is a state in which the shortest distance between the user's hand and the virtual object is zero. "Embedding" is a state in which the user's hand is interfering with the area of the virtual object.

In the AR system 100 according to the present disclosure, a display indicating the gripping method is displayed in the vicinity of the virtual object so that the gripping method of the virtual object can be understood when the user's hand approaches the virtual object. Based on the displayed gripping method, the user comes to have a virtual object so as not to deviate from the method of holding an object in the real world, and secures reality.

It is difficult to completely perform all the gripping methods that can be done with a real object for a virtual object. For example, even when the virtual object is pinched and held by the thumb and index finger, the finger slips through, so that the force of pinching cannot be applied to the virtual object. Since it is difficult for the user to keep the thumb and index finger being picked at a stable distance, it is not possible to pick and hold a virtual object in exactly the same way as in the real world. Further, the AR system 100 can simplify the processing of the system by limiting the gripping method of the virtual object, but the user may not be able to use an arbitrary gripping method.

When trying to guide the method of grasping a virtual object by the same method as in the real space, there is a situation where the AR system 100 cannot handle it. Therefore, when the user wants to grasp the virtual object to be observed through the AR glass, it becomes difficult to understand what kind of method should be used and whether the virtual object can be grasped.

Therefore, in the AR system 100 according to the present disclosure, when the user's hand approaches the virtual object, a UI that guides the method of grasping the virtual object is displayed near the virtual object (or near the hand), and the AR system Information that guides the gripping method that can be handled in 100 is shown to the user. The guide of the gripping method is displayed by using the display unit 131 in the form of UI, for example, but character information such as a text message and voice announcement may also be performed. Therefore, the user can easily grasp the virtual object without hesitation by being guided by the UI that guides the grasping method of the virtual object.

The guides for gripping virtual objects are assumed to be the following types (1) to (3) as the types of guides for gripping virtual objects. Of course, other methods may be used to guide the user on how to grasp the virtual object.

(1) Indicates a state in which a hand is holding a virtual object.
(2) Indicates the position where the hand holds the virtual object.
(3) The movement of grasping the virtual object with the virtual hand is shown at the position of the hand.

For example, the application execution unit 601 that generates the virtual object may also use the display unit 131 to display the UI that guides the method of grasping the virtual object.

FIG. 10 shows one display example of a UI that guides a method of grasping a virtual object according to the above guide type (1). In the example shown in FIG. 10, when the user's hand approaches the cylindrical virtual object 1001, the hand in the state where the user is holding the outer circumference of the virtual object 1001 is attached to the virtual object 1001 by the dotted line in the figure. It is displayed superimposed. When it is assumed that the virtual object 1001 is a heavy object, the position where the virtual object 1001 is held is set in consideration of the position of the center of gravity of the virtual object 1001. Therefore, the user can easily grip the outer circumference of the virtual object 1001 without hesitation, guided by the UI that guides the gripping method shown in FIG.

Further, FIG. 11 shows another display example of the UI that guides the method of grasping the virtual object according to the above guide type (1). In the example shown in FIG. 11, when the user's hand approaches the cylindrical virtual object 1101, a virtual mirror 1102 is placed behind the virtual object 1101, and the user places the outer circumference of the mirror image 1103 of the virtual object 1101. The hand holding the object is displayed through the mirror 1102. When it is assumed that the virtual object 1101 is a heavy object, the position where the virtual object 1001 is held is set in consideration of the position of the center of gravity of the virtual object 1101. In the example shown in FIG. 10, the fingertip of the user is hidden behind the virtual object 1101 and cannot be seen, but in the example shown in FIG. 11, the position of each finger touching the outer circumference of the mirror image 1103 can be confirmed. Therefore, the user can be guided by the UI that guides the gripping method shown in FIG. 11 to have a deeper understanding of how to use each finger when gripping the outer circumference of the virtual object 1101, and can easily grip the virtual object 1101 without hesitation.

FIG. 12 shows one display example of the UI that guides the method of grasping the virtual object according to the above guide type (2). In the example shown in FIG. 12, when the user's hand approaches the thin cylindrical virtual object 1201, the user grips (or precisely grips) the outer circumference of the virtual object 1201 so as to be sandwiched between the thumb and the index finger. The

positions

1202 and 1203 picked by the thumb and index finger at the time are displayed superimposed on the virtual object 1201 by the dotted line in the figure. When it is assumed that the virtual object 1201 is a heavy object, the position where the virtual object 1201 is held is set in consideration of the position of the center of gravity of the virtual object 1201. Therefore, the user is guided by the UI that guides the gripping method shown in FIG. 12, and has a deeper understanding of the position where the outer circumference of the virtual object 1201 is pinched by the thumb and the index finger (or the position where the virtual object 1201 is precisely gripped) without hesitation. It can be easily grasped.

Further, FIG. 13 shows another display example of the UI that guides the method of grasping the virtual object according to the above guide type (2). In the example shown in FIG. 13, when the user's hand approaches the thick cylindrical virtual object 1301, the user grips (or grips) the outer circumference of the virtual object 1301 with all fingers. The finger positions 1302 to 1306 are superimposed and displayed on the virtual object 1301 by the dotted line in the figure. When it is assumed that the virtual object 1301 is a heavy object, the position of holding the virtual object 1001 is set in consideration of the position of the center of gravity of the virtual object 1301. Therefore, the user is guided by the UI that guides the gripping method shown in FIG. 13 to better understand the position (or the position where the grip strength is gripped) where the outer circumference of the virtual object 1301 is gripped by all fingers, and without hesitation. It can be easily grasped.

FIG. 14 shows one display example of the UI that guides the method of grasping the virtual object according to the above guide type (3). In the example shown in FIG. 14, when the user's hand approaches the thin cylindrical virtual object 1401, the virtual thumb and index finger are once spread as shown by

reference numbers

1402 and 1403 in the figure. An animation of closing and grasping (or precisely grasping with a fingertip) the virtual object 1401 so as to sandwich it is displayed superimposed on the actual thumb and index finger. Therefore, the user is guided by the UI that guides the gripping method shown in FIG. 14, and once the thumb and index finger are spread and then closed, the user picks the outer periphery of the virtual object 1201 (or precisely grips). Can be easily grasped without hesitation by understanding more deeply.

Further, FIG. 15 shows another display example of the UI that guides the method of grasping the virtual object according to the above guide type (3). In the example shown in FIG. 15, when the user's hand approaches the thick cylindrical virtual object 1501, the user uses all fingers on the outer circumference of the virtual object 1501 as shown by the reference number 1502 in the figure. The virtual hand movement when gripping (or gripping force) is superimposed on the actual hand and displayed. In the example shown in FIG. 15, an animation in which the wrist is slightly rotated so as to imitate the outer peripheral surface of the virtual object 1501 is superimposed and displayed on the actual hand. Therefore, the user is guided by the UI that guides the gripping method shown in FIG. 15, rotates his / her wrist so as to follow the outer circumference of the virtual object 1501, and then grips the virtual object 1501 (or grips the grip force). You can understand the operation) more deeply and easily grasp it without hesitation.

According to the AR system 100 according to the present disclosure, when the user brings his / her hand close to grip the virtual object, he / she can see the UI guiding the gripping method as shown in FIGS. 10 to 15 through the AR glass. It can grip the virtual object according to the specified gripping method.

The application execution unit 601 creates and displays a virtual object, and also displays a UI that guides how to hold the virtual object. The application execution unit 601 acquires the movement of the hand or finger when the user tries to grasp the virtual object based on the detection results of the finger position / posture detection unit 604 and the finger gesture detection unit 605, and the user virtual object. You may want to evaluate the gripping motion of.

C. Selection of gripping method There are two methods of gripping an object: precision gripping, in which the object is gripped with the thumb and index finger, and grip strength gripping, in which the object is gripped with all fingers (or with the entire finger). Furthermore, there are variations in gripping methods such as intermediate gripping using the side surface of the finger and gripping without using the thumb. Further, in order to stably grip an object with only one hand, it is necessary to sandwich the object between two or more opposite surfaces of the hands. In some cases, multiple fingers are used on one surface.

The UI that guides the gripping method guides the user to grip the virtual object by the predetermined gripping method. The UI that guides the gripping method differs depending on the method of gripping the virtual object such as pinching or grasping, and using which finger to pinch or grasp, and further differs depending on the type of guide. Which gripping method is selected to guide the user is determined at the time of designing the AR system 100, and it is assumed that the method is not switched according to each user or the user's operation. Alternatively, it is assumed that the gripping method used for each virtual object is set in advance and does not switch according to each user or the user's operation. For example, the optimum gripping method according to the size and shape of the virtual object is preset for each virtual object.

However, the gripping method for the same virtual object may be dynamically switched according to the user's operation. For example, gripping for each user or virtual object using a machine learning model learned to estimate the optimal gripping method based on user attribute information such as the user's personality, habit, age, gender, and physique. The method may be dynamically switched.

For example, it may be possible to acquire the shape of the user's hand based on the detection result of the finger gesture detection unit 605 and determine whether the user is trying to pick or grab the virtual object. When controlling the display of the UI that guides the gripping method, the application execution unit 601 selects either the UI that guides the gripping method to be picked or the UI that guides the gripping method based on the determination result of the gripping method. It may be selected and presented to the user through the AR glass.

C-1. Selection of gripping method according to approaching direction The optimum gripping method may differ depending on the direction in which the user's hand approaches the virtual object. In such a case, the UI that guides the gripping method may be dynamically switched according to the direction in which the user's hand approaches the virtual object. The application execution unit 601 detects the direction in which the user's hand approaches the virtual object based on the detection result of the finger position / posture detection unit 604, and determines the optimum gripping method according to the further approaching direction. Then, the user is presented with a UI that guides the gripping method selected based on the determination result.

For example, the virtual object 1601 in the shape of a bottle having an elongated neck as shown in FIG. 16 has a different optimum gripping method depending on the location to be gripped. When the user's hand approaches the thick body portion of the lower half of the virtual object 1601 as shown by the reference number 1602, the gripping method is suitable. Further, as shown by

reference numbers

1603 and 1604, when the user's hand approaches the elongated neck portion of the upper half, the gripping method of picking is suitable. Further, even when the user's hand approaches the elongated neck portion, the case where the user's hand approaches the side surface of the neck as shown by the reference number 1603 and the case where the user's hand approaches the side surface of the neck as shown by the reference number 1604 When approaching a part, the optimum picking method is different.

The application execution unit 601 can determine from which direction the user's hand is approaching the currently displayed virtual object based on the detection result of the finger position / posture detection unit 604. Then, based on the determination result, the application execution unit 601 selects a UI that guides the gripping method suitable for the direction in which the user's hand approaches the virtual object, and presents the UI to the user through the AR glass.

C-2. Selection of gripping method according to the user The optimum gripping method may differ even for the same virtual object depending on the user's age (infant, elderly person), physical injury, and the user's daily gripping method. In addition, the optimum guide type may differ depending on the age (infant, elderly), race, physical injury, and daily gripping method of the user. Therefore, the UI that guides the gripping method for the same virtual object may be switched for each user.

For example, when gripping a cylindrical virtual object, an adult male can grip it stably by the gripping method, but an infant with short fingers can firmly grip it with all fingers. Otherwise, it may not be possible to grip stably.

In addition, even if it is a virtual object that can be stably gripped by a healthy person user by the gripping method, if the user has a damaged finger, the gripping method must be performed by using all the fingers. If so, it may not be possible to grip it stably. In the case of a user with a missing finger, when using a UI showing a movement held by a virtual hand, it is necessary to change to a UI showing a movement using only available fingers. Moreover, even a healthy person user may have a different daily gripping method depending on his / her habits and preferences.

The user may manually input information on the user's own user attributes such as age, race, physical injury, daily gripping method, etc. into the AR system 100 by himself / herself. Alternatively, the AR system 100 may be able to acquire information on user attributes as user registration information at the start of use of the AR system 100. Further, the user attribute may be estimated by using the machine learning model from the sensor information of the first sensor unit 110 and the second sensor unit 120. Therefore, the first sensor unit 110 and the second sensor unit 120 may be equipped with sensors other than those shown in FIG. 1, such as a biological sensor.

Then, the application execution unit 601 determines the optimum gripping method based on the attribute information of the user, and presents the UI to the user through the AR glass to guide the selected gripping method based on the determination result.

D. Presenting a guide on how to grip a real object So far, the AR system 100 uses AR glasses to provide a UI that guides the gripping method so that the user can understand how to grip the virtual object when the user's hand approaches the virtual object. The embodiments presented in the above have been described. The present disclosure can be applied not only when the user grips a virtual object, but also when the user grips a real object, that is, an object existing in the real space.

The application execution unit 601 can identify a real object in the user's field of view based on the image recognition result of the captured image of the outward camera 121. Further, when the application execution unit 601 detects that the user has approached the real object based on the detection result of the finger position / posture detection unit 604 and the image recognition result of the user's hand based on the image captured by the outward camera 121. , Select the optimal gripping method and display the UI that guides the gripping method of the real object on the AR glass. Further, the application execution unit 601 may dynamically switch the UI that guides the gripping method based on the direction in which the user's hand is approaching the real object, the user's attributes, and the like.

For example, the application execution unit 601 determines the optimum gripping method so that the real object can be gripped while avoiding a dangerous place to hold, and the user guides the selected gripping method based on the determination result through the AR glass. Present to. Places that are dangerous to hold are places that cannot be gripped stably, or places that are weak and break when trying to grip. The application execution unit 601 estimates a dangerous place to hold the target real object based on the result of object recognition from the image captured by the outward camera 121. For each category of the object, a dangerous place to hold may be defined in advance based on an empirical rule or the like, and the application execution unit 601 may determine the optimum gripping method based on the definition. Alternatively, the optimum gripping method may be determined by using a machine learning model learned to estimate a dangerous place to hold based on the recognized object category, size, shape, and the like.

E. Guide display trigger In the AR system 100 according to the present disclosure, when a user's hand approaches a virtual object, a UI indicating a method of grasping the virtual object is displayed on the AR glass. However, the user does not bring his / her hand close to grasp the virtual object, and the position of the hand may happen to be close to the virtual object. If the user does not intend to grab the virtual object and the UI that guides how to grab the virtual object is displayed on the AR glass, the user does not need the guide UI or obstructs the view and gets in the way. do.

Therefore, when the user's hand approaches the virtual object, the grasping method is guided on the condition that the user is looking at the target virtual object or the user is interested in the target virtual object. The UI to be used may be displayed on the AR glass.

The application execution unit 601 can detect, for example, the line-of-sight direction of the user from the captured image of the inward camera 122, and determine whether or not the user is looking at the virtual object. Alternatively, the application execution unit 601 may be able to estimate the degree of interest of the user in the virtual object by using the machine learning model from the sensor information of the first sensor unit 110 and the second sensor unit 120. Therefore, the first sensor unit 110 and the second sensor unit 120 may be equipped with sensors other than those shown in FIG. 1, such as a biological sensor. Then, when the condition that the user is looking at the target virtual object or the user is interested in the target virtual object is satisfied, the application execution unit 601 approaches the virtual object. When it is detected that the virtual object is being gripped, a UI that guides how to grasp the virtual object is displayed on the AR glass.

F. Switching the display of the guide at the time of contact Since the virtual object does not actually exist, even if the user touches the virtual object by a method such as picking or grasping it, a realistic tactile sensation cannot be obtained and the hand slips through the virtual object.

For example, an exoskeleton type force sense presenting device may be attached to the hand to give the user a realistic tactile sensation according to the contact state with the virtual object. However, there are problems in the purchase cost and installation location of the force sense presentation device.

Therefore, in the present disclosure, when the user's finger touches the virtual object, the display of the UI that guides the gripping method of the virtual object is switched so that the user is notified or fed back that the contact has been made.

The application execution unit 601 acquires the movement of the hand or finger when the user tries to grasp the virtual object based on the detection results of the finger position / posture detection unit 604 and the finger gesture detection unit 605, and obtains the movement of the user's finger. The contact state with the virtual object can be determined. Then, the application execution unit 601 switches the display of the UI that guides the method of grasping the virtual object based on the determination result, and notifies or feeds back the contact to the user.

FIG. 17 shows an example of displaying a UI that highlights the contact points between the user's thumb and index finger and the virtual object 1701 when the user's hand touches the outer circumference of the cylindrical virtual object 1701. ing.

The application execution unit 601 indicates that the user's thumb and index finger are in contact with the virtual object 1701 and that the user's thumb, index finger and virtual object are in contact with each other based on the detection results of the finger position / orientation detection unit 604 and the finger gesture detection unit 605. When the contact points 1702 and 1703 with the 1701 are detected, the contact points 1702 and 1703 are switched to the highlighted UI, and the user is notified or fed back that the contact points have been made. By switching to the UI that displays the highlight, the user can accurately pinch the virtual object 1701 at an appropriate distance between the thumb and the index finger (the distance that matches the width of the virtual object 1701).

Further, even if the guiding UI is switched as shown in FIG. 17 to notify the user that the finger has touched the virtual object, the virtual object does not actually exist, and the user does not have a realistic tactile sensation. The user's finger may be further sunk into the virtual object by continuing the gripping operation.

Therefore, in the present disclosure, the user is notified of the degree of the finger immersing in the virtual object by gradually switching the UI that guides the method of grasping the virtual object according to the degree of the user's finger immersing in the virtual object. Try to give feedback.

FIG. 18 shows when the user's hand touches the outer circumference of the cylindrical virtual object 1801 and then the thumb and index finger are sunk into the virtual object 1801 because the user narrows the distance between the thumb and the index finger. Shows an example of displaying a UI that emphasizes a highlight indicating a contact point between the user's thumb and index finger and the virtual object 1801.

When the application execution unit 601 detects that the user's thumb and index finger are sunk into the virtual object 1801 based on the detection results of the finger position / orientation detection unit 604 and the finger gesture detection unit 605, the thumb, index finger, and virtual Switch to a UI that highlights the highlights of

contact points

1802 and 1803 with the object 1801 to notify or feed back to the user that the finger is sunk into the virtual object 1801. In addition, the application execution unit 601 gradually switches to a UI that emphasizes highlights according to the degree of immersion. The user can visually understand that the virtual object 1801 is being excessively pressed with the thumb and forefinger by gradually switching the highlights indicated by the UI that guides the grasping method, and the user can grasp correctly. It can be modified to work.

G. Processing procedure for presenting a UI that guides the gripping method As described in Sections B to F above, the AR system 100 according to the present disclosure has a virtual object so as not to deviate from the method of holding an object in the real world. Provides the user with a guide on how to grab a virtual object.

FIG. 19 shows in the form of a flowchart a processing procedure for presenting a UI that guides a user how to grasp a virtual object in the AR system 100. For example, the application execution unit 601 plays a central role in this processing procedure.

First, the application execution unit 601 acquires the position of the user's hand based on the detection result of the finger position / posture detection unit 604 (step S1901). The application execution unit 601 shall constantly monitor the relative position between the displayed virtual object and the hand of the user who is trying to grasp the virtual object.

Then, the application execution unit 601 checks whether the user's hand approaches the virtual object, that is, whether the shortest distance between the user's hand and the virtual object is equal to or less than a predetermined value (step S1902).

Note that the user did not bring his hand close to the virtual object in an attempt to grasp it, and the position of his hand may happen to be close to the virtual object. Therefore, in step S1902, the application execution unit 601 adds the user's hand on the condition that the user is looking at the target virtual object or the user is interested in the target virtual object. May be checked to see if it has approached the virtual object.

Here, when the shortest distance between the user's hand and the virtual object becomes equal to or less than a predetermined value (Yes in step S1902), the application execution unit 601 uses the UI to guide the user on how to grasp the virtual object. Is determined (step S1903).

In step S1903, the application execution unit 601 selects the type of guide for grasping the virtual object. Further, the application execution unit 601 has selected a gripping method preset for the virtual object or a gripping method selected based on user attributes such as the user's personality, habit, age, gender, and physique. The UI for guiding is determined by the guide type. Further, the application execution unit 601 may select a gripping method according to the direction in which the user's hand approaches the virtual object, and determine a UI for guiding the gripping method.

Then, based on the result determined in step S1903, the application execution unit 601 displays a UI for guiding the method of grasping the virtual object approaching by the user's hand near the virtual object using the display unit 131 (the display unit 131 is used). Step S1904).

After that, the application execution unit 601 acquires the detection results of the finger position / orientation detection unit 604 and the finger gesture detection unit 605 for the movement of the hand or finger when the user tries to grasp the virtual object (step S1905). Determine the contact state between the user's finger and the virtual object.

Then, when the user's finger touches the virtual object (Yes in step S1906), the application execution unit 601 switches the display of the UI that guides the gripping method of the virtual object, and notifies or feeds back the contact. (Step S1907).

In step S1907, the application execution unit 601 uses the display unit 131 to display, for example, a UI that highlights the contact point between the user's finger and the virtual object. Further, when the user's finger is sunk into the virtual object, the application execution unit 601 gradually switches the highlight display according to the degree of sunk.

H. Virtual hand display at the time of contact As shown in Section F above, the user is notified of the contact state by switching the display of the UI that guides the gripping method according to the contact state between the user's finger and the virtual object. Or you can give feedback. However, since the virtual object does not exist, the user still has no realistic tactile sensation.

Therefore, in the present disclosure, when the user's finger touches the virtual object, the virtual finger is displayed. Specifically, when the user is trying to grasp two opposing faces of a virtual object, each of the first finger approaching one face and the second finger approaching the other face On the other hand, the first virtual finger and the second virtual finger are displayed. At that time, the opening amount of the first virtual finger and the second virtual finger is made wider than the opening degree of the actual first finger and the second finger, and the actual first finger and the second finger are opened. Each virtual finger is displayed so that the first virtual finger and the second virtual finger come into contact with the virtual object and are in a gripped state when the fingers of the finger touch.

FIG. 20 shows an example of displaying a virtual finger when the user's finger touches the virtual object. FIG. 20 is a display example when the user sandwiches and grips two opposing surfaces of the virtual object 2001 between the thumb and the index finger, and the position 2002 of the virtual thumb and the position 2003 of the virtual index finger are shown by dotted lines, respectively. There is. The actual thumb and index finger are drawn with solid lines. As can be seen from FIG. 20, the opening amount of the virtual thumb 2002 and the virtual index finger 2003 is wider than the opening degree of the actual thumb and index finger. Then, when the actual thumb and index finger come into contact with each other, the virtual thumb 2002 and the virtual index finger 2003 come into contact with the virtual object 2001 and are in a gripped state.

The application execution unit 601 can acquire the movements of the thumb and index finger when the user tries to grasp the virtual object 2001 based on the detection results of the finger position posture detection unit 604 and the finger gesture detection unit 605. Then, the application execution unit 601 displays the virtual thumb 2002 and the virtual index finger 2003 for each of the thumb approaching one surface and the index finger approaching the other surface of the virtual object 2001.

At that time, the application execution unit 601 makes the opening amount of the virtual thumb 2002 and the virtual index finger 2003 wider than the opening degree of the actual thumb and index finger. Therefore, when the actual thumb and index finger come into contact with each other, the user visually observes that the virtual thumb 2002 and the virtual index finger 2003 come into contact with the virtual object 2001 in the virtual space displayed by the display unit 131 and are in a grasped state. Recognize. At this time, a contact force acts between the actual thumb and index finger, but the user recognizes the tactile sensation that his / her thumb and index finger receive from the virtual object 2001.

Since the virtual object 2001 does not actually exist, even if the user performs a method of picking or grasping the virtual object 2001, the hand slips through the virtual object 2001, and the user cannot obtain a realistic tactile sensation. Therefore, the application execution unit 601 displays the virtual thumb 2002 and the virtual index finger 2003 at positions near the user's thumb and index finger so as to sandwich the virtual object 2001. This allows the user to feel the contact between the thumb and index finger. Although the thumb and index finger are actually in contact with each other, the user can see the virtual object 2001 by the virtual thumb 2002 and the virtual index finger 2003 in the virtual space displayed by the display unit 131 (or seen through the AR glass). Since the sandwiched image is viewed, it is possible to obtain the feeling of actually sandwiching and holding the virtual object 2001.

In short, in the present disclosure, when there is an inconsistency between the movement of the body in the visual sense and the movement of the body felt by oneself, the visual information becomes predominant and a pseudo tactile sensation is generated. The illusion, or "visual-tactile interaction," can be used to give the user a tactile sensation when grasping a virtual object.

Further, as shown in FIG. 20, in order to make the user recognize that the method of grasping the virtual object such that the actual fingers come into contact with each other when the virtual finger touches the virtual object is a very natural and smooth movement. , When the user's hand approaches the virtual object, the display of the virtual finger that is wider than the actual finger opening is started.

FIG. 21 shows an example of displaying a virtual finger when the user's finger approaches the virtual object. FIG. 21 is a display example when the user brings his / her hand close to the virtual object 2101 with the intention of sandwiching and grasping the two opposing surfaces of the virtual object 2101 with the thumb and the index finger, and the position of the virtual thumb. The 2102 and the virtual index finger position 2103 are shown by dotted lines, respectively. The actual thumb and index finger are drawn with solid lines. As can be seen from FIG. 21, the opening amount of the virtual thumb 2102 and the virtual index finger 2103 is wider than the opening degree of the actual thumb and index finger.

The application execution unit 601 acquires the position of the user's hand based on the detection result of the finger position / posture detection unit 604, and when the shortest distance between the hand and the virtual object 2101 is equal to or less than a predetermined value, the user's hand Detects a state in which is approaching the virtual object 2101. Then, the application execution unit 601 displays the virtual thumb 2102 and the virtual index finger 2103 for each of the thumb approaching one surface and the index finger approaching the other surface of the virtual object 2001.

At that time, the application execution unit 601 makes the opening amount of the virtual thumb 2102 and the virtual index finger 2103 wider than the opening degree of the actual thumb and index finger. After that, when the actual thumb and index finger come into contact with each other, the virtual thumb 2102 and the virtual index finger 2103 come into contact with the virtual object 2101 in the virtual space displayed by the display unit 131 to be in a gripped state. The application execution unit 601 is a very natural and smooth operation in which the virtual thumb 2102 and the virtual index finger 2103 try to grasp the virtual object 2101 from the time when the user's hand enters the approaching state to the time when the user's hand shifts to the contacting state. Is displayed. Therefore, the user can easily grasp the virtual object 2101 without hesitation, guided by the UI of the guide displaying the virtual thumb 2102 and the virtual index finger 2103.

The amount of virtual finger opening displayed when the user's hand approaches the virtual object is set based on the thickness of the virtual object to be grasped. By making the opening amount of the virtual finger wider than the opening degree of the actual finger by the thickness of the virtual object, when the actual finger is closed, the virtual finger is just at the position where the virtual object is sandwiched.

However, when trying to grasp a thick virtual object, if the opening amount of the virtual finger is widened by the thickness of the virtual object, the virtual finger will be spread unnaturally, which is impossible in reality. It ends up. Therefore, the opening amount of the virtual finger displayed when the user's hand approaches the virtual object does not necessarily have to match the thickness of the virtual object to be gripped. Spread the virtual finger slightly wider than the actual finger position, and change the virtual finger closing width slightly with respect to the actual finger closing width.

FIG. 22 shows another display example of the virtual finger when the user's finger approaches the virtual object. In FIG. 22, similar to the display example shown in FIG. 21, when the user brings his / her hand closer to the virtual object 2201 with the intention of sandwiching and grasping the two opposing surfaces of the virtual object 2101 between the thumb and the index finger. As a display example, the position 2202 of the virtual thumb and the position 2203 of the virtual index finger are shown by dotted lines, respectively. The actual thumb and index finger are drawn with solid lines. As can be seen from FIG. 22, the opening amount of the virtual thumb 2202 and the virtual index finger 2203 is wider than the opening degree of the actual thumb and index finger. However, the total value d1 + d2 of the difference d1 of the opening amount of the virtual thumb 2202 from the actual thumb and the difference d2 of the opening amount of the virtual index finger 2203 with the actual index finger is the thickness of the virtual object 2201 to be gripped. Less than d. That is, d> d1 + d2 is established.

The application execution unit 601 acquires the position of the user's hand based on the detection result of the finger position / posture detection unit 604, and when the shortest distance between the hand and the virtual object 2101 is equal to or less than a predetermined value, the user's hand Detects a state in which is approaching the virtual object 2101. Then, the application execution unit 601 displays the virtual thumb 2202 and the virtual index finger 2203 for each of the thumb approaching one surface and the index finger approaching the other surface of the virtual object 2001.

At that time, the application execution unit 601 makes the opening amount (d1 + d2) of the virtual thumb 2202 and the virtual index finger 2203 wider than the actual opening degree of the thumb and index finger, but smaller than the thickness d of the virtual object 2201. After that, when the actual thumb and index finger come into contact with each other, the virtual thumb 2202 and the virtual index finger 2203 come into contact with the virtual object 2101 in the virtual space displayed by the display unit 131, and are in a gripped state. The application execution unit 601 is a very natural and smooth operation in which the virtual thumb 2202 and the virtual index finger 2203 try to grasp the virtual object 2101 from the time when the user's hand enters the approaching state to the time when the user's hand shifts to the contacting state. Is displayed. At this time, the application execution unit 601 changes the closing width of the virtual thumb 2202 and the virtual index finger 2203 to be smaller than the closing width of the actual thumb and index finger. For example, when the actual opening amount of the thumb and the index finger is narrowed by 1 cm, the opening amount of the virtual thumb 2202 and the virtual index finger 2203 is narrowed by 0.2 cm, and the movement is performed by changing the magnification. This allows the virtual thumb 2202 and the virtual index finger 2203 to just fit when the real thumb and index finger are closed, without having to spread the virtual thumb 2202 and the virtual index finger 2203 unnaturally. It can be positioned so as to sandwich the virtual object 2201.

I. Processing procedure for displaying a virtual hand As described in Section H above, in the AR system 100 according to the present disclosure, when a user's finger touches a virtual object, the virtual hand is made wider than the actual finger opening degree. By displaying the finger, it is possible to give the user a tactile sensation of a virtual object by utilizing the visual-tactile interaction.

FIG. 23 shows a processing procedure for displaying a virtual hand to the user in the AR system 100 in the form of a flowchart. This processing procedure is mainly executed by, for example, the application execution unit 601 when the user's hand approaches the virtual object.

The application execution unit 601 acquires the width of the virtual object being displayed (step S2301).

In the present embodiment, since the application execution unit 601 creates a virtual object by itself, the application execution unit 601 can acquire the width of the virtual object based on the setting information at the time of generating the virtual object. In the case of virtual objects having different widths depending on the gripping position, the application execution unit 601 determines the direction in which the user's hand approaches the virtual object based on the detection result of the finger position / posture detection unit 604. , The width of the virtual object may be obtained based on the approaching direction.

Further, the application execution unit 601 acquires the actual finger width (the distance between the thumb and the index finger) used by the user to grasp the virtual object (step S2302).

The application execution unit 601 can acquire the actual finger width based on the detection result of the finger gesture detection unit 605. Further, the application execution unit 601 may acquire the actual finger width based on the addition recognition result of the captured image of the outward camera 121.

Next, the application execution unit 601 calculates the virtual finger width based on the current finger width (step S2303).

The application execution unit 601 calculates the width of the virtual finger so that the virtual finger is exactly at the position where the virtual object is sandwiched in the state where the actual finger used by the user to grasp the virtual object is in contact with the user. ..

Then, the application execution unit 601 displays the virtual finger corresponding to each actual finger in the vicinity of the actual finger used by the user to grasp the virtual object by using the display unit 131 (step S2304).

In the virtual space displayed by the display unit 131 (or seen through the AR glass), the user actually holds the virtual object in order to see the image in which the virtual finger is just sandwiching the virtual object. You can get a sense.

J. Virtual hand display timing In the above, the timing for starting the virtual finger display is the time when the user's hand approaches the virtual object. In the AR system 100, the display of the virtual finger may be started when the shortest distance between the user's hand and the virtual object falls within a predetermined value. As a predetermined value, for example, 50 cm may be set in advance.

K. Adjustment of gripping force of virtual object In this disclosure, the opening amount of the virtual finger is made wider than the opening degree of the actual finger, and when the actual finger touches, the virtual object is opened with the virtual finger in the virtual space. It is in a gripping state (see, for example, FIG. 20). At this time, the user recognizes the contact force between the fingers as a tactile sensation received from the virtual object. That is, according to the present disclosure, when there is an inconsistency between the movement of the body in the visual sense and the movement of the body felt by oneself, the visual information becomes predominant and a pseudo tactile sensation occurs. By utilizing the "visual-tactile interaction" that occurs, it is possible to give the user a tactile sensation when grasping the virtual object.

Furthermore, the user may adjust the magnitude of the force for grasping the virtual object by strongly closing the actual finger (strongly pressing the thumb and index finger against each other). For example, when the weight of a virtual object can be set, if the user simply picks a heavy virtual object with a weak gripping force, the virtual object slides off the finger, but when the user is picking or grasping with a strong gripping force. It is possible to realize the expression that a virtual object can be lifted.

L. Application to remote control For example, there is known a master-slave system in which an operator operates a controller on the master side to drive a robot at an output end on the slave side to perform remote work. Remote work has been introduced in various industries such as remote surgery and remote construction.

In the master-slave system, it is assumed that the operator performs operations such as picking or grasping an object that is not at hand with a remote robot. The remote control performed by the operator in the master-slave system is treated as the same as the operation in which the user picks or grabs the virtual object through the AR glass in the AR system 100, and the present disclosure can be applied. Therefore, since the master side displays the UI that guides the gripping method of the object in the remote place to the operator, the operator is guided by the guide of the gripping method and easily grips the object that is not at hand without hesitation. Can be done.

In the master-slave system, not a virtual object but a real object installed in a remote location is the target of grasping. In the AR system 100, the gripping method is set in advance for each virtual object, or the gripping method is selected based on the user attributes such as the user's personality, habit, age, gender, and physique, and the gripping method is guided according to a predetermined guide type. The UI to do is determined. On the other hand, in the master-slave system, an object to be gripped is detected in advance at a remote location, and a UI for guiding the gripping method and gripping method of the object is determined in advance based on the detection result.

Therefore, when the operator starts remote control on the master side, it is determined in a short time whether or not the object existing at the slave side is the object of gripping, and if it is the object of gripping, the operator is operated in a short time. Can be presented with a UI that guides how to grip the object. The operator can easily grasp an object that is not at hand by being guided by the guide of the grasping method without hesitation.

FIG. 24 shows a configuration example of the remote control system 2400 to which the present disclosure is applied. The illustrated remote control system 2400 includes a master device 2410 operated by an operator and a slave device 2420 including a robot 2421 to be remotely controlled.

The master device 2410 includes a controller 2411, a display unit 2412, a master control unit 2413, and a communication unit 2414.

The controller 2411 is used by the operator to input a command for remotely controlling the robot 2421 on the slave device 2420 side. In the present embodiment, the controller 2411 is used by being attached to the operator's hand as shown in FIG. 5, and is a device for inputting the position and orientation of the operator's fingers and the gestures of the fingers as operation commands for the robot 2421. I'm assuming. However, the controller 2411 may be a camera or the like that captures the operator's hand, and may recognize the position and orientation of the operator's fingers and the gesture of the fingers from the captured image of the hand.

The display unit 2412 is composed of, for example, AR glasses, but may be a display device such as a general liquid crystal display. The display unit 2412 displays the virtual object in the real space on which the operator's fingers are projected according to the control by the master control unit 2413. The virtual object referred to here is a virtual object corresponding to a remote real object that the remotely controlled robot 2421 is trying to grasp. The virtual object is displayed at a position where the relative position with respect to the operator's hand coincides with the relative position between the robot 2421 and the object. When the robot 2421 approaches the object on the slave device 2420 side according to the operation of the controller 2411 by the operator, a UI for guiding the method of grasping the virtual object is displayed near the virtual object (or near the operator's hand) on the display unit 2412. Is displayed.

When the master control unit 2413 acquires the position and orientation of the operator's fingers and the gestures of the fingers based on the input signal from the controller 2411, the master control unit 2413 converts the robot 2421 into an operation command for remote control via the communication unit 2414. The operation command is transmitted to the slave device 2420.

Further, the master control unit 2413 receives an image of the operation status of a remote object by the robot 2421 taken by the camera 2422 from the slave device 2420 via the communication unit 2414. Then, the master control unit 2413 controls the display unit 2412 so that the virtual object is displayed in the real space on which the operator's fingers are projected. The virtual object is a virtual object corresponding to a remote real object that the remotely controlled robot 2421 is trying to grasp. The virtual object is placed at a position where the relative position with respect to the operator's hand coincides with the relative position between the robot 2421 and the object.

Further, in the master control unit 2413, when the robot 2421 approaches the object on the slave device 2420 side according to the operation of the controller 2411 by the operator, the UI for guiding the method of grasping the virtual object is located near the virtual object (or near the operator's hand). indicate. The UI that guides the gripping method of the virtual object corresponds to the UI that guides the gripping method of the actual object by the robot 2421. Therefore, the operator is guided by the UI that guides the gripping method of the virtual object and performs the operation of gripping the virtual object with his / her own fingers, so that the robot 2421 grips the actual object on the slave device 2420 side. You can enter commands.

The communication unit 2414 is a functional module for interconnecting with the slave device 2420 side. The communication medium between the master device 2410 and the slave device 2420 may be either wired or wireless, and is not limited to a specific communication standard.

The slave device 2420 includes a robot 2421, a camera 2422, a slave control unit 2423, and a communication unit 2424. The slave device 2420 is interconnected with the master device 2410 side via the communication unit 2424, receives an operation command of the robot 2421 from the master device 2410, and transmits a captured image of the camera 2422 to the master device 2410. ..

The operation command sent from the master device 2410 is a command for driving the robot 2421 according to the position and orientation of the operator's fingers and the gesture of the fingers. The slave control unit 2423 interprets the operation command received from the master device 2420 and controls the drive of the robot 2421 so that the robot 2421 reproduces the position and orientation of the operator's fingers and the gestures of the fingers. FIG. 25 shows how the operator is approaching the virtual object with his / her hand on the master device 2410 side. FIG. 26 shows how the robot 2421 is approaching an object on the slave device 2420 side so as to follow the movement of the operator's hand.

The camera 2422 captures the operation status of the object by the robot 2421. The slave control unit 2423 encodes the captured image of the camera 2422 and controls the communication unit 2424 to transmit the captured image to the master device 2410 in a predetermined transmission format. As described above, on the master device 2410 side, the display unit 2412 displays the virtual object corresponding to the object in the real space on which the operator's fingers are projected. The virtual object is arranged at a position where the relative position with respect to the operator's hand coincides with the relative position between the robot 2421 and the object.

FIG. 27 shows a processing procedure in the form of a flowchart on the master device 2410 side for presenting a UI that guides the operator how to grasp the virtual object. This processing procedure is mainly carried out by the master control unit 2413.

First, the master control unit 2413 acquires the position of the operator's hand based on the detection result of the controller 2411 (step S2701).

Then, the master control unit 2413 checks whether the operator's hand approaches the virtual object, that is, whether the shortest distance between the operator's hand and the virtual object is equal to or less than a predetermined value (step S2702).

The virtual object is placed at a position where the relative position with respect to the operator's hand coincides with the relative position between the robot 2421 and the object. Therefore, when the operator's hand approaches the virtual object, the robot 2421 is approaching the object on the slave device 2420 side.

Then, when the shortest distance between the operator's hand and the virtual object becomes equal to or less than a predetermined value (Yes in step S2702), the master control unit 2413 provides the operator with a UI used to guide the method of grasping the virtual object. Determine (step S2703).

In step S2703, the master control unit 2413 selects the type of guide for grasping the virtual object. Further, the master control unit 2413 is based on a gripping method preset based on a category of a remote object corresponding to a virtual object, or a user attribute such as an operator's personality, habit, age, gender, and physique. The UI for guiding the selected gripping method according to the selected guide type is determined. Further, the master control unit 2413 may select the gripping method according to the direction in which the operator's hand approaches the virtual object, and determine the UI for guiding the gripping method.

Then, based on the result determined in step S2703, the master control unit 2413 displays a UI for guiding the gripping method of the virtual object approaching by the operator's hand near the virtual object using the display unit 2412 (the display unit 2412). Step S2704).

After that, the master control unit 2413 acquires the movement of the hand or finger from the controller 2411 when the operator is trying to grasp the virtual object (step S2705), and determines the contact state between the operator's finger and the virtual object.

The virtual object is placed at a position where the relative position with respect to the operator's hand coincides with the relative position between the robot 2421 and the object. Therefore, the contact state between the operator's hand and the virtual object is the same as the contact state between the robot 2421 and the real object on the slave device 2420 side.

Then, when the operator's finger touches the virtual object (Yes in step S2706), the master control unit 2413 switches the display of the UI that guides the gripping method of the virtual object, and touches the virtual object, in other words. The operator is notified or fed back that the robot 2421 has come into contact with the object (step S2707).

In step S2707, the master control unit 2413 uses the display unit 2412 to display, for example, a UI that highlights the contact point between the operator's finger and the virtual object. Further, when the operator's finger is sunk into the virtual object, the master control unit 2413 gradually switches the highlight display according to the degree of sunk.

The UI that guides the gripping method of the virtual object corresponds to the UI that guides the gripping method of the actual object by the robot 2421. Therefore, the operator remotely controls the robot 2421 on the slave device 2420 side by performing the operation of grasping the virtual object with his / her fingers guided by the UI that guides the grasping method of the virtual object, and the object that is not at hand. Can be easily grasped without hesitation.

The present disclosure has been described in detail with reference to the specific embodiment. However, it is self-evident that a person skilled in the art can modify or substitute the embodiment without departing from the gist of the present disclosure.

Although the present specification has mainly described embodiments in which the present disclosure is applied to an AR system, the gist of the present disclosure is not limited to this. For example, the present disclosure can be similarly applied to a VR system that perceives a virtual space as reality, an MR system that intersects reality and virtual, and a master-slave remote system.

In short, the present disclosure has been described in the form of an example, and the contents of the present specification should not be interpreted in a limited manner. In order to judge the gist of this disclosure, the scope of claims should be taken into consideration.

Note that this disclosure can also have the following structure.

(1) An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Information processing device.

(2) The acquisition unit acquires the position and posture of the hand based on the sensor information from the sensor attached to the hand, or includes a sensor attached to the hand.
The information processing device according to (1) above.

(3) The control unit controls the display device so that the information is displayed on either the virtual object or the vicinity of the hand.
The information processing device according to any one of (1) and (2) above.

(4) The control unit displays the display device so as to display the information including at least one of a gripping method of picking the virtual object with the thumb and one other finger or a gripping method of grasping the virtual object with the whole hand. Control,
The information processing device according to any one of (1) to (3) above.

(5) The control unit is in a state where the hand is holding the virtual object, a position where the hand is holding the virtual object, and a movement in which the virtual hand is holding the virtual object at the position of the hand. Control the display device to display the information indicating at least one.
The information processing device according to any one of (1) to (4) above.

(6) The acquisition unit further acquires the shape of the hand and obtains the shape of the hand.
The control unit selects the information based on the shape of the hand.
The information processing device according to any one of (1) to (5) above.

(7) The control unit selects the information based on the direction in which the hand approaches the virtual object.
The information processing device according to any one of (1) to (5) above.

(8) The control unit selects the information based on the attributes of the user.
The information processing device according to any one of (1) to (5) above.

(8-1) The user's attributes include at least one of age, race, physical injury, and daily gripping method.
The information processing device according to (8) above.

(9) The control unit controls the display of the information based on the state of the user when the hand approaches the virtual object.
The information processing device according to any one of (1) to (8) above.

(9-1) The state of the user includes at least one of the line-of-sight direction of the user or the degree of interest of the user in the virtual object.
The information processing device according to (9) above.

(10) The control unit controls the display of the information based on the contact state between the hand and the virtual object.
The information processing device according to any one of (1) to (9) above.

(11) The control unit controls the display of the information so that the contact point indicates that the hand and the virtual object have come into contact with each other.
The information processing device according to (10) above.

(12) The control unit controls the display of the information so as to indicate to the contact point that the hand has sunk into the virtual object.
The information processing device according to any one of (10) and (11) above.

(13) The control unit controls the display device so as to display the virtual hand when the hand touches the virtual object.
The information processing device according to any one of (1) to (12) above.

(14) When the user is trying to grasp the two opposing surfaces of the virtual object, the control unit has a first finger approaching one surface and a second finger approaching the other surface. The display device is controlled to display the first virtual finger and the second virtual finger for each of the fingers.
The information processing device according to (13) above.

(15) The control unit makes the opening amount of the first virtual finger and the second virtual finger wider than the actual opening degree of the first finger and the second finger, and actually The display operation of the display device so that when the first finger and the second finger come into contact with each other, the first virtual finger and the second virtual finger come into contact with the virtual object and are in a gripped state. To control,
The information processing device according to (14) above.

(16) The control unit displays the virtual hand in which the opening amount of the virtual finger trying to grasp the virtual object is wider than the actual opening degree of the finger when the hand approaches the virtual object. Control to start,
The information processing device according to any one of (13) to (15) above.

(17) The control unit controls the opening amount of the virtual finger to be wider than the actual opening degree of the finger based on the thickness of the virtual object when the hand approaches the virtual object. do,
The information processing device according to (16) above.

(18) An acquisition step for acquiring the position and posture of the user's hand, and
A control step that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Have,
In the control step, the display device is controlled so as to display information on a method of grasping the virtual object when the hand approaches the virtual object.
Information processing method.

(19) Acquisition unit for acquiring the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space.
Written in computer readable format so that the computer works as
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Computer program.

(20) A display device that superimposes and displays virtual objects in real space,
An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of the display device,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Augmented reality system.

100 ... AR system, 110 ... first sensor 111 ... gyro sensor, 112 ... acceleration sensor, 113 ... orientation sensor 120 ... second sensor unit, 121 ... outward camera 122 ... inward camera, 123 ... microphone, 124 ... Gyro sensor 125 ... Accelerometer, 126 ... Orientation sensor 131 ... Display unit, 132 ... Speaker 133 ... Vibration presentation unit 134 ... Communication unit, 140 ... Control unit, 150 ... Storage unit 300 ... AR system, 301 ... AR glass, 302 … Sensor 400… AR system, 401… AR glass, 402… Controller 403… Information terminal 500… Controller, 501, 502, 503… IMU
511, 512, 513 ... Band 601 ... Application execution unit, 602 ... Head position posture detection unit 603 ... Output control unit, 604 ... Finger position posture detection unit 605 ... Finger gesture detection unit 2400 ... Remote control system, 2410 ... Master device 2411 ... controller, 2412 ... display unit, 2413 ... master control unit 2414 ... communication unit, 2420 ... slave device, 2421 ... robot 2422 ... camera, 2423 ... slave control unit, 2424 ... communication unit

Claims

An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Information processing device.
The acquisition unit acquires the position and posture of the hand based on the sensor information from the sensor attached to the hand, or includes a sensor attached to the hand.
The information processing device according to claim 1.
The control unit controls the display device so that the information is displayed on either the virtual object or the vicinity of the hand.
The information processing device according to claim 1.
The control unit controls the display device to display the information including at least one of a gripping method of picking the virtual object with the thumb and one other finger or a gripping method of gripping the virtual object with the whole hand.
The information processing device according to claim 1.
The control unit is at least one of a state in which the hand is holding the virtual object, a position where the hand is holding the virtual object, and a movement in which the virtual hand is holding the virtual object at the position of the hand. Control the display device to display the information indicating
The information processing device according to claim 1.
The acquisition unit further acquires the shape of the hand and
The control unit selects the information based on the shape of the hand.
The information processing device according to claim 1.
The control unit selects the information based on the direction in which the hand approaches the virtual object.
The information processing device according to claim 1.
The control unit selects the information based on the attributes of the user.
The information processing device according to claim 1.
The control unit controls the display of the information based on the state of the user when the hand approaches the virtual object.
The information processing device according to claim 1.
The control unit controls the display of the information based on the contact state between the hand and the virtual object.
The information processing device according to claim 1.
The control unit controls the display of the information so that the contact point indicates that the hand and the virtual object have come into contact with each other.
The information processing device according to claim 10.
The control unit controls the display of the information so as to indicate to the contact point that the hand has sunk into the virtual object.
The information processing device according to claim 10.
The control unit controls the display device so as to display the virtual hand when the hand touches the virtual object.
The information processing device according to claim 1.
When the user is trying to grasp the two opposing surfaces of the virtual object, the control unit has a first finger approaching one surface and a second finger approaching the other surface, respectively. The display device is controlled so as to display the first virtual finger and the second virtual finger.
The information processing device according to claim 13.
The control unit makes the opening amount of the first virtual finger and the second virtual finger wider than the actual opening degree of the first finger and the second finger, so that the actual first finger is opened. The display operation of the display device is controlled so that when the finger and the second finger come into contact with each other, the first virtual finger and the second virtual finger come into contact with the virtual object and are in a gripped state. ,
The information processing device according to claim 14.
When the hand approaches the virtual object, the control unit starts displaying the virtual hand in which the opening amount of the virtual finger trying to grasp the virtual object is wider than the actual opening degree of the finger. To control,
The information processing device according to claim 13.
The control unit controls the opening amount of the virtual finger to be wider than the actual opening degree of the finger based on the thickness of the virtual object when the hand approaches the virtual object.
The information processing device according to claim 16.
The acquisition step to acquire the position and posture of the user's hand,
A control step that controls the display operation of a display device that superimposes and displays virtual objects in real space,
Have,
In the control step, the display device is controlled so as to display information on a method of grasping the virtual object when the hand approaches the virtual object.
Information processing method.
Acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of a display device that superimposes and displays virtual objects in real space.
Written in computer readable format so that the computer works as
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Computer program.
A display device that superimposes and displays virtual objects in real space,
An acquisition unit that acquires the position and posture of the user's hand,
A control unit that controls the display operation of the display device,
Equipped with
The control unit controls the display device so as to display information on how to hold the virtual object when the hand approaches the virtual object.
Augmented reality system.