WO2020195292A1 - 感覚器オブジェクトを表示する情報処理装置 - Google Patents

感覚器オブジェクトを表示する情報処理装置 Download PDF

Info

Publication number
WO2020195292A1
WO2020195292A1 PCT/JP2020/005471 JP2020005471W WO2020195292A1 WO 2020195292 A1 WO2020195292 A1 WO 2020195292A1 JP 2020005471 W JP2020005471 W JP 2020005471W WO 2020195292 A1 WO2020195292 A1 WO 2020195292A1
Authority
WO
WIPO (PCT)
Prior art keywords
information processing
user
virtual object
distance
processing device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/005471
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
友久 田中
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Priority to JP2021508231A priority Critical patent/JPWO2020195292A1/ja
Priority to US17/435,556 priority patent/US20220049947A1/en
Publication of WO2020195292A1 publication Critical patent/WO2020195292A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G01B11/026—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness by measuring distance between sensor and object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G01B11/022—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness by means of tv-camera scanning
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/37—Details of the operation on graphic patterns
    • G09G5/377—Details of the operation on graphic patterns for mixing or overlaying two or more graphic patterns
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
    • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0141—Head-up displays characterised by optical features characterised by the informative content of the display
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Definitions

  • This disclosure relates to an information processing device, an information processing method, and a recording medium. More specifically, the present invention relates to an output signal control process according to a user's operation.
  • AR Augmented Reality
  • VR Virtual Reality
  • MR Mated Reality
  • object composition there is a technology that can easily convey whether or not a subject exists within an appropriate range by acquiring depth information of the subject included in the captured image and executing effect processing. Has been done. Further, there is known a technique capable of recognizing the hand of a user wearing a head-mounted display (HMD, Head Mounted Display) or the like with high accuracy.
  • HMD Head Mounted Display
  • the user may be required to have some kind of interaction such as touching a virtual object superimposed on the real space.
  • the virtual image distance of a display generally tends to be fixed at a constant distance. Therefore, even when the stereoscopic display is performed by changing the display positions of the right eye image and the left eye image, the virtual image distance of the display does not change. For this reason, there may be a contradiction between the display mode of the virtual object and the characteristics of human vision. Such a problem is commonly known as a congestion control contradiction. This congestion adjustment contradiction makes it difficult for the user to properly recognize the sense of distance to the virtual object displayed at a short distance or a long distance. For example, the user may try to touch the virtual object but cannot reach it, or conversely, the user may reach deeper than the virtual object.
  • the present disclosure proposes an information processing device, an information processing method, and a recording medium that can improve the user's spatial recognition in a technique of superimposing a virtual object on a real space.
  • the information processing apparatus of one form according to the present disclosure is a distance between a real object operated by a user in the real space and a virtual object superimposed on the real space in the display unit.
  • the acquisition unit that acquires the change of the real object based on the detection result of the sensor that detects the position of the real object, and the sensory organ object that represents the sensory organ of the virtual object for recognizing the real space are displayed on the display unit.
  • it includes an output control unit that continuously changes a predetermined region of the sensory organ object according to a change in the distance acquired by the acquisition unit.
  • the information processing device According to the information processing device, the information processing method, and the recording medium according to the present disclosure, it is possible to improve the user's spatial recognition in the technique of superimposing virtual objects on the real space.
  • the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.
  • FIG. 2 is a second diagram showing an outline of information processing according to the first embodiment of the present disclosure. It is a 3rd figure which shows the outline of the information processing which concerns on 1st Embodiment of this disclosure.
  • FIG. 4 is a fourth diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • FIG. 5 is a fifth diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • FIG. 6 is a sixth diagram showing an outline of information processing according to the first embodiment of the present disclosure. It is a figure for demonstrating the output control process which concerns on 1st Embodiment of this disclosure.
  • FIG. 1 is a first diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the information processing according to the first embodiment of the present disclosure is executed by the information processing apparatus 100 shown in FIG.
  • the information processing device 100 is an information processing terminal for realizing so-called AR technology and the like.
  • the information processing device 100 is a wearable display that is worn and used on the head of the user U01.
  • the information processing device 100 in the present disclosure may be more specifically referred to as an HMD, an AR glass, or the like.
  • the information processing device 100 has a display unit 61 which is a transmissive display.
  • the information processing apparatus 100 superimposes on the real space and displays the superimposed object represented by CG (Computer Graphics) or the like on the display unit 61.
  • the information processing apparatus 100 displays the virtual object V01 as a superposed object.
  • the display FV 11 imitates the information displayed on the display unit 61 (that is, the information visually recognized by the user U01).
  • the user U01 can simultaneously visually recognize a real object in addition to the display FV11 via the display unit 61.
  • the information processing device 100 may have a configuration for outputting a predetermined output signal in addition to the display unit 61.
  • the information processing device 100 may have a speaker or the like for outputting sound.
  • the virtual object V01 is arranged with reference to the global coordinate system associated with the real space based on the detection result of the sensor 20 described later.
  • the user U01 information processing device 100
  • the virtual object V01 is fixed at the first coordinates (x1, y1, z1).
  • the information processing apparatus 100 recognizes that the virtual object V01 still exists in the first coordinates (x1, y1, z1) so that the user recognizes it.
  • At least one of the position, orientation, and size of the virtual object V01 on the display unit 61 is changed.
  • the user U01 can perform an interaction such as touching the virtual object V01 or picking up the virtual object V01 by using an arbitrary input means in the real space.
  • the arbitrary input means is an object operated by the user and is an object that the information processing apparatus 100 can recognize in space.
  • any input means is a part of the body such as a user's hand or foot, a controller held by the user, or the like.
  • the user U01 uses his / her hand H01 (see FIG. 2 and below) as the input means.
  • the fact that the hand H01 touches the virtual object V01 means that, for example, the hand H01 exists in a predetermined coordinate space recognized by the information processing apparatus 100 as the user U01 touching the virtual object V01.
  • the user U01 can visually recognize the real space that is visually recognized through the display unit 61 and the virtual object V01 that is superimposed on the real space. Then, the user U01 uses the hand H01 to execute an interaction that touches the virtual object V01.
  • the virtual image distance of the display is generally fixed at a constant value. Therefore, for example, when the virtual image distance is fixed at 3 m, when the virtual object V01 is displayed within a few tens of centimeters within reach from the user U01, the virtual object V01 having a virtual image distance of 3 m is several tens of centimeters. There is a contradiction of fusion to the distance of. This contradiction is commonly known in AR technology as a congestion control contradiction.
  • the user U01 may not reach the virtual object V01 even if he / she thinks he / she has touched it, or conversely, he / she may put out the hand H01 deeper than the virtual object V01. Further, it is difficult for the user U01 to determine where to move the hand H01 to recognize the interaction when the interaction with the virtual object V01 is not recognized by the AR device, and it is difficult to correct the position.
  • OST display Optical See-Through Display
  • the user can use a virtual object (V01) having a virtual image distance of 3 m and a fusion distance of several tens of cm, and a real object (FIG. 1) having a virtual image distance of several tens of cm and a fusion distance.
  • V01 virtual object
  • FOG. 1 real object
  • the hand H01 must be visually recognized at the same time. Therefore, if a display having a fixed virtual image distance is used, the user U01 cannot focus on the virtual object V01 and the hand H01 at the same time when the hand H01 directly interacts with the virtual object V01.
  • VST display Video See-Through Display
  • the real object is replaced as a display object and has the same virtual image distance of 3 m as the virtual object. That is, when the OST display is used, the user U01 becomes more difficult to recognize the position of the virtual object V01 in the depth direction as compared with the case where the VST display is used.
  • the information processing device 100 executes the information processing described below in order to improve the recognition of the space in the AR technology.
  • the information processing device 100 includes a real object (hand H01 in the example of FIG. 1) operated by the user U01 in the real space and a virtual object (virtual object in the example of FIG. 1) displayed on the display unit 61. Acquires the change in distance from the object V01). The change in distance is determined by the acquisition unit 32 based on the position of the real object detected by the sensor 20 described later.
  • the information processing device 100 further displays a sensory organ object representing the sensory organ of the virtual object V01.
  • the indicator E01 imitating a human eyeball may be regarded as corresponding to a sensory organ object.
  • the sensory organ object has a predetermined region that changes continuously in response to the above-mentioned change in distance.
  • the black eye EC01 which is a black eye portion, may be regarded as corresponding to a predetermined region.
  • a predetermined area that continuously changes according to the above-mentioned change in distance may be referred to as a second display area, and an area displayed adjacent to the outside of the second display area may be referred to as a first display area. is there.
  • the second display area is narrower than the first display area, but the area of the predetermined area is not limited to this.
  • the information processing device 100 causes the user U01 to recognize the approach of the hand H01 of the user U01 by changing the display mode of the black eye EC01 corresponding to the pupil in the indicator E01.
  • the indicator E01 reduces the area of the black eye EC01 so as to adjust the focus according to the approach of the real object in order to reproduce the natural behavior of the living thing.
  • the information processing device 100 in the present disclosure can solve at least a part of the problem of congestion adjustment contradiction in the AR technology, and can improve the spatial recognition of the user U01.
  • the outline of the information processing according to the present disclosure will be described along the flow with reference to FIGS. 1 to 7.
  • the information processing apparatus 100 displays the indicator E01 on the surface of the virtual object V01 (more specifically, on the spatial coordinates set as the surface of the virtual object V01).
  • the indicator E01 is composed of a pair of white eye EP01 and black eye EC01, and is displayed so that the black eye EC01 is superimposed on the white eye EP01.
  • the display FV11 the user U01 visually recognizes that the indicator E01 is superimposed and displayed on the virtual object V01.
  • the information processing device 100 performs display control processing that imitates a situation in which the virtual object V01 is "looking at the user U01".
  • the white-eyed EP01 and the black-eyed EC01 may be displayed in association with the virtual object V01.
  • the black eye EC01 may be provided in the white eye EP01 so as to be included in the surface of the virtual object V01, and may form a part of the virtual object V01.
  • the indicators of the present disclosure are not limited to this, and various display forms may be adopted.
  • the virtual object V01 is in the global coordinate system as seen from the user U01 based on the position information of the information processing device 100 itself (in other words, the position information of the head of the user U01).
  • the position, orientation, and size of the virtual object V01 are controlled on the display unit 61 so as to be recognized at a predetermined position.
  • SLAM simultaneous localization and mapping
  • the information processing apparatus 100 recognizes the hand H01 of the user U01 based on a recognition technique different from the self-position estimation technique described above, for example, an image recognition technique.
  • the information processing device 100 can recognize the position and posture of the user U01, it may not be able to recognize the position and posture of the hand H01.
  • the information processing device 100 controls the black eye EC01 so as to face the head of the user U01, while ignoring the movement of the hand H01 that is not properly detected by the sensor 20. That is, the display of the indicator E01 and the black eye EC01 does not change with respect to the movement of the hand H01. Details of such processing will be described later.
  • the information processing device 100 does not display the indicator E01 instead of looking at the head of the user U01, or displays the white-eyed EP01 and the black-eyed EC01 as concentric circles. Display processing may be performed.
  • FIG. 2 is a second diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the user U01 executes an interaction in which the virtual object V01 superimposed on the real space is touched by the hand H01.
  • the information processing device 100 acquires the position of the hand H01 raised by the user U01 in space.
  • the information processing device 100 recognizes the hand H01 existing in the real space that the user U01 sees through the display unit 61 by using a sensor such as a recognition camera that covers the line-of-sight direction of the user U01. Then, the position of the hand H01 is acquired. Further, the information processing apparatus 100 sets an arbitrary coordinate HP01 used when measuring the distance between the hand H01 and the virtual object V01. Further, the information processing device 100 acquires the position of the virtual object V01 superimposed on the real space by recognizing the real space displayed in the display unit 61 as the coordinate space. Then, the information processing device 100 acquires the distance between the user's hand H01 and the virtual object V01.
  • FIG. 3 is a third diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the relationship between the user's hand H01, the distance L acquired by the acquisition unit 32, and the virtual object V01 is schematically shown.
  • the information processing device 100 When the information processing device 100 recognizes the hand H01, the information processing device 100 sets an arbitrary coordinate HP01 included in the recognized hand H01. For example, the coordinate HP01 is set at the substantially center of the recognized hand H01. Alternatively, in the recognized hand H01 area, the portion of the hand H01 closest to the virtual object V01 may be set in the coordinates HP01. The update frequency of the coordinate HP01 may be set to be lower than the detection frequency of the signal value of the sensor 20 so that the fluctuation of the signal value of the sensor 20 is absorbed. Further, the information processing apparatus 100 sets the coordinates in the virtual object V01 that are recognized as having been touched by the user's hand.
  • the information processing apparatus 100 sets not only the coordinates of only one point but also a plurality of coordinates in order to have a certain degree of spatial expanse. This is because it is difficult for the user U01 to accurately touch the coordinates of one point in the virtual object V01 by hand, so it is easy to set a certain spatial range and allow the user U01 to "touch" the virtual object V01 to some extent. To make it.
  • the information processing device 100 has a distance between the coordinate HP01 and an arbitrary coordinate set in the virtual object V01 (it may be any specific coordinate, or it may be the center point, the center of gravity, or the like of a plurality of coordinates). Get L.
  • FIG. 4 is a fourth diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the indicator E01 is composed of two overlapping display areas, a white eye EP01 and a black eye EC01.
  • the white eye EP01 has a lighter color than the black eye EC01, has a wider area than the black eye EC01 so as to include the black eye EC01, and has a translucent aspect.
  • the black eye EC01 has a darker color than the white eye EP01 and has a narrower region than the white eye EP01.
  • the black eye EC01 is, for example, a sphere having a radius half that of the white eye EP01.
  • the pupil of the indicator E01 is represented as a black eye EC01, but the present disclosure is not limited to this.
  • the color of the predetermined region of the indicator E01 corresponding to the pupil does not have to be black, and it may be reproduced with various shapes and colors that the organism can have.
  • the indicator E01 may reproduce the eyeball of a generally recognized virtual character instead of the eyeball of an actual creature.
  • the point C01 is the center of the white eye EP01.
  • the point C02 is a point where the white eye EP01 and the black eye EC01 meet.
  • the direction connecting the points C01 and C02 and moving from the point C01 to the point C02 that is, the direction in which the indicator E01 "sees" the user U01.
  • the direction from the point C01 to the point C02 is the direction indicated by the eyeball-shaped indicator E01.
  • a straight line connecting C01 and the coordinate HP01 is set as the optical axis of the eyeball-shaped indicator E01 so that the optical axis passes through the substantially center of the black eye EC01 and the plane represented by the black eye EC01 is substantially vertical.
  • the display of the indicator E01 may be controlled.
  • the information processing device 100 controls the hand H01 of the user U01 as if the indicator E01 is looking at it by changing the display mode of the black eye EC01.
  • the user U01 visually recognizes the indicator E01 and determines whether his / her hand H01 is recognized by the information processing apparatus 100, or whether his / her hand H01 is appropriately directed in the direction of the virtual object V01.
  • FIG. 5 is a fifth diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the user U01 raises the hand H01 within the range of the angle of view FH01 that the display unit 61 can display.
  • the information processing device 100 recognizes the user's hand H01. Further, the information processing apparatus 100 acquires the direction of the coordinate HP01 on the hand H01 and the point C01 which is the center of the indicator E01. Then, the information processing device 100 moves the black eye EC01 in the direction in which the hand H01 approaches the virtual object V01. Further, the information processing device 100 acquires the distance L between the hand H01 and the virtual object V01. Then, the information processing device 100 changes the size of the black eye EC01 based on the distance L.
  • the user U01 can confirm an image in which the indicator E01 is visually recognizing the hand H01 extended to the virtual object V01. As a result, the user U01 can grasp that his / her hand H01 is recognized and from what direction the hand H01 is approaching the virtual object V01.
  • FIG. 6 is a sixth diagram showing an outline of information processing according to the first embodiment of the present disclosure.
  • the user U01 brings the hand H01 closer to the virtual object V01 as compared with the situation of FIG.
  • the user U01 brings the hand H01 close to the range where the distance between the virtual object V01 and the hand H01 is less than 50 cm.
  • the information processing apparatus 100 continuously changes the size of the black eye EC01 based on the change in the distance L between the point C01 and the coordinate HP01. Specifically, the information processing apparatus 100 changes the radius of the black eye EC01 so that the smaller the value of the distance L, the larger the black eye EC01.
  • the user U01 can visually recognize that the black eye EC01 is displayed larger than that of FIG. Therefore, the user U01 can determine that the hand H01 is closer to the virtual object V01. Further, due to the change in the display mode, the user U01 has an impression that the indicator E01 has his eyes wide open, so that it can be more intuitively determined that the hand H01 is approaching the virtual object V01.
  • FIG. 7 is a diagram for explaining an output control process according to the first embodiment of the present disclosure.
  • the graph shown in FIG. 7 shows the relationship between the distance L between the point C01 and the coordinates HP01 and the size of the black eye EC01.
  • the size (radius) of the black eye EC01 is obtained by multiplying the "radius of the white eye EP01" by the "coefficient m", for example.
  • the information processing apparatus 100 continuously displays the display so that the radius of the black eye EC01 gradually increases (coefficient m> 0.5) in inverse proportion to the distance L. Change.
  • the information processing apparatus 100 can perform an effective effect as if the eyes were wide open by changing the display mode of the black eye EC01 as shown in the graph shown in FIG.
  • the change in the numerical value shown in FIG. 7 is an example, and if the change shown in FIG. 6 can be given to the display mode of the black eye EC01, the setting of the coefficient m and the radius of the black eye EC01 are shown in FIG. It is not limited to the example shown in 7.
  • the information processing apparatus 100 includes a real object (for example, hand H01) operated by the user U01 in the real space and a virtual object (for example, virtual object V01) superimposed on the real space on the display unit 61. Gets the change in distance L between and. Then, the information processing apparatus 100 has a first display area (for example, white-eyed EP01) superimposed on the virtual object and displayed, and a second display area (for example, black-eyed EC01) superimposed on the first display area. The display is displayed on the display unit 61, and the display mode of the second display area is continuously changed according to the change of the acquired distance L.
  • a real object for example, hand H01
  • a virtual object for example, virtual object V01
  • the information processing apparatus 100 has a first display area (for example, white-eyed EP01) superimposed on the virtual object and displayed, and a second display area (for example, black-eyed EC01) superimposed on the first display area.
  • the display is displayed on the display unit 61
  • the information processing device 100 superimposes and displays an eyeball-shaped indicator E01 in which a white eye EP01 and a black eye EC01 are paired on a virtual object V01, and changes the display mode to change the direction in which the hand H01 is headed. It recognizes the approach of the hand H01 to the virtual object V01. As a result, the information processing apparatus 100 can improve the recognizability of the user U01 with respect to the virtual object V01 superimposed on the real space, which is difficult for the user U01 to recognize in AR technology or the like. It is empirically known that the display imitating the eyeball has higher human cognitive ability than other displays.
  • the user U01 can more intuitively grasp the movement of the hand H01 more intuitively than the inorganic indicator simply indicating the distance and the direction, and does not involve a large load. It is possible to grasp the movement of the hand H01. That is, the information processing device 100 can improve usability in a technique using an optical system such as AR.
  • FIG. 8 is a diagram showing the appearance of the information processing device 100 according to the first embodiment of the present disclosure.
  • the information processing device 100 includes a sensor 20, a display unit 61, and a holding unit 70.
  • the holding portion 70 has a configuration corresponding to a spectacle frame. Further, the display unit 61 has a configuration corresponding to a spectacle lens. The holding unit 70 holds the display unit 61 so that the display unit 61 is located in front of the user's eyes when the information processing device 100 is attached to the user.
  • the sensor 20 is a sensor that detects various environmental information.
  • the sensor 20 has a function as a recognition camera for recognizing the space in front of the user's eyes.
  • the sensor 20 may be a so-called stereo camera provided in each of the display units 61.
  • the sensor 20 is held by the holding unit 70 so that the user's head faces the direction (that is, the front of the user). Based on this configuration, the sensor 20 recognizes a subject (that is, a real object located in the real space) located in front of the information processing device 100. Further, the sensor 20 acquires an image of the subject located in front of the user, and based on the parallax between the images captured by the stereo camera, from the information processing device 100 (in other words, the position of the user's viewpoint) to the subject. It becomes possible to calculate the distance of.
  • the configuration and method are not particularly limited as long as the distance between the information processing device 100 and the subject can be measured.
  • the distance between the information processing device 100 and the subject may be measured based on a method such as multi-camera stereo, moving parallax, TOF (Time Of Flight), Structured Light, or the like.
  • TOF is the distance (depth) to the subject based on the measurement result by projecting light such as infrared rays onto the subject and measuring the time until the posted light is reflected by the subject and returned for each pixel. ) Is a method of obtaining an image (so-called distance image).
  • Structured Light is a distance image that includes the distance (depth) to the subject based on the change in the pattern obtained from the imaging result by irradiating the subject with a pattern with light such as infrared rays and imaging it. Is a method of obtaining.
  • the moving parallax is a method of measuring the distance to the subject based on the parallax even in a so-called monocular camera. Specifically, by moving the camera, the subjects are imaged from different viewpoints, and the distance to the subject is measured based on the parallax between the captured images. At this time, by recognizing the moving distance and the moving direction of the camera by various sensors, it is possible to measure the distance to the subject with higher accuracy.
  • the method of the sensor 20 (for example, a monocular camera, a stereo camera, etc.) may be changed as appropriate depending on the distance measurement method.
  • the sensor 20 may detect not only the information in front of the user but also the information of the user himself / herself.
  • the sensor 20 is held by the holding unit 70 so that the user's eyeball is positioned within the imaging range when the information processing device 100 is attached to the user's head. Then, the sensor 20 recognizes the direction in which the line of sight of the right eye is directed based on the image of the eyeball of the user's right eye captured and the positional relationship between the right eye and the right eye. Similarly, the sensor 20 recognizes the direction in which the line of sight of the left eye is directed based on the image of the eyeball of the user's left eye captured and the positional relationship between the left eye and the left eye.
  • the sensor 20 may have a function of detecting various information related to the user's movement such as the orientation, inclination, movement and moving speed of the user's body. Specifically, the sensor 20 detects information on the user's head and posture, movements of the user's head and body (acceleration and angular velocity), visual field direction, viewpoint movement speed, and the like as information on the user's movement. To do.
  • the sensor 20 functions as various motion sensors such as a 3-axis acceleration sensor, a gyro sensor, and a speed sensor, and detects information related to the user's movement.
  • the sensor 20 detects components in the yaw direction, the pitch direction, and the roll direction as the movement of the user's head, thereby detecting the components of the user's head. Detects changes in at least one of position and orientation.
  • the sensor 20 does not necessarily have to be provided in the information processing device 100, and may be, for example, an external sensor connected to the information processing device 100 by wire or wirelessly.
  • the information processing apparatus 100 may have an operation unit that accepts input from the user.
  • the operation unit is composed of input devices such as touch panels and buttons.
  • the operation unit may be held at a position corresponding to the temple of the glasses.
  • the information processing device 100 may be provided with an output unit (speaker or the like) for outputting a signal such as voice in appearance.
  • the information processing device 100 includes a control unit 30 (see FIG. 9) and the like that execute information processing according to the present disclosure.
  • the information processing device 100 recognizes a change in the position and posture of the user in the real space according to the movement of the user's head. Further, the information processing apparatus 100 uses the so-called AR technology based on the recognized information so that the virtual content (that is, the virtual object) is superimposed on the real object located in the real space. The content is displayed on 61.
  • the information processing device 100 may estimate the position and orientation of its own device in the real space based on, for example, SLAM technology, or may use the estimation result for the display processing of the virtual object.
  • SLAM is a technology that performs self-position estimation and environment map creation in parallel by using an imaging unit such as a camera, various sensors, an encoder, and the like.
  • an imaging unit such as a camera, various sensors, an encoder, and the like.
  • the three-dimensional shape of the captured scene (or subject) is sequentially restored based on the captured moving image. Then, by associating the restored result of the captured scene with the detection result of the position and orientation of the imaging unit, a map of the surrounding environment can be created and the imaging unit in the environment (sensor 20 in the example of FIG. 8, in other words, information processing).
  • the position and orientation of the device 100) are estimated.
  • the position and orientation of the information processing device 100 As described above, various information is detected by using various sensor functions such as an acceleration sensor and an angular velocity sensor of the sensor 20, and relative changes are made based on the detection results. It is possible to estimate as information indicating. If the position and orientation of the information processing device 100 can be estimated, the method is not necessarily limited to the method based on the detection results of various sensors such as an acceleration sensor and an angular velocity sensor.
  • Examples of a head-mounted display (HMD) applicable as the information processing device 100 include an optical see-through type HMD, a video see-through type HMD, and a retinal projection type HMD.
  • the see-through type HMD uses, for example, a half mirror or a transparent light guide plate to hold a virtual image optical system composed of a transparent light guide portion or the like in front of the user's eyes and display an image inside the virtual image optical system. Therefore, the user wearing the see-through type HMD can see the outside scenery while viewing the image displayed inside the virtual image optical system.
  • the see-through type HMD is based on, for example, AR technology, and is a virtual object with respect to an optical image of the real object located in the real space according to the recognition result of at least one of the position and the posture of the see-through type HMD. Images can be superimposed.
  • the see-through type HMD there is a so-called glasses-type wearable device in which a portion corresponding to a lens of glasses is configured as a virtual image optical system.
  • the information processing device 100 shown in FIG. 8 corresponds to an example of a see-through type HMD.
  • the video see-through type HMD when the video see-through type HMD is worn on the user's head or face, it is worn so as to cover the user's eyes, and a display unit such as a display is held in front of the user's eyes. Further, the video see-through type HMD has an imaging unit for imaging the surrounding landscape, and displays an image of the landscape in front of the user captured by the imaging unit on the display unit. With such a configuration, it is difficult for the user wearing the video see-through type HMD to directly see the external scenery, but the external scenery can be confirmed by the image displayed on the display unit. Further, the video see-through type HMD may superimpose a virtual object on an image of an external landscape according to the recognition result of at least one of the position and orientation of the video see-through type HMD based on, for example, AR technology. ..
  • the projection unit In the retinal projection type HMD, the projection unit is held in front of the user's eyes, and the image is projected from the projection unit toward the user's eyes so that the image is superimposed on the external landscape.
  • an image is directly projected from the projection unit onto the retina of the user's eye, and the image is imaged on the retina. With such a configuration, even a user with myopia or hyperopia can view a clearer image.
  • the user wearing the retinal projection type HMD can see the external landscape in the field of view while viewing the image projected from the projection unit.
  • the retinal projection type HMD is virtual with respect to the optical image of the real object located in the real space according to the recognition result of at least one of the position and the posture of the retinal projection type HMD based on, for example, AR technology.
  • the image of the object can be superimposed.
  • the information processing device 100 may be configured as an HMD called an immersive HMD.
  • the immersive HMD is worn so as to cover the user's eyes, and a display unit such as a display is held in front of the user's eyes. Therefore, it is difficult for the user wearing the immersive HMD to directly see the external landscape (that is, the real space), and only the image displayed on the display unit is in the field of view.
  • the immersive HMD controls to display both the captured real space and the superimposed virtual object on the display unit. That is, in the immersive HMD, the virtual object is not superimposed on the transparent real space, but the virtual object is superimposed on the captured real space, and both the real space and the virtual object are displayed on the display.
  • the information processing according to the present disclosure can be realized even with such a configuration.
  • the information processing system 1 includes an information processing device 100.
  • FIG. 9 is a diagram showing a configuration example of the information processing device 100 according to the first embodiment of the present disclosure.
  • the information processing device 100 includes a sensor 20, a control unit 30, a storage unit 50, and an output unit 60.
  • the senor 20 is a device or element that detects various information related to the information processing device 100.
  • control unit 30 for example, a program (for example, an information processing program according to the present disclosure) stored inside the information processing apparatus 100 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like is stored in a RAM (Random Access Memory). ) Etc. are executed as a work area. Further, the control unit 30 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • control unit 30 has a recognition unit 31, an acquisition unit 32, and an output control unit 33, and realizes or executes the information processing functions and actions described below.
  • the internal configuration of the control unit 30 is not limited to the configuration shown in FIG. 9, and may be any other configuration as long as it performs information processing described later.
  • the control unit 30 may be connected to a predetermined network by wire or wirelessly using, for example, a NIC (Network Interface Card) or the like, and may receive various information from an external server or the like via the network.
  • NIC Network Interface Card
  • the recognition unit 31 performs recognition processing of various information. For example, the recognition unit 31 controls the sensor 20 and detects various information using the sensor 20. Then, the recognition unit 31 performs various information recognition processes based on the information detected by the sensor 20.
  • the recognition unit 31 recognizes where the user's hand is in space. Specifically, the recognition unit 31 recognizes the position of the user's hand based on the image captured by the recognition camera, which is an example of the sensor 20. For such hand recognition processing, the recognition unit 31 may use various known techniques related to sensing.
  • the recognition unit 31 analyzes the captured image acquired by the camera included in the sensor 20 and performs the recognition process of the real object existing in the real space. For example, the recognition unit 31 sets the image feature amount extracted from the captured image as the image feature amount of a known real object (specifically, an object operated by the user such as a user's hand) stored in the storage unit 50. Match. Then, the recognition unit 31 identifies the real object in the captured image and recognizes the position in the captured image. Further, the recognition unit 31 analyzes the captured image acquired by the camera included in the sensor 20 and acquires the three-dimensional shape information in the real space.
  • a known real object specifically, an object operated by the user such as a user's hand
  • the recognition unit 31 performs a stereo matching method for a plurality of images acquired at the same time, an SfM (Structure from Motion) method, a SLAM method, etc. for a plurality of images acquired in chronological order, thereby performing three-dimensional real space.
  • the shape may be recognized and three-dimensional shape information may be acquired. Further, when the recognition unit 31 can acquire the three-dimensional shape information in the real space, the recognition unit 31 may recognize the three-dimensional position, shape, size, and posture of the real object.
  • the recognition unit 31 is not limited to recognizing the real object, and may recognize the user information about the user and the environmental information about the environment in which the user is placed based on the sensing data detected by the sensor 20.
  • the user information includes, for example, behavior information indicating the user's behavior, movement information indicating the user's movement, biological information, gaze information, and the like.
  • the behavior information is information indicating the current behavior of the user, for example, while stationary, walking, running, driving a car, climbing stairs, etc., and is recognized by analyzing sensing data such as acceleration acquired by the sensor 20. Will be done.
  • the motion information is information such as movement speed, movement direction, movement acceleration, approach to the position of the content, etc., and is recognized from sensing data such as acceleration acquired by the sensor 20 and GPS data.
  • the biological information is information such as the user's heart rate, body temperature sweating, blood pressure, pulse, respiration, blinking, eye movement, brain wave, etc., and is recognized based on the sensing data by the biological sensor included in the sensor 20.
  • the gaze information is information related to the user's gaze such as the line of sight, the gaze point, the focus, and the congestion of both eyes, and is recognized based on the sensing data by the visual sensor included in the sensor 20.
  • the environmental information includes, for example, information such as surrounding conditions, location, illuminance, altitude, temperature, wind direction, air volume, and time.
  • Information on the surrounding situation is recognized by analyzing the sensing data from the camera or microphone included in the sensor 20.
  • the location information may be information indicating the characteristics of the place where the user is, such as indoors, outdoors, underwater, or a dangerous place, or the user of the place such as a home, a company, a familiar place, or a place to visit for the first time. It may be information that shows the meaning for.
  • the location information is recognized by analyzing the sensing data of the camera, microphone, GPS sensor, illuminance sensor, etc. included in the sensor 20. Further, information on illuminance, altitude, temperature, wind direction, air volume, and time (for example, GPS time) may also be recognized based on sensing data acquired by various sensors included in the sensor 20.
  • the acquisition unit 32 acquires a change in the distance between the real object operated by the user in the real space and the virtual object which is a virtual object superimposed on the real space in the display unit 61.
  • the acquisition unit 32 acquires information about the user's hand sensed by the sensor 20 as a real object. That is, the acquisition unit 32 determines the distance between the user's hand and the virtual object based on the spatial coordinate position of the user's hand recognized by the recognition unit 31 and the spatial coordinate position of the virtual object displayed on the display unit 61. Get the change.
  • the acquisition unit 32 sets an arbitrary coordinate HP01 included in the recognized hand H01. Further, the acquisition unit 32 sets the coordinates in the virtual object V01 that are recognized as having been touched by the user's hand. Then, the acquisition unit 32 acquires the distance L between the coordinates HP01 and the arbitrary coordinates set in the virtual object V01. For example, the acquisition unit 32 acquires a change in the distance L in real time for each frame (for example, 30 times per second or 60 times per second) imaged by the sensor 20.
  • FIG. 10 is a first diagram for explaining information processing according to the first embodiment of the present disclosure.
  • FIG. 10 shows the angle of view at which the information processing apparatus 100 recognizes an object as viewed from the position of the user's head.
  • the area FV01 indicates a range in which the sensor 20 (recognition camera) can recognize the object. That is, the information processing device 100 can recognize the spatial coordinates of any object included in the area FV01.
  • FIG. 11 is a second diagram for explaining information processing according to the first embodiment of the present disclosure.
  • FIG. 11 schematically shows the relationship between the area FV01 showing the angle of view covered by the recognition camera, the area FV02 which is the display area of the display (display unit 61), and the area FV03 showing the viewing angle of the user. ing.
  • the acquisition unit 32 can acquire the distance between the real object and the virtual object when the real object exists inside the area FV01.
  • the acquisition unit 32 cannot recognize the real object when the real object exists outside the area FV01, the acquisition unit 32 cannot acquire the distance between the real object and the virtual object.
  • the output control unit 33 which will be described later, may output to notify the user that the real object cannot be recognized. As a result, the user can grasp that the hand is visible in his / her field of view, but the information processing device 100 does not recognize the hand.
  • FIG. 12 is a third diagram for explaining information processing according to the first embodiment of the present disclosure.
  • the area FV04 covered by the recognition camera is wider than the area FV03 showing the viewing angle of view of the user.
  • the area FV05 shown in FIG. 12 indicates a display area of the display when the range covered by the recognition camera is wide.
  • the information processing device 100 can perform a predetermined output (feedback) indicating that the user's hand has been recognized. Therefore, the user can avoid a situation in which he / she feels uneasy whether or not his / her hand is recognized, or the user does not recognize the hand even if the operation is performed.
  • the acquisition unit 32 may acquire the position information indicating the position of the real object by using the sensor 20 having a detection range exceeding the angle of view of the display unit 61. That is, even when the angle of view of the display does not include the real object, the acquisition unit 32 can show the recognition result of the user's hand in the three-dimensional space by the indicator E01 of the virtual object.
  • the acquisition unit 32 may acquire the user's head position information when the position information indicating the position of the real object cannot be acquired.
  • the output control unit 33 may output to indicate that the real object cannot be recognized.
  • the output control unit 33 may control the indicator E01 to display the initial state without giving any particular change.
  • the acquisition unit 32 may acquire not only the actual object but also the position information in the display unit 61 of the virtual object.
  • the output control unit 33 changes the mode of the output signal according to the approach of the virtual object from within the angle of view of the display unit 61 to the vicinity of the boundary between the inside and outside of the angle of view of the display unit 61. You may let me.
  • the acquisition unit 32 may acquire information indicating that the real object has transitioned from a state that cannot be detected by the sensor 20 to a state that can be detected by the sensor 20. Then, the output control unit 33 may give some feedback when the information indicating that the real object has transitioned to the detectable state by the sensor 20 is acquired. For example, the output control unit 33 may output a sound effect indicating that when the sensor 20 newly detects the user's hand. Alternatively, the output control unit 33 may perform processing such as displaying the hidden indicator E01 when the sensor 20 newly detects the user's hand. As a result, the user can dispel the anxiety about whether or not his / her hand is recognized.
  • the output control unit 33 displays the first display area superimposed on the virtual object and the second display area displayed superimposed on the first display area on the display unit 61, and is displayed by the acquisition unit 32.
  • the display mode of the second display area is continuously changed according to the change of the acquired distance.
  • the output control unit 33 superimposes and displays the first display area and the second display area on the surface of the virtual object. For example, the output control unit 33 sets the first display area and the second display area so that the arbitrary coordinates constituting the surface of the virtual object and the center of the first display area and the second display area overlap. Indicator E01) is displayed. Further, the output control unit 33 does not necessarily display the indicator E01 on the surface of the virtual object, but may display it so as to cut into the inside of the virtual object.
  • the output control unit 33 may perform various processes as a change in the display mode of the second display area.
  • the output control unit 33 continuously changes the size of the second display area according to the change in the distance acquired by the acquisition unit 32.
  • the output control unit 33 continuously changes the radius of the black eye EC01 according to the change in the distance acquired by the acquisition unit 32.
  • the output control unit 33 can make an impressive change in the display mode, such as increasing the black eye EC01 as the user's hand approaches.
  • the output control unit 33 stops the control of continuously changing the display mode of the second display area when the distance between the real object and the virtual object becomes equal to or less than a predetermined threshold value (second threshold value).
  • second threshold value a predetermined threshold value
  • the output control unit 33 stops the feedback that continuously changes the size of the black eye EC01 when the distance L becomes 0.
  • the output control unit 33 outputs, for example, a specific sound effect indicating that the user's hand touches the virtual object, or outputs a display process indicating that the user's hand touches the virtual object. You may do it.
  • the output control unit 33 may change the display mode of the first display area or the second display area based on the position information of the real object acquired by the acquisition unit 32. For example, the output control unit 33 displays the indicator E01 when the recognition unit 31 recognizes the real object or the acquisition unit 32 acquires the distance between the real object and the virtual object. It may be. As a result, the user can easily grasp that his / her hand has been recognized.
  • the output control unit 33 may move the second display area so that the real object faces the direction in which the real object approaches the virtual object, based on the position information of the real object acquired by the acquisition unit 32. That is, the output control unit 33 sets a predetermined region corresponding to the pupil so as to be substantially perpendicular to the straight line (optical axis) connecting the position of the real object and the position of the sensory organ object detected by the sensor. It may be considered to be moved.
  • the output control unit 33 acquires a vector connecting the coordinates indicating the center point of the black eye EC01 and the coordinates indicating the real object, and moves the center point of the black eye EC01 by an arbitrary distance in the direction of the vector. Etc. may be performed.
  • the user can visually recognize the black-eyed EC01 as if he / she is looking at this hand when the hand moved by himself / herself heads toward the virtual object, so that his / her hand is recognized.
  • the output control unit 33 may control the black eye EC01 so as to be inscribed in the white eye EP01 even when the black eye EC01 moves most. As a result, the output control unit 33 can prevent the black eye EC01 from moving to the outside of the white eye EP01.
  • the output control unit 33 continuously changes the size of the radius of the black eye EC01 according to the approach of the hand as described above, but the position of the black eye EC01 may be adjusted thereafter.
  • the center coordinate of the black eye EC01 is M
  • the radius after the change is r
  • the coordinate (origin) of the center point of the white eye EP01 is O
  • the radius is R
  • the black eye EC01 after the movement is set.
  • the coordinates of the center point of are expressed by the following equation (1).
  • the output control unit 33 can display as if the wide open black eye EC01 is looking at the user's hand.
  • the output control unit 33 displays the first display area or the second display area based on the user's head position information acquired by the acquisition unit 32.
  • the mode may be changed.
  • the output control unit 33 specifies the coordinates indicating the user's head based on the user's head position information. For example, the output control unit 33 specifies arbitrary coordinates near the center of the spectacle frame on the appearance of the information processing device 100 as coordinates indicating the user's head. Then, the output control unit 33 moves the position of the center of the black eye EC01 based on the vector connecting the center of the indicator E01 and the coordinates indicating the user's head. As a result, the output control unit 33 can display as if the eyeball of the indicator E01 is looking at the user. Further, the user can grasp that his / her hand is not recognized by the information processing apparatus 100 while the eyeball is looking at the user.
  • the output control unit 33 may perform the above output control process based on, for example, predefined information.
  • the output control unit 33 refers to the storage unit 50 and performs output control processing based on a definition file in which various output control methods described above, calculation methods such as the above equation (1), and the like are stored. Do.
  • the storage unit 50 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.
  • the storage unit 50 is a storage area for temporarily or permanently storing various types of data.
  • the storage unit 50 may store data for the information processing apparatus 100 to execute various functions (for example, an information processing program according to the present disclosure). Further, the storage unit 50 may store data (for example, a library) for executing various applications, management data for managing various settings, and the like.
  • the output unit 60 has a display unit 61 and an acoustic output unit 62, and is controlled by the output control unit 33 to output various information.
  • the display unit 61 is a display or the like for displaying a virtual object superimposed on a transparent real space.
  • the acoustic output unit 62 is a speaker or the like for outputting a predetermined audio signal.
  • FIG. 13 is a flowchart showing a flow of processing according to the first embodiment of the present disclosure.
  • the information processing device 100 first determines whether or not the position of the user's hand can be acquired by using the sensor 20 (step S101).
  • the information processing apparatus 100 acquires the coordinate HP01 indicating the current position of the hand (step S102).
  • the information processing apparatus 100 substitutes the coordinates HP01 indicating the position of the hand into the variable “target coordinates” (step S103).
  • the variable is a variable for executing the information processing according to the first embodiment, and is, for example, a value (coordinate) used for calculating the distance and the direction from the indicator E01.
  • step S101 when the position of the user's hand cannot be acquired (step S101; No), the information processing apparatus 100 acquires the coordinates C indicating the position of the head based on the current user's head position information (step S104). ). Then, the information processing apparatus 100 substitutes the coordinates C indicating the position of the head into the variable “target coordinates” (step S105).
  • the information processing apparatus 100 obtains the distance L between the target coordinate T and the center position of the indicator E01 (step S106). Further, the information processing apparatus 100 obtains a coefficient m from the distance L based on, for example, the graph shown in FIG. 7 (step S107).
  • the information processing apparatus 100 updates the radius of the black eye EC01 of the indicator E01 based on the obtained coefficient m (step S108). Further, the information processing apparatus 100 updates the center position of the black eye EC01 of the indicator E01 based on the above equation (1) (step S109).
  • the information processing apparatus 100 shows an example of displaying one indicator E01 on a virtual object.
  • the information processing device 100 may display a plurality of indicators on the virtual object. This point will be described with reference to FIGS. 14 and 15.
  • FIG. 14 is a first diagram for explaining information processing according to the second embodiment of the present disclosure. As shown in FIG. 14, the information processing apparatus 100 displays two indicators E01 and an indicator E02 on the surface of the virtual object V01.
  • the user U01 is visually recognized as having a pair of eyeballs on the virtual object V01.
  • the display control process for each of the black eyes of the indicator E01 and the indicator E02 is performed in the same manner as in the first embodiment.
  • FIG. 15 is a second diagram for explaining information processing according to the second embodiment of the present disclosure.
  • the information processing apparatus 100 specifies the coordinate HP01 indicating the position of the hand H01 as in the first embodiment. Then, the distances between the center points of the specified coordinate HP01 and the indicator E01 and the center points of the specified coordinate HP01 and the indicator E02 are acquired.
  • the information processing device 100 changes the display mode of each of the black eyes of the indicator E01 and the indicator E02.
  • the user U01 can recognize the indicator E01 and the indicator E02 as the movement of the eyeball with congestion like the human eye.
  • the display control process imitating congestion is realized by the difference between the direction and distance from the coordinate HP01 to the center point of the indicator E01 and the direction and distance from the coordinate HP01 to the center point of the indicator E02.
  • the information processing apparatus 100 displays a plurality of sets of the first display area and the second display area side by side on the surface of the virtual object.
  • the information processing device 100 according to the second embodiment can display the movement of the human eyeball more, so that the intuitive recognition of the movement of the hand H01 can be further improved. ..
  • FIG. 16 is a diagram showing a configuration example of the information processing system 2 according to the third embodiment of the present disclosure.
  • the information processing system 2 according to the third embodiment includes an information processing device 100 and a controller CR01. The description of the configuration common to the first embodiment or the second embodiment will be omitted.
  • the controller CR01 is an information device connected to the information processing device 100 by a wired or wireless network.
  • the controller CR01 is, for example, an information device held and operated by a user wearing the information processing apparatus 100, and detects the movement of the user's hand and the information input from the user to the controller CR01.
  • the controller CR01 controls the built-in sensors (for example, various motion sensors such as a 3-axis acceleration sensor, a gyro sensor, and a speed sensor) to detect the three-dimensional position and speed of the controller CR01. .. Then, the controller CR01 transmits the detected three-dimensional position, speed, and the like to the information processing device 100.
  • the controller CR01 may transmit the three-dimensional position of its own device detected by an external sensor such as an external camera. Further, the controller CR01 may transmit information that is paired with the information processing device 100, position information (coordinate information) of the own device, and the like based on a predetermined communication function.
  • the information processing device 100 recognizes not only the user's hand but also the controller CR01 operated by the user as a real object. Then, the information processing device 100 changes the display mode of the second display area (for example, black eye EC01) based on the change in the distance between the controller CR01 and the virtual object. That is, the acquisition unit 32 according to the third embodiment acquires the change in the distance between the user's hand or the controller HR01 operated by the user sensed by the sensor 20 and the virtual object. The information processing device 100 acquires the position information of the controller CR01 by using the sensor 20, and performs a process of changing the display mode of the first display area and the second display area based on the acquired position information. May be good.
  • FIG. 17 is a diagram for explaining information processing according to the third embodiment of the present disclosure.
  • the relationship between the controller CR01 operated by the user, the distance L acquired by the acquisition unit 32, and the virtual object V01 is schematically shown.
  • the acquisition unit 32 specifies an arbitrary coordinate HP02 included in the recognized controller CR01.
  • the coordinate HP02 is a preset recognition point of the controller CR01, and is a point that can be easily recognized by the sensor 20 by emitting some kind of signal (infrared signal or the like), for example.
  • the acquisition unit 32 has a distance L between the coordinate HP02 and an arbitrary coordinate set in the virtual object V01 (any specific coordinate may be used, or the center point or the center of gravity of a plurality of coordinates may be used). To get.
  • the information processing apparatus 100 recognizes not only the user's hand but also some object such as the controller CR01 operated by the user, and executes feedback based on the recognized information. Good. That is, the information processing device 100 is not limited to the hand, and may recognize an object that can be recognized by using the sensor 20 such as the controller CR01 and perform information processing according to the present disclosure.
  • the indicator E01 may further have a display area (not shown) representing the eyelids as a display area different from the black eye EC01 and the white eye EP01.
  • the display area of the eyelid is increased when the distance between the real object and the virtual object becomes equal to or less than a predetermined threshold (second threshold value) and then the distance between the real object and the virtual object further decreases.
  • a third threshold value equal to or lower than the second threshold value may be set in order to determine the distance between the real object and the virtual object.
  • the threshold value for changing the display area of the eyelid may be referred to as a first threshold value. According to such control, the user can grasp the recognition result of the distance between the virtual object and the real object more stepwise and naturally by reproducing the operation of the pupil contraction and the eyelid closing of the virtual object. ..
  • the description has focused on the stepwise notification of the distance between the real object and the virtual object to the user, but the present disclosure is not limited to the above example.
  • the indicator E01 may act to close the eyelids before the complete contraction of the pupil.
  • indicator E01 may complete the action of closing the eyelids after complete contraction of the pupil.
  • only the display area of the eyelid may be changed without changing the display area of the pupil.
  • the information processing device 100 shows an example in which a processing unit such as a control unit 30 is built-in.
  • the information processing device 100 may be separated into, for example, a glasses-type interface unit, a calculation unit including a control unit 30, and an operation unit that receives an input operation or the like from a user.
  • the information processing apparatus 100 is a so-called AR glass when the display unit 61 has transparency and is held in the line-of-sight direction of the user.
  • the information processing device 100 may be a device that communicates with the display unit 61, which is an external display, and controls the display on the display unit 61.
  • the information processing device 100 may use an external camera installed in another place as a recognition camera instead of the sensor 20 provided in the vicinity of the display unit 61.
  • a camera may be installed on the ceiling of a place where the user acts, for example, so that the entire movement of the user wearing AR goggles can be imaged.
  • the information processing device 100 may acquire an image captured by a camera installed outside via a network and recognize the position of the user's hand or the like.
  • the information processing device 100 determines the state of the user for each frame.
  • the information processing apparatus 100 does not necessarily have to determine the states of all the frames.
  • the information processing apparatus 100 may smooth several frames and determine the states of each of several frames.
  • the information processing device 100 may use not only the camera but also various sensing information for recognizing the real object. For example, when the real object is the controller CR01, the information processing device 100 may recognize the position of the controller CR01 based on the speed and acceleration measured by the controller CR01, the information on the magnetic field generated by the controller CR01, and the like. Good.
  • each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. It can be integrated and configured.
  • the recognition unit 31 and the acquisition unit 32 shown in FIG. 9 may be integrated.
  • FIG. 18 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100.
  • the computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600.
  • Each part of the computer 1000 is connected by a bus 1050.
  • the CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200, and executes processing corresponding to various programs.
  • the ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, a program that depends on the hardware of the computer 1000, and the like.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by the program.
  • the HDD 1400 is a recording medium for recording an information processing program according to the present disclosure, which is an example of program data 1450.
  • the communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet).
  • the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.
  • the input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000.
  • the CPU 1100 receives data from an input device such as a keyboard or mouse via the input / output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. Further, the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media).
  • the media is, for example, an optical recording medium such as DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
  • an optical recording medium such as DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk)
  • a magneto-optical recording medium such as MO (Magneto-Optical disk)
  • tape medium such as DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk)
  • MO Magneto-optical disk
  • the CPU 1100 of the computer 1000 realizes the functions of the recognition unit 31 and the like by executing the information processing program loaded on the RAM 1200. To do. Further, the information processing program according to the present disclosure and the data in the storage unit 50 are stored in the HDD 1400. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, but as another example, these programs may be acquired from another device via the external network 1550.
  • the present technology can also have the following configurations.
  • (1) The change in the distance between the real object operated by the user in the real space and the virtual object superimposed on the real space on the display unit is acquired based on the detection result of the sensor that detects the position of the real object.
  • Acquisition department and A sensory organ object representing the sensory organ of the virtual object for recognizing the real space is displayed on the display unit, and a predetermined area of the sensory organ object is displayed according to a change in the distance acquired by the acquisition unit.
  • the output control unit that continuously changes Information processing device equipped with.
  • (2) The sensory organ object represents the centerpiece of the virtual object.
  • (3) The predetermined area represents the pupil of the virtual object.
  • the output control unit continuously reduces the area of the pupil of the virtual object in accordance with the decrease in the distance acquired by the acquisition unit.
  • the information processing device according to (3) above.
  • the sensory organ object includes the eyelids of the virtual object.
  • the output control unit determines that the distance between the eyeball of the virtual object and the real object is equal to or less than the first threshold value based on the detection result of the sensor, the output control unit determines the display area of the eyelid of the virtual object. increase, The information processing device according to (4) above.
  • the output control unit continuously changes the predetermined region when the distance between the eyeball of the virtual object and the real object becomes equal to or less than the second threshold value based on the detection result of the sensor.
  • the information processing apparatus according to any one of (2) to (5) above.
  • the sensory organ object includes the eyelids of the virtual object. After the control of continuously changing the predetermined area is stopped, the distance between the eyelid of the virtual object and the real object is equal to or less than the second threshold value based on the detection result of the sensor. The display area of the eyelid of the virtual object is increased based on the determination that the value is equal to or less than the threshold value.
  • the information processing device according to (6) above.
  • the sensor has a detection range that exceeds the angle of view of the display unit.
  • the output control unit continuously changes the predetermined area based on the change in the distance between the real object and the virtual object located outside the angle of view of the display unit (1) to (7). ) Is described in any of the information processing devices.
  • the output control unit moves the predetermined region so as to be substantially perpendicular to the straight line connecting the position of the real object and the position of the sensory organ object detected by the sensor (2) to (8).
  • the information processing device according to any one of. (10) When the acquisition unit cannot acquire the position information indicating the position of the real object, the acquisition unit acquires the head position information of the user.
  • the information processing device according to any one of (1) to (9) above, wherein the output control unit changes the predetermined region based on the head position information acquired by the acquisition unit.
  • the acquisition unit is described in any one of (1) to (10) above, which acquires a change in the distance between the user's hand sensed by the sensor or the controller operated by the user and the virtual object. Information processing device.
  • the information processing apparatus according to any one of (1) to (11), further comprising the display unit having optical transparency and being held in the line-of-sight direction of the user.
  • the computer The change in the distance between the real object operated by the user in the real space and the virtual object superimposed on the real space on the display unit is acquired based on the detection result of the sensor that detects the position of the real object.
  • a sensory organ object representing the sensory organ of the virtual object for recognizing the real space is displayed on the display unit, and a predetermined area of the sensory organ object is continuously displayed in response to a change in the acquired distance.
  • Information processing method that changes to.
  • Information processing system 100 Information processing device 20 Sensor 30 Control unit 31 Recognition unit 32 Acquisition unit 33 Output control unit 50 Storage unit 60 Output unit 61 Display unit 62 Acoustic output unit CR01 controller

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Optics & Photonics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2020/005471 2019-03-26 2020-02-13 感覚器オブジェクトを表示する情報処理装置 Ceased WO2020195292A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2021508231A JPWO2020195292A1 (https=) 2019-03-26 2020-02-13
US17/435,556 US20220049947A1 (en) 2019-03-26 2020-02-13 Information processing apparatus, information processing method, and recording medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-058265 2019-03-26
JP2019058265 2019-03-26

Publications (1)

Publication Number Publication Date
WO2020195292A1 true WO2020195292A1 (ja) 2020-10-01

Family

ID=72608997

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/005471 Ceased WO2020195292A1 (ja) 2019-03-26 2020-02-13 感覚器オブジェクトを表示する情報処理装置

Country Status (3)

Country Link
US (1) US20220049947A1 (https=)
JP (1) JPWO2020195292A1 (https=)
WO (1) WO2020195292A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023041004A (ja) * 2022-02-24 2023-03-23 株式会社mediVR リハビリテーション支援装置、その方法およびプログラム
WO2024225009A1 (ja) * 2023-04-28 2024-10-31 ソニーグループ株式会社 撮像装置、表示制御方法、プログラム
EP4478158A4 (en) * 2022-02-09 2025-05-07 Sony Group Corporation INFORMATION PROCESSING DEVICE AND INFORMATION PROCESSING METHOD

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023174066A (ja) * 2022-05-27 2023-12-07 キヤノン株式会社 画像処理装置、画像処理方法、およびプログラム
CN119830541B (zh) * 2024-12-18 2025-10-03 西北工业大学深圳研究院 一种基于gsfm信号的水下探通导一体化波形设计

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017213070A1 (ja) * 2016-06-07 2017-12-14 ソニー株式会社 情報処理装置および方法、並びに記録媒体
JP2018057554A (ja) * 2016-10-04 2018-04-12 トヨタ自動車株式会社 音声対話装置及びその制御方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2156869A1 (en) * 2008-08-19 2010-02-24 Sony Computer Entertainment Europe Limited Entertainment device and method of interaction
KR101760804B1 (ko) * 2011-11-23 2017-07-24 인텔 코포레이션 복수의 뷰, 디스플레이 및 피직스를 가진 제스쳐 입력
JP2015526309A (ja) * 2012-08-31 2015-09-10 リシンク ロボティクス インコーポレイテッド 安全ロボット動作のためのシステムおよび方法
US10269180B2 (en) * 2013-04-16 2019-04-23 Sony Corporation Information processing apparatus and information processing method, display apparatus and display method, and information processing system
JP6596883B2 (ja) * 2015-03-31 2019-10-30 ソニー株式会社 ヘッドマウントディスプレイ及びヘッドマウントディスプレイの制御方法、並びにコンピューター・プログラム
WO2017199662A1 (ja) * 2016-05-20 2017-11-23 Ｇｒｏｏｖｅ Ｘ株式会社 自律行動型ロボット及びコンピュータプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017213070A1 (ja) * 2016-06-07 2017-12-14 ソニー株式会社 情報処理装置および方法、並びに記録媒体
JP2018057554A (ja) * 2016-10-04 2018-04-12 トヨタ自動車株式会社 音声対話装置及びその制御方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4478158A4 (en) * 2022-02-09 2025-05-07 Sony Group Corporation INFORMATION PROCESSING DEVICE AND INFORMATION PROCESSING METHOD
US12554317B2 (en) 2022-02-09 2026-02-17 Sony Group Corporation Information processing device and information processing method
JP2023041004A (ja) * 2022-02-24 2023-03-23 株式会社mediVR リハビリテーション支援装置、その方法およびプログラム
WO2024225009A1 (ja) * 2023-04-28 2024-10-31 ソニーグループ株式会社 撮像装置、表示制御方法、プログラム

Also Published As

Publication number Publication date
US20220049947A1 (en) 2022-02-17
JPWO2020195292A1 (https=) 2020-10-01

Similar Documents

Publication Publication Date Title
JP7283506B2 (ja) 情報処理装置、情報処理方法、及び情報処理プログラム
EP3571673B1 (en) Method for displaying virtual image, storage medium and electronic device therefor
CN110647237B (zh) 在人工现实环境中基于手势的内容共享
US12332452B2 (en) Geometry modeling of eyewear devices with flexible frames
US11861062B2 (en) Blink-based calibration of an optical see-through head-mounted display
JP7190434B2 (ja) 外部条件に基づくウェアラブルディスプレイデバイスの自動制御
US20210303258A1 (en) Information processing device, information processing method, and recording medium
TWI597623B (zh) 可穿戴基於行爲的視覺系統
WO2020195292A1 (ja) 感覚器オブジェクトを表示する情報処理装置
CN105393192B (zh) 用于近眼显示器的网状分层菜单显示配置
CN104956252B (zh) 用于近眼显示设备的外围显示器
US9122053B2 (en) Realistic occlusion for a head mounted augmented reality display
US9041622B2 (en) Controlling a virtual object with a real controller device
CN107577045B (zh) 用于头戴式显示器的预测跟踪的方法、装置及存储介质
CN105393284B (zh) 基于人类身体数据的空间雕刻
US20200322595A1 (en) Information processing device and information processing method, and recording medium
KR20220120649A (ko) 인공 현실 콘텐츠의 가변 초점 디스플레이를 갖는 인공 현실 시스템
CN110646938A (zh) 近眼显示器系统
CN112567287A (zh) 具有帧调制功能的增强现实显示
JP7676416B2 (ja) 中立アバタ
WO2013155217A1 (en) Realistic occlusion for a head mounted augmented reality display
KR20240091224A (ko) 사용자의 표현을 생성 및 디스플레이하기 위한 디바이스들, 방법들, 및 그래픽 사용자 인터페이스들
KR20230025697A (ko) 기하학적 위험 검출을 갖는 시각 장애인 지원 아이웨어
KR20240008359A (ko) 오디오 향상 증강 현실
CN116529652A (zh) 增强现实空间音频体验

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20777235

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021508231

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20777235

Country of ref document: EP

Kind code of ref document: A1