WO2019130900A1

WO2019130900A1 - Information processing device, information processing method, and program

Info

Publication number: WO2019130900A1
Application number: PCT/JP2018/042527
Authority: WO
Inventors: 浩丈市川; 諒介村田; 広幸安賀; 俊逸小原
Original assignee: ソニー株式会社
Priority date: 2017-12-26
Filing date: 2018-11-16
Publication date: 2019-07-04

Abstract

[Problem] To recommend an information processing device, information processing method, and program capable of highly conveniently correcting a parameter relating to a display of a virtual object based on a result of a depth sensing. [Solution] Provided is an information processing device comprising: an acquisition part for acquiring a result of a depth sensing of a real object associated with a user's viewing position; and a display control part for causing a display part associated with the user to display an object for correction for correcting a parameter relating to the display of a virtual object based on the result of the depth sensing of the real object.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM

The present disclosure relates to an information processing device, an information processing method, and a program.

Conventionally, various technologies related to AR (Augmented Reality) have been developed. In the AR, various types of information (for example, a virtual object) can be presented to the user in association with the position of the user in the real space.

For example, in Patent Document 1 below, a gesture area associated with at least one operation target is set based on the reference positions of a plurality of operation targets operable by the user, and then, via the gesture interface in the gesture area It is described to control the operation of each operation target.

JP 2014-186361 A

By the way, in the scene where a virtual object is displayed based on the result of depth sensing, the case where the value of the parameter regarding the display of the virtual object concerned may arise may arise. However, in the technology described in Patent Document 1, it is not considered to correct the value of the parameter related to the display of the virtual object.

Therefore, the present disclosure proposes a novel and improved information processing apparatus, an information processing method, and a program that can highly conveniently correct parameters related to display of a virtual object based on the result of depth sensing.

According to the present disclosure, an acquisition unit for acquiring a result of depth sensing of a real object corresponding to a viewpoint position of a user, and a correction for correcting a parameter related to display of a virtual object based on the result of depth sensing of the real object An information processing apparatus is provided, comprising: a display control unit configured to display an object for display on a display unit corresponding to the user.

Further, according to the present disclosure, for acquiring a result of depth sensing of a real object corresponding to a user's viewpoint position, and correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object. There is provided an information processing method including a processor displaying a correction object on a display unit corresponding to the user.

Further, according to the present disclosure, an acquisition unit that acquires a result of depth sensing of a real object corresponding to a viewpoint position of a user, and a parameter related to display of a virtual object based on the result of depth sensing of the real object There is provided a program for causing a correction object for correction to function as a display control unit that causes the display unit corresponding to the user to display the correction object.

As described above, according to the present disclosure, parameters relating to display of a virtual object based on the result of depth sensing can be corrected with high convenience. In addition, the effect described here is not necessarily limited, and may be any effect described in the present disclosure.

It is an explanatory view showing an example of composition of an information processing system common to each embodiment of this indication. It is a figure showing the example where the occlusion field of the virtual object by the hand is correctly displayed. It is a figure showing the example where the occlusion field of the virtual object by the hand is shifted and displayed. It is a block diagram showing an example of functional composition of eyewear 10 concerning a 1st embodiment. FIG. 7 is a diagram schematically illustrating an example of correction of parameters related to a display position of a virtual object based on movement of a depth object in the screen coordinate system according to the first embodiment. FIG. 6 is a diagram showing a specific example of the flow of a correction start instruction and a correction end instruction. FIG. 6 is a diagram showing a specific example of the flow of a correction start instruction and a correction end instruction. FIG. 6 is a diagram showing a specific example of the flow of a correction start instruction and a correction end instruction. FIG. 6 is a diagram showing a specific example of the flow of a correction start instruction and a correction end instruction. FIG. 6 is a view showing a display example of a depth object and four correction objects according to the first embodiment. FIG. 6 is a view showing a display example of a depth object and four correction objects according to the first embodiment. It is a figure showing the flow of processing concerning a 1st embodiment. It is a figure showing a concrete example of amendment of a parameter about a display position of a virtual object concerning application example 1 of a 1st embodiment. It is a figure showing a concrete example of amendment of a parameter about a display position of a virtual object concerning application example 1 of a 1st embodiment. It is a figure showing a concrete example of amendment of a parameter about a display position of a virtual object concerning application example 1 of a 1st embodiment. It is a figure showing a concrete example of amendment of a parameter about a display position of a virtual object concerning application example 2 of a 1st embodiment. It is a figure showing a concrete example of amendment of a parameter about a display position of a virtual object concerning application example 2 of a 1st embodiment. FIG. 17 is a diagram schematically illustrating an example of correction of a parameter related to a display position of a virtual object based on movement of a depth object in a camera coordinate system according to an application example 4 of the first embodiment. It is a figure showing an example of classification of a kind of case where accuracy of a result of depth sensing is low. It is a block diagram showing an example of functional composition of eyewear 10 concerning a 2nd embodiment. It is a figure showing a concrete example of a parameter set concerning a 2nd embodiment. It is the figure which showed the example of a display of the animation corresponding to each parameter set concerning a 2nd embodiment. It is the figure which expanded and showed the moving image 90c shown to FIG. 15A. It is a figure showing a part of flow of processing concerning a 2nd embodiment. It is a figure showing a part of flow of processing concerning a 2nd embodiment. It is a figure showing an example of hardware constitutions of eyewear 10 common to each embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different alphabets to the same reference numerals. For example, a plurality of components having substantially the same functional configuration are distinguished as required by the display unit 124 a and the display unit 124 b. However, when it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numerals will be given. For example, when there is no need to distinguish between the display unit 124 a and the display unit 124 b, the display unit 124 is simply referred to as the display unit 124.

In addition, the “mode for carrying out the invention” will be described in the order of items shown below.
1. Configuration of information processing system First embodiment 3. Second embodiment 4. Hardware configuration 5. Modified example

<< 1. Information Processing System Configuration >>
The present disclosure can be implemented in various forms as will be described in detail in, for example, "2. first embodiment" and "3. second embodiment". First, a configuration example of an information processing system common to each embodiment of the present disclosure will be described with reference to FIG. As shown in FIG. 1, an information processing system common to the embodiments includes an eyewear 10, a server 20, and a communication network 22.

<1-1. Eyewear 10>
The eyewear 10 is an example of an information processing apparatus according to the present disclosure. Eyewear 10 may control the display of content that includes one or more virtual objects. For example, the eyewear 10 allows one or more virtual objects to be displayed on the display unit 124 described later while making the real object (for example, the user's hand, etc.) around the user wearing the eyewear 10 visible to the user. Display.

Here, the content is, for example, AR content or VR (Virtual Reality) content. Also, the virtual object may be a 2D object or a 3D object. The eyewear 10 can also receive the content, for example, from an external device such as the server 20 via the communication network 22 described later, or can be stored in advance (in its own device) It may be

As shown in FIG. 1, eyewear 10 may be a head-mounted device that includes a display 124. For example, the eyewear 10 may be an AR glass, a video see-through HMD (Head Mounted Display), or a shield HMD.

In the example illustrated in FIG. 1, the eyewear 10 includes, as the display unit 124, a right side display unit 124a and a left side display unit 124b described later. In this case, the eyewear 10 can display the predetermined content on the right side display unit 124a and the left side display unit 124b. For example, the eyewear 10 first generates an image for the right eye and an image for the left eye based on the predetermined content, and displays the image for the right eye on the right display unit 124 a and the image for the left eye Is displayed on the left side display unit 124b.

{1-1-1. Right display section 124a, left display section 124b}
As shown in FIG. 1, the right side display unit 124 a and the left side display unit 124 b may be configured as a transmissive display device. In this case, the right side display unit 124a can project an image by using at least a partial area of the right-eye lens (or the goggle-type lens) included in the eyewear 10 as a projection plane. Furthermore, the left display unit 124b can project an image by using at least a partial area of the left eye lens (or the goggle type lens) included in the eyewear 10 as a projection plane.

As a modification, the display unit 124 may be configured as a non-transmissive display device. For example, the right side display unit 124 a and the left side display unit 124 b may be configured to include an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diode), and the like. In this case, the eyewear 10 has a camera, and can sequentially display on the display unit 124 an image in front of the user captured by the camera. Thus, the user can view the scenery in front of the user through the video displayed on the display unit 124.

{1-1-2. Display example of virtual object}
Here, with reference to FIG. 2, a display example of a virtual object by the eyewear 10 will be described. FIG. 2 is a view showing an example in which a virtual object 40 is displayed on the display screen 30 of the display unit 124. As shown in FIG. Although details will be described later, the eyewear 10 has a depth sensor, and senses individual real objects in the real space (in the example shown in FIG. 2, the user's hand 2 etc.) using the depth sensor. obtain. Thereafter, the eyewear 10 overlaps the individual real object with the virtual object 40 based on comparison of the result of the depth sensing by the depth sensor with the position information in the real space corresponding to the display position of the virtual object 40. (In other words, the presence or absence of occlusion of the virtual object 40 by the respective real objects). For example, if it is determined that the hand 2 and the virtual object 40 overlap (in other words, part of the virtual object 40 is occluded by the hand 2), as shown in FIG. The eyewear 10 hides the portion of the virtual object 40 that corresponds to the overlapping area. In the example illustrated in FIG. 2, in the virtual object 40 which is a plane, the area where the user's hand 2 overlaps is hidden.

<1-2. Server 20>
The server 20 is a device that manages various types of content (such as AR content and VR content). Also, the server 20 can communicate with other devices via the communication network 22. For example, when receiving an acquisition request for content from the eyewear 10, the server 20 transmits the content corresponding to the acquisition request to the eyewear 10.

<1-3. Communication network 22>
The communication network 22 is a wired or wireless transmission path of information transmitted from a device connected to the communication network 22. For example, the communication network 22 may include a telephone network, the Internet, a public network such as a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), a WAN (Wide Area Network), etc. . Also, the communication network 22 may include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

<1-4. Organize issues>
The configuration of the information processing system common to each embodiment has been described above. By the way, for example, as shown in FIG. 3, a shielded area of the virtual object 40 (in other words, one of the virtual objects 40) by one or more real objects (in the example shown in FIG. The non-display area by the above real object may be displayed shifted from the position of the target (for example, the position intended by the developer) due to various causes. FIG. 3 is a view showing an example in which the shielded area of the virtual object 40 by the user's hand 2 is shifted and displayed for some reason in the same situation as the example shown in FIG.

Here, as an example of the various causes, first, an error related to calibration of depth sensing by the eyewear 10 can be mentioned. The said error may arise, for example, when the precision of the sensing by the depth sensor which eyewear 10 has later mentioned later is low. For example, the error may occur because an incorrect value is set as the value of the internal parameter of the depth sensor. Furthermore, the error may occur in the case where the calibration of the display unit 124 is not properly performed. Generally, the installation position of the depth sensor in the eyewear 10 and the installation position of the display unit 124 may be different. Therefore, the error may occur if calibration (correction or the like) is not appropriately performed according to the difference between the installation position of the depth sensor and the installation position of the display unit 124.

Another example of the various causes is that the viewpoint position of the user wearing the eyewear 10 changes. Usually, the interocular distance is different for each user, and the individual difference in the interocular distance may cause the error. Also, usually, the internal rotation angle may be different for each user. For example, while the human eye tends to turn inward when looking at an object, the magnitude of this tendency may differ from user to user. And the said difference | error may arise by the individual difference of the said internal rotation angle.

Furthermore, the error may also occur due to the slippage of the eyewear 10 (that is, the user does not properly wear the eyewear 10).

Therefore, for example, in order to correct the displacement of the shielding area of the virtual object due to the real object as shown in FIG. 3, it is necessary to perform appropriate correction according to these errors. However, it is extremely difficult to make a complete correction automatically. Therefore, it is desirable that the user wearing the eyewear 10 can easily and accurately correct the displacement of the shielding area of the virtual object due to the real object (for example, a hand or the like).

Then, the eyewear 10 which concerns on each embodiment came to be created in view of the said situation. The eyewear 10 according to each embodiment acquires the result of depth sensing of at least one real object corresponding to the viewpoint position of the user wearing the eyewear 10, and performs depth sensing of the at least one real object. At least one correction object can be displayed on the display unit 124 for correcting parameters related to the display of at least one virtual object based on the result. Therefore, the user can easily and appropriately correct the value of the parameter related to the display of the at least one virtual object. The contents of each embodiment will be sequentially described in detail below.

<< 2. First embodiment >>
<2-1. Configuration>
First, the first embodiment will be described. First, the configuration of the eyewear 10 according to the first embodiment will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the eyewear 10. As shown in FIG. 4, the eyewear 10 includes a control unit 100, a communication unit 120, a sensor unit 122, a display unit 124, an input unit 126, and a storage unit 128. In addition, description is abbreviate | omitted about the same content as said description.

{2-1-1. Sensor unit 122}
The sensor unit 122 may include, for example, a depth sensor (for example, a stereo camera or a time of flight sensor), an image sensor (camera), a microphone, and the like. For example, when the depth sensor is a stereo camera, the depth sensor may be a left camera that performs depth sensing on the front left side of the user wearing the eyewear 10 and a right camera that performs depth sensing on the front right side of the user. Including. Here, the left camera is an example of the left depth camera according to the present disclosure. In addition, the right camera is an example of the right depth camera according to the present disclosure.

The individual sensors included in the sensor unit 122 may constantly sense, may periodically sense, or in a specific case (for example, when an instruction from the control unit 100 is given, etc.) You may sense only in).

{2-1-2. Input unit 126}
The input unit 126 receives various inputs from the user wearing the eyewear 10. The input unit 126 can be configured to include an input device 160 described later. For example, the input unit 126 includes one or more physical buttons.

{2-1-3. Control unit 100}
The control unit 100 can be configured to include, for example, processing circuits such as a central processing unit (CPU) 150 and a graphics processing unit (GPU) described later. The control unit 100 centrally controls the operation of the eyewear 10. Further, as illustrated in FIG. 4, the control unit 100 includes a sensing result acquisition unit 102, a correction unit 104, and a display control unit 106.

{2-1-4. Sensing result acquisition unit 102}
The sensing result acquisition unit 102 is an example of an acquisition unit according to the present disclosure. The sensing result acquisition unit 102 acquires the result of depth sensing of one or more real objects corresponding to the viewpoint position of the user wearing the eyewear 10, for example, by reception or readout processing. For example, the sensing result acquisition unit 102 acquires the result of the depth sensing of the one or more real objects by the depth sensor included in the sensor unit 122 by reading from the sensor unit 122.

As a modification, in the case where one or more depth sensors (not shown) are installed in the environment where the user is located, the sensing result acquisition unit 102 determines the result of the depth sensing by the one or more depth sensors as the one or more depth sensors It may be acquired by receiving from. In this case, for example, the sensing result acquisition unit 102 first causes the communication unit 120 to transmit a sensing result acquisition request to at least one of the one or more depth sensors. Then, when the communication unit 120 described later receives the result of the depth sensing from the one or more depth sensors, the sensing result acquisition unit 102 may acquire the result of the depth sensing from the communication unit 120.

Here, the one or more real objects may basically be real objects located in a space corresponding to the field of view of the user. However, the present invention is not limited to such an example, and the one or more real objects may include one or more real objects located in a predetermined space corresponding to the outside (e.g., behind the user) of the user.

{2-1-5. Correction unit 104}
When predetermined instruction information of the user wearing the eyewear 10 is acquired, the correction unit 104 corrects the value of the parameter related to the display of one or more virtual objects based on the instruction information. For example, the correction unit 104 corrects the value of the parameter related to the display of the one or more virtual objects based on the instruction information of the user acquired after the depth sensing result is acquired by the sensing result acquisition unit 102. Here, the parameters related to the display of the one or more virtual objects include the parameters related to the display position of the one or more virtual objects.

For example, the correction unit 104 is a parameter related to the display position of the one or more virtual objects based on the instruction information of the user for the one or more correction objects displayed on the display unit 124 under the control of the display control unit 106 described later. Correct the Here, the instruction information of the user may be information acquired while the result of the depth sensing is acquired and the one or more correction objects are displayed on the display unit 124. In this case, when the instruction information of the user is acquired, the correction unit 104 may correct the parameter related to the display position of the one or more virtual objects based on the instruction information of the user. The specific contents of the user's instruction information will be described later.

(2-1-5-1. Correction example of parameter)
The contents of the correction by the correction unit 104 will be described in more detail below. For example, the correction unit 104 sets the depth object indicating the shape of one or more real objects specified based on the result of depth sensing acquired by the sensing result acquisition unit 102 to the instruction information of the user in the screen coordinate system. By moving based on the value, the value of the parameter related to the display position of the one or more virtual objects may be corrected. Here, the depth object may be an object that indicates the entire shape of each of the one or more real objects, or the shape of the outer peripheral portion (such as an outline) of each of the one or more real objects. It may be an object to be emphasized. Alternatively, the depth object may be an object that emphasizes and indicates a part of edges extracted from a captured image of the one or more real objects.

FIG. 5 is a diagram schematically showing an example in which the depth object 50 is moved in the screen coordinate system in order to correct the value of the parameter regarding the display position of the one or more virtual objects. In the example illustrated in FIG. 5, the depth object 50 is an object indicating the shape of the user's hand specified based on the result of depth sensing. For example, as illustrated in FIG. 5, when the user's instruction information for instructing to move the position of the depth object 50 a to the position of the depth object 50 b is acquired, the correction unit 104 corrects the one or more items. The value of the parameter related to the display position of the virtual object is corrected by the movement amount of the depth object 50 indicated by the instruction information.

More specifically, in the example illustrated in FIG. 5, regarding the right display unit 124 a and the left display unit 124 b, the values of the parameters related to the display position of the one or more virtual objects may be separately corrected. For example, when the instruction information of the user corresponding to each of the right side display unit 124a and the left side display unit 124b is obtained, the correction unit 104 first performs parallel movement of the depth object 50 in each of the left and right screen coordinate systems. Scaling is performed separately based on these instruction information. Then, the correction unit 104 separately corrects the values of the parameters regarding the display position of the one or more virtual objects based on the result of the parallel movement and the scaling of the depth object 50.

Here, the above contents will be described in more detail. For example, with respect to each of the right side display unit 124a and the left side display unit 124b, the correction unit 104 performs coordinate conversion of each vertex of the depth object 50 on the basis of Equation (1) below to perform the one or more virtual Correct the parameter values related to the display position of the object separately.

Here, v is a vertex position in the local coordinate system. That is, the result of the depth sensing may be the position of the vertex represented in the three-dimensional coordinate system with the corresponding depth sensor (right or left camera) as the origin. M is a model matrix. Specifically, M may be a matrix for translating, rotating, or scaling the result of the depth sensing. V is a viewing transformation. Specifically, V may be a matrix for conversion to the camera coordinate system of the virtual camera in order to display the vertex v on the right side display unit 124 a or the left side display unit 124 b. P is a projection matrix. Specifically, P may be a matrix for normalizing three directions (vertical, horizontal, and depth) to generate a screen coordinate system. v 'is the vertex position (coordinates) after conversion. Specifically, v 'may be the position (coordinates) of v in the coordinate system in which the three directions are normalized.

In the example shown in FIG. 5, for each of the right side display unit 124a and the left side display unit 124b, the correction unit 104 determines the value of one or more parameters in the matrix P in Equation (1) based on the instruction information of the user. By correcting separately, the value of the parameter related to the display position of the one or more virtual objects is corrected.

The above correction is performed for each eye. Therefore, while correction is being performed on either the left or the right, it is usually necessary for the user to close the eyes on the other side or hide the eyes on the other side become. However, these methods may cause high user load or stains on the eyewear 10. Therefore, while correction is performed on either the left or the right, the display control unit 106 described later sets the display color of the entire display area of the display unit 124 on the other side to a predetermined color (for example, black). It may be set, or a predetermined image may be displayed on the entire display area of the display unit 124 on the other side. Alternatively, in the case where one or more light adjustment elements are provided in each of the two display units 124, and while correction is performed for one of the two, the display control unit 106 is configured to With regard to the display unit 124 of the above, one or more light control elements of the display unit 124 may be controlled such that the entire display area is displayed in black. Alternatively, in the case where each of the two display units 124 is provided with a shutter unit (physical blindfold) that can be opened and closed in a predetermined direction (such as in the vertical direction), correction is performed on one of them. While the display control unit 106 is operating, the shutter unit of the display unit 124 on one of the two sides is open, and the shutter unit of the display unit 124 on the other side is closed. The shutter unit may be controlled.

(2-1-5-2. Instruction information)
Below, the content of the said user's instruction information is demonstrated in more detail. The instruction information of the user may include a correction start instruction for starting correction of a parameter related to display of the one or more virtual objects, or may end correction of a parameter related to the display of the one or more virtual objects. A correction end instruction may be included, or an instruction of the correction amount of the parameter when the correction is performed by the correction unit 104 may be included. The instruction information of the user may include one or more of these three types of instructions.

-Operation on Input Unit 126 Hereinafter, specific examples of the correction start instruction and the correction end instruction will be described. For example, the correction start instruction and / or the correction end instruction may be that a predetermined operation of the user on the input unit 126 (for example, a predetermined physical button or the like) is detected. As one example, when it is detected that a predetermined physical button (included in input unit 126) is pressed for a long time, correction unit 104 acquires the detection result as the correction start instruction or the correction end instruction. You may According to this method, there is an advantage that the user's operation mistake is small, and the load of determination of the operation by the eyewear 10 is small.

-Speech or The correction start instruction and / or the correction end instruction may be a result of speech recognition of a predetermined speech of the user. For example, when it is recognized that the user has issued a predetermined voice command (for example, “Start calibration.” Or the like) for starting the correction, the correction unit 104 instructs the correction result to be the correction start instruction. Get as. Alternatively, when it is recognized that the user has issued a predetermined voice command (for example, “End calibration.” Or the like) for ending the correction, the correction unit 104 instructs the correction end to the correction end. It may be acquired as According to this method, the user can instruct hands-free start and end of correction.

-Movement of a hand or the correction start instruction and / or the correction end instruction may be a recognition result of the user's hand movement. For example, when a predetermined gesture using a hand is recognized to instruct start and / or end of the correction, the correction unit 104 instructs the correction result to be the correction start instruction or the correction end instruction. Get as. The predetermined gesture may be, for example, transforming the hand into the shape of "goo" and then performing the transformation of the hand into the shape of "par" twice in total. Alternatively, the predetermined gesture may be to align the palms of both hands, or may be to align the fingertips of predetermined fingers of both hands.

Such predetermined gestures are significantly different from hand movements for other operations, and thus have the advantage that the eyewear 10 is easy to recognize. In addition, when the predetermined gesture is the correction start instruction, immediately after performing the predetermined gesture, it is possible to extend or withdraw the hand performing the gesture in a predetermined direction. It may be further defined as an instruction. Thereby, the user can indicate the correction amount with high continuity of operation.

-Line-of-sight change Alternatively, the correction start instruction and / or the correction end instruction may be an instruction acquired based on the line-of-sight information of the user. For example, when it is recognized that the user has closed one eye and it is recognized that the time during which the one eye is closed has continued for a predetermined time or longer, the correction unit 104 starts the correction of the recognition result. It may be acquired as an instruction or the correction end instruction. Note that the line-of-sight information of the user may be information acquired based on, for example, a captured image of the eyeball of the user, or may be acquired based on the detection result of the posture (direction or the like) of the eyewear 10 It may be information. In the latter case, the change in the user's line of sight may be identified indirectly based on the detection result.

-Movement of a hand with respect to a stereo camera Alternatively, when the depth sensor included in the sensor unit 122 is a stereo camera, the correction start instruction and / or the correction end instruction is performed in front of at least one of the stereo cameras. Holding the hand (that is, hiding the camera with the hand) may be used. Usually, it is easy to recognize whether the hand is held in front of the camera. Therefore, according to this method, it is possible to prevent misrecognition regarding the instruction of start and end of correction with high accuracy.

-Combination or The correction start instruction and / or the correction end instruction may be a combination of any two or more of the plurality of types of operations described above. For example, while these instructions are long pressed on a predetermined physical button (included in the input unit 126), the hand is deformed into the shape of "goo" and then the hand is deformed into the shape of "par" It may be. Alternatively, these instructions may be, while uttering a predetermined voice command, transforming the hand into the shape of "goo" and then transforming the hand into the shape of "par". Alternatively, these instructions may be to align the palms of both hands while uttering a predetermined voice command. Alternatively, these instructions may be that the user closes one eye while uttering a predetermined voice command. Alternatively, these instructions may be that the user stretches at least one hand ahead of the user while closing one's eyes. Alternatively, these instructions may be to utter a predetermined voice command while holding a hand in front of one of the stereo cameras. Alternatively, these instructions may be to extend the other hand forward of the user while holding one hand in front of one of the stereo cameras.

As described above, by using a combination of a plurality of types of operations as the correction start instruction and / or the correction end instruction, the eyewear 10 can perform these instructions and other operations (normal operation), More accurate distinction is possible. In particular, since it is very easy to recognize that the camera is hidden (that is, the hand is held in front of the camera), the combination of holding the hand in front of the camera and other operations By being used as an instruction of, misrecognition can be prevented with higher accuracy.

Specific Example Here, a specific example of the flow of the correction start instruction and the correction end instruction will be described with reference to FIGS. 6A to 6D. In the example shown in FIGS. 6A to 6D, the eyewear 10 includes stereo cameras (right camera 122a and left camera 122b) as depth sensors. FIG. 6A is a diagram showing an example in which the user wearing the eyewear 10 performs a normal operation (that is, an operation other than these instructions) using the left hand 2a. After the timing shown in FIG. 6A, as shown in FIG. 6B, the user holds the right hand 2b in front of both the right camera 122a and the right display unit 124a as the correction start instruction. Thereafter, as shown in FIG. 6C, the user shifts the right hand 2b from the front of the right camera 122a, and keeps holding the right hand 2b only in front of the right display 124a. Thereafter, as shown in FIG. 6D, the user holds the right hand 2b again in front of both the right camera 122a and the right display unit 124a as the correction end instruction.

In the example shown in FIGS. 6A to 6D, the correction unit 104 first detects the detection result of the position of the left hand 2a at the timing shown in FIG. 6B and the detection result of the position of the left hand 2a at the timing shown in FIG. Calculate the difference. Then, the correction unit 104 may determine the correction amount of the parameter related to the display position of the one or more virtual objects according to the difference.

{2-1-6. Display control unit 106}
The display control unit 106 controls the display of the display unit 124. For example, the display control unit 106 displays, on the display unit 124, one or more correction objects for correcting a parameter related to the display position of one or more virtual objects based on the result of depth sensing acquired by the sensing result acquisition unit 102. Let As one example, the one or more correction objects may be displayed by the number of different correction directions with respect to the display position of the one or more virtual objects. Then, each of the one or more correction objects may be an object for the user to designate the correction amount of the display position of the one or more virtual objects in the correction direction corresponding to the correction object. .

Furthermore, the display control unit 106 can cause the display unit 124 to display a depth object based on the result of depth sensing acquired by the sensing result acquisition unit 102 in association with the one or more correction objects. For example, the display control unit 106 causes the display unit 124 to simultaneously display the depth object and the one or more correction objects.

(2-1-6-1. Specific example)
The above functions will now be described in more detail with reference to FIGS. 7A and 7B. FIG. 7A is a diagram showing an example in which the depth object based on the result of the depth sensing of the user's right hand and the four correction objects are displayed on the display unit 124 simultaneously. As shown in FIG. 7A, for example, the display control unit 106 corrects the correction amount of the display position of the depth object 52 indicating the contour of the right hand 2b specified based on the result of the depth sensing of the right hand 2b The display screen 30 displays four correction objects 54 for the user to instruct. More specifically, the display control unit 106 causes the depth object 52 to be displayed at the display position in the display screen 30 corresponding to the result of the depth sensing of the contour portion of the right hand 2 b. At the same time, the display control unit 106 makes one correction object 54 for the user to designate the correction amount of the display position of the depth object 52 in each of four directions in the upper, lower, right and left directions in the display screen 30 Display in the vicinity of 52.

When a predetermined operation (for example, tapping with the left hand 2a) is performed on any of the four correction objects 54, the correction unit 104 corrects the operation object correction object 54 and the corresponding operation. Depending on the content, parameters relating to the display position of the virtual object may be corrected. For example, when it is detected that a predetermined physical button (included in the input unit 126) has been pressed for a predetermined time or longer, the correction unit 104 acquires the detection result as a correction start instruction, and Switch the current mode from the normal mode to the correction mode. Subsequently, as shown in FIG. 7A, the display control unit 106 causes the display screen 30 to simultaneously display the depth object 52 and the four correction objects 54. Thereafter, the user taps one or more of the four correction objects 54 as many as necessary so that the display position of the depth object 52 and the position of the outline of the right hand 2b coincide with each other. For example, when one of the four correction objects 54 is tapped, the offset amount of the display position of the depth object 52 in the correction direction corresponding to the correction object 54 is added by a predetermined value.

By repeating the user's operation as described above, for example, as shown in FIG. 7B, the display position of the depth object 52 and the position of the contour of the right hand 2b can substantially coincide. Thereafter, the user holds the predetermined physical button for a long time. If it is detected that the predetermined physical button has been pressed and held for a predetermined time or more, the correction unit 104 acquires the detection result as a correction end instruction, and the current mode is determined from the correction mode. The mode can be switched to the normal mode. That is, the correction mode ends.

When the display screen 30 shown in FIGS. 7A and 7B is displayed, the depth sensor included in the sensor unit 122 can perform depth sensing in real time. Then, each time a result of the depth sensing is newly obtained, the display control unit 106 is based on the result of the depth sensing newly obtained and the latest offset amount of the display position of the depth object 52 in each direction. Thus, the display position of the depth object 52 may be updated sequentially. Thereby, during the correction mode, the user can confirm in detail whether the offset amount of the display position of the depth object 52 in each direction is appropriate by freely moving the right hand 2b.

-Effect 1
In order to recognize the presence or absence of a tap on the correction object 54, other recognition processing such as speech recognition is not necessary. Therefore, as described above, by using the tap for the correction object 54 as the correction method of the display position of the depth object 52, the load of recognition processing of the correction method by the eyewear 10 can be reduced. Furthermore, compared with, for example, pressing of a predetermined physical button included in the input unit 126, the user can intuitively operate the gesture operation (tap or the like) on the four correction objects 54 vertically and horizontally. In addition, in the gesture operation, there is also an advantage that a desired operation can be performed without depending on the installation location of the predetermined physical button or the like.

-Effect 2
Further, as described above, by using the pressing of the predetermined physical button as a trigger of the mode transition, there is an advantage that the mode is transitioned as the user intends. In general, pressing the predetermined physical button is unlikely to cause a user's erroneous operation. Therefore, according to this transition method, mode transition can be performed as intended by the user.

(2-1-6-2. Modified Example 1)
As a modification, in the examples shown in FIGS. 7A and 7B, instead of the tap on the correction object 54, the display position of the depth object 52 may be changed based on the recognition result of the user's speech. For example, when it is recognized that the user has uttered “right” once, the display control unit 106 controls the display so that the depth object 52 moves to the right in the display screen 30 at the first speed. You may In this case, when it is recognized that the user has uttered “right” once more, the display control unit 106 changes the moving speed of the depth object 52 to a second speed faster than the first speed. You may In addition, when it is recognized that the user has uttered “stop”, the display control unit 106 may stop the movement of the depth object 52.

(2-1-6-3. Modified Example 2)
As another modification, the one or more correction objects may be objects for correcting parameter values in a binary search. For example, the one or more correction objects shift the display position of the depth object on the display unit 124 toward the one end side by a predetermined distance or more with respect to the predetermined direction (for example, the horizontal direction of the display unit 124) Hereinafter, the first object is also referred to) and the result of shifting the display position of the depth object to the opposite end side with respect to the predetermined direction by the predetermined distance or more (hereinafter also referred to as a second object) May be In this case, for example, the display control unit 106 first causes the display unit 124 to simultaneously display the first object and the second object. Subsequently, the display control unit 106 is configured to occlude the corresponding virtual object by the amount of deviation from the target position of either the first object or the second object (that is, the corresponding real object (eg, a hand)). The user is made to select whether the shift amount of the position of the area is smaller. After that, the display control unit 106 changes, for example, the distance between the first object and the second object, the direction in which the first object and the second object shift from the initial display position, and the like each time. Repeat the above process several times.

By using the binary search parameter correction method as described above, it is possible to correct the parameter values regarding the display position of the one or more virtual objects more efficiently. For example, it can be expected that the number of operations performed by the user until the value of the parameter is corrected to an appropriate value can be reduced. As a result, the load on the user may be reduced.

{2-1-7. Communication unit 120}
The communication unit 120 may be configured to include a communication device 166 described later. The communication unit 120 transmits and receives information to and from another device by, for example, wireless communication and / or wired communication. For example, the communication unit 120 can receive various contents from the server 20.

{2-1-8. Storage unit 128}
The storage unit 128 may be configured to include a storage device 164 described later. The storage unit 128 stores various data such as one or more virtual objects and one or more contents, and various software.

<2-2. Flow of processing>
The configuration of the first embodiment has been described above. Next, an example of the flow of processing according to the first embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing the flow of processing according to the first embodiment. Here, an example in which a normal mode (that is, a mode other than the correction mode) is set as the mode in the initial state will be described.

As shown in FIG. 8, first, the sensing result acquisition unit 102 of the eyewear 10 acquires the result of depth sensing by the depth sensor (included in the sensor unit 122), and then, the predetermined result is obtained for the result of the depth sensing. Perform recognition processing of Then, the display control unit 106 causes the display unit 124 to display, for example, one or more virtual objects included in the currently activated content based on the result of the recognition process. For example, the display control unit 106 applies the offset amount (hereinafter also referred to as a correction offset amount) related to the correction of the value of the parameter related to the display of the one or more virtual objects to the recognition result. Are displayed on the display unit 124 (S101).

Thereafter, the correction unit 104 determines whether a correction start instruction (hereinafter also referred to as a calibration start trigger) by the user wearing the eyewear 10 is detected. While the start trigger of the calibration is not detected (S103: No), the eyewear 10 repeats the processing after S101 again.

On the other hand, when the calibration start trigger is detected (S103: Yes), first, the correction unit 104 switches the current mode from the normal mode to the correction mode. Subsequently, the display control unit 106 cancels the display of the one or more virtual objects. Furthermore, the display control unit 106 may cause the display unit 124 to display a display indicating that the mode for correction has been switched. After that, the correction unit 104 sets the current correction offset amount stored in the storage unit 128 as a new offset value (that is, initializes the new offset value) (S105).

Subsequently, the control unit 100 acquires the latest depth sensing result by the depth sensor, and performs the predetermined recognition processing on the depth sensing result.

Subsequently, the display control unit 106 generates the depth object corresponding to the result of the depth sensing by applying the new offset value to the recognition result (that is, the recognition result at the current point in time). Then, the display control unit 106 causes the display unit 124 to display one or more correction objects and the depth object together (S107).

After that, the correction unit 104 determines whether a correction end instruction (hereinafter also referred to as a calibration end trigger) by the user is detected (S109). If the calibration termination trigger is detected (S109: Yes), the correction unit 104 determines the current correction offset amount stored in the storage unit 128 (set in S105 or S115). Change (update) to the new offset value. (In other words, the storage content of the storage unit 128 is updated) (S111). After that, the eyewear 10 repeats the processing after S101 again.

On the other hand, when the termination trigger of the calibration is not detected (S109: No), next, the correction unit 104 instructs the user regarding the change of the new offset value (as instruction information of the user) It is determined whether it has been detected (S113). When the instruction of the user is not detected (S113: No), the eyewear 10 repeats the processing of S107 and thereafter again.

On the other hand, when the instruction of the user is detected (S113: Yes), the correction unit 104 changes (updates) the new offset value by a value corresponding to the detection result (S115). After that, the eyewear 10 repeats the processing after S107 again.

<2-3. Effect>
As described above, the eyewear 10 according to the first embodiment acquires the result of depth sensing of at least one real object corresponding to the viewpoint position of the user wearing the eyewear 10, and The display unit 124 displays at least one correction object for correcting parameters related to display of at least one virtual object based on a result of depth sensing of one real object. Therefore, the user can easily and appropriately correct the value of the parameter related to the display of the at least one virtual object.

<2-4. Application example>
The first embodiment is not limited to the example described above. Hereinafter, application examples of the first embodiment will be described in “2-4-1. Application Example 1” to “2-4-4. Application Example 4”. Note that all components included in the eyewear 10 according to each application are the same as the example shown in FIG. In the following, only components having functions different from those described above will be described, and descriptions of the same content will be omitted.

{2-4-1. Application example 1}
First, Application Example 1 according to the first embodiment will be described. In the above description (for example, FIG. 8 and the like), the example in which the depth sensing result is updated in real time even in the correction mode (that is, the example in which the depth sensing is continued even in the correction mode) has been described.

As described later, in the application example 1, the result of depth sensing is not updated during the correction mode. Furthermore, in the application example 1, the values of the parameters related to the display of the one or more virtual objects are updated according to the difference in the result of the depth sensing of the user's hand before and after the switching between the normal mode and the correction mode. Can.

(2-4-1. Correction section 104)
The correction unit 104 according to the application example 1 includes the first position information of the user's hand when the user's correction start instruction is acquired, and the number of the user's hand when the correction end instruction is acquired. In accordance with the difference from the position information of 2, the correction amount of the parameter related to the display position of the one or more virtual objects (that is, the above-mentioned offset amount for correction) is determined. For example, the first position information of the user's hand is a position corresponding to the result of the depth sensing of the user's hand when the correction start instruction of the user is acquired by the depth sensor (included in the sensor unit 122) It is information. The second position information of the user's hand is position information corresponding to the result of depth sensing of the user's hand when the correction end instruction of the user is acquired by the depth sensor.

In addition, the correction unit 104 (a part of the user's hand at these timings (instead of using the relationship of positional information of the user's hand at the time of acquisition of the correction start instruction and acquisition of the correction end instruction) For example, parameters related to the display position of the one or more virtual objects may be corrected using the relationship of position information of a predetermined finger tip or the like.

Here, the above-mentioned function of the correction unit 104 will be described in more detail with reference to FIGS. 9A to 9C. In addition, below, as shown, for example in FIG. 3, it presupposes that the shielding area | region of a certain virtual object (illustration omitted) by the user's hand 2 is shifted and displayed from the position of a target.

As shown in FIG. 9A, first, the user performs a correction start instruction (for example, an utterance of a predetermined voice command (“Start calibration” or the like)). Thereafter, when the correction unit 104 acquires the correction start instruction, the position information 60a corresponding to the result of the depth sensing of the user's hand 2 (for example, a fingertip) by the depth sensor is the first position of the user's hand. Acquire as information. Then, the correction unit 104 switches the current mode from the normal mode to the correction mode. As described above, in the application example 1, the result of depth sensing is not updated during the correction mode.

Thereafter, the user moves the hand 2 so that the shielding area of the virtual object (not shown) by the user's hand 2 substantially matches the position of the target. Then, as shown in FIG. 9B, the user performs a correction end instruction (for example, an utterance of a predetermined voice command (“End calibration” or the like)). After that, as illustrated in FIG. 9B, the correction unit 104 performs positional information 60b corresponding to the result of the depth sensing of the user's hand 2 (for example, a fingertip) by the depth sensor at the time of obtaining the correction end instruction. Acquire as second position information of the user's hand. Next, the correction unit 104 calculates the difference ("d" shown in FIG. 9B) between the position information 60a and the position information 60b, and determines the above-mentioned correction offset amount according to the difference. Then, the correction unit 104 corrects the value of the parameter related to the display position of the one or more virtual objects by the determined offset amount. As a result, as shown in FIG. 9C, the position of the hand depth object 50 substantially matches the position of the target. Therefore, the shielded area of the virtual object by the user's hand 2 substantially matches the position of the target. After that, the correction unit 104 switches the current mode from the correction mode to the normal mode.

(2-4-1. Display control unit 106)
-Display example 1
Next, the function of the display control unit 106 according to Application Example 1 will be described. Generally, for example, when the user moves the hand from the state where the shielding area of the virtual object by the user's hand is shifted as shown in FIG. 3 to the target position (as shown in FIG. 2) The user load is high because there are few clues for the user. For example, it may lead to an increase in the number of trials and an increase in calibration time.

Therefore, the display control unit 106 according to the application example 1 may cause the display unit 124 to display only the outer peripheral portion (for example, an outline) of the hand at the time of obtaining the correction start instruction, for example.
Alternatively, the display control unit 106 may cause the display unit 124 to display an image in which the edge of the captured image of the hand at the time of acquisition of the correction start instruction is emphasized. This may reduce the load on the user as the user has more clues to be aligned (e.g., outlines of the hand, eyebrows on the hand, etc.).

-Display example 2
In addition, moving the hand from the state where the shielding area of the virtual object by the user's hand is shifted to the target position (strict position) usually causes a heavy load on the user. Therefore, the display control unit 106 may control the display so that the shielding area of the virtual object by the user's hand is blurred, for example, at the time of obtaining the correction start instruction. This can inform the user that they do not have to be exactly the same.

-Display example 3
Furthermore, as in the example shown in FIGS. 9A to 9C, the parameter related to the display position of the one or more virtual objects is corrected using the positional relationship of a part of the user's hand (for example, a fingertip of a predetermined finger). In the case where the display control unit 106 determines that only the part is illuminated, the display control unit 106 may cause the display unit 124 to highlight the part.

-Modified Example Note that depending on the positional relationship between the depth sensor and the user's eyes, there may exist an area in the real space that can be viewed by the user but is difficult to sense by the depth sensor. Therefore, the display control unit 106 may control the display by the display unit 124 so that a space in which the accuracy of the sensing by the depth sensor is high is noticeable. For example, the display control unit 106 may display the display area corresponding to the space in a predetermined display color.

{2-4-2. Application example 2}
Next, application example 2 according to the first embodiment will be described. As described later, in the second application, parameters relating to the display position of one or more virtual objects can be corrected based on a change in position information (for example, the position of the head) of the user.

(2-4-2-1. Correction unit 104)
The correction unit 104 according to the application example 2 displays one or more depth objects corresponding to the result of the depth sensing acquired by the sensing result acquisition unit 102 and when the correction start instruction of the user is acquired. Correction of parameters related to the display position of the one or more virtual objects according to the difference between the first position information of the user and the second position information of the user when the correction end instruction of the user is acquired Determine the amount (that is, the offset amount for correction). For example, the first position information of the user is position information of the eyewear 10 when the correction start instruction of the user is acquired by the depth sensor (included in the sensor unit 122). The second position information of the user is the position information of the eyewear 10 when the correction end instruction of the user is acquired by the depth sensor. For example, the amount of change in the position information of the eyewear 10 sensed between the time when the correction start instruction is obtained and the time when the correction end instruction is obtained and the first position information. The second position information of the user may be identified.

Specific Example Now, with reference to FIGS. 10A and 10B, the above functions will be described in more detail. As shown in FIG. 10A, first, it is assumed that the user wearing the eyewear 10 first issues a correction start instruction (for example, an utterance of a predetermined voice command (“Start calibration.” Or the like)). Thereafter, the correction unit 104 switches the current mode from the normal mode to the correction mode. In this case, as shown in FIG. 10A, the display control unit 106 sets the contour of each real object corresponding to the result of the depth sensing by the depth sensor (included in the sensor unit 122) at the time of obtaining the correction start instruction. The depth object 52 shown is displayed on the display screen 30. That is, the display control unit 106 emphasizes an edge (as a depth object 52) and causes the display unit 124 to display the result of the depth sensing of the entire environment by the depth sensor. In Application Example 2, the result of depth sensing is not updated during the correction mode.

Thereafter, as shown in FIG. 10B, the user moves or moves the head in the room such that the depth object 52 and the position of each real object in the display screen 30 substantially coincide with each other. Then, the user performs a correction end instruction (for example, an utterance of a predetermined voice command (“End calibration.” Or the like)). Next, the correction unit 104 calculates the difference between the position information of the user at the time of acquisition of the correction start instruction and the position information of the user at the time of acquisition of the correction end instruction, and according to the difference. The above-mentioned offset amount for correction is determined. Then, the correction unit 104 corrects the value of the parameter related to the display position of the one or more virtual objects by the determined correction offset amount. After that, the correction unit 104 switches the current mode from the correction mode to the normal mode.

As described above, according to Application Example 2, the parameter related to the display position of the one or more virtual objects is corrected based on the detection result of the change of the user's position information or the change of the user's head movement. This makes it possible to correct the displacement of the shielding in the entire field of view of the user. Therefore, it is particularly effective in the case where occlusion of a virtual object by an object other than the user's hand occurs.

{2-4-3. Application example 3}
Next, Application Example 3 according to the first embodiment will be described. Usually, when the real object exists near the eyewear 10 and when it exists far from the eyewear 10, the shift amount of the shielding area of the virtual object by the real object may be different.

Therefore, the correction unit 104 according to the application example 3 includes a first real object existing within a predetermined distance from the eyewear 10 and a second real object located farther than the predetermined distance from the eyewear 10 , The parameters relating to the display of one or more virtual objects may be separately corrected. For example, the correction unit 104 may use one or more virtual object display parameters based on the result of the first real object depth sensing and one or more virtual objects based on the second real object depth sensing result. Parameters related to display are separately corrected based on the user's instruction information described above. Thereby, according to the distance of each real object from eyewear 10, the gap amount of the occlusion field of the virtual object by each real object can be amended appropriately.

{2-4-4. Application example 4}
Further, in the above description, an example in which the correction unit 104 separately corrects values of parameters regarding display positions of the one or more virtual objects with respect to the right side display unit 124a and the left side display unit 124b has been mainly described. That is, the example in which the correction unit 104 two-dimensionally corrects the parameter has been described.

Next, Application 4 according to the first embodiment will be described. According to this application example 4, it is possible to three-dimensionally correct the values of the parameters related to the display position of the one or more virtual objects. For example, as shown in FIG. 11, first, the control unit 100 determines the three-dimensional shape of the real object based on the sensing result of a certain real object (for example, the user's hand) by the depth sensor (included in the sensor unit 122). To generate mesh data 70 of Then, the display control unit 106 causes the display unit 124 to display the mesh data 70 during the correction mode. The mesh data 70 is an example of a depth object according to the present disclosure.

When the user's instruction information for moving the mesh data 70 is obtained during the correction mode, as shown in FIG. 11, the correction unit 104 performs the mesh information 70 in the camera coordinate system as the instruction information. For example, it moves to the front and rear, right and left based on. Then, the correction unit 104 corrects the value of the parameter related to the display position of the one or more virtual objects according to the movement amount of the mesh data 70.

For example, the correction unit 104 changes the value of one or more parameters in the matrix M in the above equation (1) based on the instruction information of the user to obtain the parameters related to the display position of the one or more virtual objects. Change the value

<< 3. Second embodiment >>
The first embodiment has been described above. As described above, the eyewear 10 according to the first embodiment may correct the value of the parameter regarding the display position of one or more virtual objects.

<3-1. Background>
Next, a second embodiment will be described. First, the background that led to the creation of the second embodiment will be described. When a stereo camera is used as the depth sensor, the eyewear 10 is the result of depth sensing by the left camera of the stereo cameras (hereinafter also referred to as left image) and the depth sensing by the right camera of the stereo cameras. The depth values of one or more real objects may be measured using the result (hereinafter also referred to as right image) and a predetermined algorithm. Here, the predetermined algorithm includes estimating which pixel in the right image corresponds to a pixel in the left image. Here, several hundreds of types of depth parameters may be used in the predetermined algorithm. The plurality of types of depth parameters can define, for example, a search range, a threshold, and the like in the predetermined algorithm. The plurality of types of depth parameters are examples of the “parameter related to the display of one or more virtual objects” according to the present disclosure.

More specifically, the plurality of types of depth parameters may include a plurality of types of parameters regarding calculation of a matching score between the result of depth sensing by the left depth camera and the result of depth sensing by the right depth camera. For example, the matching score may be calculated using a plurality of types of calculation methods. The plurality of types of parameters related to calculation of the matching score may include a first parameter related to the application ratio of each of the plurality of types of calculation methods, and a second parameter related to the threshold value of the matching score. For example, the first parameter is a matching score calculation ratio parameter, and the second parameter is a disparity penalty parameter. The matching score calculation ratio parameter is a parameter relating to the blending ratio of the score based on the difference absolute value at the time of calculation of the matching score and the score based on the Hamming distance. The disparity penalty parameter is a threshold of the matching score.

Generally, depending on the value of each of the plurality of types of depth parameters, for example, the level of accuracy of the sensing result, the type of excellent / unfavorable condition, and the like may change. For example, in the case where a virtual object is occluded by the user's hand, among the plurality of types of depth parameters, types of depth parameters that can accurately recognize the three-dimensional shape of the hand are various environmental conditions (for example, It may differ depending on the real object located behind the hand, the brightness of the environment, the color of the hand, and the like. FIG. 12 is a diagram (Table 80) illustrating classification examples of types of cases in which the accuracy of the depth sensing result is low. As shown in FIG. 12, there may be multiple types of cases where the accuracy of the depth sensing result is low.

Furthermore, even if the result of depth sensing is the same, how to feel the quality of the result of depth sensing may differ for each user. In the example shown in FIG. 12, a user may feel more uncomfortable with “protruding” than “performing” (for example, a high frequency component in the temporal direction that is noticeable around the outline of the object). Also, another user may feel more uncomfortable with "perforated" than with "outside". Therefore, it is desirable that the values of each of the plurality of types of depth parameters can be set appropriately and easily according to the sense (preference) of the user who uses the eyewear 10.

As described later, according to the second embodiment, the user using the eyewear 10 can easily set desired values for each of the plurality of types of depth parameters.

<3-2. Configuration>
Next, the configuration of the eyewear 10 according to the second embodiment will be described. FIG. 13 is a block diagram showing an example of the functional configuration of the eyewear 10 according to the second embodiment. As shown in FIG. 13, the eyewear 10 according to the second embodiment further includes a parameter set DB 130 in the storage unit 128 as compared to the first embodiment shown in FIG. 4. In the following, only the contents different from the first embodiment will be described, and the description of the same contents will be omitted.

{3-2-1. Parameter set DB130}
The parameter set DB 130 can store in advance a combination of a plurality of different types of values of the plurality of types of depth parameters (hereinafter also referred to as “a plurality of parameter sets”). The plurality of types of depth parameters include, for example, a matching score calculation ratio parameter and a disparity penalty parameter. Furthermore, for each parameter set, the ratio of each of these two parameters may be different.

FIG. 14 is a diagram showing the relationship between the ratio of each of the matching score calculation ratio parameter and the disparity penalty parameter, and the type of the case in which the accuracy of the depth sensing result is low as shown in FIG. 12. As shown in FIG. 14, when the value of the matching score calculation ratio parameter changes, the occurrence frequency of “punch” and “perforation” (shown in FIG. 12) may change. In addition, when the value of the parallax penalty parameter changes, the occurrence frequency of "outside" and "perforation" may change.

Further, FIG. 14 shows an example of two types of parameter sets 82. As shown in FIG. 14, in the parameter set 82a, the value of the matching score calculation ratio parameter is small, and the value of the disparity penalty parameter is large. Therefore, the parameter set 82 a is desirable for a user who feels more uncomfortable than “overwhelming” than “overwhelming”. Further, in the parameter set 82b, the value of the matching score calculation ratio parameter is large, and the value of the disparity penalty parameter is small. Therefore, the parameter set 82 b is desirable for a user who feels more uncomfortable than “perforated” than “abrupt”.

{3-2-2. Display control unit 106}
The display control unit 106 according to the second embodiment is configured such that the left image and the right side sensed by the stereo camera (included in the sensor unit 122) for each of the plurality of types of parameter sets stored in the parameter set DB 130 Each of one or more correction objects is displayed on the display unit 124 as an object indicating the result of using the parameter set for an image.

For example, each of the plurality of types of parameter sets may be uniquely associated with each of one or more correction objects. In this case, the display control unit 106 first obtains the result of the depth sensing of the hand of the user wearing the eyewear 10 when the parameter set associated with each of the one or more correction objects is used. And a captured image of the user's hand are combined to generate a moving image. Then, the display control unit 106 causes the display unit 124 to display each generated moving image.

FIG. 15A is a diagram showing a display example of a moving image (moving image 90) corresponding to each parameter set. FIG. 15B is an enlarged view of the moving image 90c shown in FIG. 15A. As shown in FIG. 15A, the display control unit 106 causes the display screen 30 to simultaneously display a predetermined number of moving images 90, such as six. As mentioned above, the parameter sets associated with each of the predetermined number of animations 90 are different from one another.

As shown in FIG. 15B, in each moving image 90, a hand-captured image 900 and an image of occlusion of a virtual object by the hand are combined. A noise 902 corresponding to the result of depth sensing when the parameter set corresponding to the moving image 90 is used is shown in the image of the shielding.

For example, the display control unit 106 causes the display screen 30 to simultaneously display a predetermined number of moving images 90 out of all the generated moving images 90 (as correction objects). As an example, the display control unit 106 first extracts a predetermined number of moving images 90 from all the moving images 90, and causes the display screen 30 to simultaneously display the predetermined number of moving images 90. Then, whenever any of the predetermined number of moving images 90 is selected by the user, the display control unit 106 simultaneously displays a predetermined number of undisplayed moving images 90 among all the moving images 90. Display on 30 The display control unit 106 can repeat this process until the undisplayed moving image 90 disappears.

Alternatively, the display control unit 106 may cause the display unit 124 to display a predetermined number of moving images 90 different from one another among all the moving images 90 while switching at predetermined time intervals.

The one or more correction objects may exist by the number of all parameter sets stored in the parameter set DB 130, or the number of partial parameter sets stored in the parameter set DB 130. It may only exist.

{3-2-3. Correction unit 104}
The correction unit 104 according to the second embodiment uses the parameter set associated with each of at least one correction object selected by the user among the one or more correction objects displayed on the display unit 124. Based on the parameters related to the display of one or more virtual objects.

For example, when one or more correction objects are displayed on the display unit 124 and at least one correction object is selected from the one or more correction objects, the correction unit 104 first Information indicating the type of the corresponding correction object selected by the user is acquired as instruction information of the user. Then, based on the parameter set associated with each of the corresponding correction objects selected by the user indicated by the user's instruction information, the correction unit 104 indicates the parameter related to the display of the one or more virtual objects. to correct. For example, the correction unit 104 corrects the values of the parameters related to the display of the one or more virtual objects to values corresponding to the values of the individual depth parameters included in the parameter set associated with each of the corresponding correction objects. Do.

<3-3. Flow of processing>
The configuration of the second embodiment has been described above. Next, an example of the flow of processing according to the second embodiment will be described with reference to FIG. 16 and FIG. 16 and 17 are flowcharts showing a part of the flow of processing according to the second embodiment.

As shown in FIG. 16, first, the control unit 100 of the eyewear 10 performs “correction processing regarding display position of virtual object” (S201). The “correction process regarding the display position of the virtual object” may be substantially the same as the process flow according to the first embodiment (that is, the process of S101 to S115). Thereby, the value of the parameter regarding the display position of the virtual object can be appropriately corrected.

Thereafter, the display control unit 106 causes the display unit 124 to display a character string for instructing the user to move the hand. For example, the display control unit 106 causes the display unit 124 to display character strings such as “Adjust the recognition accuracy. Move your hand in front of your eyes.” And “Start” (S203).

Subsequently, the stereo camera (included in the sensor unit 122) starts depth sensing of the user's hand according to the control of the control unit 100 (S205).

Thereafter, the user moves the hand according to the instruction displayed in S203 (S207).

After that, when a predetermined time has elapsed from S205, the display control unit 106 causes the display unit 124 to display a character string (for example, "stop" or the like) for instructing to stop moving the hand. Then, the stereo camera ends the depth sensing of the user's hand according to the control of the control unit 100 (S209).

Thereafter, the display control unit 106 causes each of the plurality of types of parameter sets stored in the parameter set DB 130 to be a result of depth sensing (one or more left images and one or more right images) between S205 and S209. A moving image in which the result of application to the image and the captured image of the user's hand are combined is generated (S211).

Here, the flow of processing after S211 will be described with reference to FIG. As shown in FIG. 17, after S211, the display control unit 106 extracts a plurality of undisplayed moving images from all the moving images generated in S211 (S221).

Subsequently, the display control unit 106 causes the display unit 124 to simultaneously display the plurality of moving images extracted in S221 (S223).

Thereafter, the user selects one of the plurality of moving images being displayed. Then, the correction unit 104 acquires identification information of the selected moving image as instruction information of the user (S225).

Subsequently, the correction unit 104 determines whether the display of all the moving images generated in S211 is finished (S227). If there is an undisplayed moving image (S227: No), the processing after S221 is repeated again.

On the other hand, when the display of all the moving images is completed (S227: Yes), the correction unit 104 uses the parameter set corresponding to each of the instruction information of all the users acquired each time S225 is performed. And correct parameters related to display of one or more virtual objects (S229).

(Modification)
The flow of processing according to the second embodiment is not limited to the example described above. For example, the above-described “correction process regarding display position of virtual object” may be performed last (that is, after S229) instead of being performed first.

<3-4. Effect>
As described above, the eyewear 10 according to the second embodiment includes at least one of the one or more correction objects displayed on the display unit 124 selected by the user wearing the eyewear 10. The parameters related to the display of one or more virtual objects are corrected based on the parameter set associated with each of the correction objects. For example, the eyewear 10 uses the value of each depth parameter included in the parameter set associated with each of the at least one correction object selected by the user when displaying the one or more virtual objects. Set as the value of each depth parameter. Therefore, the user can easily set desired values for each of the plurality of types of depth parameters.

<3-5. Modified example>
The second embodiment is not limited to the example described above. In general, there may be more than one type of parameter (occluding filter parameter) in the algorithm that produces the occluding mesh, eg, meshing or prediction. Therefore, the eyewear 10 may perform substantially the same processing as the correction processing of the value of each depth parameter described above in order to correct one or more shielding filter parameters. Here, the one or more occlusion filter parameters are examples of the “parameter related to the display of one or more virtual objects” according to the present disclosure.

<< 4. Hardware configuration >>
Next, a hardware configuration example of the eyewear 10 common to each embodiment will be described with reference to FIG. As illustrated in FIG. 18, the eyewear 10 includes a CPU 150, a read only memory (ROM) 152, a random access memory (RAM) 154, a bus 156, an interface 158, an input device 160, an output device 162, a storage device 164, and , Communication device 166.

The CPU 150 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the eyewear 10 according to various programs. The CPU 150 also realizes the function of the control unit 100 in the eyewear 10. The CPU 150 is configured of a processor such as a microprocessor.

The ROM 152 stores programs used by the CPU 150, control data such as calculation parameters, and the like.

The RAM 154 temporarily stores, for example, a program executed by the CPU 150, data in use, and the like.

The bus 156 is configured of a CPU bus and the like. The bus 156 connects the CPU 150, the ROM 152, and the RAM 154 to one another.

The interface 158 connects the input device 160, the output device 162, the storage device 164, and the communication device 166 to the bus 156.

The input device 160 includes, for example, an input unit such as a touch panel, a button, a switch, a lever, and a microphone for inputting information by a user, and an input control circuit that generates an input signal based on an input by the user and outputs it to the CPU 150 Configured The input device 160 can function as the input unit 126.

The output device 162 includes a display such as an LCD or an OLED, or a display such as a projector. Output device 162 may also include an audio output device such as a speaker. The output device 162 can function as the display unit 124.

The storage device 164 is a device for storing data. The storage device 164 includes, for example, a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, or a deletion device that deletes data recorded in the storage medium. The storage device 164 can function as the storage unit 128.

The communication device 166 is a communication interface configured by, for example, a communication device (for example, a network card or the like) for connecting to the communication network 22 or the like. Further, the communication device 166 may be a wireless LAN compatible communication device, an LTE (Long Term Evolution) compatible communication device, or a wire communication device performing communication by wire. The communication device 166 can function as the communication unit 120.

<< 5. Modified example >>
The preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the present disclosure is not limited to such examples. It is obvious that those skilled in the art to which the present disclosure belongs can conceive of various changes or modifications within the scope of the technical idea described in the claims. It is naturally understood that these also fall within the technical scope of the present disclosure.

<5-1. Modification 1>
In each embodiment mentioned above, although the information processor concerning this indication explained an example which is eyewear 10, it is not limited to this example. For example, when a device of another type has functions such as the sensing result acquisition unit 102, the display control unit 106, and the correction unit 104 according to each embodiment, the information processing device is the other type of device. It may be an apparatus. As one example, the other type of device may be the server 20. Alternatively, the other type of device may be a general-purpose PC (Personal Computer), a tablet type terminal, a game machine, a mobile phone such as a smartphone, a portable music player, a speaker, a projector, a wearable device such as a smart watch or earphone, It may be a device (such as a car navigation device) or a robot (such as a humanoid robot or an autonomous vehicle).

<5-2. Modification 2>
The steps in the process flow of each embodiment described above may not necessarily be processed in the order described. For example, each step may be processed in an appropriate order. Also, each step may be processed partially in parallel or individually instead of being processed chronologically. Also, some of the described steps may be omitted or additional steps may be added.

Moreover, according to each embodiment described above, it is also possible to provide a computer program for causing hardware such as the CPU 150, the ROM 152, and the RAM 154 to exhibit the same function as each component of the eyewear 10 according to each embodiment. is there. Also, a storage medium in which the computer program is recorded may be provided.

In addition, the effects described in the present specification are merely illustrative or exemplary, and not limiting. That is, the technology according to the present disclosure can exhibit other effects apparent to those skilled in the art from the description of the present specification, in addition to or instead of the effects described above.

The following configurations are also within the technical scope of the present disclosure.
(1)
An acquisition unit for acquiring a result of depth sensing of a real object corresponding to the viewpoint position of the user;
A display control unit that causes a display unit corresponding to the user to display a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object;
An information processing apparatus comprising:
(2)
The information processing apparatus according to (1), further including: a correction unit that corrects a parameter related to display of the virtual object based on instruction information of the user on the correction object.
(3)
The information processing apparatus according to (2), wherein the user instruction information is information acquired while the depth sensing result is acquired and the correction object is displayed.
(4)
The information processing apparatus according to (3), wherein the real object is located in a space corresponding to a field of view of the user.
(5)
The display control unit causes the display unit to display a depth object indicating a shape of the real object, which is specified based on a result of depth sensing of the real object, together with the correction object. The information processing apparatus according to
(6)
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The real object includes at least the user's hand,
The information processing apparatus according to (5), wherein the instruction information of the user includes a recognition result of the movement of the user's hand with respect to the correction object.
(7)
The correction object includes a plurality of correction objects
The plurality of correction objects respectively correspond to different correction directions with respect to the display position of the virtual object,
The information processing apparatus according to (6), wherein the plurality of correction objects are objects for instructing a correction amount of a display position of the virtual object in the correction direction.
(8)
The information processing apparatus according to (4), wherein the correction object includes a depth object indicating a shape of the real object, which is identified based on a result of depth sensing of the real object.
(9)
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The real object includes at least the user's hand,
The user's instruction information includes a correction start instruction for instructing start of correction of parameters related to display of the virtual object, and a correction end instruction for instructing end of correction of parameters related to display of the virtual object. ,
The correction unit is configured to calculate a difference between the first position information of the user's hand when the correction start instruction is acquired and the second position information of the user's hand when the correction end instruction is acquired. The information processing apparatus according to (8), wherein the correction amount of the parameter related to the display position of the virtual object is determined according to.
(10)
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The information processing apparatus according to (8) or (9), wherein the instruction information of the user is information based on line-of-sight information of the user.
(11)
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The user's instruction information includes a correction start instruction for instructing start of correction of parameters related to display of the virtual object, and a correction end instruction for instructing end of correction of parameters related to display of the virtual object. ,
The correction unit is configured to display, after the display of the correction object, the first position information of the user when the correction start instruction is acquired and the number of the user when the correction end instruction is acquired. The information processing apparatus according to (8), wherein the correction amount of the parameter related to the display position of the virtual object is determined according to the difference between the position information and the second position information.
(12)
The correction object includes a plurality of correction objects
The depth sensing result includes a plurality of depth sensing results based on different combinations of a plurality of types of parameters related to depth sensing,
Each of the plurality of correction objects indicates a corresponding one of the plurality of depth sensing results,
The information according to any one of (3) to (11), wherein the user's instruction information is information on at least one correction object selected by the user from the plurality of correction objects. Processing unit.
(13)
The information processing according to (12), wherein the correction unit corrects a parameter related to display of the virtual object based on a combination of the plurality of parameters associated with the selected at least one correction object. apparatus.
(14)
The plurality of types of parameters are a result of first depth sensing of the real object by the left depth camera corresponding to the viewpoint position of the user, and a second of the real object by the right depth camera corresponding to the left depth camera The information processing apparatus according to (13), including a plurality of types of parameters related to calculation of a matching score with the result of depth sensing.
(15)
The matching score is calculated using a plurality of types of calculation methods.
The plurality of types of parameters related to calculation of the matching score are a first parameter related to an application ratio of each of the plurality of types of calculation methods at the time of calculation of the matching score, and a second parameter related to a threshold of the matching score The information processing apparatus according to (14), including
(16)
The real object includes at least the user's hand,
The correction object is a plurality of moving images in which a plurality of depth sensing results on the real object when the different combinations are used with respect to the plurality of types of parameters and a captured image of the user's hand are combined The information processing apparatus according to any one of (13) to (15), including
(17)
The information processing apparatus according to any one of (13) to (16), wherein the display control unit causes the display unit to simultaneously display the plurality of correction objects.
(18)
The correction object includes at least a first correction object and a second correction object,
When the correction object is displayed, the display control unit controls the display unit to display the first correction object and the display unit to display the second correction object for a predetermined time. The information processing apparatus according to any one of (13) to (17), wherein switching is performed at intervals.
(19)
Obtaining a result of depth sensing of a real object corresponding to the viewpoint position of the user;
Displaying a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object on a display unit corresponding to the user;
Information processing methods, including:
(20)
Computer,
An acquisition unit for acquiring a result of depth sensing of a real object corresponding to the viewpoint position of the user;
A display control unit that causes a display unit corresponding to the user to display a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object;
Program to function as.

10 eyewear 20 server 22 communication network 100 control unit 102 sensing result acquisition unit 104 correction unit 106 display control unit 120 communication unit 122 sensor unit 124 display unit 126 input unit 128 storage unit 130 parameter set DB

Claims

An acquisition unit for acquiring a result of depth sensing of a real object corresponding to the viewpoint position of the user;
A display control unit that causes a display unit corresponding to the user to display a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object;
An information processing apparatus comprising:
The information processing apparatus according to claim 1, further comprising a correction unit configured to correct a parameter related to display of the virtual object based on instruction information of the user on the correction object.
The information processing apparatus according to claim 2, wherein the user instruction information is information acquired while the result of the depth sensing is acquired and the correction object is displayed.
The information processing apparatus according to claim 3, wherein the real object is located in a space corresponding to a field of view of the user.
The display control unit causes the display unit to display a depth object indicating a shape of the real object, which is specified based on a result of depth sensing of the real object, together with the correction object. Information processor as described.
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The real object includes at least the user's hand,
The information processing apparatus according to claim 5, wherein the instruction information of the user includes a recognition result of the movement of the user's hand with respect to the correction object.
The correction object includes a plurality of correction objects
The plurality of correction objects respectively correspond to different correction directions with respect to the display position of the virtual object,
The information processing apparatus according to claim 6, wherein the plurality of correction objects are objects for instructing a correction amount of a display position of the virtual object in the correction direction.
The information processing apparatus according to claim 4, wherein the correction object includes a depth object indicating a shape of the real object, which is identified based on a result of depth sensing of the real object.
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The real object includes at least the user's hand,
The user's instruction information includes a correction start instruction for instructing start of correction of parameters related to display of the virtual object, and a correction end instruction for instructing end of correction of parameters related to display of the virtual object. ,
The correction unit is configured to calculate a difference between the first position information of the user's hand when the correction start instruction is acquired and the second position information of the user's hand when the correction end instruction is acquired. The information processing apparatus according to claim 8, wherein the correction amount of the parameter related to the display position of the virtual object is determined in accordance with.
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The information processing apparatus according to claim 8, wherein the instruction information of the user is information based on line-of-sight information of the user.
The parameter related to display of the virtual object includes a parameter related to the display position of the virtual object,
The user's instruction information includes a correction start instruction for instructing start of correction of parameters related to display of the virtual object, and a correction end instruction for instructing end of correction of parameters related to display of the virtual object. ,
The correction unit is configured to display, after the display of the correction object, the first position information of the user when the correction start instruction is acquired and the number of the user when the correction end instruction is acquired. The information processing apparatus according to claim 8, wherein the correction amount of the parameter related to the display position of the virtual object is determined according to a difference between the position information and the second position information.
The correction object includes a plurality of correction objects
The depth sensing result includes a plurality of depth sensing results based on different combinations of a plurality of types of parameters related to depth sensing,
Each of the plurality of correction objects indicates a corresponding one of the plurality of depth sensing results,
The information processing apparatus according to claim 3, wherein the instruction information of the user is information on at least one correction object selected by the user from among the plurality of correction objects.
The information processing apparatus according to claim 12, wherein the correction unit corrects a parameter related to display of the virtual object based on a combination of the plurality of parameters associated with the selected at least one correction object. .
The plurality of types of parameters are a result of first depth sensing of the real object by the left depth camera corresponding to the viewpoint position of the user, and a second of the real object by the right depth camera corresponding to the left depth camera The information processing apparatus according to claim 13, comprising a plurality of types of parameters related to calculation of a matching score with a result of depth sensing of.
The matching score is calculated using a plurality of types of calculation methods.
The plurality of types of parameters related to calculation of the matching score are a first parameter related to an application ratio of each of the plurality of types of calculation methods at the time of calculation of the matching score, and a second parameter related to a threshold of the matching score The information processing apparatus according to claim 14, comprising:
The real object includes at least the user's hand,
The correction object is a plurality of moving images in which a plurality of depth sensing results on the real object when the different combinations are used with respect to the plurality of types of parameters and a captured image of the user's hand are combined The information processing apparatus according to claim 13, comprising:
The information processing apparatus according to claim 13, wherein the display control unit causes the display unit to simultaneously display the plurality of correction objects.
The correction object includes at least a first correction object and a second correction object,
When the correction object is displayed, the display control unit controls the display unit to display the first correction object and the display unit to display the second correction object for a predetermined time. The information processing apparatus according to claim 13, wherein switching is performed at intervals.
Obtaining a result of depth sensing of a real object corresponding to the viewpoint position of the user;
Displaying a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object on a display unit corresponding to the user;
Information processing methods, including:
Computer,
An acquisition unit for acquiring a result of depth sensing of a real object corresponding to the viewpoint position of the user;
A display control unit that causes a display unit corresponding to the user to display a correction object for correcting a parameter related to display of a virtual object based on a result of depth sensing of the real object;
Program to function as.