WO2017081915A1

WO2017081915A1 - Image processing device, image processing method and program

Info

Publication number: WO2017081915A1
Application number: PCT/JP2016/075878
Authority: WO
Inventors: 浩尚後藤
Original assignee: ソニー株式会社
Priority date: 2015-11-10
Filing date: 2016-09-02
Publication date: 2017-05-18
Also published as: JP2017091190A

Abstract

The present invention improves visibility. An image processing device provided with a visual field information acquisition unit, an eyeball information acquisition unit, and a display control unit. The visual field information acquisition unit acquires visual field information pertaining to an image included in the visual field of a person on whom the image processing device is worn. The eyeball information acquisition unit acquires eyeball information pertaining to the eyeball of the person. The display control unit exercises control for making an object to be displayed displayed superimposed on the image included in the visual field of the person. The display control unit also exercises control for changing the display mode of the object to be displayed in the image included in the visual field of the person on the basis of the visual field information and the eyeball information.

Description

Image processing apparatus, image processing method, and program

This technology relates to an image processing apparatus. Specifically, the present invention relates to an image processing apparatus and an image processing method that handle an image to be displayed, and a program that causes a computer to execute the method.

Conventionally, there is an image processing apparatus that provides an image to a user while being worn on a part of the user's body. For example, there is a glasses-type wearable device that can display a display target object on an external object.

Also, for example, a technique for displaying a display target object on coordinates fixed with respect to an external object has been proposed. For example, a virtual image display that detects a change in the posture of the wearer's head and adjusts the screen position of the image based on the virtual image to change in a direction opposite to the direction in which the posture of the wearer changes. An apparatus has been proposed (see, for example, Patent Document 1).

JP 2013-225042 A

In the above-described conventional technology, an image based on a virtual image can be adjusted according to a change in the posture of the wearer of the virtual image display device by adjusting the posture so that the posture of the wearer is changed in the opposite direction. .

Here, it is also assumed that the relationship between the eyeball of the wearer wearing the virtual image display device and the display unit of the virtual image display device changes. In this case, it is preferable to improve the visibility in consideration of the relationship between the eyeball of the wearer and the display unit of the virtual image display device.

This technology was created in view of such a situation, and aims to improve visibility.

The present technology has been made to solve the above-described problems, and a first aspect of the present technology is view information acquisition for acquiring view information related to an image included in the view of a person wearing the image processing apparatus. An eyeball information acquisition unit that acquires eyeball information related to the eyeball of the person, a display target object superimposed on an image included in the field of view, and the field of view based on the field of view information and the eyeball information. An image processing apparatus including a display control unit that performs control to change a display mode of the display target object in an included image, an image processing method thereof, and a program for causing a computer to execute the method. This brings about the effect | action of changing the display mode of the display target object in the image contained in a visual field based on visual field information and eyeball information.

In the first aspect, the eyeball information acquisition unit may determine the position of the eyeball of the person and the line of sight of the eyeball based on images generated by a plurality of imaging units provided in the eyeball direction of the person. The eyeball information may be acquired by detecting the direction. Thereby, based on the image produced | generated by the several imaging part provided in the person's eyeball direction, the effect | action of acquiring eyeball information by detecting the position of the person's eyeball and the gaze direction of the eyeball is brought about.

In the first aspect, the image processing apparatus further includes a display unit that provides an image included in the field of view of the person to the eyes of the person, and the plurality of imaging units includes the display unit that displays the display target object. You may make it provide in the edge part in a display surface. This brings about the effect | action that the some imaging part provided in the edge part in the display surface of the display part which displays a display target object is used.

In the first aspect, the visual field information acquisition unit extracts feature points of an object included in the visual field based on images generated by a plurality of imaging units provided in the visual line direction of the person. Information regarding the position of the object included in the field of view may be acquired as the field of view information by performing the feature point extraction processing to be performed and the depth detection processing for detecting the depth of the object included in the field of view. Thus, by performing feature point extraction processing and depth detection processing based on images generated by a plurality of imaging units provided in the line-of-sight direction of a person, information on the position of an object included in the visual field can be obtained. As an effect to get as.

In the first aspect, the image processing apparatus further includes a display unit that provides an image included in the field of view of the person to the eyes of the person, and the display control unit includes position information regarding the position of the display unit and the field of view. You may make it perform control which changes the said display mode based on information and the said eyeball information. This brings about the effect | action of changing a display mode based on the positional information regarding the position of a display part, visual field information, and eyeball information.

In the first aspect, the display unit is arranged on the display surface based on a refraction image output from an image output unit arranged at an edge of the display surface that transmits an image included in the field of view of the person. The refraction display unit that displays the display target object, wherein the display control unit is based on position information regarding a display position where the display target object is virtually displayed, the visual field information, and the eyeball information. You may make it perform control which changes the said display mode. This brings about the effect | action that a display mode is changed based on the positional information regarding the display position which displays a display target object virtually, visual field information, and eyeball information.

In the first aspect, the display control unit changes the display mode by changing at least one of a display position, a display angle, and a display size of the display target object on the display surface of the display unit. May be changed. Accordingly, there is an effect that the display mode is changed by changing at least one of the display position, the display angle, and the display size of the display target object on the display surface.

In the first aspect, the display control unit may control the sharpness of the display target object based on the view information and the eyeball information. This brings about the effect | action of controlling the sharpness of a display target object based on visual field information and eyeball information.

In the first aspect, the display control unit may control the sharpness of the display target object based on a three-dimensional position in the field of view where the display target object is to be displayed. . This brings about the effect | action of controlling the sharpness of a display target object based on the three-dimensional position in the visual field which should display a display target object.

In the first aspect, the display control unit may control the sharpness by performing a blurring process on the display target object. This brings about the effect that the sharpness is controlled by performing the blur process on the display target object.

In the first aspect, the image processing apparatus further includes a posture information acquisition unit that acquires posture information related to a change in posture of the image processing device, and the display control unit includes the visual field information, the eyeball information, and the posture. Control for changing the display mode may be performed on the basis of the posture information related to the change of the display. This brings about the effect | action of changing a display mode based on visual field information, eyeball information, and attitude | position information.

According to the present technology, an excellent effect that visibility can be improved can be achieved. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a top view showing an example of the appearance composition of image processing device 100 in a 1st embodiment of this art. It is a block diagram showing an example of functional composition of image processing device 100 in a 1st embodiment of this art. It is a figure which shows typically the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows typically the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows typically the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows an example of the procedure in the case of converting the local coordinate system of the display target object into the display coordinate system of the glass by the image processing apparatus 100 in the first embodiment of the present technology. It is a figure which shows typically the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. It is a figure which shows the example of a display of the display target object by the image processing apparatus 100 in 1st Embodiment of this technique. 6 is a flowchart illustrating an example of a processing procedure of a stabilizer process performed by the image processing apparatus 100 according to the first embodiment of the present technology. It is a figure which shows typically an example of the blurring process by the image processing apparatus 100 in 2nd Embodiment of this technique. It is a figure showing an example of a display at the time of performing blur processing by image processing device 100 in a 2nd embodiment of this art. It is a figure which shows the example of a display (comparative example) before implementing blurring process with the image processing apparatus 100 in 2nd Embodiment of this technique. 14 is a flowchart illustrating an example of a processing procedure of blur processing by the image processing apparatus 100 according to the second embodiment of the present technology.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First Embodiment (Example in which display mode of display target object is corrected based on view information and eyeball information)
2. Second Embodiment (Example of controlling the sharpness of a display target object based on visual field information and eyeball information)

<1. First Embodiment>
[External configuration example of image processing apparatus]
FIG. 1 is a top view illustrating an example of an external configuration of the image processing apparatus 100 according to the first embodiment of the present technology. In FIG. 1, the relationship between the eyeball (right eye) 11 and the eyeball (left eye) 12 when the image processing apparatus 100 is mounted on a human face is shown in a simplified manner.

The image processing apparatus 100 includes an outward imaging unit (R) 101, an outward imaging unit (L) 102, an inward imaging unit (R) 103, an inward imaging unit (L) 104, and a display unit (R). ) 181, a display unit (L) 182, infrared light emitting devices 183 to 186, and a bridge 190.

The image processing apparatus 100 can be, for example, a transmissive glasses-type electronic device (for example, a glasses-type wearable device) using a hologram light guide plate type optical technology. For example, the image processing apparatus 100 can include various sensing functions such as an image sensor, an acceleration sensor, a gyroscope, an electronic compass, an illuminance sensor, and a microphone. The image sensor is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device Image Sensor) image sensor. Further, the image processing apparatus 100 can provide information according to the user's situation by utilizing position information or the like by GPS (Global Positioning System).

The infrared light emitting devices 183 to 186 are infrared light emitting devices that emit infrared light. For example, the infrared

light emitting devices

183 and 184 can be provided at both ends of the display surface of the display portion (R) 181, and the infrared

light emitting devices

185 and 186 can be provided at both ends of the display surface of the display portion (L) 182. . However, the infrared light emitting devices may be arranged at other positions where the imaging operation can be appropriately performed. Moreover, you may make it set it as 3 or more about the installation number of each infrared rays light-emitting device in right and left.

The outward imaging unit (R) 101 and the outward imaging unit (L) 102 are imaging units that image subjects included in the field of view of the person wearing the image processing apparatus 100. That is, the outward image capturing unit (R) 101 and the outward image capturing unit (L) 102 are a plurality of image capturing units provided in the line-of-sight direction of a person.

The inward imaging unit (R) 103, the inward imaging unit (L) 104, and the infrared light emitting devices 183 to 186 capture the eyes of a person wearing the image processing device 100 (for example, the eyeball and its surroundings). The imaging unit is configured. That is, the inward imaging unit (R) 103 and the inward imaging unit (L) 104 are a plurality of imaging units provided in the direction of the human eyeball. Further, the inward imaging unit (R) 103 receives the light reflected by the infrared light emitted from the infrared

light emitting devices

183 and 184, so that the position of the eyeball (right eye) 11 and the direction of the line of sight can be calculated. Become. Similarly, the inward imaging unit (L) 104 receives the light reflected by the infrared light emitted from the infrared

light emitting devices

185 and 186, so that the position of the eyeball (left eye) 12 and the direction of the line of sight can be calculated. It becomes. Further, for example, the inward imaging unit (R) 103 is provided at an end of the display surface of the display unit (R) 181, and the inward imaging unit (L) 104 is an end of the display unit (L) 182 on the display surface. It can be provided in the part.

The display unit (R) 181 and the display unit (L) 182 are glass display units for providing various images to a person wearing the image processing apparatus 100. For example, when adopting the hologram optical technique, a hologram optical element having high transparency can be used as the display unit (R) 181 and the display unit (L) 182 without using a half mirror that blocks the visual field.

Further, as the display unit (R) 181 and the display unit (L) 182, in addition to the hologram element type display unit, a half mirror type display unit, a video transmission type display unit (for example, an external camera instead of a see-through display) Or a retina projection type display unit may be used.

Here, the hologram optical elements (display unit (R) 181 and display unit (L) 182) are optical elements for displaying the image light emitted from the optical engine by the hologram light guide plate technology. Here, the hologram light guide plate technology is a technology that, for example, incorporates hologram optical elements at both ends of a glass plate to propagate the image light emitted from the optical engine through a very thin glass plate to the eyes. is there.

Thus, the display unit (R) 181 and the display unit (L) 182 are provided with a display surface that transmits an image included in the person's field of view, and provide the image included in the person's field of view to the eyes of the person. Further, when the image processing apparatus 100 is transmissive glasses, a refracted video output from an optical engine arranged at the edge of the glass portion (display surface) of the display unit (R) 181 and the display unit (L) 182 is provided. As a result, an image is formed on the glass portion. That is, when the display unit (R) 181 and the display unit (L) 182 are refractive display units, the display surface is based on the refraction image output from the image output unit arranged at the edge of the display surface. Displays the display target object.

The bridge 190 is a member that connects the display unit (R) 181 and the display unit (L) 182. The bridge 190 corresponds to, for example, a bridge for glasses.

In FIG. 1, each of the outward imaging unit (R) 101, the outward imaging unit (L) 102, the inward imaging unit (R) 103, and the inward imaging unit (L) 104 is illustrated for ease of explanation. An example in which the display unit (R) 181 and the display unit (L) 182 are arranged outside is shown. However, these imaging units may be arranged at other positions where the imaging range (including the imaging direction (optical axis direction)) can be appropriately set.

[Configuration example of image processing apparatus]
FIG. 2 is a block diagram illustrating a functional configuration example of the image processing apparatus 100 according to the first embodiment of the present technology.

The image processing apparatus 100 includes an outward imaging unit (R) 101, an outward imaging unit (L) 102, an inward imaging unit (R) 103, an inward imaging unit (L) 104, and an image processing unit 111. To 114, the visibility information acquisition unit 120, the space model creation unit 130, the DB (database) 140, the eyeball information acquisition unit 150, the display control unit 160, the display processing unit 170, and the display unit (R) 181. And a display unit (L) 182 and a posture information acquisition unit 195.

The outward imaging unit (R) 101 and the outward imaging unit (L) 102 are a plurality of cameras facing the visual field side used for SLAM (SimultaneousaneLocalization and Mapping). Although FIG. 2 shows an example in which two outward imaging units are provided, three or more outward imaging units may be provided.

The inward imaging unit (R) 103 and the inward imaging unit (L) 104 generate a plurality of images used to detect the position, movement, line of sight, etc. of the eyeball of the person wearing the image processing apparatus 100. It is an inward camera. 2 shows an example in which two inward imaging units are provided, but three or more inward imaging units may be provided. Although not shown in FIG. 2, a device for photographing the movement of the eyeball (for example, an infrared light emitting device (for example, four for each imaging unit)) includes an inward imaging unit (R) 103 and an inner imaging unit (R) 103. It is assumed to be included in the system of the orientation imaging unit (L) 104.

The image processing unit 111 performs various types of image processing (for example, POST processing) on the image data generated by the outward imaging unit (R) 101. The image data subjected to the image processing is subjected to the view information acquisition unit 120. Output to.

The image processing unit 112 performs various image processing (for example, POST processing) on the image data generated by the outward imaging unit (L) 102, and the image data subjected to the image processing is subjected to the visual field information acquisition unit 120. Output to.

The image processing unit 113 performs various types of image processing (for example, POST processing) on the image data generated by the inward imaging unit (R) 103, and the image data subjected to the image processing is used as the eyeball information acquisition unit 150. Output to.

The image processing unit 114 performs various types of image processing (for example, POST processing) on the image data generated by the inward imaging unit (L) 104, and the image data subjected to the image processing is subjected to the eyeball information acquisition unit 150. Output to.

The view information acquisition unit 120 uses the image data output from the

image processing units

111 and 112 to display the view information (objects included in the imaging range of the outward imaging unit (R) 101 and the outward imaging unit (L) 102). Information). Then, the view information acquisition unit 120 outputs the extracted view information (for example, the feature point of the object, the depth of the object) to the space model creation unit 130.

For example, the visual field information acquisition unit 120 extracts feature points of an object by surrounding feature point extraction processing. The feature point of the object can be, for example, the vertex of the object.

For example, the view information acquisition unit 120 can extract feature points from the entire image (frame) corresponding to the image data output from the

image processing units

111 and 112. For example, for the first image among the images (frames) corresponding to the image data output from the

image processing units

111 and 112, the view information acquisition unit 120 extracts feature points from the entire image. In addition, for an image (frame) other than the first image, feature points are extracted from a newly imaged region compared with an image corresponding to the immediately preceding image (frame). Note that, for example, a point having a strong edge gradient in the vertical direction and the horizontal direction can be extracted as the feature point. This feature point is a strong feature point for optical flow calculation, and can be obtained using edge detection. Here, an example is shown in which feature points are extracted from the entire image for the first image, and feature points are extracted from a newly captured region portion compared to the previous image for images other than the first image. However, for each image other than the first image, feature points may be extracted from the entire image according to the processing capability and the like.

Also, for example, the visual field information acquisition unit 120 detects the depth of the object by depth detection processing. Note that the depth of the object is, for example, a distance in the depth direction (gaze direction).

For example, the field-of-view information acquisition unit 120 includes an image generated by the outward imaging unit (R) 101 and the outward imaging unit (L) 102 and each information (for example, a lens position and a focus position) when the image is generated. ) To calculate the subject distance for the image. For example, a depth map (depth map (Depth Map)) can be generated, and subject distances related to each region can be obtained based on the depth map. Here, the depth map is a map composed of data representing the subject distance. As a method for generating the depth map, for example, a TOF (Time of Flight) method, a blur amount analysis (Depth from Defocus), or the like can be used. For example, the TOF method is a method of calculating the distance to the subject based on the light delay time until the light emitted from the light source is reflected by the object and reaches the sensor, and the speed of the light.

Further, for example, as the image pickup device of each image pickup unit, for example, an image pickup device that receives transmitted light that has passed through different portions of the exit pupil and performs focus detection (phase difference AF (Auto-Focus)) using a phase difference detection method is used. be able to. An image sensor that performs this phase difference AF outputs a phase difference detection signal together with an image signal (analog signal). Therefore, the field-of-view information acquisition unit 120 can calculate the subject distance of each region in the image corresponding to the image signal output from each image sensor based on the phase difference detection signal output from each image sensor. .

Also, for example, the visibility information acquisition unit 120 can simultaneously estimate the position of the image processing apparatus 100 and create an environment map using SLAM technology.

As described above, the visual field information acquisition unit 120 acquires visual field information (external information) related to an image included in the visual field of the person wearing the image processing apparatus 100. For example, the visual field information acquisition unit 120 extracts feature points of objects included in the visual field based on images generated by the outward imaging unit (R) 101 and the outward imaging unit (L) 102. And depth detection processing for detecting the depth of an object included in the field of view. The visual field information acquisition unit 120 acquires information regarding the position of the object included in the visual field as the visual field information. The visibility information acquisition unit 120 is an example of a line-of-sight information acquisition unit described in the claims.

The DB (database) 140 is a database that stores information (spatial model information) for displaying virtual objects (space models) on the display unit (R) 181 and the display unit (L) 182. The spatial model information is supplied to the spatial model creation unit 130. Also, a spatial model created in the past (for example, an object specified by feature points and depth) is stored in the DB 140, and this spatial model is compared with a spatial model created thereafter. Also good.

The space model creation unit 130 displays objects on the display unit (R) 181 and the display unit (L) 182 based on the view information output from the view information acquisition unit 120 and the space model information supplied from the DB 140. Information (spatial model display information) is created. Then, the space model creation unit 130 supplies the created space model display information to the display control unit 160.

The eyeball information acquisition unit 150 detects eyeball information using the image data output from the

image processing units

113 and 114, and outputs the detected eyeball information to the display control unit 160. Here, the eyeball information is, for example, the position of the eyeball, the line of sight, the position of the iris in the eyeball (for example, the iris center) and size, the position and size of the pupil in the eyeball, and the eye axis angle. The iris is a portion where the eyeball is colored. Further, the pupil is a portion (a portion called “black eyes”) existing in the middle of the iris. Further, as the position of the eyeball, for example, a position on the eyeball, a three-dimensional position, a distance from the display surface of the display unit (R) 181 and the display unit (L) 182 can be obtained.

For example, the eyeball information acquisition unit 150 can detect eyeball information (for example, the position of the eyeball, the position and size of the iris on the eyeball, and the position and size of the pupil on the eyeball) using an image recognition technique. For example, the eyeball information acquisition unit 150 can detect the line of sight using the detected eyeball information (for example, each movement of the position of the eyeball, the position of the iris on the eyeball, and the position of the pupil on the eyeball).

As an eyeball detection method, for example, a detection method by matching a template in which the luminance distribution information of the eyeball is recorded with an actual image (see, for example, Japanese Patent Application Laid-Open No. 2004-133737), an eyeball color part included in image data, A detection method based on the feature amount of the eyeball or the like can be used.

Further, other eyeball detection methods may be used. For example, the image data output from the

image processing units

113 and 114 may be binarized and black eyes may be detected based on the binarized data on one screen. For example, the image data output from the

image processing units

113 and 114 is binarized, and the black pixel and the white pixel of each line in the horizontal direction (left-right direction) are determined. Then, in the horizontal direction, the number of white pixels in the predetermined range continues after the number of white pixels in the predetermined range, and the number of white pixels in the predetermined range continues after the continuous black pixels. The line is extracted as a candidate line for the black eye region. Then, in the vertical direction (vertical direction), a region where a number of lines within a predetermined range are continuous can be extracted as the extracted candidate lines, and the region can be detected as a black eye region.

Also, the eye may be detected using an eye movement measurement device. For example, the position of the black eye portion can be specified by applying infrared light from an infrared LED (Light Emitting Diode) to the eyeball and detecting the reflected light with a light receiver.

Also, for example, the eyeball can be detected using another infrared light emitting device. For example, when line-of-sight recognition is realized by pupil detection processing and bright spot detection processing, infrared rays can be used in order not to be affected by external light and to prevent specular reflection from the cornea.

Also, as a means of improving the accuracy of eye-gaze detection, when obtaining the corneal curvature center, it is possible to detect a maximum of four infrared light source reflections (bright spots, red circles).

It should be noted that these eyeball information detection methods (for example, eye gaze detection) are merely examples, and high-precision eye gaze detection can be realized by applying a plurality of techniques other than these.

Thus, the eyeball information acquisition unit 150 acquires eyeball information related to the human eyeball. For example, the eyeball information acquisition unit 150 detects the position of the human eyeball and the eye gaze direction based on the images generated by the inward imaging unit (R) 103 and the inward imaging unit (L) 104. Get eyeball information.

The display control unit 160 performs control for displaying each image (for example, a display target object) on the display unit (R) 181 and the display unit (L) 182. For example, the display control unit 160 performs display position calculation processing, image effect processing, and image superposition based on the spatial model display information output from the spatial model creation unit 130 and the eyeball information output from the eyeball information acquisition unit 150. Perform processing.

The display control unit 160 is realized by, for example, a host CPU (Central Processing Unit). Note that some of the above-described units can be realized by the host CPU.

The display processing unit 170 performs various image processing on the images displayed on the display unit (R) 181 and the display unit (L) 182 based on the control of the display control unit 160. For example, the display control unit 160 performs control to change the display mode of the display target object based on the field-of-view information, the eyeball information, and the posture information regarding the posture change.

The posture information acquisition unit 195 detects a change in posture of the image processing device 100 by detecting acceleration, movement, inclination, and the like of the image processing device 100, and controls display of posture information regarding the detected posture change. Output to the unit 160. Note that various sensors such as a gyro sensor and an acceleration sensor can be used as the posture information acquisition unit 195, for example.

The image processing apparatus 100 also has two functions (stabilizer and depth correction) for the display object.

Here, the stabilizer is a function for displaying the display object at a desired position in the external coordinates. In other words, the stabilizer is a function that makes a display object appear at an expected position in external coordinates. A display example by this stabilizer is shown in FIG.

[Display example with stabilizer]
FIG. 3 is a diagram schematically illustrating a display example of the display target object by the image processing apparatus 100 according to the first embodiment of the present technology.

FIG. 3 shows an example in which the relationship among the image processing apparatus 100, the external objects A (201) and B (202), and the display target object X (203) is viewed from above. The top view of the image processing apparatus 100 is the same as the example shown in FIG. However, in FIG. 3, the eyeball (right eye) 11 and the eyeball (left eye) 12 are moved in the direction of viewing the display target object X (203) displayed on the display unit (R) 181 and the display unit (L) 182. An example is shown.

External objects A (201) and B (202) are actually existing objects. In FIG. 3, the external objects A (201) and B (202) are shown as

rectangular objects

201 and 202 for ease of explanation.

The display target object X (203) virtually indicates an object displayed on the display unit (R) 181 and the display unit (L) 182 by the stabilizer. FIG. 3 shows an example in which the display target object X (203) is displayed between the external objects A (201) and B (202).

Here, as described above, in the transmission glasses, images are formed on the glass portions of the display portion (R) 181 and the display portion (L) 182 by the refractive image output of the optical engine arranged at the edge of the glass portion. It is. For this reason, in FIG. 3, virtual

display screen positions

211 and 212 correspond to the actual display positions of the display unit (R) 181 and the display unit (L) 182.

The virtual

display screen positions

211 and 212 are virtual display positions for displaying the display target object X (203) when the display target object X (203) is displayed on the display unit (R) 181 and the display unit (L) 182 by the stabilizer. Display position.

Also, the display position of the display target object X (203) at the virtual display screen position 211 is indicated by a display position 213. The display position 214 indicates the display position of the display target object X (203) at the virtual display screen position 212.

[Display example before correction when image processing apparatus moves (comparative example)]
FIG. 4 is a diagram schematically illustrating a display example of a display target object by the image processing apparatus 100 according to the first embodiment of the present technology.

FIG. 4 shows an example in which the position of the eyeball and the image processing apparatus 100 is moved for some reason when the display target object X is displayed as shown in FIG. 3 (display example before correction). . This example is shown as a comparative example in FIG.

In FIG. 4, the image processing apparatus 100 before movement, the eyeball (right eye) 11, and the eyeball (left eye) 12 are indicated by dotted lines. The positions of the image processing apparatus 100, the eyeball (right eye) 11, and the eyeball (left eye) 12 indicated by the dotted lines correspond to the positions shown in FIG. In addition, the image processing apparatus 100 after movement, the eyeball (right eye) 11, and the eyeball (left eye) 12 are indicated by solid lines.

In FIG. 4, the virtual

display screen positions

211 and 212 before the movement are indicated by normal dotted lines. The positions of the virtual

display screen positions

211 and 212 indicated by the normal dotted lines correspond to the positions shown in FIG. Further, the virtual

display screen positions

211 and 212 after the movement are indicated by thick dotted lines.

As shown in FIG. 4, the virtual

display screen positions

211 and 212 change according to the change of the position of the image processing apparatus 100. The positions of the eyeball (right eye) 11 and the eyeball (left eye) 12 also change. For this reason, the display target object X (205) may be displayed at an unintended position (the same position as the object 202).

Therefore, the first embodiment of the present technology shows an example in which the position of the display target object X is appropriately displayed even when the position of the image processing apparatus 100 or the position of the eyeball is changed.

[Example of display after correction when image processing device moves]
FIG. 5 is a diagram schematically illustrating a display example of a display target object by the image processing apparatus 100 according to the first embodiment of the present technology.

FIG. 6 is a diagram illustrating an example of a procedure when the local coordinate system of the display target object is converted into the display coordinate system of the glass by the image processing apparatus 100 according to the first embodiment of the present technology.

FIG. 5 shows an example in which the position of the display target object X is appropriately displayed by moving the display position of the display target object X in the example shown in FIG.

For example, a line segment extending in a direction (line-of-sight direction) perpendicular to (or substantially perpendicular to) the display surfaces of the display unit (R) 181 and the display unit (L) 182, and the display target object X (204) A line segment extending from the position where the position is to be arranged is indicated by a line segment 30.

The distance between the line segment 30 in the horizontal direction and the eyeball (right eye) 11 (iris center) is W1, and the distance between the display surface of the display unit (R) 181 and the eyeball (right eye) 11 in the horizontal direction (line segment 30). The upper distance) is L1. Further, the distance (distance on the line segment 30) between the display surface of the display unit (R) 181 and the display target object X (204) in the horizontal direction is L2. Further, the distance (distance on the line segment 30) between the display surface of the display unit (R) 181 in the horizontal direction and the virtual display screen position 211 is L3.

The distances L1 and W1 can be obtained based on the eyeball information output from the eyeball information acquisition unit 150. The distances L2 and L3 can be acquired based on the display target object X.

In this case, the display position 221 in the horizontal direction of the display target object X (204) at the virtual display screen position 211 can be obtained by the following equation.
W2 = {(L2-L3) / (L2 + L1)} W1

Here, W2 is the distance from the intersection of the virtual display screen position 211 and the line segment 30 to the display position 221.

Thus, the display position 221 (W2) in the horizontal direction of the display target object X (204) at the virtual display screen position 211 can be obtained by the trigonometric function formula. Further, the display position 221 in the vertical direction of the display target object X (204) at the virtual display screen position 211 can be similarly obtained. Further, the display position 222 in the horizontal direction and the vertical direction of the display target object X (204) at the virtual display screen position 212 can be similarly obtained.

Further, when the relationship between the eyeball and the image processing apparatus 100 is rotated with the line-of-sight direction as the rotation axis, the display mode of the display target object X (204) at the virtual display screen position is changed according to the rotation. To do. For example, the display target object X is rotated and displayed in the direction opposite to the rotation direction.

Also, the display size of the display target object X (204) at the virtual display screen position is changed according to the change in the distance L1.

Each of these processes is calculated by a determinant. An example of this procedure (steps S701 to S705) is shown in FIG. For example, the view coordinate system changes as the eyeball moves. In this case, assuming that the view transformation matrix V is multiplied by the movement vector of the eyeball corresponding to the movement and multiplied by the view transformation matrix V, the view transformation matrix changes from V → V ′. Further, the parameters for calculating the projection transformation matrix P and the screen transformation matrix S also change due to the change of the viewpoint. Therefore, due to the movement of the eyeball, the projection conversion matrix changes from P → P ′, and the screen conversion matrix changes from S → S ′. Further, the display coordinates of the glass can be finally calculated based on these values.

Thus, by performing the correction process, the position of the display target object X can remain at the position before the movement.

[Example of correction when the line of sight of a person moves]
FIG. 7 is a diagram schematically illustrating a display example of a display target object by the image processing apparatus 100 according to the first embodiment of the present technology. FIG. 7 shows an example of a stabilizer for the movement of the line of sight of a person wearing the image processing apparatus 100.

FIG. 7 shows an example in which the eyeball 20 is looking at the display target object 303 displayed at the display position 301 in the virtual display screen position 300 (line of sight 305). In this case, it is assumed that the eyeball 20 moves to the left and the line of sight moves. Thus, when the eyeball 20 moves, when the eyeball 20 sees the display target object displayed at the display position 301 in the virtual display screen position 300, the display target object 304 becomes visible (line segment 306). ).

Therefore, when the eyeball 20 moves to the left side and the line of sight moves, the display position 301 in the virtual display screen position 300 is moved to the display position 302 by performing the above-described correction processing. Thereby, even when the eyeball 20 moves, the eyeball 20 can see the display target object 303 displayed at the display position 302 in the virtual display screen position 300 (line segment 307).

As described above, when the person wearing the image processing apparatus 100 moves his / her line of sight to the left and right, the display target object can be appropriately displayed by performing the above-described correction processing.

[Coordinate conversion example]
Here, a coordinate conversion process for displaying the display target object on the screen (on the display screen of the display unit (R) 181 and the display unit (L) 182) will be described.

Since the information of the display target object is a local coordinate, it is necessary to perform coordinate conversion processing in order to display it on the screen.

For example, a general process necessary for displaying 3D object data on a 2D screen (for example, transmissive glasses) will be described. For example, local coordinates → world coordinate conversion processing, world coordinates → view coordinate conversion processing, view coordinates → projection coordinate conversion processing, and projection coordinates → screen coordinate conversion processing are general processes.

Here, the local coordinate → world coordinate conversion process is a process of converting where each object is arranged in the outside world.

Also, the world coordinate → view coordinate conversion process is a process for converting where the camera (image processing apparatus 100) is located in the outside world.

Further, the view coordinate → projection coordinate conversion process is a process for converting where the screen (on the display screen of the display unit (R) 181 and the display unit (L) 182) is arranged.

Also, the projection coordinate → screen coordinate conversion process is a process for converting where the display target object is displayed on the screen.

However, in reality, it is also necessary to perform projective transformation (processing to put in shadows of other objects depending on the context), texture mapping, lighting, and the like. However, only the processing limited to the coordinate system is shown here for ease of explanation.

For example, when the eye position (camera position) of a person wearing the image processing apparatus 100 is fixed with respect to the glasses (display unit (R) 181 and display unit (L) 182), Assume the case of moving with respect to the glass. In this case, the eyeball position is calculated based on the images generated by the inward camera (inward imaging unit (R) 103, inward imaging unit (L) 104), and the movement of the eye position relative to the glass is grasped. can do. For this reason, in the world coordinate → view coordinate conversion process described above, based on the movement of the eye position relative to the glass, the fixed eye position (camera position) and the glass (display unit (R) 181, display unit ( L) The relationship with the position of 182) is corrected.

Also, in the above-described view coordinates → projection coordinate conversion processing and projection coordinates → screen coordinate conversion processing, coordinate conversion is performed in consideration of the eye position and screen position of the person wearing the image processing apparatus 100.

Here, in the view coordinate conversion, a conversion matrix can be obtained based on the coordinate information of the world coordinates of the display target object, the world coordinates of the eyes, and the upward coordinates of the glass.

[Display example of display target object]
8 to 10 are diagrams illustrating display examples of display target objects by the image processing apparatus 100 according to the first embodiment of the present technology. 8 to 10, only the image processing apparatus 100 and a part of the periphery thereof are shown in a rectangle.

FIG. 8 shows a relationship between a frame 401 that holds the display unit (L) 182 of the image processing apparatus 100 and an arrow (display target object) 402 displayed on the display unit (L) 182. As shown in FIG. 8, what actually exists can be seen through the lens of the display unit (L) 182 of the image processing apparatus 100. In FIG. 8, squares are shown as an example of what actually exists for ease of explanation.

Further, an arrow (display target object) 402 can be displayed on the display unit (L) 182 of the image processing apparatus 100.

FIG. 9 shows a display example when the image processing apparatus 100 is tilted. FIG. 9 a shows a display example when the arrow (display target object) 402 is corrected according to the inclination of the image processing apparatus 100. FIG. 9B shows a display example when correction according to the inclination of the image processing apparatus 100 is not performed.

As shown in FIG. 9b, when the correction process is not performed, the arrow (display target object) 402 is tilted according to the tilt of the image processing apparatus 100.

On the other hand, as shown in FIG. 9A, when the correction process is performed, the arrow (display target object) 402 is not tilted because the arrow (display target object) 402 is corrected according to the tilt of the image processing apparatus 100. .

FIG. 10 shows a display example when the image processing apparatus 100 moves horizontally. FIG. 10 a shows a display example when the arrow (display target object) 402 is corrected in accordance with the horizontal movement of the image processing apparatus 100. FIG. 10B shows a display example when the correction according to the horizontal movement of the image processing apparatus 100 is not performed.

As shown in FIG. 10b, when the correction process is not performed, the arrow (display target object) 402 moves in accordance with the horizontal movement of the image processing apparatus 100.

On the other hand, as shown in a of FIG. 10, when the correction process is performed, the arrow (display target object) 402 is corrected in order to correct the arrow (display target object) 402 in accordance with the horizontal movement of the image processing apparatus 100. Does not move.

Thus, by performing the correction process, it is possible to make the external object appear as if the arrow (display target object) 402 is fixed.

[Operation example of image processing apparatus (stabilizer processing example)]
FIG. 11 is a flowchart illustrating an example of a processing procedure of a stabilizer process performed by the image processing apparatus 100 according to the first embodiment of the present technology.

First, the visual field information acquisition unit 120 extracts visual field information using images generated by the outward imaging unit (R) 101 and the outward imaging unit (L) 102 (step S801). For example, the visual field information acquisition unit 120 acquires an external singular point and an acceleration sensor value as visual field information (step S801). In this example, the view information is obtained by using the images generated by the outward imaging unit (R) 101 and the outward imaging unit (L) 102, but the view information is obtained by other methods. You may make it acquire. For example, an acceleration sensor value may be acquired from the posture information acquisition unit 195 (for example, an acceleration sensor). Note that step S801 is an example of a view information acquisition procedure described in the claims.

Subsequently, the space model creation unit 130 calculates the absolute coordinates of the external object based on the view information extracted by the view information acquisition unit 120 (step S802). Further, the space model creation unit 130 determines the display coordinates of the display target object X on the absolute coordinates based on the view information extracted by the view information acquisition unit 120 and the display target object information stored in the DB 140. Confirm (step S802).

In this way, the absolute coordinates of the external object are calculated by multi-SLAM using a plurality of outward cameras, and the display coordinates of the display target object X on the absolute coordinates are determined.

Subsequently, the eyeball information acquisition unit 150 extracts eyeball information using images generated by the inward imaging unit (R) 103 and the inward imaging unit (L) 104 (step S803). For example, the eyeball information acquisition unit 150 obtains the absolute coordinates of the iris center point in the eyeball and the line-of-sight vector of the eyeball as eyeball information (step S803). Step S803 is an example of an eyeball information acquisition procedure described in the claims.

Subsequently, the display control unit 160 determines the display position, display size, and display direction of the display target object X at the virtual

display screen positions

211 and 212 based on each acquired information (step S804). For example, the display control unit 160 displays the display target at the virtual

display screen positions

211 and 212 based on the absolute coordinates of the iris center point, the display coordinates of the display target object X, and the absolute coordinates of the virtual

display screen positions

211 and 212. The display position, display size, and display direction of the object X are determined.

Subsequently, the display control unit 160 performs correction processing on the display target object X based on the determined contents (display position, display size, display direction of the display target object X at the virtual display screen positions 211 and 212) ( Step S805). Steps S804 and S805 are an example of a control procedure described in the claims.

Subsequently, it is determined whether or not to end the display of the display target object (step S806). When the display of the display target object is ended (step S806), the stabilizer processing operation is ended. On the other hand, when the display of the display target object is not terminated (step S806), the process returns to step S801.

As described above, the display control unit 160 can display the display target object so as to overlap the image included in the field of view of the person. In addition, the display control unit 160 can perform control to change the display mode of the display target object in the image included in the person's view based on the view information and the eyeball information. For example, the display control unit 160 changes at least one of the display position, the display angle, and the display size of the display target object on the display surface of the display unit (R) 181 and the display unit (L) 182. Thus, the display mode can be changed.

For example, the display control unit 160 performs control to change the display mode of the display target object based on position information regarding the positions of the display unit (R) 181 and the display unit (L) 182, view information, and eyeball information. It can be carried out. Further, for example, the display control unit 160 displays the display target object based on the position information regarding the display position (for example, virtual display screen position) where the display target object is virtually displayed, the visual field information, and the eyeball information. Control to change the aspect can be performed. For example, the display mode of the display target object can be changed based on the relative relationship between the positions specified by these pieces of information.

<2. Second Embodiment>
In 1st Embodiment of this technique, the example which correct | amends the display mode of a display target object based on visual field information and eyeball information was shown.

In the second embodiment of the present technology, an example in which the sharpness of a display target object is controlled based on view information and eyeball information will be described. That is, an example of performing blur processing (focus processing, depth correction processing) in the depth direction based on detection of a person's line of sight is shown. Note that in the second embodiment of the present technology, only an example of performing blur processing is shown, but stabilization and blur processing may be performed simultaneously.

Note that the configuration of the image processing apparatus according to the second embodiment of the present technology is substantially the same as that of the image processing apparatus 100 shown in FIGS. For this reason, about the part which is common in 1st Embodiment of this technique, the code | symbol same as 1st Embodiment of this technique is attached | subjected, and some of these description is abbreviate | omitted.

[Example of blur processing based on gaze detection]
FIG. 12 is a diagram schematically illustrating an example of a blur process performed by the image processing apparatus 100 according to the second embodiment of the present technology.

FIG. 12 shows an example of blurring the display target object X when the person wearing the image processing apparatus 100 is not in focus on the display target object X (blur processing example based on line-of-sight detection). Specifically, a case where a person wearing the image processing apparatus 100 sees a distance and a case where the person sees a distance are detected from the line of sight of the person. An example in which the display target object X is subjected to the blur process in accordance with the depth when the display target object X is not in focus (an example of the blur process based on line-of-sight detection) is shown.

Here, the relationship of blur processing between the subject depth and the line of sight will be described. In the second embodiment of the present technology, an example in which blur processing is performed in consideration of the following will be described.

[Bokeh processing example according to visual acuity (processing applied regardless of line of sight)]
In the second embodiment of the present technology, an example in which blur processing according to visual acuity is always processed regardless of the focal length is shown. When performing the blur process according to the visual acuity, it is necessary to input (or adjust) the visual acuity of the person wearing the glass (image processing apparatus 100) in advance.

For example, assume a case where blur processing is performed on a person with a vision of 1.0. In this case, a 30-mm Gaussian Blur blur process is applied to a person with a visual acuity of 1.0 with a target object 10 m ahead. Here, Gaussian Blur (Gaussian blurring) is an example of image processing for performing blur processing.

For example, it is defined that a person with a visual acuity of 1.0 can recognize a 15 cm Landolt ring ("C" mark) 10 m ahead. For example, when the resolution capable of recognizing a 15 cm Landolt ring is set to a level multiplied by 30 mm Gaussian Blur, Kg = 10000/30 = 333 when expressed by the same calculation formula as the visual acuity coefficient.

Therefore, when the distance to the display target object X is X1 cm, the diameter of the Gaussian blur required for visual acuity 1.0 is x / 333. In this example, for ease of explanation, a comparatively simple calculation example has been shown. However, in practice, it is preferable to use a Gaussian blur closer to human vision.

[Example of blur processing based on the distance X1 to the display target object X and the focal position X1 ′ (processing that varies depending on the line of sight)]
For example, when the performance of the human eye is represented by an index of a camera (for example, a digital still camera), the F value of the human eye is said to be about 1.0. In addition, it is said that the human eye changes its focal length in the range of about 28 mm to 130 mm depending on the situation.

Therefore, based on the focal length of 50 mm and the F value of 1.0, “focal length of 50 mm / F value of 1.0 = effective diameter of 50” is set as the standard effective diameter, and blur processing suitable for the focal length can be performed by visual line detection. .

For example, it is defined as follows according to the distance X1 to the display target object X.

For example, when X1 ≦ A1 (for example, A1 is about 10 cm), an unadjustable process is performed. In this non-adjustable processing, for example, tele blur processing is performed regardless of the line-of-sight detection result. In this case, the blur process is increased as the value approaches 0.

For example, when A1 <X1 ≦ A2 (for example, A2 is 20 to 50 cm), macro processing is performed. In this macro process, the blur process by the next line-of-sight detection is performed as an additional process of the adjustment impossible process described above.
(1) When the focal length matches the display target object X, the blur process is not performed. However, the blur process corresponding to the above-described visual acuity is performed.
(2) When the focal length is shorter than the display target object X, blur processing corresponding to an effective diameter of 50 to 85 is performed. In other words, the blur process is roughly performed.
(3) When the focal length is farther than the display target object X, blur processing corresponding to an effective diameter of 28 to 50 is performed. That is, the blur process is performed with a small amount.

If A2 <X1, standard processing is performed. In this standard process, for example, a blur process corresponding to an effective diameter of 50 is performed as an additional process of the adjustment impossible process described above.

Here, it is assumed that the person wearing the image processing apparatus 100 is looking with one eye by meditating one eye or hiding one eye with a hand. In this case, blur processing (one-eye blur processing) when viewing with one eye is performed.

In the one-eye blur process, for example, the macro process and the standard process described above are performed with a blur process corresponding to an effective diameter of 35.

As described above, by performing the blur processing, for example, even when only the display target object exists in the field of view (for example, darkness), the distance to the display target object can be correctly recognized. For this reason, it is possible to correctly recognize the size of the display target object.

[Comparison example when blur processing is performed]
Here, with reference to FIG. 13 and FIG. 14, an outline of the effect of the above-described blur processing is shown.

FIG. 13 is a diagram illustrating a display example when the blur processing is performed by the image processing apparatus 100 according to the second embodiment of the present technology.

FIG. 14 is a diagram illustrating a display example (comparative example) before the blur processing is performed by the image processing apparatus 100 according to the second embodiment of the present technology.

FIGS. 13 and 14 show examples (images 600 and 610) of images (images entering a human eye) displayed on either the display unit (R) 181 or the display unit (L) 182. FIG. The imaging range corresponding to the

images

600 and 610 includes an object (cone) 601 disposed relatively close to the person (front side position) and a position relatively far from the person (back side position). ) (Object (cylinder)) 602 arranged in the above. Further, it is assumed that the line of sight of the person is focused on the object (cone) 601 but the object (cylinder) 602 appears out of focus.

In such a case, it is assumed that the display target object (sphere) is arranged and displayed at substantially the same position (back position) as the object (cylinder) 602.

For example, as shown in FIG. 14, it is assumed that the display target object (sphere) 611 is displayed without performing the blur process. In this case, although the object (cylinder) 602 looks out of focus without being focused, the display target object (sphere) 611 appears to be in focus. In this way, one object (object (cylinder) 602) arranged at substantially the same position (back side position) appears to be blurred, and the other object (display target object (sphere) 611) is in focus. If you see it, it will make you feel strange.

Therefore, as shown in FIG. 13, it is assumed that the above-described blur processing is performed to display the display target object (sphere) 603. In this case, the object (cylinder) 602 and the display target object (sphere) 611 look out of focus without being in focus. In this way, two objects (object (cylinder) 602 and display target object (sphere) 611) arranged at substantially the same position (back side position) appear to be blurred in the same way, so they are seen. Does not give a sense of incongruity to people.

[Operation Example of Image Processing Device (Bokeh Processing Example)]
FIG. 15 is a flowchart illustrating an example of a processing procedure of blur processing by the image processing apparatus 100 according to the second embodiment of the present technology.

First, based on the eyeball information from the eyeball information acquisition unit 150, the display control unit 160 allows the person wearing the image processing apparatus 100 to display the display unit (R) 181 and the display unit (L) 182 with both eyes. It is determined whether or not the user is watching (step S811). That is, the display control unit 160 determines whether or not the person wearing the image processing apparatus 100 is looking at both the display unit (R) 181 and the display unit (L) 182 (step S811).

When the person wearing the image processing apparatus 100 is not looking with both eyes (step S811), the display control unit 160 determines that the person wearing the image processing apparatus 100 is the display unit (R) 181 and the display. It is determined whether or not the part (L) 182 is viewed with one eye (step S812). That is, the display control unit 160 determines whether or not the person wearing the image processing apparatus 100 is looking at one of the display unit (R) 181 and the display unit (L) 182 (step). S812). When a person wearing the image processing apparatus 100 does not see the display unit (R) 181 and the display unit (L) 182 with one eye (that is, both the display unit (R) 181 and the display unit (L) 182) (Step S812), it is not necessary to perform blur processing or the like. Therefore, the blur processing operation is terminated.

When the person wearing the image processing apparatus 100 is looking at one of the display unit (R) 181 and the display unit (L) 182 (step S812), the display control unit 160 displays the one eye. The blur process is performed (step S813). For example, the above-described macro processing and standard processing are performed with blur processing corresponding to an effective diameter of 35.

When the person wearing the image processing apparatus 100 is looking with both eyes (step S811), the display control unit 160 determines whether or not the distance X1 to the display target object X is A1 or less ( Step S814). When the distance X1 to the display target object X is A1 or less (step S814), the display control unit 160 performs an adjustment impossible process (step S815). In this adjustment impossible processing, for example, tele blur processing is performed regardless of the line-of-sight detection result (step S815).

When the distance X1 to the display target object X is longer than A1 (step S814), the display control unit 160 determines whether or not the distance X1 to the display target object X is A2 or less (step S816). . When the distance X1 to the display target object X is longer than A2 (step S816), the display control unit 160 performs standard processing (step S817). In this standard process, for example, a blur process corresponding to an effective diameter of 50 is performed as an additional process of the adjustment impossible process described above.

When the distance X1 to the display target object X is A2 or less (step S816), the display control unit 160 determines whether or not the focal length matches the display target object X (step S818). If the focal length matches the display target object X (step S818), the display control unit 160 performs a blur process according to the visual acuity (step S819).

When the focal length does not match the display target object X (step S818), the display control unit 160 determines whether the focal length is closer than the display target object X (step S820). When the focal length is shorter than the display target object X (step S820), the display control unit 160 performs a blur process corresponding to an effective diameter of 50 to 85 (step S822). In other words, the blur process is roughly performed.

When the focal distance is not closer than the display target object X (when the focal distance is farther than the display target object X) (step S820), the display control unit 160 performs a blur process corresponding to an effective diameter of 28 to 50. (Step S821). That is, the blur process is performed with a small amount.

As described above, the display control unit 160 can control the sharpness of the display target object based on the visual field information and the eyeball information. For example, the display control unit 160 can control the sharpness of the display target object based on a three-dimensional position (for example, a distance to the subject) in the field of view where the display target object is to be displayed. For example, the display control unit 160 can control the sharpness of the display target object by performing a blur process.

Suppose here that, for example, the display target object is displayed on coordinates fixed to the external object. For example, a method (first method) of displaying by using an acceleration sensor to move the display position of the display target object in the direction opposite to the operation direction of the image processing apparatus is conceivable. Further, for example, a method (second method) of calculating and displaying the coordinate axis of the display target object by calculating the absolute position of the display unit using SLAM can be considered.

However, in the first method, there is a possibility that the display target object appears to vibrate only by moving the face a little. In the first method, it is also assumed that the accuracy of the acceleration sensor is low. In this case, there is a possibility that the display target object cannot be set to an appropriate display position. Even if 100% accuracy is obtained as the accuracy of the acceleration sensor, the accuracy for the external object is effective only for the display unit. Also in this case, there is a possibility that the display target object appears to vibrate only by moving the face a little.

For this reason, in the first method, it is difficult to see the display target object, and there is a possibility that symptoms such as intoxication appear when looking at the display target object.

In the second method, when the displayed image is always in focus and there is no target object that is the starting point (for example, darkness), the display target object is in front of the eyes (for example, virtual There is a risk of falling into the illusion displayed at the position of the display unit. In the second method, SLAM that can accurately determine the position of the external object is an effective means, but the accuracy is effective only for the display unit. Also in this case, there is a possibility that the display target object appears to vibrate only by moving the face a little.

For this reason, in the second method, it is difficult to see the display target object, and there is a possibility that symptoms such as intoxication appear when looking at the display target object.

Here, for example, when a person wearing the image processing apparatus is walking, the display unit may vibrate due to the walking vibration. In this case, it is assumed that the display unit changes with respect to the position of the visual recognition unit (for example, the left and right eyeball positions). In addition, for example, when a person wearing the image processing apparatus moves eyebrows, when looking near and looking far away (or vice versa), when shifting his / her line of sight, the eyeball position is Move. In this case, it is assumed that the position of the visual recognition unit (for example, the left and right eyeball positions) changes with respect to the display unit.

However, the first method and the second method do not take into consideration that the relationship between the position of the visual recognition unit and the display unit changes. Therefore, it is important to accurately display the display target object on the coordinate axis fixed with respect to the external object in consideration of the change in the relationship between the position of the visual recognition unit and the display unit.

Therefore, in the first embodiment of the present technology, each piece of information (for example, position information) related to an external object, an image processing device, and an eyeball is obtained using a plurality of outward cameras and a plurality of inward cameras. Detect and use and correct the display image appropriately. Thereby, blurring of the display image can be reduced. In addition, the position of the virtual screen can be appropriately controlled.

This can prevent the display target object from shaking. Also, the visibility can be dramatically improved. In addition, a stronger sense of reality can be obtained. In addition, it is possible to reduce the discomfort that the brain receives from the display. Thereby, it is possible to alleviate the fatigue of viewing for a long time, and to prevent sickness. As a result, a comfortable application of the transmissive wearable glass can be realized.

In other words, the display target object can be displayed in a state of being blended into the outside world without a sense of incongruity. In addition, visibility can be improved, 3D sickness can be prevented, and fatigue during long-time viewing can be reduced.

In other words, it is possible to visually display the display target object on the coordinates fixed with respect to the external object more stably and naturally.

Further, according to the second embodiment of the present technology, the sharpness of the display target object can be appropriately controlled based on the target depth of the display target object.

Thus, according to the embodiment of the present technology, the visibility can be improved.

Note that the above-described embodiment is an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.

It should be noted that the effects described in this specification are merely examples, and are not limited, and other effects may be obtained.

In addition, this technique can also take the following structures.
(1)
A field-of-view information acquisition unit that acquires field-of-view information about an image included in the field of view of a person wearing the image processing apparatus;
An eyeball information acquisition unit for acquiring eyeball information related to the eyeball of the person;
Display control for displaying a display target object superimposed on an image included in the field of view and changing the display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information An image processing apparatus.
(2)
The eyeball information acquisition unit detects the position of the eyeball of the person and the line-of-sight direction of the eyeball based on images generated by a plurality of imaging units provided in the eyeball direction of the person. The image processing apparatus according to (1), wherein:
(3)
A display unit for providing an image included in the field of view of the person to the eyes of the person;
The image processing device according to (2), wherein the plurality of imaging units are provided at an end of a display surface of the display unit that displays the display target object.
(4)
The visual field information acquisition unit extracts a characteristic point of an object included in the visual field based on images generated by a plurality of imaging units provided in the visual line direction of the person, and the visual field The image according to any one of (1) to (3), wherein information regarding a position of an object included in the field of view is acquired as the field of view information by performing depth detection processing for detecting a depth of the object included in the field of view. Processing equipment.
(5)
A display unit for providing an image included in the field of view of the person to the eyes of the person;
In any one of (1) to (4), the display control unit performs control to change the display mode based on position information regarding the position of the display unit, the visual field information, and the eyeball information. The image processing apparatus described.
(6)
The display unit displays the object to be displayed on the display surface based on a refraction image output from an image output unit disposed at an edge of the display surface that transmits an image included in the field of view of the person. A display unit,
The display control unit performs control to change the display mode based on position information regarding a display position for virtually displaying the display target object, the visual field information, and the eyeball information. The image processing apparatus described.
(7)
The display control unit changes the display mode by changing at least one of a display position, a display angle, and a display size of the display target object on the display surface of the display unit. The image processing apparatus according to any one of 6).
(8)
The image processing apparatus according to any one of (1) to (7), wherein the display control unit controls a sharpness of the display target object based on the visual field information and the eyeball information.
(9)
The image processing apparatus according to (8), wherein the display control unit controls the sharpness of the display target object based on a three-dimensional position in the field of view where the display target object is to be displayed.
(10)
The image processing apparatus according to (8) or (9), wherein the display control unit controls the sharpness by performing blurring processing on the display target object.
(11)
A posture information acquisition unit that acquires posture information related to a change in posture of the image processing apparatus;
The display control unit according to any one of (1) to (10), wherein the display control unit performs control to change the display mode based on the visual field information, the eyeball information, and posture information regarding the change in posture. Image processing apparatus.
(12)
A view information acquisition procedure for acquiring view information regarding an image included in the view of a person to whom the image processing apparatus is attached;
Eyeball information acquisition procedure for acquiring eyeball information relating to the eyeball of the person;
A control procedure for superimposing a display target object on an image included in the field of view and changing a display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information. Image processing method.
(13)
A view information acquisition procedure for acquiring view information regarding an image included in the view of a person to whom the image processing apparatus is attached;
Eyeball information acquisition procedure for acquiring eyeball information relating to the eyeball of the person;
A control procedure for causing a display target object to overlap and display an image included in the field of view, and changing a display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information. A program to be executed.

100 Image processing apparatus 101 Outward imaging unit (R)
102 Outward imaging unit (L)
103 Inward imaging unit (R)
104 Inward imaging unit (L)
111 to 114 Image processing unit 120 Visibility information acquisition unit 130 Spatial model creation unit 140 DB (database)
150 Eyeball Information Acquisition Unit 160 Display Control Unit 170 Display Processing Unit 181 Display Unit (R)
182 Display (L)
183 to 186 Infrared light emitting device 190 Bridge 195 Attitude information acquisition unit

Claims

A field-of-view information acquisition unit that acquires field-of-view information about an image included in the field of view of a person wearing the image processing apparatus;
An eyeball information acquisition unit for acquiring eyeball information related to the eyeball of the person;
Display control for displaying a display target object superimposed on an image included in the field of view and changing the display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information An image processing apparatus.
The eyeball information acquisition unit detects the position of the eyeball of the person and the line-of-sight direction of the eyeball based on images generated by a plurality of imaging units provided in the eyeball direction of the person. The image processing device according to claim 1, wherein
A display unit for providing an image included in the field of view of the person to the eyes of the person;
The image processing apparatus according to claim 2, wherein the plurality of imaging units are provided at an end portion of a display surface of the display unit that displays the display target object.
The visual field information acquisition unit extracts a characteristic point of an object included in the visual field based on images generated by a plurality of imaging units provided in the visual line direction of the person, and the visual field The image processing apparatus according to claim 1, wherein information regarding a position of an object included in the field of view is acquired as the field of view information by performing depth detection processing for detecting a depth of the object included in the field of view.
A display unit for providing an image included in the field of view of the person to the eyes of the person;
The image processing apparatus according to claim 1, wherein the display control unit performs control to change the display mode based on position information regarding the position of the display unit, the visual field information, and the eyeball information.
The display unit displays the object to be displayed on the display surface based on a refraction image output from an image output unit disposed at an edge of the display surface that transmits an image included in the field of view of the person. A display unit,
The said display control part performs control which changes the said display mode based on the positional information regarding the display position which displays the said display target object virtually, the said visual field information, and the said eyeball information. Image processing device.
The image according to claim 1, wherein the display control unit changes the display mode by changing at least one of a display position, a display angle, and a display size of the display target object on a display surface of the display unit. Processing equipment.
The image processing apparatus according to claim 1, wherein the display control unit controls a sharpness of the display target object based on the visual field information and the eyeball information.
The image processing apparatus according to claim 8, wherein the display control unit controls the sharpness of the display target object based on a three-dimensional position in the field of view where the display target object is to be displayed.
The image processing apparatus according to claim 8, wherein the display control unit controls the sharpness by performing a blurring process on the display target object.
A posture information acquisition unit that acquires posture information related to a change in posture of the image processing apparatus;
The image processing apparatus according to claim 1, wherein the display control unit performs control to change the display mode based on the visual field information, the eyeball information, and posture information regarding the change in posture.
A view information acquisition procedure for acquiring view information regarding an image included in the view of a person to whom the image processing apparatus is attached;
Eyeball information acquisition procedure for acquiring eyeball information relating to the eyeball of the person;
A control procedure for superimposing a display target object on an image included in the field of view and changing a display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information. Image processing method.
A view information acquisition procedure for acquiring view information regarding an image included in the view of a person to whom the image processing apparatus is attached;
Eyeball information acquisition procedure for acquiring eyeball information relating to the eyeball of the person;
A control procedure for causing a display target object to overlap and display an image included in the field of view, and changing a display mode of the display target object in the image included in the field of view based on the field of view information and the eyeball information. A program to be executed.