JP2020021012A - Image processing apparatus and program - Google Patents

Abstract

To reduce the fatigue of an observer who wears a retina projection device employing a see-through system.SOLUTION: An image processing apparatus is connected to a retina projection device that projects an image on the retina of an observer, comprises means that calculates a gaze distance indicating the distance between a gazing point gazed by a user who wears the retina projection device and the observer, comprises means that converts vertex information on three-dimensional data into pixel information on a two-dimensional space, comprises means that sets, to the pixel information, a gaze area including the gazing point and a non-gaze area other than the gaze area on the basis of the gaze distance, comprises means that executes blur processing on pixel information on non-gaze pixels in the non-gaze area, and comprises means that outputs the pixel information after the blur processing is executed to the retina projection device.SELECTED DRAWING: Figure 10

Description

  The present invention relates to an image processing device and a program.

2. Description of the Related Art In recent years, attention has been focused on a retinal projection device that projects an image on a retina of an observer.
For example, Patent Literature 1 discloses a glasses-type display that projects an image generated by an image processing device onto a retina. The user of the spectacle-type display of Patent Literature 1 can observe the image projected on the retina even when moving the eyeball.

  In particular, a retinal projection device that employs a see-through method for simultaneously observing an image of a real space and an image generated by an image processing device is applicable to AR (Augmented Reality) and MR (Mixed Reality). Expected.

JP 2015-169920 A

  Generally, when observing the real space, a human turns his or her gaze to a specific point of gaze. Therefore, an image at a position distant from the periphery of the gazing point in the image of the real space looks blurred.

  However, in Patent Literature 1, since the gaze point of the observer is not considered, even if the image processing apparatus executes the blur processing, the blur of the image generated by the image processing apparatus and the real space observed by the observer Does not match the blur of the image. This difference in blur causes observer fatigue.

  That is, in the related art, in a retinal projection apparatus that employs a see-through method, fatigue caused by mismatch between a blur of a real space image and a blur of an image generated by an image processing apparatus is large.

  An object of the present invention is to reduce fatigue of an observer wearing a retinal projection device employing a see-through method.

One embodiment of the present invention provides
An image processing device connected to a retinal projection device that projects an image on a viewer's retina,
A means for calculating a gaze distance indicating a distance between a gazing point at which the user wearing the retinal projection device is gazing and the observer,
Means for converting vertex information of three-dimensional data into pixel information of two-dimensional space;
Based on the gaze distance, for the pixel information, a gaze area including the gaze point, a non-gaze area other than the gaze area, comprising means for setting,
For the pixel information of the non-gaze pixel of the non-gaze region, comprising means for performing a blur process,
Comprising means for outputting pixel information after the blur processing has been performed to the retinal projection device,
An image processing device.

  ADVANTAGE OF THE INVENTION According to this invention, the fatigue of the observer who wears the retinal projection apparatus which employs a see-through system can be reduced.

It is a figure showing appearance of a retinal projection device of this embodiment. FIG. 2 is a diagram illustrating a user view of the retinal projection device of FIG. 1. FIG. 2 is a diagram illustrating a configuration of the retinal projection device of FIG. 1. FIG. 4 is a schematic diagram illustrating an example of the light source in FIG. 3. FIG. 4 is a schematic diagram of a transmission type two-sided corner reflector array mirror which is an example of the transmission type mirror of FIG. 3. FIG. 4 is an explanatory diagram of the operation principle of the transmission mirror of FIG. 3. FIG. 4 is an explanatory diagram of a first viewpoint of an operation principle of the transmission mirror of FIG. 3. FIG. 4 is an explanatory diagram of a second viewpoint of the operation principle of the transmission mirror of FIG. 3. FIG. 5 is an explanatory diagram of a third viewpoint of the operation principle of the transmission mirror of FIG. 3. 5 is a flowchart of image processing according to the embodiment. It is explanatory drawing of the specific processing of the point of gaze of FIG. It is explanatory drawing of the area setting and blur processing of FIG.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for describing the embodiments, the same components are denoted by the same reference numerals in principle, and the repeated description thereof will be omitted.

  In the following description, “light transmittance” means a property of transmitting light (in particular, light emitted from the light source 13 (FIG. 1)), but the magnitude of the transmittance does not matter.

(1) Outline of Retina Projection Apparatus An outline of the retina projection apparatus of the present embodiment will be described. FIG. 1 is a diagram illustrating an appearance of a retinal projection device according to the present embodiment. FIG. 2 is a diagram showing a user view of the retinal projection device of FIG.

The retinal projection device 10 of FIG. 1 is configured to project an image on the retina of the observer U (FIG. 2).
As shown in FIG. 1, the retinal projection device 10 includes a frame 10a, a transmission mirror 11, an eyepiece mirror 12, a light source 13, and a light transmission plate 14.

A pair of light sources 13 and a pair of transmission mirrors 11 are embedded in the left and right frames 10a, respectively.
The light transmitting plate 14 is a material having light transmitting properties (for example, plastic or glass). The eyepiece mirror 12 is embedded in each of the pair of light transmitting plates 14.

  The optical system of the retinal projection device 10 includes a light source 13, a transmission mirror 11, and an eyepiece mirror 12. With this optical system, an image of light from the light source 13 is projected on the retina of the observer U.

As shown in FIG. 2, the retinal projection device 10 can be connected to the image processing device 30. For example, the image processing device 30 is a computer configured to generate image data of the three-dimensional image IMG and transmit the image data to the retinal projection device 10. The light source 13 emits light corresponding to the image transmitted from the image processing device 30. In this case, the view (hereinafter referred to as “observer view”) UV of the observer U wearing the retinal projection device 10 looks as if the three-dimensional image IMG is displayed in the air.
Further, since the light transmitting plate 14 transmits light reflected on the physical object existing around the observer U, the observer U sees the three-dimensional image IMG and the image of the physical object in a superimposed manner. .

  As described above, the retinal projection device 10 has a function of observing the three-dimensional image IMG projected in the air (a function as a so-called aerial display) and a function of superimposing and observing the three-dimensional image IMG and the image of the physical object ( (A function as a display for AR and MR).

  In the following description, as shown in FIG. 2, the direction of the vertex side Uht of the observer U when the retinal projection device 10 is mounted is referred to as “upper UP”, and the direction opposite to the upper UP is referred to as “lower LO”. ”, The right direction to the observer U is referred to as“ right side R ”, the left direction to the observer is referred to as“ left HL ”, the forward direction to the observer is referred to as“ front FR ”, and the observer backward. The direction is referred to as “rear RR”.

(2) Configuration of Retina Projection Apparatus The configuration of the retinal projection apparatus of the present embodiment will be described. FIG. 3 is a diagram showing a configuration of the retinal projection device of FIG.

  As shown in FIG. 3, the retinal projection device 10 can be connected to the image processing device 30. The retinal projection device 10 includes a transmission mirror 11, an eyepiece mirror 12, and a light source 13.

  The light source 13 is configured to emit light OP0 (for example, parallel light) that goes straight in a radial direction. Light OP0 emitted from the light source 13 corresponds to the original image. The original image is generated by the image processing device 30 connected to the retinal projection device 10.

The transmission mirror 11 has a first surface 11a, a second surface and 11b.
The first surface 11a faces the space on the light source 13 side (that is, the second space SP2).
The second surface 11b faces a space on the opposite side of the first surface 11a (that is, the first space SP1).

The transmission mirror 11 is disposed on the light source 13 side with respect to the eyepiece mirror 12. That is, the transmission mirror 11 is located between the light source 13 and the eyepiece mirror 12 in the optical system of the retinal projection device 10.
The transmissive mirror 11 is configured to generate the transmitted light OP1 without distorting the wavefront of light by reflecting the radially incident light OP0 incident on the transmissive mirror 11 and transmitting the reflected light in an inverse radial manner. You. The transmitted light OP <b> 1 travels straight in a reverse radial manner with the transmission mirror 11 as a starting point, and converges at the focus FP at a target position with respect to the light source 13 with the transmission mirror 11 as a symmetric axis.

The transmission mirror 11 includes, for example, the following.
・ Optical Metamaterial Mirror ・ Transmission Type Dihedral Corner Reflector Array Mirror

The eyepiece mirror 12 is disposed on the eyeball UE side of the observer U with respect to the transmission mirror 11. That is, the eyepiece mirror 12 is located between the transmission mirror 11 and the eyeball UE in the optical system of the retinal projection device 10.
As shown in FIG. 3A, the eyepiece mirror 12 reflects the light transmitted through the transmission mirror 11 toward the observer U, thereby forming an image corresponding to the light OP0 emitted from the light source 13 on the retina of the observer U. Is configured to be projected onto
In other words, as shown in FIG. 3B, the eyepiece mirror 12 is configured to transfer the virtual light source equivalent to the light source 13 to the focal point FP in space by reflecting the transmitted light of the transmission mirror 11.

  The image processing device 30 is configured to generate an image and transmit the generated image to the retinal projection device 10.

(2-1) Configuration of Light Source The configuration of the light source of the present embodiment will be described. FIG. 4 is a schematic diagram illustrating an example of the light source of FIG.

FIG. 4A is a schematic diagram of a first example of the light source 13.
As shown in FIG. 4A, a first example of the light source 13 is a laser projector 131 that emits light that goes straight radially.
The laser projector 131 is configured to emit the visible light laser OP0. The laser projector 131 includes, for example, a semiconductor laser that emits a visible light laser OP0 of an arbitrary color (for example, at least one of red, green, and blue).
The visible light laser OP0 emitted from the laser projector 131 travels radially straight from the laser projector 131 as a starting point.

FIG. 4B is a schematic diagram of a second example of the light source 13.
As shown in FIG. 4B, the second example of the light source 13 includes a display 132a and a pinhole 132b.

The display 132a is configured to emit light corresponding to the original image. The display 132a is, for example, one of the following.
-Liquid crystal display-Organic EL (Electro Luminescence) display.

Light OP0 of the display 132a is focused on the pinhole 132b.
The light OP0 converged by the pinhole 132b travels radially straight from the pinhole 132b.

FIG. 4C is a schematic diagram of a third example of the light source 13.
As shown in FIG. 4C, a third example of the light source 13 is a laser projector 133 that emits light that travels straight in a reverse radial manner.
The laser projector 133 is configured to emit the visible light laser OP0. The laser projector 131 includes, for example, a semiconductor laser that emits a visible light laser OP0 of an arbitrary color (for example, at least one of red, green, and blue).
The visible light laser OP0 emitted from the laser projector 133 travels straight in a reverse radial manner starting from the laser projector 133.
The laser projector 133 may be an SLM type or a laser scan type.

  As described above, the light source 13 may be a light source that emits light in which a light emitting source (for example, the laser projector 131 in FIG. 4A) goes straight in a radial direction, or may be a light source (for example, the display 132a in FIG. 4B). (For example, the pinhole 132b in FIG. 4B) may generate light that goes straight in a radial direction, or the light emitting source (for example, the laser projector 133 in FIG. 4C) may go straight in a reverse radial direction. It may emit light that emits light.

(2-2) Configuration of Transmission Mirror The configuration of the transmission mirror of the present embodiment will be described. FIG. 5 is a schematic diagram of a transmission type two-sided corner reflector array mirror which is an example of the transmission type mirror of FIG.

FIG. 5A shows the appearance of a transmission type dihedral corner reflector array mirror which is an example of the transmission type mirror 11.
As shown in FIG. 5A, the transmission mirror 11 has a first layer 11e including a first surface 11a and a second layer 11f including a second surface 11b. The first layer 11e and the second layer 11f are stacked in the Y direction. The first surface 11a faces in a direction opposite to the second surface 11b in the stacking direction (Y direction) of the first layer 11e and the second layer 11f.

FIG. 5B is an enlarged view of a region I in FIG. 5A.
As shown in FIG. 5B, the transmission mirror 11 has a plurality of first micromirror sheets 11ha and a plurality of second micromirror sheets 11hb.

The plurality of first micromirror sheets 11ha of the first layer 11e are arranged at a pitch p along the X direction. The reflection surface 11haa of each first micromirror sheet 11ha faces in the arrangement direction (X direction) of the first micromirror sheets 11ha.
Light incident on the first layer 11e travels while being reflected in the X direction.

The plurality of second micromirror sheets 11hb of the second layer 11f are arranged at a pitch p along the Z direction. The reflection surface 11hba of each second micromirror sheet 11hb faces in the arrangement direction (Z direction) of the second micromirror sheet 11hb.
Light incident on the second layer 11f travels while being reflected in the Z direction.

  In other words, the transmission mirror 11 is configured to transmit incident light in the Y direction while reflecting incident light in the X and Z directions. In other words, the transmission mirror 11 functions as an optical element that makes incident light recursive in the X and Z directions and specularly reflects in the Y direction.

  When the eyepiece mirror 12 is a transmissive dihedral corner reflector array mirror, the configuration of the eyepiece mirror 12 is the same as the configuration in FIG.

(3) Operation Principle of Retinal Projection Device The operation principle of the retinal projection device of the present embodiment will be described.

(3-1) Operation Principle of Transmission Mirror The operation principle of the transmission mirror of the present embodiment will be described. FIG. 6 is an explanatory diagram of the operation principle of the transmission mirror of FIG.

As shown in FIG. 6, the space is divided into a first space SP1 and a second space SP2 by an XZ plane along the plane of the transmission mirror 11.
In addition, the first space SP1 and the second space SP2 are divided into four quadrants (first quadrant Q1 to fourth quadrant Q4) by the XZ plane and the Y axis along the normal line of the transmission mirror 11. You.

The second space SP2 includes a first quadrant Q1 and a fourth quadrant Q4.
The first quadrant Q1 is a space where the incident light OP0 travels (that is, a space where the light source 13 is arranged). The first quadrant Q1 includes the Y axis (that is, the optical axis of the transmission mirror 11).
The fourth quadrant Q4 is a space opposite to the first quadrant Q1 with respect to the Y axis.

The first space SP1 includes a second quadrant Q2 and a third quadrant Q3.
The second quadrant Q2 is a space where the transmitted light OP1 travels (that is, a space including the eyeball UE of the observer U). The second quadrant Q2 includes the Y axis (that is, the optical axis of the transmission mirror 11).
The third quadrant Q3 is a space opposite to the second quadrant Q2 with respect to the Y axis.

  The first quadrant Q1 and the second quadrant Q2 are located on opposite sides with respect to the transmission mirror 11 (XZ plane) and on the same side with reference to the normal (Y axis) of the transmission mirror 11. .

  When the light emitted from the light source 13 directly enters the transmission mirror 11, the light source 13 is arranged in the first quadrant Q1.

(3-1-1) First Aspect of Operation Principle of Transmission Mirror In the first aspect, the operation of the transmission mirror 11 is considered from the viewpoint that the transmission mirror 11 is configured to change the traveling direction of the incident light OP0. The principle will be described. FIG. 7 is an explanatory diagram of a first viewpoint of the operation principle of the transmission mirror of FIG.

  As shown in FIG. 7, the light OP0 emitted from the light source 13 travels radially straight at a spread angle θ0 toward the transmission mirror 11, and enters the transmission mirror 11 from the first quadrant Q1.

  The transmission mirror 11 reflects the incident light OP0 in the XZ direction and transmits it in the Y direction, thereby emitting the transmitted light OP1 that travels straight in the reverse radial direction at the emission angle θ0 to the second quadrant Q2. The transmitted light OP <b> 1 travels straight in a reverse radial direction from the transmission mirror 11 as a starting point, and is focused at a focal point FP in the second quadrant Q <b> 2.

  As described above, the transmission mirror 11 functions as an optical element that changes the incident light OP0 radially incident from the first quadrant Q1 to the transmitted light OP1 traveling in the second quadrant Q2 in a reverse radial manner.

  When the transmission mirror 11 is a transmission dihedral corner reflector array mirror, the focal point FP has a plane-symmetric relationship with the light source 13 with respect to the transmission mirror 11. In this case, the transmission mirror 11 functions as a plane-symmetric imaging element configured to form incident light at a plane-symmetric position with respect to the light source 13.

(3-1-2) Second Perspective of Operation Principle of Transmission Mirror In a second perspective, the operation principle of transmission mirror 11 is described from the viewpoint that transmission mirror 11 is configured to transfer light source 13. I do. FIG. 8 is an explanatory diagram of a second viewpoint of the operation principle of the transmission mirror of FIG.

  As shown in FIG. 8, the light OP0 emitted from the light source 13 travels radially toward the transmission mirror 11 at a divergence angle θ0, and enters the transmission mirror 11 from the first quadrant Q1.

  The transmissive mirror 11 transmits the incident light OP0 in the second quadrant Q2 by reflecting the incident light OP0 in the XZ direction and transmitting it in the reverse radial direction at the divergence angle θ0 to the second quadrant Q2. The transmitted light OP <b> 1 travels straight in a reverse radial direction from the transmission mirror 11 as a starting point, and is focused at a focal point FP in the second quadrant Q <b> 2. The transmitted light OP <b> 1 focused at the focal point FP forms a real image of the light source 13. This phenomenon is equivalent to the virtual light source VOS corresponding to the light source 13 being formed.

  As described above, the transmission mirror 11 is a real image of the light source 13 arranged in the second space SP2 (first quadrant Q1), and transmits the virtual light source VOS equivalent to the light source 13 to the first space SP1 (second quadrant Q1). It functions as an optical element for transferring to Q2).

  When the transmission mirror 11 is a transmission dihedral corner reflector array mirror, the focal point FP has a plane-symmetric relationship with the light source 13 with respect to the transmission mirror 11. In this case, the transmission mirror 11 functions as a plane-symmetric transfer element configured to transfer the virtual light source VOS to a plane-symmetric position with respect to the light source 13.

(3-1-3) Third Viewpoint of Operation Principle of Transmission Mirror In the third view, the operation principle of transmission mirror 11 will be described from the viewpoint of the optical characteristics of transmission mirror 11. FIG. 9 is an explanatory diagram of a third viewpoint of the operation principle of the transmission mirror of FIG.

  As shown in FIG. 9, the transmission mirror 11 acts as an interface between the first space SP1 and the second space SP2. The transmission mirror 11 is inclined so as to form an angle α0 (0 ° <α0 <360 °) with respect to the optical axis of the light source 13.

  Light OP0 emitted from the light source 13 in the first quadrant Q1 of the second space SP goes straight radially toward the transmission mirror 11, and from the first quadrant Q1 the transmission mirror 11 (that is, the first space SP1). (An interface of the second space SP2) at an incident angle α0.

The transmission mirror 11 emits the transmitted light OP1 to the second quadrant Q2 by transmitting the incident light OP0 in the Y direction while reflecting the incident light OP0 in the XZ direction. The transmitted light OP1 travels straight in a reverse radial manner with the transmission mirror 11 as a starting point at an emission angle α1 (0 ° <α1 <360 °), and converges at a focal point FP in the second quadrant Q2.
When the transmission mirror 11 is a transmission dihedral corner reflector array mirror, α0 = α1.

  Both the first space SP1 and the second space SP2 are filled with air. Therefore, the refractive index as the physical property of the first space SP1 (that is, the actual refractive index of the first space SP1) is the refractive index as the physical property of the second space SP2 (that is, the actual refractive index of the second space SP2). Is equal to

However, the transmission mirror 11 acts as an interface between the first space SP1 and the second space SP2, thereby giving the first space SP1 an apparent refractive index n1. The refractive index n1 is expressed by Expression 1 according to Snell's law.
· N1 · · · the apparent refractive index of the first space SP1 · n2 · · · the refractive index as a physical property of the second space SP2 · α0 · · · incidence angle of light OP0 · α1 · · · emission angle of light OP1

When light that does not pass through the transmission mirror 11 travels through the first space SP1, the light has a refractive index as a physical property of the first space SP1. The refractive index as a physical property is generally a positive value.
On the other hand, when the transmitted light OP1 generated by the transmission mirror 11 travels in the first space SP1, the apparent refractive index n1 acts on the transmitted light OP1. Since the transmitted light OP1 can be regarded as the light obtained by refracting the incident light OP0 incident on the transmission mirror 11 from the first quadrant Q1 toward the second quadrant Q2, the apparent refractive index n1 is a negative value. is there.

  As described above, the transmission mirror 11 is configured to function as an interface that acts as the medium having the negative refractive index n1 in the first space SP1 in which the transmitted light OP1 travels.

Since the transmission mirror 11 has the optical characteristics described in any of the first to third aspects of the operation principle, no distortion of the wavefront occurs in the transmission mirror 11. Accordingly, light having no wavefront distortion enters the eyeball UE.
Thereby, distortion of the wavefront due to the optical system of the retinal projection device 10 can be prevented. As a result, an area where the observer U can clearly observe the three-dimensional image IMG is expanded.

  When the focus FP is centered on the lens of the eye UE, a virtual light source VOS is formed in the eye UE. As a result, the substantial distance between the retina and the light source 13 becomes extremely short. Thus, the viewing angle can be increased without increasing the size of the retinal projection device 10 and without distorting the wavefront.

(4) Image Processing Image processing of the present embodiment will be described. FIG. 10 is a flowchart of the image processing according to the present embodiment. FIG. 11 is an explanatory diagram of the specific processing of the point of regard in FIG. FIG. 12 is an explanatory diagram of the area setting and blur processing of FIG.

  As illustrated in FIG. 10, the image processing device 30 executes the specification of the point of regard (S100).

As a first example of Step S100, the image processing device 30 is connected to a refractive index measurement device. The refractive index measurement device is configured to measure the corneal refractive index of the eyeball UE of the observer wearing the retinal projection device 10 (see FIG. 11A).
The image processing device 30 calculates gazing point information based on the corneal refractive index measured by the refractive index measuring device. The fixation point information includes the following information.
-Three-dimensional coordinates of the gazing point NP based on the position of the eyeball UE (hereinafter referred to as "gazing coordinates") {xnp, ynp, znp}
A distance from the eyeball UE to the gazing point NP (hereinafter referred to as “gazing distance”) fnp
In the first example of step S100, a gaze point is specified by a combination of the gaze coordinates and the gaze distance.

As a second example of Step S100, the image processing device 30 is connected to a depth sensor. The depth sensor is configured to acquire depth image data in the gaze direction (the V direction in FIG. 11A) of the observer wearing the retinal projection device 10. The depth image data includes depth information D11 to D33 (FIG. 11B) related to the depth of each pixel in the image space of the depth image data.
The image processing device 30 calculates the depth information D22 of the center pixel of the depth image data acquired by the depth sensor as the gaze distance fnp.
In the second example of step S100, the gazing point is specified by the gazing distance.

As a third example of Step S100, the image processing device 30 is connected to a camera. The camera is configured to acquire image data in a gaze direction of an observer wearing the retinal projection device 10.
The image processing device 30 extracts a feature amount of the image data acquired by the camera.
The image processing device 30 estimates an image object (hereinafter, referred to as a “gaze image object”) that the observer gazes among image objects included in the image data, based on the extracted feature amount.
The image processing device 30 calculates the estimated value of the depth of the gaze image object as the gaze distance fnp.
The image processing device 30 calculates the coordinates of the gaze image object as the gaze coordinates {xnp, ynp, znp}.
In the third example of step S100, the gaze point is specified by a combination of the gaze coordinates and the gaze distance.

As a fourth example of Step S100, the image processing device 30 is connected to a plurality of eye trackers. As shown in FIG. 11C, the plurality of eye trackers are, for example, a pair of eye trackers (for example, an eye tracker that measures eye track information on the line of sight of the left eye LEEYE and an eye tracker that measures eye track information on the line of sight of the right eye REEYE). is there.
The image processing device 30 calculates the fixation coordinates {xnp, ynp, znp} and the fixation distance fnp of the fixation point NP at which the plurality of gaze directions intersect based on the plurality of eye track information measured by the plurality of eye trackers.
In the fourth example of step S100, the gaze point is specified by the combination of the gaze coordinates and the gaze distance.

After step S100, the image processing device 30 performs rasterization (S101).
Specifically, the image processing device 30 acquires three-dimensional data from a memory built in the image processing device 30 or a device connected to the image processing device 30. The three-dimensional data includes, for example, the following information.
-Includes vertex information on the vertices of an object (hereinafter referred to as "3D object") in a 3D space.
Line information on a line connecting the vertices Surface information on the surface of the three-dimensional object The image processing device 30 converts the vertex information of each vertex of the three-dimensional object into pixel information.

  After step S101, the image processing device 30 executes region setting (S102).

As a first example of step S102, when the first example or the fourth example of step S100 is executed, the image processing device 30 sets the fixation point pixel corresponding to the fixation coordinates {xnp, ynp, znp} in the image space. decide.
The image processing device 30 sets an area within a predetermined range from the point of interest pixel as a point of interest, and sets an area other than the point of interest as a non-point of interest.

When the second example or the third example of step S100 is executed as the second example of step S102, the image processing device 30 determines the center pixel in the image space as the point of interest pixel.
As illustrated in FIG. 12A, the image processing device 30 sets an area within a predetermined range from the gazing point pixel as the gazing area NA, and sets an area other than the gazing area as the non-gazing area NNA.

After step S102, the image processing device 30 executes a blur process (S103).
Specifically, the image processing device 30 calculates the non-gaze distance between the gazing point pixel and the non-gazing point pixel in the image space.
The image processing device 30 determines a blur coefficient indicating a blur level for each non-gaze point pixel based on a combination of the gaze distance fnp and the non-gaze distance.
The image processing device 30 applies a blur filter having the determined blur coefficient to the pixel information of each non-gaze point pixel.
That is, the image processing device 30 performs a blur process on the pixel information of the non-gaze area.

  As a result of step S103, a two-dimensional image (FIG. 12B) blurred according to the combination of the gaze distance fnp and the non-gaze distance is obtained.

  As shown in FIG. 12B, the blur filter is not applied to the image object IMG1 included in the watch area NA based on the watch point NP. On the other hand, a blur filter according to the non-gaze distance is applied to the image objects IMG2 to IMG3 included in the non-gaze area NNA. Specifically, since the non-gaze distance of the image object IMG2 is shorter than the non-gaze distance of the image object IMG3, the blur level of the blur filter applied to the image object IMG2 is the blur level of the blur filter applied to the image object IMG3. Lower than level.

The image object IMG1 in FIG. 12B is the image obtained in step S102.
The image object IMG2 is an image obtained by applying a relatively low-level blur filter to the image obtained in step S102.
The image object IMG3 is an image obtained by applying a relatively high-level blur filter to the image obtained in step S102.

After step S103, the image processing device 30 outputs image data (S104).
Specifically, the image processing device 30 outputs the image data of the two-dimensional image obtained in step S103 to the light source 13.

The light source 13 emits light corresponding to the image data output from the image processing device 30.
As a result, the observer wearing the retinal projection device 10 can observe the two-dimensional image obtained in step S103.

  According to the present embodiment, in step S103, blur processing is performed in consideration of the gaze distance fnp. Therefore, for the observer wearing the retinal projection device 10, the blur of the image in the real space substantially matches the blur of the image projected by the retinal projection device 10. Thereby, the fatigue of the observer can be reduced.

  In addition, according to the present embodiment, the depth of the image in the real space substantially matches the depth of the image projected by the retinal projection device 10. This makes it possible to accurately represent the positional relationship between the real space image and the image in the three-dimensional space.

(5) Modification A modification of the present embodiment will be described. The modification is a modification relating to the specification of the point of regard (S100) in FIG.

In a modified example, in the second example of step S100, the image processing device 30 is connected to a depth sensor and an eye tracker.
The eye tracker is configured to measure eye track information about the line of sight of the observer.
The image processing device 30 calculates the gaze coordinates {xnp, ynp, znp} based on the eye track information measured by the eye tracker.

  After step S101, similarly to the first example of step S102, the image processing device 30 determines a gazing point pixel corresponding to the gazing coordinates {xnp, ynp, znp}.

  According to the modification, in the case where the point of interest is specified based on the measurement result of the depth sensor, the point of interest pixel can be determined with higher accuracy.

(6) Summary This embodiment will be summarized.

A first aspect of the present embodiment is:
An image processing device 30 connected to a retinal projection device 10 that projects an image on a viewer's retina,
Means for calculating a gaze distance indicating a distance between a gaze point watched by a user wearing the retinal projection device 10 and an observer (for example, a computer that executes the process of step S100);
Means for converting vertex information of the three-dimensional data into pixel information of a two-dimensional space (for example, a computer that executes the process of step S101);
Means (for example, a computer that executes the process of step S102) for setting a gaze area including a gaze point and a non-gaze area other than the gaze area with respect to the pixel information based on the gaze distance;
Means for executing a blur process on the pixel information of the non-gaze pixel in the non-gaze region (for example, a computer for executing the process of step S103);
Means for outputting the pixel information after the blur processing has been executed to the retinal projection device 10 (for example, a computer executing the processing of step S104);
An image processing device 30.

  According to the first aspect, in the two-dimensional space of the pixel information, the blur processing is performed on the pixels in the area that the observer is not watching. Thereby, the fatigue of the observer wearing the retinal projection device adopting the see-through method can be reduced.

  Further, according to the first aspect, the depth of the image in the real space substantially matches the depth of the image projected by the retinal projection device 10. This makes it possible to accurately represent the positional relationship between the real space image and the image in the three-dimensional space.

A second aspect of the present embodiment is:
The means for performing blur processing is
The gaze coordinates of the gaze point and the coordinates of each non-gaze pixel in the non-gaze area, based on the calculated gaze distance between the gaze point and each non-gaze pixel,
For the pixel information of each non-gaze pixel, execute blur processing at the blur level according to the combination of the gaze distance and the non-gaze distance,
An image processing device 30.

  According to the second aspect, the blur processing at the blur level according to the distance from the area where the observer is gazing is executed. Thereby, the fatigue of the observer wearing the retinal projection device adopting the see-through method can be further reduced.

A third aspect of the present embodiment is:
The calculating means calculates the gaze distance and the gaze coordinates of the gaze point based on the corneal refractive index of the observer,
The setting means sets a gaze area and a non-gaze area based on the gaze coordinates,
An image processing device 30.

  According to the third aspect, the gaze coordinates are calculated based on the corneal refractive index. Accordingly, it is possible to reduce an error between the gazing region and the region where the observer is actually gazing. As a result, it is possible to further reduce the fatigue of the observer wearing the retinal projection device employing the see-through method.

A fourth aspect of the present embodiment is:
The calculating means calculates the gaze distance based on depth information of the depth image data of the observer's field of view,
An image processing device 30.

  According to the fourth aspect, the gaze distance is calculated based on the depth information. Thus, even when a relatively inexpensive depth sensor is used, the above-described image processing device 30 can be realized.

A fifth aspect of the present embodiment is:
The calculating means calculates a gaze distance based on a feature amount of the image data of the observer's field of view,
An image processing device 30.

  According to the fifth aspect, even when a relatively inexpensive image sensor is used, the above-described image processing device 30 can be realized.

A sixth aspect of the present embodiment is:
The calculating means calculates the gaze distance based on the eye track information on the line of sight of the observer,
An image processing device 30.

  According to the sixth aspect, the gaze distance is calculated based on the eye track information. Thus, even when a relatively inexpensive eye tracker is used, the above-described image processing device 30 can be realized.

A seventh aspect of the present embodiment is:
The calculating means calculates gaze coordinates of the gaze point based on the eye track information,
The setting means sets a gaze area and a non-gaze area based on the gaze coordinates,
An image processing device 30.

  According to the seventh aspect, the gaze coordinates are calculated based on the eye track information. Accordingly, it is possible to reduce an error between the gazing region and the region where the observer is actually gazing. As a result, it is possible to further reduce the fatigue of the observer wearing the retinal projection device employing the see-through method.

An eighth aspect of the present embodiment is:
Means for calculating, based on eye track information on the line of sight of the observer's eyes, calculates the gaze distance,
An image processing device 30.

  According to the eighth aspect, the gaze distance is calculated based on the line of sight of both eyes. Accordingly, it is possible to reduce an error between the gazing region and the region where the observer is actually gazing. As a result, it is possible to further reduce the fatigue of the observer wearing the retinal projection device employing the see-through method.

A ninth aspect of the present embodiment is:
A program for causing a computer to function as each of the above-described units.

(7) Other Modification Examples Other modification examples will be described.

  In the present embodiment, as an example of the light source 13, the laser projectors 131 and 133 (FIGS. 4A and 4C) and the combination of the display 132a and the pinhole 132b (FIG. 4B) are illustrated. However, the light source 13 is not limited to these as long as it outputs light that travels straight radially.

  In the example of the application, the example in which the retinal projection device 10 is connected to the image processing device 30 has been described. The present embodiment is not limited to this. The image processing device 30 may be built in the retinal projection device 10.

  The pair of light sources 13 may emit light OP corresponding to the parallax image. In this case, a parallax image is projected on the retina of the observer U. Thereby, the observer U can stereoscopically view the image corresponding to the light OP0 emitted from the light source 13.

  When the light source 13 is a laser projector 131 or 133, the laser projector may be an SLM (Spatial Light Modulator) type or a laser scan type.

  In the present embodiment, an example has been described in which one light source 13 has one laser projector 131 (FIG. 4A) or one display 132a (FIG. 4B). However, the present embodiment is not limited to this. One light source 13 may include a plurality of laser projectors 131 or a plurality of displays 132a (that is, a plurality of light emitting elements). In this case, the viewing angle of the observer U can be enlarged.

  In the present embodiment, as shown in FIG. 7, an example has been shown in which the transmission mirror 11 forms an image of the transmission light OP1 on the focal point FP at a position that is plane-symmetric with respect to the incident light OP0 with respect to the transmission mirror 11. . However, the transmission mirror 11 of the present embodiment is not limited to this. The transmission mirror 11 may form an image of the transmission light OP1 on the focal point FP of the second quadrant Q2 (including the optical axis of the transmission mirror 11) that is not plane-symmetric with respect to the transmission mirror 11.

In the present embodiment, the principle of the retinal projection device 10 has been described based on the first to third viewpoints. However, the principle of the retinal projection device 10 of the present embodiment is not limited to this. For example, the retinal projection device 10 may include a roof mirror array instead of the transmission mirror 11.
As an example, when the light source 13 is arranged in the first quadrant Q1 of FIG. 6, the incident light OP0 is reflected in the fourth quadrant Q4. As described above, the roof mirror array functions as an optical element that retroreflects the incident light OP0 radially incident from the first quadrant Q1 back to the fourth quadrant Q4.

  As mentioned above, although embodiment of this invention was described in detail, the scope of this invention is not limited to said embodiment. Further, the above-described embodiment can be variously improved or changed without departing from the gist of the present invention. Further, the above embodiments and modified examples can be combined.

10: Retinal projection device 10a: Frame 11: Transmission mirror 12: Eyepiece mirror 13: Light source 14: Light transmission plate 30: Computer 30 Image processing device 30: Image processing device 131: Laser projector 132a: Display 132b: Pinhole 133: Laser projector

Claims (9)

An image processing device connected to a retinal projection device that projects an image on a viewer's retina,
A means for calculating a gaze distance indicating a distance between a gazing point at which the user wearing the retinal projection device is gazing and the observer,
Means for converting vertex information of three-dimensional data into pixel information of two-dimensional space;
Based on the gaze distance, for the pixel information, a gaze area including the gaze point, a non-gaze area other than the gaze area, comprising means for setting,
For the pixel information of the non-gaze pixel of the non-gaze region, comprising means for performing a blur process,
Comprising means for outputting pixel information after the blur processing has been performed to the retinal projection device,
Image processing device. The means for performing the blur processing includes:
The gaze coordinates of the gaze point and the coordinates of each non-gaze pixel in the non-gaze area, based on the non-gaze distance between the gaze point and each non-gaze pixel,
For the pixel information of each non-gaze pixel, execute blur processing of a blur level according to the combination of the gaze distance and the non-gaze distance,
The image processing device according to claim 1. The calculating means calculates the gazing distance based on the corneal refractive index of the observer, and gazing coordinates of the gazing point,
The setting means sets the gaze area and the non-gaze area based on the gaze coordinates,
The image processing device according to claim 1. The calculating means calculates the gaze distance based on depth information of depth image data of the observer's field of view,
The image processing device according to claim 1. The calculating means calculates the gaze distance based on a feature amount of image data of the field of view of the observer,
The image processing device according to claim 1. The calculating means calculates the gaze distance based on eye track information on the line of sight of the observer,
The image processing device according to claim 5. The calculating means calculates gaze coordinates of the gaze point based on the eye track information,
The setting means sets the gaze area and the non-gaze area based on the gaze coordinates,
The image processing device according to claim 6. The calculating means calculates the gaze distance based on eye track information on the line of sight of the observer's eyes,
The image processing device according to claim 1.   A program for causing a computer to function as each unit according to claim 1.