US11320784B2 - Method and apparatus for processing holographic image - Google Patents
Method and apparatus for processing holographic image Download PDFInfo
- Publication number
- US11320784B2 US11320784B2 US16/717,348 US201916717348A US11320784B2 US 11320784 B2 US11320784 B2 US 11320784B2 US 201916717348 A US201916717348 A US 201916717348A US 11320784 B2 US11320784 B2 US 11320784B2
- Authority
- US
- United States
- Prior art keywords
- plane
- eye lens
- fourier transform
- focal length
- observer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 230000004438 eyesight Effects 0.000 claims description 49
- 230000002207 retinal effect Effects 0.000 claims description 48
- 230000003247 decreasing effect Effects 0.000 claims description 38
- 210000001747 pupil Anatomy 0.000 claims description 6
- 210000001508 eye Anatomy 0.000 description 88
- 238000010586 diagram Methods 0.000 description 10
- 210000001525 retina Anatomy 0.000 description 10
- 210000005252 bulbus oculi Anatomy 0.000 description 5
- 239000000470 constituent Substances 0.000 description 4
- 206010020675 Hypermetropia Diseases 0.000 description 3
- 230000004305 hyperopia Effects 0.000 description 3
- 201000006318 hyperopia Diseases 0.000 description 3
- 230000004379 myopia Effects 0.000 description 3
- 208000001491 myopia Diseases 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/16—Processes or apparatus for producing holograms using Fourier transform
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
- G03H1/0808—Methods of numerical synthesis, e.g. coherent ray tracing [CRT], diffraction specific
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/0402—Recording geometries or arrangements
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/22—Processes or apparatus for obtaining an optical image from holograms
- G03H1/2202—Reconstruction geometries or arrangements
- G03H1/2205—Reconstruction geometries or arrangements using downstream optical component
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/571—Depth or shape recovery from multiple images from focus
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/0402—Recording geometries or arrangements
- G03H2001/0441—Formation of interference pattern, not otherwise provided for
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
- G03H1/0808—Methods of numerical synthesis, e.g. coherent ray tracing [CRT], diffraction specific
- G03H2001/0816—Iterative algorithms
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/04—Processes or apparatus for producing holograms
- G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
- G03H1/0808—Methods of numerical synthesis, e.g. coherent ray tracing [CRT], diffraction specific
- G03H2001/0825—Numerical processing in hologram space, e.g. combination of the CGH [computer generated hologram] with a numerical optical element
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/22—Processes or apparatus for obtaining an optical image from holograms
- G03H1/2202—Reconstruction geometries or arrangements
- G03H1/2205—Reconstruction geometries or arrangements using downstream optical component
- G03H2001/221—Element having optical power, e.g. field lens
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
- G03H1/22—Processes or apparatus for obtaining an optical image from holograms
- G03H1/2249—Holobject properties
- G03H2001/2252—Location of the holobject
- G03H2001/2255—Holobject out of Fourier or hologram planes
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H2210/00—Object characteristics
- G03H2210/40—Synthetic representation, i.e. digital or optical object decomposition
- G03H2210/44—Digital representation
- G03H2210/441—Numerical processing applied to the object data other than numerical propagation
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
- G03H2210/00—Object characteristics
- G03H2210/40—Synthetic representation, i.e. digital or optical object decomposition
- G03H2210/45—Representation of the decomposed object
- G03H2210/454—Representation of the decomposed object into planes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
Definitions
- Example embodiments of the present disclosure relate to methods and apparatuses for processing a holographic image, and more particularly, to methods and apparatuses for processing computer-generated holograms (CGHs).
- CGHs computer-generated holograms
- a hologram is a type of three-dimensional (3D) spatial representation technology that has no limitation of the field of view and little stereoscopic fatigue since an object is reproduced in a 3D space by controlling an amplitude and phase of light. Accordingly, devices that realize high resolution holograms in real time have been developed by using a complex spatial light modulator (SLM) capable of simultaneously controlling the amplitude and phase of light.
- SLM complex spatial light modulator
- a hologram may be displayed on a 3D space by using an interference pattern of an object wave and a reference wave.
- a computer-generated hologram (CGH) technology has been utilized.
- the CGH technology may provide a hologram on a flat panel display by processing an interference pattern for reproducing a holographic video.
- a method of generating digital holograms for example, a CGH technology, generates holograms by approximating optical signals and calculating interference patterns generated through mathematical operations.
- a method of generating a digital hologram represents a finished hologram by calculating data that constitute a 3D object based on the fact that the 3D object consists of a collection of various data such as 3D points, polygons, or depth data.
- Example embodiments provide methods and apparatuses for processing a holographic image.
- a method of processing a holographic image including obtaining image data with respect to a three-dimensional (3D) object, obtaining interference patterns in a computer-generated hologram (CGH) plane by performing a Fourier transform on the image data, and generating a CGH with respect to the 3D object based on the interference patterns, wherein the Fourier transform is performed based on a focal length of an eye lens of an observer.
- 3D three-dimensional
- CGH computer-generated hologram
- the Fourier transform may include a primary Fourier transform for calculating propagation of light waves from a retinal plane to an eye lens plane of the observer, and a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane to the CGH plane based on the focal length of the eye lens of the observer.
- the method may further include storing vision information of at least one user in a memory, and reading vision information of a user matching vision information of the observer from the stored vision information of the at least one user.
- the vision information of the at least one user may include at least one of a focal length of an eye lens of the at least one user, a refractive power of a lens for correction of the at least one user, a vision of the at least one user, a distance from a retinal plane to the eye lens plane of the at least one user, and a radius of a pupil of the at least one user.
- the Fourier transform may be performed based on a distance from a retinal plane to an eye lens plane of the observer and the focal length of the eye lens of the observer.
- the method may further include obtaining a distance between an eye lens of the observer and a CGH plane, wherein the Fourier transform is performed based on the distance between the eye lens of the observer and the CGH plane.
- the method may further include reconstructing holographic images for each of an increased focal length and a decreased focal length by performing a Fourier transform while increasing or decreasing a focal length of an eye lens, and receiving a command for selecting one of the reconstructed holographic images, wherein the Fourier transform is performed based on the increased focal length or the decreased focal length corresponding to the selected reconstructed holographic image.
- the method may further include reconstructing holographic images based on an increase or a decrease in at least one of a focal length and a distance from the retinal plane to an eye lens plane by performing a Fourier transform while increasing or decreasing the at least one of the focal length and the distance from the retinal plane to the eye lens plane, and receiving a command for selecting one of the reconstructed holographic images, wherein the Fourier transform is performed based on the increased at least one of the focal length and the distance from the retinal plane to the eye lens plane or the decreased at least one of the focal length and the distance from the retinal plane to the eye lens plane corresponding to the selected reconstructed holographic image.
- the method may further include reconstructing holographic images for each increased or decreased distance between an eye lens of the observer and a CGH plane after performing the Fourier transform while increasing or decreasing the distance between the eye lens of the observer and the CGH plane, and receiving a command for selecting one of the reconstructed holographic images, wherein the Fourier transform is performed based on the increased distance between the eye lens of the observer or the decreased distance between the eye lens of the observer and a CGH plane corresponding to the selected reconstructed holographic image.
- a computer-readable recording medium on which a program for executing the method of processing a holographic image on a computer is recorded, the method including obtaining image data with respect to a three-dimensional (3D) object, obtaining interference patterns in a computer-generated hologram (CGH) plane by performing a Fourier transform on the image data, and generating a CGH with respect to the 3D object based on the interference patterns, wherein the Fourier transform is performed based on a focal length of an eye lens of an observer.
- 3D three-dimensional
- CGH computer-generated hologram
- an apparatus for processing a holographic image including a memory configured to store at least one program, and a processor configured to process a holographic image by executing the at least one program, wherein the processor is configured to obtain image data with respect to a three-dimensional (3D) object, obtain interference patterns in a computer-generated hologram (CGH) plane by performing a Fourier transform on the image data, and generate a CGH with respect to the 3D object based on the interference patterns, wherein the Fourier transform is performed based on a focal length of an eye lens of an observer.
- 3D three-dimensional
- CGH computer-generated hologram
- the Fourier transform may include a primary Fourier transform for calculating the propagation of light waves from a retinal plane to the eye lens plane of the observer, and a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane to the CGH plane by reflecting the focal length of the eye lens of the observer.
- the memory may be further configured to store vision information of at least one user
- the processor may be further configured to read vision information of a user matching the observer among the stored vision information of the at least one user stored in the memory.
- the vision information of the at least one user may include at least one of a focal length of an eye lens of the at least one user, a refractive power of a lens for correction of the at least one user, a vision of the at least one user, a distance from a retinal plane to the eye lens plane of the at least one user, and a radius of a pupil of the at least one user.
- the Fourier transform may be performed based on a distance from a retinal plane to an eye lens plane of the observer and a focal length of the eye lens of the observer.
- the processor may be further configured to obtain a distance between an eye lens of an observer and a CGH plane, and the Fourier transform may be performed based on the distance between the eye lens of the observer and the CGH plane.
- the apparatus for processing a holographic image may further include a display panel configured to display the generated CGH, and a user interface configured to receive a command of the observer.
- the display panel may be further configured to display a CGH generated by Fourier transform performed while increasing and decreasing the focal length of the eye lens and reconstructing holographic images for each of increased focal length and the decreased focal length
- the user interface may be further configured to receive a command for selecting any one of the reconstructed holographic images
- the processor may be further configured to perform the Fourier transform based on the increased focal length or the decreased focal length corresponding to the selected reconstructed holographic image.
- the display panel may be further configured to reconstruct holographic images based on increase or decrease in at least one of a focal length and a distance from the retinal plane to an eye lens plane of the observer by displaying a CGH generated by performing the Fourier transform while increasing or decreasing the at least one of the focal length and the distance from the retinal plane to the eye lens plane
- the user interface may be further configured to receive a command for selecting any one of the reconstructed holographic images
- the processor may be further configured to perform the Fourier transform based on the increased at least one of a focal length and a distance between a retinal plane to an eye lens or the decreased at least one of a focal length and a distance between a retinal plane to an eye lens corresponding to the selected reconstructed holographic image.
- the display panel may be further configured to reconstruct holographic images for each of increased or decreased distance from an eye lens of an observer to a CGH plane by displaying a CGH generated by performing a Fourier transform while increasing or decreasing the distance between the eye lens of the observer and the CGH plane
- the user interface may be further configured to receive a command for selecting any one of the reconstructed holographic images
- the processor may be further configured to perform the Fourier transform based on an the increased distance between an eye lens of an observer and a CGH plane or the decreased distance between an eye lens of an observer and a CGH plane corresponding to the selected reconstructed holographic image.
- FIG. 1 is a diagram for explaining a processing principle of a CGH according to an example embodiment
- FIG. 2 is a diagram for explaining a principle of vision correction of a user
- FIG. 3A is a diagram for describing a process of converting input image data according to an example embodiment
- FIG. 3B is a diagram illustrating a Fourier operation process in the process of converting input image data of FIG. 3A ;
- FIG. 4 is a block diagram of a hardware configuration of a holographic image display system according to an example embodiment
- FIG. 5 is a flowchart of a method of processing a holographic image, according to an example embodiment.
- FIG. 6 is a flowchart of a method of processing a holographic image, according to an example embodiment.
- the expression, “at least one of a, b, and c,” should be understood as including only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.
- FIG. 1 is a diagram for explaining a processing principle of a computer-generated hologram (CGH) according to an example embodiment.
- CGH computer-generated hologram
- An observer may recognize a 3D object 20 in a space through an eye ball 10 .
- the observer may see the 3D object 20 in a space since light reflected from the 3D object 20 is refracted through an eye lens 11 on a front face of the eye ball 10 and focused on a retina 12 on the back side of the eye ball 10 . That is, when an anterior focus of the eye lens 11 of the observer is corresponded to the 3D object 20 , it may be assumed that an image on the anterior focus has an imaging focus on the retina 12 .
- a complex light wave field in a display panel 30 may be calculated by inversely propagating an image formed on the retina 12 towards the display panel 30 , and thus, a CGH interference pattern for expressing a CGH may be obtained.
- the CGH may be classified into a point cloud method, a polygon method, a depth map (or layer-based) method according to a generation method.
- a surface of an object is expressed with a number of points and an interference pattern at each point is calculated, and thus, a precise depth may be expressed.
- the amount of calculations is greatly increased depending on the number of points.
- the depth map method is a layer-based method and generates a CGH using a 2D intensity image and depth data, and the amount of calculations may be determined according to the resolution of an image.
- the depth map method is a method of calculating a CGH after approximating a 3D object to multi-depth and modeling, the efficiency of CGH calculation may be higher than that of other methods. Also, a CGH may be generated by using only 2D intensity information and depth information such as a general picture.
- Fourier transform in a CGH processing is an operation for obtaining a distribution of diffracted images obtained by Fresnel diffraction of an image and corresponds to generalized Fresnel transform (GFT) or Fresnel transform.
- the Fourier transform may include an inverse Fourier transform (iFT), a Fast Fourier transform (FFT), an inverse Fast Fourier transform, GFT, and the like, which are operations using the Fourier transform.
- FIG. 2 is a diagram for explaining a principle of vision correction of a typical user. Hyperopia or myopia of an observer is caused because of limited ability of muscles to alter a focal length of the lens. Accordingly, as shown in FIG. 2 , for example, when an observer has hyperopia, in a naked eye, the eye lens 11 focuses an image with respect to an object beyond the retina 12 , and thus, a convex lens as a correction lens 40 is generally used to focus the image with respect to the object on the retina 12 .
- the eye lens 11 When the observer has myopia, in a naked eye, the eye lens 11 focuses an image with respect to an object before reaching the retina 12 , and thus, a concave lens as the correction lens 40 is generally used to focus the image with respect to the object on the retina 12 .
- the method of processing a holographic image reflects a vision, for example, a focal length of an eye lens, when a CGH displayed by the display panel 30 is calculated unlike the general vision correction method by using the correction lens 40 described with reference to FIG. 2 .
- a vision for example, a focal length of an eye lens
- the method of processing a holographic image reflects a vision, for example, a focal length of an eye lens, when a CGH displayed by the display panel 30 is calculated unlike the general vision correction method by using the correction lens 40 described with reference to FIG. 2 .
- a 3D object that is, a holographic image
- a virtual reality (VR) machine an augmented reality (AR) machine, or a mixed reality (MR) machine
- an eye ball of an observer is located close to a display panel.
- a numerical aperture (NA) of a lens has to have an unrealistically large value in order to form an image in the pupil of the eye ball, which may be difficult to implement.
- the method of processing a holographic image according to the example embodiment enables a viewer to watch a clearer image on the display panel 30 without a corrective lens.
- FIG. 3A is a diagram illustrating a process of converting input image data according to an example embodiment.
- FIG. 3B is a diagram illustrating a Fourier operation process included in the process of converting input image data of FIG. 3A .
- (x1, y1) represents a coordinate in a spatial light modulator (SLM) plane.
- SLM spatial light modulator
- a CGH is displayed on a display surface of an SLM 155 (refer to FIG. 4 )
- an SLM plane may be understood as a CGH plane.
- the coordinate (u, v) is a coordinate of an eye lens plane, and more strictly, a coordinate plane right in front of an eye lens.
- the coordinate (x2, y2) is a coordinate in a retinal plane of an observer.
- Equation 1 an image F(x2, y2) formed on a retinal plane is inversely propagated from the observer's retinal plane to the eye lens plane by a primary Fourier transform and, as shown in Equation 2, a focal length of the eye lens is propagated from the eye lens plane to an SLM plane by reflecting a focal length of the eye lens by a secondary Fourier transform).
- Equation 2 a complex light wave field at the SLM plane may be calculated, and in this way, a CGH interference pattern for expressing a CGH may be obtained.
- FFT may be represented by Equation 3 below.
- the depth map method is a method of calculating a CGH after approximating a 3D object to a multi-depth and modeling the multi-depth.
- a space where a hologram is reconstructed may be set to be divided into a predetermined number of depth layers.
- the number of depth layers may be any number that may be changed according to settings by a user.
- the number of depth layers may be 256 or another number.
- a CGH is calculated via a double stage Fourier transform.
- the double stage Fourier transform may denote a primary Fourier transform for calculating a propagation of light waves from a retinal plane to an eye lens plane of an observer with respect to an image, that is, a depth image, F (x2, y2) in which a depth layer is in the observer's retinal plane, and a secondary Fourier transform for calculating a propagation of light waves from the eye lens plane to a CGH plane or an SLM plane.
- the Fourier transform in CGH processing is an operation to obtain a distribution of a diffraction image obtained by Fresnel diffraction of the image, which corresponds to a GFT or Fresnel transform.
- the Fourier transform may include iFT, FFT, iFFT, GFT, Fresnel transform, etc. which are operations using the Fourier transform.
- a primary Fourier transform may be performed to calculate the propagation of light waves from the retinal plane to an eye lens plane.
- the primary Fourier transform may be embodied by Equation 4 below.
- g ⁇ ( u , v ) e - j ⁇ ⁇ ⁇ ⁇ ( 1 d 1 + 1 d 2 + 1 f ) ⁇ ( u 2 + v 2 ) ⁇ e j ⁇ ⁇ ⁇ ⁇ d 1 ⁇ ( u ⁇ x 1 + v ⁇ y 1 ) ⁇ ⁇ ⁇ e - j ⁇ ⁇ ⁇ ⁇ d 2 ⁇ ( x 2 2 + y 2 2 ) ⁇ F ⁇ ( x 2 , y 2 ) ⁇ e j ⁇ ⁇ ⁇ ⁇ d 2 ⁇ ( x 2 ⁇ u + y 2 ⁇ v ) ⁇ d ⁇ x 2 ⁇ d ⁇ y 2 [ Equation ⁇ ⁇ 4 ]
- Equation 4 it is seen that a phase profile of an eye lens plane is applied, for example, multiplied, to a result of Fourier transform of an image, that is, a depth image, F(x2, y2) in a plane.
- a focal length f of an eye lens, the distance d1 between the SLM plane and the front face of the eye lens, and the distance d2 between the front face of an eye lens of an observer and the retinal plane are reflected to the phase profile of the eye lens plane.
- a focal length f of an eye lens may vary from person to person since muscle control capacity is limited. Accordingly, a focal length f of an eye lens is read or converted from an observer's vision information, and the focal length f is reflected to a phase profile of the eye lens plane.
- the distance d1 between the SLM plane and the front face of the eye lens may be a value generally expected in a using environment of an observer. However, in some cases, it may be possible to reflect an actual environment to the phase profile by measuring the distance d1 between the SLM plane and the front face of an eye lens, or to allow the observer to select or determine the distance d1.
- a difference of the distance d2 between the front face of the eye lens and the retinal plane does not vary substantially between people, and thus, a standard value or a representative value may be used. However, embodiments are not limited thereto.
- the distance d2 between the front face of the lens and the retinal plane may be read from vision information of an observer and the distance d2 may be reflected to the phase profile of the eye lens plane.
- a secondary Fourier transform is performed with respect to the primary Fourier transform to calculate propagation of light waves from an eye lens plane to a CGH plane or SLM plane.
- the secondary Fourier transform may be embodied by Equations 5 and 6 below.
- G ⁇ ( x 1 , y 1 ) 1 ( j ⁇ ⁇ ⁇ ⁇ ⁇ d 1 ) ⁇ ( i ⁇ ⁇ ⁇ ⁇ d 2 ) ⁇ e j ⁇ ⁇ ⁇ ⁇ d 1 ⁇ ( x 1 2 + y 1 2 ) ⁇ iFFT ⁇ ( g ) [ Equation ⁇ ⁇ 5 ]
- iFFT ⁇ ( g ) ⁇ ⁇ [ e - j ⁇ ⁇ ⁇ ⁇ ( 1 d 1 + 1 d 2 + 1 f ) ⁇ ( u 2 + v 2 ) ⁇ e j ⁇ ⁇ ⁇ ⁇ d 1 ⁇ ( u ⁇ x 1 + v ⁇ y 1 ) ⁇ ⁇ ⁇ e j ⁇ ⁇ ⁇ ⁇ d 2 ⁇ ( x 2 2 + y 2 2 ) ⁇ F ⁇ ( x 2 , y 2
- the secondary Fourier transform is performed by including a phase profile of an eye lens plane to which a focal length f of the eye lens is reflected.
- FIG. 4 is a block diagram of a hardware configuration of a holographic image display system 100 according to an example embodiment.
- the holographic image display system 100 includes an apparatus for processing a holographic image 110 and a display panel 150 .
- the apparatus for processing a holographic image 110 includes a processor 112 , a memory 114 , and a user interface 116 .
- the apparatus for processing a holographic image 110 may further include other general purpose components in addition to the components shown in FIG. 4 .
- the processor 112 may correspond to a processor provided in various kinds of computing devices, such as a personal computer (PC), a server, a television (TV), a mobile device (smartphone, tablet device, etc.), an embedded device, an autonomous vehicle, a wearable device, an AR device, or an Internet of Things (IoT) device.
- the processor 112 may correspond to a processor, such as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a neural processing unit (NPU), etc., but is not limited thereto.
- the processor 112 performs an overall function for controlling the apparatus for processing a holographic image 110 equipped with the processor 112 .
- the processor 112 may control the apparatus for processing a holographic image 110 as a whole by executing programs stored in the memory 114 .
- the processor 112 may control the display of a holographic image on the display panel 150 through controlling image processing by the apparatus for processing a holographic image 110 .
- the processor 112 may include, for example, a controller that generally controls CGH processing operations and a plurality of processing cores that perform Fourier operations, for example, FFT operations.
- Fourier operations such as FFT operations correspond to operations that occupy a large portion of the CGH processing operations, and thus, the processor 112 may accelerate the CGH processing through parallel processing of a plurality of processing cores.
- Each of the plurality of processing cores may be realized by, for example, circuit logic gates in which a shift register, a multiplier, etc. are combined to perform FFT operations.
- the number of processing cores in the processor 112 may be variously changed considering various factors, such as the performance of the processor 112 , the resolution of the display panel (holographic image display apparatus) 150 , the image resolution, etc.
- the plurality of FFT cores may perform FFT operations in parallel to perform a double stage Fourier transform including a primary Fourier transform and a secondary Fourier transform.
- the holographic image display apparatus (the display panel) 150 may correspond to a device capable of displaying a holographic image in a 3D space based on a CGH generated by the apparatus for processing a holographic image 110 .
- the holographic image display apparatus 150 may include a hardware module, such as an SLM 155 for reproducing a hologram and may include various kinds of display panels, such as an liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. That is, the holographic image display apparatus 150 may include various hardware modules and hardware configurations for displaying a holographic image in addition to the apparatus for processing a holographic image 110 for generating a CGH.
- the apparatus for processing a holographic image 110 may be a separate independent device realized outside the holographic image display apparatus 150 .
- the holographic image display apparatus 150 may receive CGH data generated by the external apparatus for processing a holographic image 110 and may display a holographic image based on the received CGH data. That is, the realization method of the apparatus for processing a holographic image 110 and the display panel 150 is not limited by any one of the embodiments.
- the memory 114 is hardware that stores various data processed in the processor 112 , for example, the memory 114 may store CGH data processed in the processor 112 and CGH data to be processed. Also, the memory 114 may include vision information of at least one user.
- the vision information of the user may include at least one of a focal length of an eye lens, a refractive power (diopter) of a lens for correction, a vision of a user, a distance from a retinal plane to an eye lens plane, and a pupil radius.
- the memory 114 may store various applications, for example, a hologram playback application, a web browsing application, a game application, a video application, etc. to be driven by the processor 112 .
- applications for example, a hologram playback application, a web browsing application, a game application, a video application, etc. to be driven by the processor 112 .
- the memory 114 may include at least one of a volatile memory or a nonvolatile memory.
- the nonvolatile memory includes read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), a flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc.
- the volatile memory includes dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, FRAM, etc.
- the memory 114 may include a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF) memory, a secure digital (SD) memory, a micro secure digital (micro-SD) memory, a mini-secure digital (mini-SD) memory, an extreme digital (xD) memory, or a memory stick.
- HDD hard disk drive
- SSD solid state drive
- CF compact flash
- SD secure digital
- micro-SD micro secure digital
- mini-SD mini-secure digital
- xD extreme digital
- xD extreme digital
- a user interface 116 is a device configured to receive a user's (observer's) command, receive vision information of the user (observer), or display a state of the use of a holographic image display system.
- the user interface 116 may include a physical button, a joggle, a touch screen, etc.
- the user interface 116 may include a port that performs input/output with an external electronic device.
- the processor 112 reads 2D intensity images and depth data with respect to a 3D object from which a holographic image is to be generated.
- the processor 112 obtains a plurality of depth images of a predetermined resolution with respect to a predetermined number of depth layers from the depth data of the 3D object.
- the predetermined resolution may be variously changed by a user's settings or an input image format.
- the processor 112 calculates a CGH through a double stage Fourier transform.
- the processor 112 performs a primary Fourier transform on each of depth images of different depth layers projected on a retina plane to calculate the propagation of light waves from a retinal plane to an eye lens plane.
- the processor 112 may perform a primary Fourier transform, for example, in the same manner as Equation 4 described above.
- the processor 112 reads a focal length f of an eye lens of an observer from vision information stored in the memory 114 .
- the memory 114 may store a lookup table for converting refractive power (diopter) of a lens for correction or a user's vision to a focal length f of an eye lens.
- a focal length f of an eye lens may be converted from the refractive power (diopter) of the lens for correction or the user's vision by using the lookup table.
- the processor 112 may use a value generally expected in an observer's usage environment as a distance d1 between an SLM plane and a front face of an eye lens in a process of primary Fourier transform, or, it may be possible to reflect an actual environment to the phase profile by measuring the distance d1 between the SLM plane and the front face of an eye lens, or to allow the observer to select or determine the distance d1.
- the processor 112 may use a standard value or a representative value as the distance d2 between a front face of an eye lens and a retinal plane in a primary Fourier transform, or, optionally, the distance d2 between a front face of an eye lens and a retinal plane may be read from the viewer's vision information and the distance d2 may be reflected to a phase profile of the eye lens plane.
- the processor 112 performs a secondary Fourier transform with respect to the primary Fourier transform to calculate the propagation of light waves from an eye lens plane to a CGH plane or SLM plane.
- the processor 112 may perform the secondary Fourier transform, for example, in the same manner as in Equations 5 and 6 as described above.
- the processor 112 performs the secondary Fourier transform including the phase profile of the eye lens plane to which the focal length f of the eye lens is reflected.
- the processor 112 obtains an interference pattern for each of depth images of different depth layers projected on a retinal plane on a CGH plane or SLM plane through the calculation process described above.
- FIG. 5 is a flowchart of a method of processing a holographic image according to an example embodiment.
- the processor 112 may read user vision information stored in the memory 114 ( FIG. 4 ).
- the memory 114 may store at least one user's vision information.
- the processor may list at least one user stored in the memory 114 and select at least one user from the listed users.
- the memory 114 may store identification information of at least one user, and the holographic image display system may automatically match an observer currently being used in the stored user list by identifying the user through recognition of the identification information of the user.
- the identification information of the user may include, for example, at least one of a face, a voiceprint, and a fingerprint.
- the processor 112 ( FIG. 4 ) reads 2D intensity images and depth data with respect to a 3D object from which a holographic image is to be generated.
- the processor 112 performs a primary Fourier transform with respect to an image of the 3D object from a retinal plane to an eye lens plane based on user's vision information.
- the processor 112 performs a secondary Fourier transform on the image that is primarily transformed into an SLM plane (or a CGH plane) based on user's vision information.
- an interference pattern of CGH is obtained based on a result of performing the secondary Fourier transform.
- FIG. 6 is a flowchart of a method of processing a holographic image according to an example embodiment.
- the processor 112 ( FIG. 4 ) reads 2D intensity images and depth data with respect to a 3D object from which a holographic image is to be generated.
- the processor 112 performs a primary Fourier transform on an image of the 3D object from a retinal plane to an eye lens plane based on a set focal length.
- the processor 112 performs a secondary Fourier transform on the image that is primarily transformed into the eye lens plane to an SLM plane or a CGH plane based on the set focal length.
- a CGH is generated by obtaining an interference pattern in the SLM plane or CGH plane based on the result of performing the secondary Fourier transform.
- the generated CGH is displayed on a display panel.
- an observer confirms whether the holographic image for his vision is being reconstructed or not through a user interface.
- the user may not select the reconstructed holographic image but may reset the focal length by varying, such as increasing or decreasing, a focal length set by a predetermined amount in operation 607 , and then operations 602 to 606 may be repeated.
- a holographic image suitable for the observer's own vision is reconstructed, the corresponding holographic image is selected and a CGH with a controlled focus selected is displayed on a display panel in operation 608 .
- the example embodiment is a case that holographic images corresponding to the increased or decreased focal length are sequentially displayed according to the increase or decrease of the focal length, and an observer selects a holographic image from the sequentially displayed holographic images, but is not limited thereto.
- the holographic images corresponding to the increased and decreased focal lengths may be stored in a memory and may be displayed simultaneously in a thumbnail format to allow an observer to select an optimal holographic image suitable for his or her vision.
- the example embodiments may be implemented as a computer-readable recording program and may be realized in general computers that execute the program by using computer-readable recording media. Also, the structure of the data used in the above example embodiments may be recorded on a non-transitory computer-readable recording medium through various means.
- the computer-readable medium may be magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROMs or DVDs), and transmission media such as Internet transmission media.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Holo Graphy (AREA)
Abstract
Description
g=˜iFFT(F) [Equation 1]
G=˜iFFT(g) [Equation 2]
Claims (18)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020190080316A KR20210004232A (en) | 2019-07-03 | 2019-07-03 | Method and apparatus for processing holographic image |
KR10-2019-0080316 | 2019-07-03 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210003967A1 US20210003967A1 (en) | 2021-01-07 |
US11320784B2 true US11320784B2 (en) | 2022-05-03 |
Family
ID=74065708
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/717,348 Active 2040-07-27 US11320784B2 (en) | 2019-07-03 | 2019-12-17 | Method and apparatus for processing holographic image |
Country Status (2)
Country | Link |
---|---|
US (1) | US11320784B2 (en) |
KR (1) | KR20210004232A (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20220107753A (en) * | 2021-01-26 | 2022-08-02 | 삼성전자주식회사 | Display apparatus including visual correction lens |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090180165A1 (en) * | 2005-12-22 | 2009-07-16 | See Real Technologies S.A. | Method for the Compensation of an Inhomogeneous Brightness Perception in Holographically Reconstructed Scenes |
KR20150068276A (en) | 2013-12-11 | 2015-06-19 | 엘지전자 주식회사 | Head Mounted Display and controlling method |
US9619021B2 (en) | 2013-01-09 | 2017-04-11 | Lg Electronics Inc. | Head mounted display providing eye gaze calibration and control method thereof |
KR20180007547A (en) | 2016-07-13 | 2018-01-23 | 삼성전자주식회사 | Method and apparatus for processing three dimensional image |
US20180129105A1 (en) | 2016-11-10 | 2018-05-10 | Samsung Electronics Co., Ltd. | Liquid crystal light deflector |
US20180181065A1 (en) | 2016-12-23 | 2018-06-28 | Samsung Electronics Co., Ltd. | Holographic display and operating method thereof |
US20180284693A1 (en) | 2017-03-30 | 2018-10-04 | Samsung Electronics Co., Ltd. | Holographic display apparatus having steerable viewing window |
US20180321426A1 (en) | 2017-05-02 | 2018-11-08 | SoliDDD Corp. | Voltage adjustable near-eye optical lens |
US20190049732A1 (en) | 2017-08-09 | 2019-02-14 | Samsung Electronics Co., Ltd. | Optical window system and see-through type display apparatus including the same |
US20190113761A1 (en) | 2017-10-13 | 2019-04-18 | Samsung Electronics Co., Ltd. | Beam expander and display including the same |
-
2019
- 2019-07-03 KR KR1020190080316A patent/KR20210004232A/en not_active Application Discontinuation
- 2019-12-17 US US16/717,348 patent/US11320784B2/en active Active
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090180165A1 (en) * | 2005-12-22 | 2009-07-16 | See Real Technologies S.A. | Method for the Compensation of an Inhomogeneous Brightness Perception in Holographically Reconstructed Scenes |
US9619021B2 (en) | 2013-01-09 | 2017-04-11 | Lg Electronics Inc. | Head mounted display providing eye gaze calibration and control method thereof |
KR20150068276A (en) | 2013-12-11 | 2015-06-19 | 엘지전자 주식회사 | Head Mounted Display and controlling method |
US10175651B2 (en) | 2016-07-13 | 2019-01-08 | Samsung Electronics Co., Ltd. | Method and apparatus for processing three-dimensional image |
KR20180007547A (en) | 2016-07-13 | 2018-01-23 | 삼성전자주식회사 | Method and apparatus for processing three dimensional image |
US20180129105A1 (en) | 2016-11-10 | 2018-05-10 | Samsung Electronics Co., Ltd. | Liquid crystal light deflector |
US20180181065A1 (en) | 2016-12-23 | 2018-06-28 | Samsung Electronics Co., Ltd. | Holographic display and operating method thereof |
KR20180074154A (en) | 2016-12-23 | 2018-07-03 | 삼성전자주식회사 | Holographic display and method of operating the same |
US20180284693A1 (en) | 2017-03-30 | 2018-10-04 | Samsung Electronics Co., Ltd. | Holographic display apparatus having steerable viewing window |
US20180321426A1 (en) | 2017-05-02 | 2018-11-08 | SoliDDD Corp. | Voltage adjustable near-eye optical lens |
US20190049732A1 (en) | 2017-08-09 | 2019-02-14 | Samsung Electronics Co., Ltd. | Optical window system and see-through type display apparatus including the same |
CN109387941A (en) | 2017-08-09 | 2019-02-26 | 三星电子株式会社 | Optics window system and Clairvoyant type display device including it |
US20190113761A1 (en) | 2017-10-13 | 2019-04-18 | Samsung Electronics Co., Ltd. | Beam expander and display including the same |
Also Published As
Publication number | Publication date |
---|---|
US20210003967A1 (en) | 2021-01-07 |
KR20210004232A (en) | 2021-01-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Chakravarthula et al. | Wirtinger holography for near-eye displays | |
US9965836B2 (en) | Method and apparatus for performing fourier transformation | |
KR20200104068A (en) | Method and apprartus for generating deep learning-based digital hologram | |
US20230359145A1 (en) | Method and apparatus for processing three-dimensional holographic image | |
US20230205133A1 (en) | Real-time Photorealistic 3D Holography With Deep Neural Networks | |
CN106707680B (en) | A kind of holographic projection methods based on light field | |
CN107622521B (en) | Method and apparatus for processing three-dimensional image | |
Plesniak et al. | Reconfigurable image projection holograms | |
Matsubara et al. | Light field display simulation for light field quality assessment | |
US9811052B2 (en) | Hologram generation apparatus and method | |
Chakravarthula et al. | Computing high quality phase-only holograms for holographic displays | |
US11320784B2 (en) | Method and apparatus for processing holographic image | |
US20240027961A1 (en) | Method and apparatus for generating computer-generated hologram | |
US10930183B2 (en) | Hologram content generation apparatus, integrated hologram content control system having the same, and method for operating the hologram content generation apparatus | |
KR102501302B1 (en) | Method and apparatus for generating multi-depth hologram | |
Yang et al. | Fast precalculated triangular mesh algorithm for 3D binary computer-generated holograms | |
US11561511B2 (en) | Method and apparatus for generating hologram with wide viewing angle | |
KR20220023953A (en) | Method and system for processing computer-generated hologram | |
US11740587B2 (en) | Method and apparatus for generating computer-generated hologram | |
Kazempourradi et al. | Fast computer-generated hologram computation using rendered depth map image | |
CN113973198B (en) | Holographic image generation method, device, equipment and computer readable storage medium | |
Zhou | Accurate depth based post-processing for perception enhancement in real time three-dimensional graphics | |
WO2019062306A1 (en) | Holographic display method and device | |
KR20200111598A (en) | Method and apparatus for processing three dimensional holographic image | |
Ahmad et al. | Holographic interface management in the age of artificial intelligence |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KIM, YOUNG;SEO, WONTAEK;AN, JUNGKWUEN;AND OTHERS;SIGNING DATES FROM 20191126 TO 20191210;REEL/FRAME:051308/0434 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |