WO2018105786A1

WO2018105786A1 - Holographic image processing device and method

Info

Publication number: WO2018105786A1
Application number: PCT/KR2016/014400
Authority: WO
Inventors: 홍지수; 김영민; 홍성희; 강훈종; 신춘성
Original assignee: 전자부품연구원
Priority date: 2016-12-08
Filing date: 2016-12-08
Publication date: 2018-06-14
Also published as: KR20180065819A; KR101927592B1

Abstract

A holographic image processing device is disclosed. The holographic image processing device for a three-dimensional graphic object, according to the present invention, whereby a three-dimensional graphic object is realized by means of a point cloud method, comprises: a region configuration unit for dividing the visual field of a user into multiple regions; a sampling unit for sampling, per each divided region, points from a three-dimensional graphic object within the visual field of the user, wherein the point density of the sampled points is different per each divided region; and a fringe pattern generation unit for generating a fringe pattern corresponding to each of the sampled points. The present invention has an effect whereby a hologram may be generated at a high speed without degrading the resolution of a holographic reproduction image.

Description

[Revision 16.02.2017] under Rule 26 Holographic Image Processing Apparatus and Method

The present invention relates to a holographic image processing apparatus and method, and more particularly, to a holographic image processing technology capable of generating a hologram at high speed without reducing the resolution of the holographic reproduction image.

Recently, studies on stereoscopic (3D) image and image reproduction technology have been actively conducted. Stereoscopic media is expected to lead the next generation of video devices as a new concept of visual media that raises the level of visual information. Conventional 2D imaging systems provide planar images, but 3D imaging systems can be said to be the ultimate image realization technology in view of showing the actual image information of an object to an observer.

Among the methods for reproducing a stereoscopic image, a holography method is a method of recognizing a real stereoscopic image without wearing special glasses when observing a hologram produced using a laser. Accordingly, the holography method is known to be the most ideal way for the viewer to feel the three-dimensional image without fatigue and excellent stereoscopic feeling.

The holography method uses a principle of recording and reproducing an interference signal obtained by superposing an interference light (object wave) and an interference light (reference wave). The recording of an interference fringe (fringe pattern) formed by causing an object wave that collides with an object using a highly coherent laser light and meets a reference wave incident in another direction is called a hologram. When the object wave and the reference wave meet, an interference fringe is formed by the interference, and the amplitude and phase information of the object are recorded together with the interference fringe. The holography is a method of restoring a stereoscopic image recorded on a hologram to a stereoscopic image by irradiating the reference light onto the interference fringes thus recorded.

A computer generated hologram (CGH) has been developed as a method for generating a hologram by a computer for storing, transmitting and image processing. The computer-generated hologram (CGH) generates hologram interference pattern data by calculating an interference pattern between the object light and the reference light using a computer, and transmits the generated hologram interference pattern data to a spatial light modulator (SLM). Then, the reference light using the laser is irradiated to the spatial light modulator (SLM) to restore and reproduce the stereoscopic image corresponding to the hologram interference pattern implemented in the spatial light modulator.

In general, CGH generation technology is largely divided into a three-dimensional information-based point source method and an integrated image-based light field method. The point light source CGH generation technique assumes a three-dimensional graphic object as a set of three-dimensional points (Point cloud), calculates and adds a hologram pattern for each point. At this time, the more detailed or complex the 3D graphic object is, the more the number of points included in the 3D graphic object increases, which causes a problem in that the hologram cannot be generated at high speed as the amount of computation increases rapidly.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide a method and a system capable of generating holograms at high speed without reducing the resolution of the holographic playback image.

In addition, the present invention provides a real-time video hologram by providing a technique for generating a hologram at high speed.

Other objects of the present invention will become more apparent through the preferred embodiments described below.

According to an aspect of the present invention, there is provided a holographic image processing apparatus for a 3D graphic object for implementing a 3D graphic object in a point cloud method, the holographic image processing apparatus comprising: an area setting unit for dividing a user's field of view into a plurality of areas; A sampling unit for sampling points for each of the divided regions from the three-dimensional graphic object within the viewing range of the user, wherein the point density of the sampled points is different for each of the divided regions; And a fringe pattern preparing unit which creates a fringe pattern corresponding to each of the sampled points.

In example embodiments, the holographic image processing apparatus may further include a fringe pattern interpolator configured to copy at least one of the created fringe patterns to interpolate the created fringe patterns.

In an embodiment, the interpolation of the fringe patterns may be performed on an area in which the point density of the points is smaller than a preset value.

In an exemplary embodiment, interpolation of the fringe patterns may be performed by shifting the copied fringe pattern by a predetermined distance from a position of the copy target fringe pattern.

In one embodiment, the holographic image processing apparatus may further include a fringe pattern synthesis unit for synthesizing the fringe patterns.

In one embodiment, the plurality of regions may be separated by concentric circles.

In one embodiment, the point density may be lower in the area farther from the center of the concentric circles.

In one embodiment, the holographic image processing apparatus may further include a tracking unit that tracks the gaze point of the user.

In one embodiment, the plurality of areas may be divided by concentric circles around the gaze point of the user.

According to another aspect of the present invention, a holographic image processing apparatus for implementing a 3D graphic object in a point cloud method, the sampling unit for sampling points from the 3D graphic object; A fringe pattern generator which creates a fringe pattern corresponding to each of the sampled points; And a fringe pattern interpolator configured to copy at least one of the created fringe patterns to interpolate the created fringe patterns.

According to still another aspect of the present invention, there is provided a holographic image processing method for implementing a 3D graphic object in a point cloud method, the method comprising: dividing a field of view of a user into a plurality of regions; Sampling points per segmented area from a three-dimensional graphic object within the viewing range of the user, wherein the point density of the sampled points differs for each segmented area; And creating a fringe pattern corresponding to each of the sampled points.

As described above, according to embodiments of the present invention, the present invention can generate holograms at high speed without degrading the resolution of the holographic playback image.

In addition, the present invention can provide a real-time video hologram by providing a technique for generating a hologram at high speed.

1 is a flowchart illustrating a holographic image processing method according to an embodiment of the present invention.

2 is a diagram illustrating a holographic image processing flow according to an embodiment of the present invention.

3 illustrates an example of a forbidden rendering technique.

4 is a view for explaining a hole peeling technique.

5 is a view showing an image to which the hole peeling technique is applied.

6 illustrates an interpolated fringe pattern according to another embodiment of the present invention.

7 is a graph showing the relationship between time and point density for creating and synthesizing a fringe pattern according to an embodiment of the present invention.

8 is a flowchart illustrating a holographic image processing method performed in a holographic image processing apparatus according to an embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a configuration of a holographic display system according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating a holographic image processing flow according to an embodiment of the present invention.

Referring to FIG. 1, the holographic display system 100 includes a holographic image processing apparatus 110, a gaze information acquirer 120, and a display 130.

The holographic image processing apparatus 110 analyzes a 3D graphic object to create and synthesize a fringe pattern. The detailed operation and configuration of the holographic image processing apparatus 110 will be described later.

The display 130 performs a function of reproducing a 3D holographic image. The display unit 130 may be a spatial light modulator (SLM), a light emitting diode (LED) display, or the like, and any device having a function of reproducing other 3D holographic images may be applied without limitation. have. In addition, the spatial light modulator may be an acoustooptic modulator (AOM), a liquid crystal display (LCD), a digital micromirror display (DMD), or the like.

The gaze information acquirer 120 performs a function of acquiring gaze information of a user looking at the display 130. For example, the gaze information acquirer 120 may be a camera. When the gaze information acquisition unit 120 is a camera, the gaze information is image information obtained through the camera. Here, the gaze information acquisition unit 120 may be a camera array composed of a plurality of cameras. The camera array may acquire a plurality of channel images in all directions based on the display 130 using a plurality of cameras. In the camera array according to an exemplary embodiment, a plurality of cameras may be arranged in a circular shape to obtain image information at an angle of 360 degrees. In this case, each camera has channel information, and the intervals between the cameras may be uniformly arranged at a predetermined distance or at a predetermined angle. Alternatively, the camera may be concentrated in a predetermined area depending on the situation.

In addition, the gaze information acquisition unit 120 may be applied without limitation as long as the device can acquire gaze information of the user in addition to the camera. For example, the system may include a light (eg, infrared light) emitter and a light receiver, a system configured by an ultrasonic emitter and a sound pressure sensor, and the like.

However, for the sake of clarity of understanding and explanation of the present invention, the gaze information acquisition unit 120 is a camera, and the gaze information is described as an example of image information acquired through the camera.

The holographic image processing apparatus 110 includes an area setting unit 112, a sampling unit 113, and a fringe pattern preparing unit 114. The holographic image processing apparatus 110 may further include one or more of the tracking unit 111, the fringe pattern interpolation unit 115, and the fringe pattern synthesis unit 116, according to various modified embodiments. Hereinafter, the holographic image processing apparatus 110 is limited to include all the above-described configurations for convenience of clear understanding and explanation of the invention, and it will be described according to the reference numeral order.

The tracking unit 111 analyzes the image information obtained from the camera 120 to track the gaze point of the user. For example, the tracking unit 111 may track the gaze point of the user by interpreting the position of the user's pupil and the information to which the pupil is directed from the image information. At this time, the gaze point may be the center of a field of view of use.

The area setting unit 112 divides the user's view toward the display 130 into a plurality of areas. In this case, each region may have various shapes (eg, circles, polygons, etc.) and various arrangements (eg, vertical arrangement, horizontal arrangement, overlapping arrangement, rectangular arrangement, etc.). According to an embodiment, the plurality of regions may be regions separated by concentric circles having different radii. Hereinafter, the shape and arrangement of the plurality of regions will be described with reference to the example illustrated in FIG. 2, but the present invention is not limited thereto.

According to an embodiment of the present invention, as shown in FIG. 2, the user's field of view is a rectangle 220, and the field of view may be divided into two concentric circles inside the rectangle. Accordingly, the user's field of view is divided into three center regions 221, an intermediate region 222, and an outer region 223.

The sampling unit 113 performs a function of sampling points from the 3D graphic object 210, and the sampling may be downsampling. In this case, the sampling unit 113 samples the points by the

respective areas

221, 222, and 223 from the 3D graphic object 210 within the user's field of view, and the point density of the sampled points is determined by the respective areas 221,. 222, 223) may be different. As shown in FIG. 2, the point density of the sampled points may be lowered from the central area 221 to the outer area 223.

Referring to FIG. 2, a result of sampling the point density differently for each of the plurality of

regions

221, 222, and 223 based on the point light source method is shown at 230. have. In reference numeral 230, the point density of the points decreases as the user moves away from the gaze point (ie, the center of the field of view).

The fringe pattern generator 114 creates a fringe pattern corresponding to each of the sampled points. Creating a fringe pattern corresponding to each point light source based on the point light source method includes a Rayleigh Sommerfeld approach, Fresnel and Fraunhofer approximation. The method of preparation is well known in the art and will be apparent to those skilled in the art, so detailed description thereof will be omitted.

The fringe pattern synthesizing unit 116 performs a function of synthesizing the created fringe pattern. Synthesis results according to one embodiment are shown at 240. The synthesized fringe pattern of reference numeral 240 is displayed on the display unit 130, and when the reference light is irradiated thereto, the user can see the 3D holographic image.

The biggest problem in implementing a 3D holographic image is that it takes too much time to create a fringe pattern, which acts as an obstacle in reproducing the 3D holographic image in real time. In the point light source CGH generation technique, when the number of points sampled from the 3D graphic object 210 is increased, the computation time for CGH generation is increased. On the contrary, when the number of points is decreased, the computation time is also reduced.

According to one embodiment of the present invention described above, since the fringe pattern is created for the points sampled from the 3D graphic object 210, the number of sampled points is calculated to generate the CGH of the 3D graphic object 210. Significant impact on time However, as the number of points sampled from the 3D graphic object 210 decreases, a secondary problem occurs in which the resolution of the 3D holographic image decreases. However, in the exemplary embodiment of the present invention, since the downsampling is performed to decrease the point density from the center of the user's field of view to the outside, the change of the resolution actually felt by the user is insignificant compared to the decrease of the point, and the reason for this is already known. This can be seen in the Focused Rendering technology.

3 is a diagram illustrating an example of a embedded rendering technique.

Due to the distribution of receptors in the human retina, human vision is highest in the center of the field of vision (ie, the gaze point), and drops toward the periphery. Thus, the quality of the images provided within the human field of view need not all be the same. That is, the resolution of the image felt by the human being does not show a big difference when the quality of the image provided within the human field of view is provided as a whole as high as the whole from the center of the field of view and lower as the outer area.

Referring to FIG. 3, when the image 310 is divided into three

regions

321, 322, and 333, and the resolution is decreased from the center to the outside, the image is synthesized, and the image is a reference number 330. The process is called forbidden rendering. If the gaze point of the user is aimed at the center of the

images

310 and 330, the user may not feel a difference in resolution between the image of the reference numeral 310 and the image of the reference numeral 330.

As a result, according to an embodiment of the present invention in which the forbidden rendering technique is applied to the CGH generation technique, the computation speed may be considerably reduced without substantially reducing the resolution of the 3D holographic image. However, in the present embodiment, if downsampling is excessively performed at the edge (peripheral portion) of the user's view, there is a risk that the user perceives that the resolution is sharply dropped. In order to solve this problem, in another embodiment of the present invention, the holographic image processing apparatus 110 further includes a fringe pattern interpolator 115.

The fringe pattern interpolator 115 performs a function of interpolating the created fringe patterns by copying at least one of the created fringe patterns. At this time, interpolation of the fringe pattern is to shift the copied fringe pattern by a predetermined distance from the position of the fringe pattern to be copied. Referring to Figure 2 for a more detailed description, since the user's vision is the lowest in the outer region 223, if the point density in the outer region 223 is maintained only above a certain level, the user is reproduced through this three-dimensional holo The resolution difference of the graphic image is not well recognized. However, if the point density is not lowered below a certain level, a problem arises that the CGH generation time can no longer be reduced. The fringe pattern interpolator 115, which is a component for solving such a problem, refers to a technical idea of a known hole filling method.

4 is a view for explaining the hole peeling method, Figure 5 is a view showing an image to which the hole peeling method is applied, Figure 6 is a view showing an interpolated fringe pattern according to another embodiment of the present invention.

Referring to FIG. 4, the hole peeling method performs downsampling on the reference image 410 (1/2 downsampling in the embodiment shown in FIG. 4, 420), and then disappears the pixels 412 through downsampling. Copy and fill the pixel 411 adjacent to the position of. In this case, although the copied and filled pixel 411 is not the same as the pixel 412 disappeared through downsampling, the user may apply the image to the image 430 after the hole peeling technique is applied rather than the image 420 before the hole peeling technique is applied. A more dramatic effect can be achieved if the hole-filling technique is applied at the periphery of the field of vision, which is poorly perceived.

In FIG. 5, reference numeral 510 is a reference image, reference numeral 520 is an image to which the hole peeling technique is applied after 1/2 downsampling, and reference numeral 530 is an image to which the hole peeling technique is applied after 1/3 downsampling. 540 is an image to which the hole peeling technique is applied after 1/4 downsampling.

In an embodiment to which the fringe pattern interpolator 115 is applied, the hole peeling technique is applied to the fringe pattern corresponding to each of the downsampled points, rather than being applied to the downsampled points to lower the point density. This is because, if the hole-filling technique is applied to the downsampled points, the point density lowered through downsampling is returned to the original state, and the CGH generation time is not reduced if there is no change in the number of points. However, if a fringe pattern is first created for the downsampled points, and then the fringe pattern of points adjacent to the disappeared points is copied and positioned at the positions of the missing points through downsampling (see FIG. 6), a fringe pattern should be created. Since the number of points can be reduced and the fringe pattern can be filled at the position where the points disappear, the CGH generation time can be greatly reduced while reducing the possibility of the user noticing that the resolution of the 3D holographic image is greatly reduced. Especially in the periphery of the field of vision, which has poor vision, a more dramatic effect can be seen.

7 is a graph illustrating a relationship between time and point density for preparing and synthesizing a fringe pattern according to an embodiment of the present invention.

In FIG. 7, the x axis represents the density of the points to be sampled, and the y axis represents the normalized CGH calculation time. Looking at the graph in the area 710, it can be seen that the CGH calculation time is dramatically reduced as the point density of the sampled points decreases. The slight increase in CGH computation time in the graph within area 720 is due to the involvement of the process of interpolating fringe patterns according to embodiments of the present invention.

In operation S810, the gazing point of the user may be tracked by interpreting the image information acquired from the camera 120. For example, the tracking unit 111 may track the gaze point of the user by interpreting the position of the user's pupil and the information to which the pupil is directed from the image information. At this time, the gaze point may be the center of a field of view of use.

In operation S820, the holographic image processing apparatus 110 divides a user's view toward the display 130 into a plurality of regions. According to an embodiment, the plurality of regions may be regions divided by concentric circles having different radii.

In operation S830, the holographic image processing apparatus 110 performs sampling of points for each area divided from a 3D graphic object within a user's field of view. In this case, the point density of the sampled points may be different for each divided region.

In operation S840, the holographic image processing apparatus 110 generates a fringe pattern corresponding to each of the sampled points. Methods of creating a fringe pattern corresponding to each point light source based on the point light source method include a Rayleigh Sommerfeld approach, Fresnel and Fraunhofer approximation.

In operation S850, the holographic image processing apparatus 110 performs an interpolation of the created fringe patterns by copying at least one of the created fringe patterns. At this time, interpolation of the fringe pattern is to shift the copied fringe pattern by a predetermined distance from the position of the fringe pattern to be copied.

In operation S860, the holographic image processing apparatus 110 performs a step of synthesizing the created fringe pattern.

Although the above-described holographic image processing method has been sequentially described from S810 to S860 in the order shown in FIG. 8, the holographic image processing method may be combined with some steps of S810 to S860 with reference to various embodiments described with reference to FIGS. 1 to 7. Embodiments may be derived and it is obvious that such combination embodiments are within the scope of the present invention.

While the above has been shown and described with respect to preferred embodiments of the present invention, the present invention is not limited to the specific embodiments described above, it is usually in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Preferred embodiments of the present invention described above are disclosed for purposes of illustration, and those skilled in the art will be able to make various modifications, changes, and additions within the spirit and scope of the present invention. Additions should be considered to be within the scope of the following claims.

Claims

In the holographic image processing apparatus for a three-dimensional graphics object that implements a three-dimensional graphics object in a point cloud method,

An area setting unit dividing a field of view of the user into a plurality of areas;

A sampling unit for sampling points for each of the divided regions from the three-dimensional graphic object within the viewing range of the user, wherein the point density of the sampled points is different for each of the divided regions; And

And a fringe pattern creating unit for creating a fringe pattern corresponding to each of the sampled points.
The method of claim 1,

And a fringe pattern interpolator configured to copy at least one of the created fringe patterns to interpolate the created fringe patterns.
The method of claim 2,

Interpolation of the fringe patterns,

The holographic image processing apparatus according to claim 1, wherein the point density of the points is smaller than a preset value.
The method of claim 2,

Interpolation of the fringe patterns,

And copying the copied fringe pattern by a predetermined distance from a position of the copy target fringe pattern.
The method according to any one of claims 1 to 4,

And a fringe pattern synthesis unit for synthesizing the fringe patterns.
The method of claim 1,

And the plurality of areas are areas separated by concentric circles.
The method of claim 6,

And the point density is lower in an area farther from a center of the concentric circle.
The method of claim 1,

Holographic image processing apparatus further comprises a tracking unit for tracking the gaze point of the user.
The method of claim 8,

And the plurality of areas are divided by concentric circles around a gaze point of a user.
A holographic image processing apparatus which implements a 3D graphic object in a point cloud method,

A sampling unit for sampling points from the 3D graphic object;

A fringe pattern generator which creates a fringe pattern corresponding to each of the sampled points; And

And a fringe pattern interpolator configured to copy at least one of the created fringe patterns to interpolate the created fringe patterns.
A holographic image processing method for implementing a 3D graphic object in a point cloud method,

Dividing a field of view of the user into a plurality of regions;

Sampling points per segmented area from a three-dimensional graphic object within the viewing range of the user, wherein the point density of the sampled points differs for each segmented area; And

And creating a fringe pattern corresponding to each of the sampled points.