KR20120054799A - A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images - Google Patents

A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images Download PDF

Info

Publication number
KR20120054799A
KR20120054799A KR1020100116102A KR20100116102A KR20120054799A KR 20120054799 A KR20120054799 A KR 20120054799A KR 1020100116102 A KR1020100116102 A KR 1020100116102A KR 20100116102 A KR20100116102 A KR 20100116102A KR 20120054799 A KR20120054799 A KR 20120054799A
Authority
KR
South Korea
Prior art keywords
image
view
depth
information
point
Prior art date
Application number
KR1020100116102A
Other languages
Korean (ko)
Inventor
김동욱
서영호
최현준
Original Assignee
광운대학교 산학협력단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 광운대학교 산학협력단 filed Critical 광운대학교 산학협력단
Priority to KR1020100116102A priority Critical patent/KR20120054799A/en
Publication of KR20120054799A publication Critical patent/KR20120054799A/en

Links

Images

Classifications

    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G03H1/0443—Digital holography, i.e. recording holograms with digital recording means
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/26—Processes or apparatus specially adapted to produce multiple sub- holograms or to obtain images from them, e.g. multicolour technique
    • G03H1/268—Holographic stereogram
    • H04N13/0018—
    • H04N13/0059—

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

A multi-view based interactive holographic restoration apparatus and system for receiving at least two multi-view depth images and restoring a holographic image corresponding to a viewer's viewpoint (hereinafter, referred to as a viewing point). An image input unit configured to receive at least two multi-view depth images; A viewpoint information acquisition unit which receives a viewer's viewpoint (hereinafter, referred to as a viewpoint); A viewpoint controller which selects at least two photographing viewpoints (hereinafter, adjacent photographing viewpoints) corresponding to viewpoints closest to the viewing viewpoint; Depth information image generation unit for generating a depth information image of the time point (hereinafter referred to as the intermediate point) corresponding to the viewing point from the multi-view depth image of the adjacent point of view, by interpolating the matching area of the depth image of the adjacent point of view ; An image information generating unit generating image information of the depth information image from the image information of the depth image at the time of adjacent photographing using the interpolated information; And a hologram generator for generating a hologram image from a depth information image (hereinafter, referred to as a mid-view depth image) having image information.
By the above apparatus and system, when the viewer's viewpoint is changed, the holographic image corresponding to the viewing viewpoint can be reproduced to reproduce the natural holographic reconstructed image, and reduce eye fatigue.

Description

A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images}

The present invention relates to a multi-view based interactive holographic restoration apparatus and system for receiving at least two multi-view depth images and restoring a holographic image corresponding to a viewpoint to be viewed.

In particular, the present invention relates to a multi-view based interactive holographic restoration apparatus and system for generating a holographic image by generating a mid-view depth image corresponding to a viewer's viewpoint from a multi-view depth image of a viewpoint closest to the viewer's viewpoint.

Recently, researches on 3D image and image reproduction technology are being actively conducted, and it is expected that the next generation display will be developed as a new concept of realistic image media that raises the level of visual information. In addition, the demand for 3D images is increasing because 3D images are more realistic, natural, and closer to the reality that humans feel.

Among the 3D image related technologies, the holography method is a method in which a viewer observes a virtual image of a virtual image while looking at the holography at a certain distance away from the front of the holography. The holography method is a method of observing holography produced using a laser, and it is possible to feel the same stereoscopic image as the real one without wearing special glasses. Therefore, the holography method is known to be the most ideal way to feel the three-dimensional image without the fatigue and excellent stereoscopic feeling.

The existing holography method has been greatly limited in its application by recording 3D information on a hologram film and restoring a 3D object using the developed film.

However, computerized holograms were made possible by calculating the interference term generated by the interference between the object wave and the reference wave. When coherent light, whose spacing is kept constant over time and space, hits and reflects on a shape, the wavefront of the reflected light varies in proportion to the shape of the object. That is, the phase change of the wavefront changes according to the shape of the object. Computer-generated holograms are obtained by calculating geometric optical methods for the phase of a wavefront that changes with the shape of such an object.

A three-dimensional model created by computer graphics is used to generate a computer generated hologram (CGH). The main reason for using the 3D model is that the information for generating CGH from the 3D model can be easily obtained, and the depth information of the image obtained using computer graphics can be easily used as the information for generating the CGH. Because there is. In particular, a holographic display technology for generating a holographic image from a depth image has been proposed.

However, since the depth image is a 3D image about a fixed viewpoint, the hologram generated therefrom is also a stereoscopic image by the fixed viewpoint. Therefore, when the viewer views at the viewing point corresponding to the depth image point of view, the viewer may see the complete stereoscopic image, but may not be able to do so when the viewer moves his position.

For example, if the viewing point is moved to the side of 90 degrees, the holographic image is seen simultaneously at the front side and the unrestored back side of the hologram. That is, the conventional holographic display technology has a disadvantage that it is not possible to freely select a viewpoint to watch.

SUMMARY OF THE INVENTION An object of the present invention is to solve the problems described above, and a multi-view interactive holographic restoration apparatus and system for reconstructing a holographic image corresponding to a viewpoint to receive at least two multi-view depth images To provide.

Particularly, an object of the present invention is to generate a multi-view image corresponding to a viewer's viewpoint, extract depth by referring to a photographing view image adjacent to the mid-view image, and generate a holographic image therefrom. A graphics restoration apparatus and system.

In order to achieve the above object, the present invention relates to a multi-view based interactive holographic restoration apparatus which receives at least two multi-view depth images and reconstructs a holographic image according to a viewer's point of view. An image input unit configured to receive at least two multi-view depth images photographed; A view point information acquisition unit which receives a view point of the viewer (hereinafter referred to as a viewing point); A view controller configured to select at least two photographing points (hereinafter, adjacent photographing points) corresponding to the viewpoints closest to the viewing point; Depth information generated from the multi-view depth image of the adjacent photographing point of time (hereinafter referred to as the intermediate point of view) corresponding to the viewing point of view, the depth information generated by interpolating the matching area of the depth image of the adjacent photographing point of view An image generator; An image information generator configured to generate image information of the depth information image from the image information of the depth image at the adjacent photographing time point by using the interpolated information; And a hologram generator for generating a hologram image from a depth information image (hereinafter, referred to as a mid-view depth image) having image information.

The present invention also provides a multi-view interactive holographic reconstruction apparatus, wherein an external variable is extracted from a calibration of a camera photographing the multi-view depth images, and the preprocessing is performed to correct the multi-view depth images using the external variable. It further comprises a wealth.

The present invention also provides a multi-view interactive holographic reconstruction apparatus, wherein the depth information image generating unit interpolates the registration area according to a position ratio of the intermediate point between the adjacent photographing points. The position of the depth information image of the intermediate view corresponding to the matching area is determined, and the depth information of the predetermined position is determined as the depth information of the matching area.

In another aspect, the present invention provides a multi-view interactive holographic restoration apparatus, wherein the depth information image generation unit finds a region where depth information overlaps on a parallel line of depth images at the adjacent photographing time point, and determines a matching area. do.

The present invention also provides a multi-view interactive holographic restoration apparatus, wherein the image information generating unit converts the image information of the position into the image information of the matching region with respect to the position of the depth information image determined by interpolating the registration region. It is characterized by.

In addition, the present invention is a multi-view interactive holographic reconstruction device, the hologram generating unit is characterized in that to generate a digital hologram image by [Equation 1] from the depth image.

[Equation 1]

Figure pat00001

Where α and j are holograms and depth images, λ is the wavelength of the reference wave,

p is the pixel pitch of the hologram,

x α and y α are the coordinates of the hologram,

x j , y j , and z j are the coordinates of the depth image.

I α is the intensity of light in the hologram, A j is the maximum size of the pixel value of the depth image.

The present invention also relates to a multi-view based interactive holographic restoration system for receiving at least two multi-view images and reconstructing a holographic image according to a viewer's point of view. Receiving a multiview image, generating a depth information image of an intermediate view corresponding to the viewer's viewpoint (hereinafter referred to as a viewing point) from the multiview depth image, extracting image information of the depth information image, and having an intermediate image information An encoding apparatus for generating a depth information image of a viewpoint (hereinafter, referred to as an intermediate viewpoint depth image); And a decoding device receiving the viewing time point and transmitting the received viewing time point to the encoding device, and receiving a mid-view depth image corresponding to the viewing time point from the encoding device to generate a hologram image.

The present invention also relates to a multi-view interactive holographic encoding apparatus, comprising: an image input unit configured to receive at least two multi-view images photographed at at least two photographing points; A view control unit which receives a viewing time point from the decoding device and selects at least two shooting time points (hereinafter, adjacent shooting time points) corresponding to the viewpoints closest to the viewing time point; Depth information generated from the multi-view depth image of the adjacent photographing point of time (hereinafter referred to as the intermediate point of view) corresponding to the viewing point of view, the depth information generated by interpolating the matching area of the depth image of the adjacent photographing point of view An image generator; An image information generator configured to generate image information of the depth information image from the image information of the depth image at the adjacent photographing time point by using the interpolated information; And an image transmitter for encoding and transmitting a depth information image (hereinafter, referred to as an intermediate view depth image) having image information.

As described above, according to the multi-view interactive holographic restoration apparatus and system according to the present invention, when the viewpoint of the viewer is changed, the problem such as the image disconnection that may occur in the process of changing the viewpoint by playing the corresponding viewpoint image By solving the problem, an effect of reproducing a natural holographic restored image is obtained. This can reduce eye strain and increase realism.

1 is a configuration diagram of an entire system for implementing the first embodiment of the present invention.
2 is a block diagram of a multi-view based interactive holographic restoration apparatus according to a first embodiment of the present invention.
3 is a block diagram of an arrangement of a multiview depth camera according to a first embodiment of the present invention.
4 is a diagram illustrating an example of obtaining a viewer's viewpoint according to the first embodiment of the present invention.
5 is a block diagram of a view point information acquisition unit according to a first embodiment of the present invention.
6 is a configuration diagram of the preprocessor according to the first embodiment of the present invention.
7 illustrates an example of obtaining overlapped regions according to the first embodiment of the present invention.
8 illustrates an example of generating a mid-view depth image from a reference image according to the first embodiment of the present invention.
9 illustrates an example of generating a mid-view depth image of a mid-view between each view of a multi-view depth image according to a first embodiment of the present invention.
10 is a configuration diagram of an entire system for implementing the second embodiment of the present invention.
11 is a block diagram of a multi-view based interactive holographic restoration system according to a second embodiment of the present invention.
Description of the Related Art [0002]
10: object 20: depth camera
25: network 30,31,32: computer terminal
40: holographic restoration device 40a: encoding device
40b: encoding device 41: video input unit
42: preprocessor 43: viewpoint information acquisition unit
44: viewpoint controller 45: depth image generator
45a: depth information image generator 45b: image information generator
45c: depth information processing unit
46: hologram generation unit 47: post-processing unit
48: spatial light modulator 49: intermediate image generating unit
51: video transmission unit
52: video receiver 60: multi-view video
61: depth image 70: holographic (restore) image

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In addition, in describing this invention, the same code | symbol is attached | subjected and the repeated description is abbreviate | omitted.

First, an example of the configuration of the entire system for implementing the first embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 1, the first embodiment of the present invention may be implemented as a program system on a computer terminal 30 which receives images 60 captured by a plurality of photographing apparatuses 20 and processes images. .

That is, the multi-view based interactive holographic restoration device 40 may be configured as a program and installed and executed in the computer terminal 30. The program installed in the computer terminal 30 may operate like one system 40. Meanwhile, as another embodiment, the holographic restoration device 40 may be implemented as a program and operate in a general-purpose computer, and may be implemented as one electronic circuit such as an ASIC (custom semiconductor). That is, it may be configured in the form of software, an FPGA chip or an electronic circuit composed of several circuit elements. Other possible forms may be implemented. However, hereinafter, the holographic restoration device 40 implemented in the computer terminal 30 will be described for convenience of description.

The plurality of photographing apparatuses 20 are multi-view depth cameras, which photograph one object 10 at a plurality of viewpoints, and are capable of capturing images and measuring depths of the images. The captured image 60 is a multiview depth image captured by the depth camera 20. In addition, the multi-view depth image 60 is directly input and stored in the computer terminal 30 and processed by the holographic restoration device 40.

Alternatively, the multi-view depth image 60 may be previously stored in a storage medium of the computer terminal 30, and may be input by reading the multi-view depth image 60 stored by the holographic restoration device 40.

Meanwhile, the holographic restoration apparatus 40 generates a mid-view depth image from the multi-view depth image 60 and generates a hologram image from the mid-view depth image.

Next, a configuration of the multi-view based interactive holographic restoration apparatus 40 according to the first embodiment of the present invention will be described with reference to FIG. 2.

As shown in FIG. 2, the holographic reconstruction device 40 includes an image input unit 41, a viewpoint information acquirer 43, a viewpoint controller 44, an intermediate viewpoint depth image generator 45, and a hologram generator ( 46). In addition, it may be configured to further include a pre-processing unit 42, a post-processing unit 47, a spatial light modulator 48.

The image input unit 41 receives at least two multi-view depth images respectively photographed at at least two photographing points.

As shown in FIG. 3, the depth camera 20 arranges a plurality of cameras, photographs an image of each viewpoint with each camera, and acquires a depth image by measuring depth. That is, not only the image information such as RGB, but also the depth information of the image is obtained from the depth camera 30. Camera arrangement is divided into a parallel arrangement, a convergent arrangement, and the like.

3A is a method of arranging multiple cameras in a direction of a point of a center, and a parallel arrangement of FIG. 3B is a method of obtaining images by arranging cameras in a line. In particular, in the convergence arrangement as shown in FIG. 3A, the camera may be disposed at an angle of 360 degrees, and the target object may be photographed in all directions to acquire images at all photographing points. At this time, it is preferable to arrange the camera intervals uniformly at a constant distance or at a constant angle.

For example, in FIG. 3A, seven cameras 20 are disposed around the object 10 to photograph the object 10 in each direction. At this time, seven multi-view depth images 60 corresponding to each camera are obtained. That is, according to the camera (or the recording timing) on a multi-view depth image by C 1 P 1, the camera (or the recording timing) on a multi-view depth image by C 2 P 2, ..., the camera (or the recording timing) C 7 A multiview depth image P 7 is obtained.

On the other hand, the viewpoint information acquisition unit 43 acquires the viewer's viewpoint (hereinafter referred to as a viewpoint). The viewing point refers to a position (or point of view) where the viewer watches the holographic image.

As shown in Figure 4a, the viewer is located at L 1 may view a graphic image (70) from the front side alone, may be slightly to the right to position the graphic image viewing position (70) alone in L 2 slightly from the right side. In addition, the holographic image 70 may be viewed at the left position L 3 .

As shown in FIG. 4B, the view point information acquisition unit 43 receives N viewer view point data from N sensing devices.

As shown in FIG. 5, the viewpoint information acquisition unit 43 is a sensor information acquisition unit 43a that receives viewer's viewpoint information from sensing devices, and corrects and analyzes sensor information for analyzing signals input from a plurality of sensors. The unit 43b includes a space for extracting viewer's viewpoint information from the analyzed sensor information and a viewpoint information extracting unit 43c.

In another embodiment, the view point information acquisition unit 43 obtains the view point by tracking the viewer's movement through infrared, multiple sensors, or the viewer's captured image. For example, techniques for tracking the position of the viewer, such as eyes of the eyes (Korean Publication No. 2010-0048747, 2009-0075536, 2010-0027976, 2005-0000369, 2007-0083575) , 2006-0106451, etc.) may be applied. Alternatively, the viewpoint information acquisition unit 43 may acquire viewpoint information by an input of a viewer. That is, the viewer may directly input a desired view point.

Also, the viewpoint controller 44 selects at least two photographing viewpoints (hereinafter, adjacent photographing viewpoints) corresponding to the viewpoints closest to the viewing viewpoint.

In the example of FIGS. 3A and 4, when the viewing point is the position L 1 , C 4 and C 5 are selected as two adjacent photographing points. It is also possible to select two or more adjacent photographing time points, such as C 3 , C 4 , and C 5 . When the viewing point is the position L 2 , C 3 and C 4 are selected as two adjacent photographing points. If the viewing point is the position L 3 , C 6 and C 7 are selected as two adjacent photographing points.

The preprocessor 42 preprocesses the selected images (or multi-view depth images) of at least two adjacent photographing time points.

As shown in FIG. 6, the preprocessor 42 selects and receives an image of a photographing time point selected by the viewpoint controller 44 (42a). In order to correct the image, the preprocessor 42 first performs a camera calibration (42b). An external variable of the camera is extracted from the camera calibration (42c), and the image is corrected using this (42d).

The intermediate view depth image generation unit 45 generates a depth information image of an intermediate point of view (a point of view corresponding to the viewing point of view) from a depth image of an adjacent photographing point of view, and extracts image information of the depth information image to finally generate an intermediate image. A viewpoint depth image is generated. As shown in FIG. 2, the depth image generator 45 includes a depth information image generator 45a, an image information generator 45b, and a depth information processor 45c.

The depth information image generating unit 45a generates a depth information image of a viewpoint (hereinafter, referred to as an intermediate viewpoint) corresponding to the viewing viewpoint from the multiview depth image of the adjacent photographing viewpoint. In particular, the depth information image of the intermediate view is generated by interpolating the matching area of the depth images of the adjacent photographing points.

The intermediate view depth information image represents depth information of each pixel, that is, an image of depth information. The intermediate view image information refers to RGB information of each pixel at the intermediate view point. The intermediate view depth image (or depth image) refers to a depth information image having image information at an intermediate view point. That is, an image including image information in the depth information image is called a mid-view depth image.

First, the depth information image generation unit 45a searches for a region where depth images of two adjacent photographing points overlap each other and selects a matching region.

In the above example, images P 4 and P 5 are depth images of two adjacent photographing time points previously selected by the viewpoint controller 44. These depth images will be referred to as first and second images, respectively. Then, regions overlapping each other in the first and second images are found.

Preferably, the depth information image generating unit 45a finds an area where the depth information overlaps on a parallel line of the depth images at the adjacent photographing time point, and selects the area as the matching area. In particular, it is preferable to define parallel lines as the same epipolar line of two images.

As shown in FIG. 7A, regions where depth information overlaps along lines e 4 and e 5 parallel to each other are found in two images P 4 and P 5 . Preferably, the two parallel lines e 4 and e 5 are defined as lines on the same epipolar with respect to the imaged object 10.

As shown in FIG. 7B, depth information overlapping each other is found on parallel lines e 4 and e 5 . In the images P 4 and P 5 , the regions p 1 4 and p 1 5 are overlapping regions, respectively, and are matched regions. In addition, the regions p 2 4 and p 2 5 are also overlapped with each other. p 1 4 and p 1 5 differ from each other by v 1 , and p 2 4 and p 2 5 differ from each other by v 2 .

Next, the depth information image generating unit 45a interpolates the registration area according to a position ratio between the intermediate time points between the adjacent photographing time points, and the depth information of the intermediate time points corresponding to the registration areas. The position of the image is determined, and the depth information of the determined position is determined as the depth information of the matching area.

As shown in FIG. 8A, an intermediate view depth information image corresponding to a viewing point L 1 may be generated from two first and second depth images P 4 and P 5 . At this time, the viewing point L 1 is closer to the shooting point C 4 of the depth image P 4 than the shooting point C 5 of the depth image P 5 . This can generate the mid-view depth information image that corresponds exactly to the viewing point L 1 by the position ratio.

That is, the depth information image generator 45a obtains the position ratio of the intermediate point corresponding to the viewing point between the photographing points of the first and second images, and moves the depth by the position ratio of the intermediate point in the first image. Generates information as midpoint depth information.

As shown in FIG. 8B, depth information of the intermediate point L 1 may be generated by interpolation from the first image P 4 . By moving the region p 1 4 on the e 4 line in the first image P 4 by the position ratio -kV 1 , depth information in the region p 1 at the intermediate point in the position can be generated. Further, by moving the area p 2 4 by the position ratio -kV 2 , depth information is generated in the area p 2 at the intermediate point in time.

At this time, to obtain the position ratio, it can be interpolated using a linear, two-dimensional curve, three-dimensional curve, and the like. As a result, it is equivalent to generating the depth information image of the intermediate point L 1 by interpolation from the depth images P 4 and P 5 .

The image information generating unit 45b determines the image information of the position as the image information of the matching region with respect to the position of the depth information image determined by interpolating the matching region. In other words, by using midpoint information (for example, derived interpolation function, position ratio, etc.) used to generate midpoint depth image, an image (RGB image) output from the camera of each viewpoint is mapped to midpoint. To generate an intermediate view image (RGB image).

In Figure 8b, the first image position P ratio a region on p 1 e 4 4 4 -kV line in the image information of the area of the middle point p 1 in the first move as the location area of the first image p 1 Create by defining the video information of 4 . In the same manner, the image information in the region p 2 at the intermediate point in time at which the region p 2 4 is moved by the position ratio -kV 2 is generated by defining the image information in the region p 2 4 of the first image. In this case, the image information of the second image may be used instead of the image information of the first image.

The image information generator 45b obtains all the RGB images of the intermediate view. This is because the CGH (computer generated hologram) must be performed for each of the R, G, and B components, and each pixel value of the mid-view image obtained as described above is used as a value of A j of Equation 1 described below.

An example of obtaining an intermediate view depth information image according to each viewing point is illustrated in FIG. 9. That is, as shown in FIG. 9, the mid-view depth information image is extracted from the images of two adjacent viewpoints, and the mid-view image information is generated. In particular, the intermediate view image information is RGB images for the intermediate view between the two viewpoints. For example, mid-view depth images corresponding to viewpoints 2-1, 2-2, 2-3, and 2-4, which are intermediate depth images, are included from two images P 2 and P 3 corresponding to viewpoint 2 and viewpoint 3. Can be generated.

Next, the depth information processor 45c corrects the generated mid-view depth image to have a more three-dimensional effect.

As an example, the depth information processor 45c extracts only an object image (object image) by separating an object that is a region of interest and a background that is an uninterested region from the depth image information obtained from the multiview image. This is performed because the image quality of the holographic reconstructed image is considerably degraded when the depth image information (light source of the image) is spread throughout the image. That is, the depth information processor 45c separates the background from the previously extracted depth information, extracts only the depth image of the object image, and generates a final depth image. At this time, the depth information processor 45c extracts the object using a contour detection technique, an area selection algorithm, and depth information.

As another example, the depth information processor 45c adjusts the depth information in order to improve the quality of the hologram when converting the hologram.

In addition, the depth information processor 45c can adjust the playback position. That is, the position of the hologram to be reproduced is adjusted through the spatial light modulator 48. In addition, the depth information processor 45c adjusts the distance between the spatial light modulator 48 and the hologram to be reproduced, thereby enabling generation of holograms suitable for various reproduction environments according to various terminals.

Meanwhile, the hologram generator 46 generates a hologram image from the mid-view depth image obtained above.

The hologram generator 46 uses [Equation 2] in which a general hologram generating formula such as [Equation 1] is arranged for high speed computation using depth image data.

[Equation 1]

Figure pat00002

Where α and j represent the pixels of the hologram and the light source of the three-dimensional object, respectively, k is the wave number of the reference wave and is defined as 2π / λ, and p is the pixel pitch of the hologram, x α and y α denotes the coordinates of the hologram, x j , y j , and z j denote the coordinates of the light source (ie, the depth image) of the 3D object. In addition, I α is the intensity of light of the hologram, A j is the maximum size of the pixel value of the depth image.

If Equation 1 is approximated to the first term after Taylor expansion, it can be summarized as Equation 2 below.

[Equation 2]

Figure pat00003

Figure pat00004

Where x and y αj αj means a - - (y j y α) (x α x j) and.

The post-processing unit 47 improves the quality of the image to be restored through noise reduction, gamma correction, and the like.

The finally obtained holographic image data is displayed on a spatial light modulator (SLM) 48 to restore and reproduce the holographic image.

For reference, the hologram image refers to a two-dimensional image obtained from the depth image by using Equation 1 or Equation 2, and the holographic image is reproduced by reconstructing the hologram image through a spatial light modulator (SLM). It is a video. Therefore, the holographic pick image is also referred to as a holographic reconstructed image.

Next, examples of the configuration of the entire system for implementing the second embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 10, the second embodiment of the present invention includes an encoding device 40a and a decoding device 40b, which are connected to each other through a network 25.

The encoding device 40a or the decoding device 40b may be configured as a program and installed and executed in the computer terminals 31 and 32. Programs installed on the computer terminals 31 and 32 may operate like one system 40a or 40b. Meanwhile, as another embodiment, the encoding device 40a and the decoding device 40b may be implemented as one electronic circuit such as an ASIC (custom semiconductor). That is, it may be configured in the form of software, an FPGA chip or an electronic circuit composed of several circuit elements. Other possible forms may be implemented. However, hereinafter, for convenience of description, the encoding apparatus 40a and the decoding apparatus 40b implemented in the computer terminal 30 will be described.

The multi-view depth image 60 may be previously stored in a storage medium of the computer terminal 31, and may be input by reading the stored multi-view depth image 60 by the encoding apparatus 40a.

Meanwhile, the encoding apparatus 40a generates a depth image from the multiview depth image 60, and transmits the extracted depth image 61 to the decoding apparatus 40b through the network 25. The decoding device 40b receives the depth image 61 and generates a hologram image from the depth image 61. In addition, the encoding device 40a reconstructs and reproduces the holographic reconstructed image (holographic image).

In this case, the network 25 includes not only a wired network such as the Internet and a wireless network such as mobile communication, but also a short range wireless network such as Bluetooth and RF.

Next, a configuration of the multi-view based interactive holographic restoration system 400 according to the second embodiment of the present invention will be described with reference to FIG. 11.

As shown in FIG. 11, the holographic reconstruction system 400 according to the second embodiment of the present invention includes an encoding device 40a and a decoding device 40b.

The encoding device 40a includes an image input unit 41, a depth image generation unit 45, a view control unit 44, and an image transmission unit 51. In addition, the preprocessing unit 42 may be further included.

In addition, the decoding apparatus 40b includes a viewpoint information obtaining unit 43, an image receiving unit 52, and a hologram generating unit 46. In addition, the post-processing unit 47, the spatial light modulator 48 may be further included.

The description of the image input unit 41, the viewpoint controller 44, the depth image generator 45, and the preprocessor 42 of the encoding device 40a has the same function as that of the corresponding holographic restoration device 40. See the description above. In addition, description of the viewpoint information acquisition part 43, the hologram generation part 46, the post-processing part 47, and the spatial light modulator 48 of the decoding apparatus 40b is also abbreviate | omitted here, and abbreviate | omits here.

However, after the view point information acquisition unit 43 of the decoding device 40b obtains the viewing point of the viewer, the view point is transmitted to the view control unit 44 of the encoding device 40a, and the view control unit 44 receives the view point. According to the viewing time point, at least two shooting time points (hereinafter, adjacent shooting time points) corresponding to the viewpoints closest to the viewing time point are selected.

The image transmitter 51 transmits the depth image 61 generated by the depth image generator 45 to the image receiver 52 of the decoding apparatus 40b. In this case, the image transmission unit 51 compresses by standard video encoding technology and then transmits the data through the wired / wireless line 25, and the image receiving unit 52 receives and decodes the compressed image.

On the other hand, the view point information acquisition unit 43 of the decoding device 40b obtains a viewing point and transmits the view point to the view control unit 44 of the encoding device 40a.

As mentioned above, although the invention made by this inventor was demonstrated concretely according to the said Example, this invention is not limited to the said Example and can be variously changed in the range which does not deviate from the summary.

The present invention is applicable to the development of a holographic restoration apparatus that receives at least two multi-view depth images and restores the holographic images according to the viewer's viewpoint.

Claims (8)

In the multi-view based interactive holographic restoration apparatus that receives at least two multi-view depth image to restore the holographic image according to the viewer's view,
An image input unit configured to receive at least two multi-view depth images respectively captured at at least two photographing points;
A view point information acquisition unit which receives a view point of the viewer (hereinafter referred to as a viewing point);
A view controller configured to select at least two photographing points (hereinafter, adjacent photographing points) corresponding to the viewpoints closest to the viewing point;
Depth information generated from the multi-view depth image of the adjacent photographing point of time (hereinafter referred to as the intermediate point of view) corresponding to the viewing point of view, the depth information generated by interpolating the matching area of the depth image of the adjacent photographing point of view An image generator;
An image information generator configured to generate image information of the depth information image from the image information of the depth image at the adjacent photographing time point by using the interpolated information; And,
Multi-view-based interactive holographic restoration apparatus comprising a hologram generator for generating a holographic image from a depth information image (hereinafter referred to as a mid-view depth image) having image information.
The method of claim 1,
A multiview-based interactive holographic reconstruction apparatus further comprising extracting an external variable from a calibration of the camera photographing the multiview depth images and correcting the multiview depth images using the external variable. .
The method of claim 1,
The depth information image generating unit interpolates the registration area according to a position ratio between the intermediate view point and the adjacent photographing view point, and determines the position of the depth information image of the intermediate view point corresponding to the registration area. A multi-view based holographic restoration apparatus, characterized in that the depth information of a predetermined position is determined as the depth information of the registration area.
The method of claim 1,
The depth information image generating unit finds a region where depth information overlaps on a parallel line of depth images at the adjacent photographing point, and determines a matching area as a multiview-based interactive holographic restoration apparatus.
The method of claim 3,
And the image information generating unit determines image information of the position as image information of the matching region with respect to the position of the depth information image determined by interpolating the matching region.
The method of claim 1,
The hologram generating unit multi-view interactive holographic reconstruction device, characterized in that for generating a hologram image by the formula [1] from the depth of the intermediate view.
[Equation 1]
Figure pat00005

Where α and j are holograms and depth images, λ is the wavelength of the reference wave,
p is the pixel pitch of the hologram,
x α and y α are the coordinates of the hologram,
x j , y j , and z j are the coordinates of the depth image.
I α is the intensity of light in the hologram, A j is the maximum size of the pixel value of the depth image.
In a multi-view based interactive holographic restoration system that receives at least two multi-view images to restore the holographic image according to the viewer's view,
Receiving at least two multi-view images respectively photographed at at least two photographing time points, generating a depth information image of an intermediate view corresponding to the viewer's view point (hereinafter, the viewing point) from the multi-view depth image, and generating the depth information. An encoding apparatus for extracting image information of an image and generating a depth information image (hereinafter, referred to as an intermediate view depth image) of an intermediate view having image information; And,
A multi-view-based interactive holo comprising a decoding apparatus for receiving the viewing time point and transmitting the received viewing time point to the encoding device, and receiving a mid-view depth image corresponding to the viewing time point from the encoding device to generate a holographic image. Graphical Restoration System.
An image input unit configured to receive at least two multi-view images photographed at at least two photographing points;
A view control unit which receives a viewing time point from the decoding device and selects at least two shooting time points (hereinafter, adjacent shooting time points) corresponding to the viewpoints closest to the viewing time point;
Depth information generated from the multi-view depth image of the adjacent photographing point of time (hereinafter referred to as the intermediate point of view) corresponding to the viewing point of view, the depth information generated by interpolating the matching area of the depth image of the adjacent photographing point of view An image generator;
An image information generator configured to generate image information of the depth information image from the image information of the depth image at the adjacent photographing time point by using the interpolated information; And,
A multi-view-based interactive holographic encoding apparatus comprising an image transmitter for encoding and transmitting a depth information image (hereinafter, referred to as an intermediate view depth image) having image information.
KR1020100116102A 2010-11-22 2010-11-22 A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images KR20120054799A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020100116102A KR20120054799A (en) 2010-11-22 2010-11-22 A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020100116102A KR20120054799A (en) 2010-11-22 2010-11-22 A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images

Publications (1)

Publication Number Publication Date
KR20120054799A true KR20120054799A (en) 2012-05-31

Family

ID=46270492

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020100116102A KR20120054799A (en) 2010-11-22 2010-11-22 A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images

Country Status (1)

Country Link
KR (1) KR20120054799A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101358431B1 (en) * 2012-07-30 2014-02-05 인텔렉추얼디스커버리 주식회사 System and method for displaying multi-view image
KR101539038B1 (en) * 2014-09-02 2015-07-24 동국대학교 산학협력단 Hole-filling method for depth map obtained from multiple depth camera
US9618902B2 (en) 2012-11-26 2017-04-11 Samsung Electronics Co., Ltd. Apparatus and method for generating hologram pattern
US9727023B2 (en) 2013-04-15 2017-08-08 Samsung Electronics Co., Ltd. Apparatus and method for generating hologram pattern

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101358431B1 (en) * 2012-07-30 2014-02-05 인텔렉추얼디스커버리 주식회사 System and method for displaying multi-view image
US9618902B2 (en) 2012-11-26 2017-04-11 Samsung Electronics Co., Ltd. Apparatus and method for generating hologram pattern
US9727023B2 (en) 2013-04-15 2017-08-08 Samsung Electronics Co., Ltd. Apparatus and method for generating hologram pattern
KR101539038B1 (en) * 2014-09-02 2015-07-24 동국대학교 산학협력단 Hole-filling method for depth map obtained from multiple depth camera

Similar Documents

Publication Publication Date Title
KR101177408B1 (en) A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user
US8462199B2 (en) Apparatus method and system for imaging
US10368047B2 (en) Six-degree of freedom video playback of a single monoscopic 360-degree video
US20220014723A1 (en) Enhancing performance capture with real-time neural rendering
CN108141578B (en) Presentation camera
US20130162861A1 (en) Image processing device for generating reconstruction image, image generating method, and storage medium
KR100897307B1 (en) Method for reproducing hologram of 3D image picked-up by integral imaging scheme and Apparatus thereof
US11308682B2 (en) Dynamic stereoscopic rendering method and processor
US20220398705A1 (en) Neural blending for novel view synthesis
WO2022182421A1 (en) Color and infra-red three-dimensional reconstruction using implicit radiance function
KR20170060725A (en) Apparatus for measuring quality of Holographic image and method thereof
KR20120054799A (en) A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by using multi-view depth images
KR101411568B1 (en) A Hologram Generating Method using Virtual View-point Depth Image Synthesis
KR20120054790A (en) A multi-view interactive holographic device and system for reconstructing holographic images adaptable at the view-point of a user by the intermediate image
KR102577707B1 (en) Apparatus and method for evaluating hologram encoding / holographic image quality for amplitude-modulated hologram
KR20140126120A (en) Method for digital hologram synthesis and apparatus therefor
KR102577686B1 (en) Holograms reconstruction method
KR101321895B1 (en) An interactive digital hologram service system for digital hologram videos
KR20140037430A (en) A system for generating high quality digital holographic video using vertical rig based on depth+rgb camera
KR20120118622A (en) Metheod and apparatus for digital hologram contents service
US9177380B2 (en) 3D video camera using plural lenses and sensors having different resolutions and/or qualities
CN109961395B (en) Method, device and system for generating and displaying depth image and readable medium
KR20140035747A (en) Ultrasound imaging apparatus and control method for the same
KR101378910B1 (en) An Intermediate View Image Generation Method for an Virtual Arbitrary View-Point Hologram Service
KR100927234B1 (en) Method, apparatus for creating depth information and computer readable record-medium on which program for executing method thereof

Legal Events

Date Code Title Description
A201 Request for examination
E601 Decision to refuse application