KR101857977B1 - Image apparatus for combining plenoptic camera and depth camera, and image processing method - Google Patents

Abstract

Dimensional image by matching a multi-view image generated by the panoramic camera and a depth image generated by the depth camera by combining a panoramic camera and a depth camera, and an image processing method for generating a three-dimensional image.

Description

TECHNICAL FIELD [0001] The present invention relates to an image device and an image processing method that combine a planar camera and a depth camera,

And a technique for processing an image generated through coupling between two cameras.

An integral imaging technique capable of acquiring one or more multi-view images is known as a method capable of acquiring a light field image. However, there is a limitation in precisely manufacturing a lens array of a plenoptic camera for acquiring multi-view images. In addition, the plannop camera has limitations in acquiring real 3D images due to limitation of spatial resolution and viewpoint resolution.

On the other hand, a color depth camera combining a color camera and a depth camera can not acquire a light field of various viewpoint images. Therefore, although a method of using a plurality of color-depth cameras has been proposed, there is a problem in making it a consumer device due to a problem such as a size of a camera system.

The imaging apparatus according to an embodiment includes a planar camera that generates an integrated image including a plurality of view images having different viewpoints, a depth camera that generates a depth image, And an image matching unit for generating an image.

The image matching unit may measure a relative position of the planar camera and the depth camera, and may match the integrated image and the depth image based on the measured relative position.

The image matching unit calculates rotation information or movement information between any one viewpoint image and the depth image among the plurality of viewpoint images and matches the integrated image and the depth image using the calculated rotation information or movement information .

The image matching unit may obtain a homography between any one view image and the depth image among the plurality of view images and may match the accumulated image and the depth image using the obtained homography.

The image matching unit may improve the shape information between the integrated image and the depth image by changing the pixel values of the decimal viewpoint images among the plurality of viewpoint images to pixel values of the majority viewpoint images, The integrated image and the depth image can be matched.

Wherein the image matching unit calculates the amount of change of the color information associated with the plurality of viewpoint images, predicts the material information, the illumination information, and the color information of the object included in the plurality of viewpoint images based on the calculated amount of change, The pixels included in the plurality of view-point images can be subdivided according to the material information, the illumination information, and the color information.

Wherein the image matching unit acquires representative light field information associated with pixels in the selected area using a light field included in the subdivided pixels in the selected area, Depth images can be matched.

The video apparatus according to another embodiment may include a plurality of multi-viewpoint images having different viewpoints, a depth camera for generating depth images, and illumination information or material information between the multi-viewpoint images, Dimensional shape information between the multi-view images and the depth image based on information or material information.

The shape capturing unit may convert the pixel values of the fractional multi-view images included in the multi-view images into pixel values of the majority multi-view images to match the illumination information or the material information of the multi-view images.

The image device may further include an image processor for matching the multi-view images with the depth images using the three-dimensional shape information.

According to another embodiment of the present invention, there is provided a display apparatus including a planar camera for generating multi-view images having different viewpoints, a depth camera for generating a depth image, material information of an object included in the multi-view images, And an image processor for matching the multi-view images with the depth image using the subdivided pixels.

The pixel segmentation unit predicts material information, illumination information, and color information of an object included in the multi-view images using the amount of change of the color information acquired from the multi-view images, and outputs the predicted material information, The pixels included in the multi-view images can be subdivided according to the information.

Wherein the image processing unit acquires representative light field information associated with pixels in the selected area by summing the light field values included in the subdivided pixels in the selected area, and based on the acquired representative light field information, And the depth image can be matched.

The image processing method includes: providing a function of a panoramic camera that generates a plurality of viewpoint images having different viewpoints; providing a function of a depth camera to generate a depth image; And implementing a camera incorporating the functions of FIG.

The real 3D image can be obtained through the combination of the panoptik camera and the depth camera.

3D shape information can be obtained from an imaging device that combines a planar camera and a depth camera.

The material information of the light field, and the light information can be easily obtained.

It is possible to implement a device that captures the light field shape with a small size camera.

The combination of the panoptik camera and the depth camera can increase the spatial and view resolution.

You can acquire a Dense light field.

1 is a block diagram showing the configuration of a video apparatus according to an embodiment of the present invention.
2 is a diagram illustrating an example of increasing the viewpoint resolution according to the present invention in comparison with the prior art.
3 is a block diagram showing the configuration of a video apparatus according to another embodiment of the present invention.
4 is a diagram showing an example of improving the shape information.
5 is a block diagram illustrating a configuration of a video apparatus according to another embodiment of the present invention.
6 is a diagram showing an example of predicting material information and illumination information in an image.
FIG. 7 is a diagram showing an example of acquiring representative light field information through pixel segmentation.
8 is a flowchart illustrating a procedure of an image processing method according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings and accompanying drawings, but the present invention is not limited to or limited by the embodiments.

1 is a block diagram showing the configuration of a video apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the imaging apparatus 100 may include a planar panoramic camera 110, a depth camera 120, and an image matching unit 130.

According to an embodiment, the imaging apparatus 100 may be a camera in which the planar panoramic camera 110 and the depth camera 120 are connected by a single body, 120 may be implemented as a camera.

A Plenoptic Camera 110 generates an integral image including a plurality of view images having different viewpoints. The planar panoramic camera 110 can generate a plurality of view images having a plurality of different view points through an array of lenses. Such an image including a plurality of viewpoint images is an integrated image.

A depth camera (120, Depth Camera) creates a depth image at a point in time.

The image matching unit 130 may generate a three-dimensional image by calibrating an accumulated image including a plurality of view images and a depth image at one view.

In an embodiment, the image matching unit 130 may measure the relative positions of the planar panoramic camera 110 and the depth camera 120. The relative position may be the distance between the plannop camera 110 and the depth camera 120 or the focal distance between the plannop camera 110 and the depth camera 120. [ The image matching unit 130 may match the integrated image with the depth image based on the measured relative position.

In another embodiment, the image matching unit 130 may calculate rotation information or movement information between any one view image and the depth image among the plurality of view images. Rotation information or translation information is one of the popular methods for matching between two different images. For reference, the image matching unit 130 can recognize a viewpoint difference between the plurality of viewpoint images according to the array configuration of the plurality of lenses of the planar panoramic camera 110. [ Accordingly, the image matching unit 130 can calculate both the rotation information and the movement information between all view images included in the integrated image and the depth image. The image matching unit 130 may match the integrated image and the depth image using the calculated rotation information or movement information.

In another embodiment, the image matching unit 130 may obtain homography between any one view image and the depth image among the plurality of view images. Homography is one of the popular methods for correcting images. The image matching unit 130 may match the depth image with the integrated image using the obtained homography.

In yet another embodiment, the image matching unit 130 may improve the shape information between the integrated image and the depth image by changing the pixel values of the decimal point view images among the plurality of view point images to pixel values of the majority viewpoint images can do. For example, if it is assumed that the plurality of viewpoint images are " 10 ", pixel values of three decimal point viewpoint images may be different from pixel values of seven majority viewpoint images. The image matching unit 130 may change the pixel values of the decimal point view images to the pixel values of the majority view images.

That is, the image matching unit 130 can maximize the consistency between the viewpoints such as photometry and texture, thereby improving the accuracy of the three-dimensional shape information. In this way, the image matching unit 130 can improve the accuracy of the depth image. The image matching unit 130 may match the depth image with the integrated image using the improved shape information.

2 is a diagram illustrating an example of increasing the viewpoint resolution according to the present invention in comparison with the prior art.

Referring to FIG. 2, a camera 210 (also referred to as a 'cdCam') combining a color camera and a depth camera can acquire three-dimensional pixels and color information obtained from a depth camera. At this time, the obtained color information represents the light ray information observed at one point in time.

On the other hand, a camera (220, also referred to as 'pdCam') combining a panoptic camera and a depth camera can acquire three-dimensional pixels obtained from a depth camera and color information at a plurality of points of view. That is, it is possible to acquire the light field information of a limited number (viewpoint resolution).

For example, the planar panoramic camera 110 has different arrays of lens arrays, and each three-dimensional pixel has a different depth. Therefore, the light field information observed in each of the three-dimensional pixels can not be regarded as being obtained from the same angle. However, the light field information can be utilized to increase the viewpoint resolution.

In yet another embodiment, the image matching unit 130 may calculate the amount of change in color information associated with the plurality of view images. The image matching unit 130 may predict the material information, the illumination information, and the color information of the object included in the plurality of view images based on the calculated amount of change. The image matching unit 130 may subdivide pixels included in the plurality of view images according to predicted material information, illumination information, and color information, respectively.

For example, the image matching unit 130 creates a first pixel by subdividing a pixel according to the predicted material information for one three-dimensional pixel, and subdivides the pixel according to the predicted illumination information to create a second pixel , The third pixel may be created by subdividing the pixel according to the predicted color information.

Then, the image matching unit 130 obtains the representative light field information representative of the three-dimensional pixel by summing the light fields included in the subdivided first pixel to the third pixel with respect to the three-dimensional pixel have. The image matching unit 130 may match the integrated image and the depth image based on the acquired representative light field information. Accordingly, the image matching unit 130 can increase the viewpoint resolution of the matched three-dimensional image by matching the accumulated image and the depth image using the representative light field information.

3 is a block diagram showing the configuration of a video apparatus according to another embodiment of the present invention.

3, the imaging apparatus 300 may include a planar panoramic camera 310, a depth camera 220, a shape capturing unit 330, and an image processing unit 340.

According to an embodiment, the imaging device 300 may be a camera that connects the planar panoramic camera 310 and the depth camera 320 by one body, or may be a camera that connects the planar panoramic camera 310 and the depth camera 320 It may be a camera with function implemented.

The planar camera 310 generates multi-view images having different viewpoints. The planar panoramic camera 310 can generate multiple viewpoint images having a plurality of different viewpoints through an array of lenses.

The depth camera 320 generates a depth image at one point in time.

The shape capturing unit 330 matches illumination information or material information between the multi-view images. In an embodiment, the shape capturing unit 330 may convert the pixel values of the fractional multi-view images included in the multi-view images into the pixel values of the majority multi-view images, so that the illumination information or the material information Can be matched.

The shape capturing unit 330 may acquire three-dimensional shape information between the multi-view images and the depth images based on the matched illumination information or material information. The shape capturing unit 330 may maximize the consistency of the illumination information or the material information, thereby improving the accuracy of the three-dimensional shape information.

The image processing unit 340 may match the depth image with the multi-view images using the three-dimensional shape information. If the accuracy of the three-dimensional shape information is improved, the accuracy of the depth image can also be improved. Accordingly, the image processing unit 340 can improve the precision of the depth image and easily process the matching of the multi-view images and the depth image.

4 is a diagram showing an example of improving the shape information.

Referring to FIG. 4, the multi-view image acquired by the planar panoramic camera 310 corresponds to the surrounding multi-view image. Accordingly, the shape capturing unit 330 can improve the accuracy of the three-dimensional shape information by processing the viewpoints of the multi-view images with high consistency.

Referring to reference numeral 410, the shape capturing unit 330 divides the pixel values (dotted arrows) of the faint multi-view images included in the multi-view images into pixel values of the majority multi-view images ). Accordingly, the shape capturing unit 330 can consistently match illumination information or material information between the multi-view images.

In this case, it can be seen that the three-dimensional shape information (dotted line circle) of the reference numeral 410 is improved like the three-dimensional shape information (solid line circle) of the reference numeral 420. In this way, the accuracy of the three-dimensional shape information can be improved, and the precision of the depth image can also be improved.

5 is a block diagram illustrating a configuration of a video apparatus according to another embodiment of the present invention.

5, the imaging device 500 may include a planar panoramic camera 510, a depth camera 520, a pixel granularity 530, and an image processing unit 540.

According to an embodiment, the imaging device 500 may be a camera in which the planar panoramic camera 510 and the depth camera 520 are connected to each other by a single body, It may be a camera with function implemented.

The planar camera 510 generates multi-view images having different viewpoints. The plannote camera 510 can generate multiple point-in-time multi-view images through a plurality of lenses array.

The depth camera 520 generates a depth image at one point in time.

The pixel subdivide unit 530 subdivides the pixels included in the multi-view images according to the material information, the illumination information, and the color information of the object included in the multi-view images. In an embodiment, the pixel segmentation unit 530 may calculate a variation amount of color information acquired from the multi-view images. The pixel segmentation unit 530 may predict the material information, the illumination information, and the color information of the object included in the multi-view images using the calculated amount of change.

6 is a diagram showing an example of predicting material information and illumination information in an image.

Referring to FIG. 6, it can be seen that the two circle colors (white) shown above are different from the two circles (black) colors shown below. In addition, the light ray (solid line arrow) included in the two circles shown above and the light ray (dotted line arrow, solid line arrow) included in the two circles shown below are different. Actual arrows indicate the color change according to the light ray, but in the drawing, it is difficult to intuitively display the color change, so that the shape change is indicated. The colors and arrow shapes of the circles indicated may have different colors or shapes depending on the material information of the object, the illumination information (illumination environment), and the color information. Accordingly, the pixel segmentation unit 530 can predict the material information, the illumination information, and the color information of the object included in the multi-view images using the calculated amount of change.

The pixel subdivider 530 may subdivide pixels included in the multi-view images according to the predicted material information, illumination information, and color information. For example, the pixel refinement unit 530 may generate a first pixel by subdividing a pixel according to the predicted material information for one three-dimensional pixel, subdivide the pixel according to the predicted illumination information to create a second pixel , The third pixel may be created by subdividing the pixel according to the predicted color information.

The image processor 540 uses the subdivided pixels to match the multi-view images with the depth images. The image processor 540 can obtain the representative light field information associated with the pixels in the selected area by summing the light field values included in the subdivided pixels in the selected area. The selected area may be a subdivided area of pixels, for example, one pixel or an area including one or more pixels.

The image processor 540 may match the depth image with the multi-view images based on the acquired representative light field information.

FIG. 7 is a diagram showing an example of acquiring representative light field information through pixel segmentation.

7, the pixel segmentation unit 530 divides one of the three-dimensional pixels into a first pixel 710, a second pixel 720, a third pixel 730, and a third pixel 740 according to material information, illumination information, . The image processor 540 combines the light field of the first pixel 710, the light field of the second pixel 720, and the light field of the third pixel 730 to generate representative light field information (740).

Accordingly, the image processing unit 540 can increase the viewpoint resolution of the matched three-dimensional image by matching the multi-view images with the depth image using the representative light field information.

8 is a flowchart illustrating a procedure of an image processing method according to an embodiment of the present invention. The image processing method of FIG. 8 can be implemented by the image device shown in FIG. 1, FIG. 3, or FIG.

Referring to FIG. 8, in step 810, the image processing method provides a function of a planar panoramic camera to generate a plurality of view images having different viewpoints.

In step 820, the image processing method provides the function of a depth camera to generate a depth image.

In step 830, the image processing method may implement a camera that combines the functions of the panoramic camera and the depth camera.

In an embodiment, the image processing method may measure a relative position of the panoramic camera and the depth camera, and may match the plurality of view images with the depth image based on the measured relative position.

In another embodiment, the image processing method may include calculating rotation information or movement information between any one of the plurality of viewpoint images and the depth image, and using the calculated rotation information or movement information, The viewpoint images and the depth images can be matched.

In still another embodiment, the image processing method may further include obtaining a homography between any one view image and the depth image among the plurality of view images, and using the obtained homography, Depth images can be matched.

In still another embodiment, the image processing method may further include converting pixel values of a plurality of viewpoint images of the plurality of viewpoint images into pixel values of a majority of viewpoint images, Dimensional shape information can be improved.

According to another embodiment, the image processing method may further include calculating a change amount of color information associated with the plurality of view-point images, and based on the calculated amount of change, the material information of the object included in the plurality of view- And color information, the pixels included in the plurality of view images can be subdivided according to each of the predicted material information, illumination information, and color information. The image processing method may obtain representative light field information associated with pixels in the selected region using a light field included in the subdivided pixels in the selected region.

The methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: Imaging device
110: Plano camera
120: Depth camera
130:

Claims (20)

A Plenoptic camera for generating an integral image including a plurality of viewpoint images whose viewpoints are different from each other;
A depth camera for generating a depth image; And
Dimensional image by calibrating the integrated image and the depth image to generate a three-
/ RTI >
Wherein the image matching unit comprises:
And a color information obtaining unit for calculating at least one of material information, illumination information, and color information of an object included in the plurality of view images based on the calculated amount of change, .
The method according to claim 1,
Wherein the image matching unit comprises:
Measuring a relative position of the planar camera and the depth camera, and matching the integrated image and the depth image based on the measured relative position. The method according to claim 1,
Wherein the image matching unit comprises:
Calculating rotation information or translation information between any one viewpoint image and the depth image of the plurality of viewpoint images and matching the integrated image and the depth image using the calculated rotation information or movement information, Imaging device. The method according to claim 1,
Wherein the image matching unit comprises:
Acquiring homography between one of the plurality of viewpoint images and the depth image, and matching the integrated image and the depth image using the acquired homography. The method according to claim 1,
Wherein the image matching unit comprises:
And changing the pixel values of the fractional view images to the pixel values of the majority view images to improve shape information between the integrated image and the depth image, Imaging device to match depth images. The method according to claim 1,
Wherein the image matching unit comprises:
And subdivides pixels included in the plurality of view images according to at least one of the predicted material information, illumination information, and color information. A Plenoptic camera for generating an integral image including a plurality of viewpoint images whose viewpoints are different from each other;
A depth camera for generating a depth image; And
Dimensional image by calibrating the integrated image and the depth image to generate a three-
And it includes a
Wherein the image matching unit comprises:
Acquiring representative light field information associated with pixels in the selected region using the light field contained in the subdivided pixels in the selected region, and matching the integrated image and the depth image based on the acquired representative light field information Video device. A panoramic camera for generating multi-view images having different viewpoints;
A depth camera for generating a depth image;
A shape capturing unit for matching illumination information or material information between the multi-view images and acquiring three-dimensional shape information between the multi-view images and the depth image based on the matched illumination information or material information; And
Dimensional image, and the depth image is matched with the multi-view images using the three-
/ RTI >
Wherein the image processing unit comprises:
And a color information obtaining unit for calculating at least one of material information, illumination information, and color information of an object included in the plurality of view images based on the calculated amount of change, .
9. The method of claim 8,
The shape capturing unit includes:
And converts the pixel values of the fractional multi-view images included in the multi-view images into pixel values of the majority multi-view images to match the illumination information or the material information of the multi-view images. delete A panoramic camera for generating multi-view images having different viewpoints;
A depth camera for generating a depth image;
A pixel subdivision unit for subdividing pixels included in the multi-view images according to material information, illumination information, and color information of an object included in the multi-view images; And
And an image processor for matching the multi-view images with the depth image using the subdivided pixels,
/ RTI > 12. The method of claim 11,
Wherein the pixel segmentation unit comprises:
Point information, estimating material information, illumination information, and color information of an object included in the multi-view images using the amount of change in color information acquired from the multi-view images, And subdivides the pixels contained in the multi-view images. 12. The method of claim 11,
Wherein the image processing unit comprises:
The method includes: acquiring representative light field information associated with pixels in the selected region by summing light field values included in the subdivided pixels in the selected region; Of the imaging device. Providing a function of a planar panoramic camera to generate a plurality of viewpoint images having different viewpoints;
Providing a function of a depth camera to generate a depth image;
A camera combining the function of the panoramic camera and the function of the depth camera, and matching the plurality of view images with the depth image;
Calculating a variation amount of color information associated with the plurality of viewpoint images; And
Estimating at least one of material information, illumination information, and color information of an object included in the plurality of view images based on the calculated amount of change;
And an image processing method.
delete 15. The method of claim 14,
Wherein the step of matching the plurality of view images with the depth image comprises:
Calculating rotation information or movement information between any one viewpoint image and the depth image among the plurality of viewpoint images; And
Matching the plurality of view images with the depth image using the calculated rotation information or movement information
And an image processing method. 15. The method of claim 14,
Wherein the step of matching the plurality of view images with the depth image comprises:
Acquiring a homography between any one viewpoint image and the depth image among the plurality of viewpoint images; And
Matching the plurality of view images with the depth image using the obtained homography;
And an image processing method. 15. The method of claim 14,
Changing pixel values of decimal point view images among the plurality of view point images into pixel values of majority viewpoint images; And
A step of improving three-dimensional shape information between the plurality of view-point images and the depth image
Further comprising the steps of: 15. The method of claim 14,
Wherein the predicting comprises:
Subdividing pixels included in the plurality of view images according to at least one of the predicted material information, illumination information, and color information
And an image processing method. 20. The method of claim 19,
Wherein the subdividing the pixels included in the plurality of view-
Acquiring representative light field information associated with a pixel in the selected region using a light field included in the subdivided pixels in the selected region
And an image processing method.

Images