WO2015050390A1

WO2015050390A1 - Method and device for processing image

Info

Publication number: WO2015050390A1
Application number: PCT/KR2014/009305
Authority: WO
Inventors: 최서영; 남동경; 조양호; 이진호
Original assignee: 삼성전자주식회사
Priority date: 2013-10-04
Filing date: 2014-10-02
Publication date: 2015-04-09
Also published as: US20160241846A1; KR20150041225A

Abstract

An image device generates a plurality of multi-view output images on the basis of an input image, and outputs the plurality of generated output images. The plurality of output images are outputted in a direction opposite to the views of the output images. The plurality of outputted output images are projected towards a left eye and a right eye of a viewer of the image device such that the viewer can recognize three-dimensional images.

Description

Image processing method and device

The following embodiments relate to image processing, and more particularly, to an image processing apparatus and an image processing method for generating a plurality of output images of a multi-view.

A method of providing a 3D stereoscopic image to a viewer of an imaging apparatus by using a stereoscopic method based on binocular disparity, a method of using a volumetric display that directly forms a stereoscopic image in space, and interference of light. And a holographic display based method.

When a 3D stereoscopic image is provided to a viewer of an imaging apparatus using a binocular parallax based stereo technique, output images having different viewpoints generated by the imaging apparatus are respectively projected to the left and right eyes of the viewer of the imaging apparatus. In this case, the output images may be separated into an image to be projected to the left of the viewer of the imaging device and an image to be projected to the right of the viewer by the 3D glasses worn by the viewer of the imaging device.

Alternatively, the output images may be separated into an image to be projected to the left of the viewer of the imaging device and an image to be projected to the right of the viewer by an optical lens which may be located in front of the display panel of the imaging device. The viewer of the imaging device may recognize a 3D stereoscopic image without wearing 3D glasses.

According to an aspect, the method may further include generating a plurality of multi-view output images based on a plurality of input images, and outputting the plurality of output images performed by the imaging apparatus. The viewpoints of the plurality of output images are different from each other, and the order from the left to the right of the regions in which the plurality of output images are projected in the space where the plurality of output images are projected is from the left of the viewpoints of the plurality of output images. An image processing method is provided, which is the reverse of the order to the right.

The plurality of output images may be projected in a continuous space.

The continuous space may be plural.

The plurality of output images may be projected into each of the plurality of consecutive spaces in the same order.

The plurality of input images may include a left input image and a right input image.

The plurality of output images may include a leftmost viewpoint output image corresponding to the left input image and a rightmost viewpoint output image corresponding to the right input image.

The rightmost viewpoint output image may be projected to the leftmost side of the plurality of output images.

The leftmost view output image may be projected to the rightmost side of the plurality of output images.

When the viewer of the imaging apparatus is located in one continuous space among the plurality of continuous spaces, a right view output image of the plurality of output images may be projected to the left eye of the viewer.

When the viewer of the imaging apparatus is located in one continuous space among the plurality of continuous spaces, a left view output image of the plurality of output images may be projected into the right eye of the viewer.

The left viewpoint output image and the right viewpoint output image may be images of viewpoints adjacent to each other among the plurality of output images.

The viewpoint of the left viewpoint output image may be a left viewpoint than the viewpoint of the right viewpoint output image.

When the viewer is located in two consecutive spaces of the plurality of consecutive spaces, the leftmost view output image may be projected to the left eye of the viewer.

When the viewer is located in two consecutive spaces of the plurality of continuous spaces, the rightmost viewpoint output image may be projected into the right eye of the viewer.

A difference between viewpoints of two adjacent output images among the plurality of output images may be within one viewpoint.

The difference between the viewpoint of the leftmost output image and the viewpoint of the rightmost output image among the plurality of output images may be one viewpoint.

The plurality of output images may include at least three output images.

The plurality of output images may include a leftmost viewpoint output image corresponding to the left input image, a rightmost viewpoint output image corresponding to the right input image, and an interpolation image.

The interpolation image may be an image generated based on an interpolation between the left input image and the right input image.

The difference between the viewpoint of the left input image and the viewpoint of the right input image may be one viewpoint.

The first time point may be determined based on an interpupillary distance (IPD) of a viewer expected for the imaging device.

In the viewpoints of the plurality of output images, the difference between the viewpoints of two adjacent output images may be constant.

The imaging device may include a plurality of pixels.

The corresponding pixels of the plurality of output images may be adjacent pixels among the plurality of pixels.

The imaging device may include a plurality of lenses.

Light emitted by the adjacent pixels may be projected through a lens located in front of the adjacent pixels of the plurality of lenses.

The lens may project the light of the adjacent pixels in a direction opposite to the order from the left to the right of the adjacent pixels.

The plurality of output images may be projected into each of a plurality of consecutive spaces in the same order.

The lens may project light of the adjacent pixels into each of the plurality of contiguous spaces.

In another aspect, an image processor and an image output unit, the image processor generates a plurality of multi-view output images based on a plurality of input images, the image output unit is a plurality of The output images are output, and the viewpoints of the plurality of output images are different from each other, and the order from the left to the right of the areas where the plurality of output images are projected in the space where the plurality of output images are projected is the plurality of outputs. An imaging apparatus is provided, which is in reverse order from the left to the right of the viewpoints of the images.

The plurality of output images may be projected in a continuous space.

The continuous space may be plural.

The image output unit may include a plurality of pixels.

The image output unit may further include a plurality of lenses.

1 illustrates an imaging device according to an exemplary embodiment.

2 illustrates an image processing method according to an exemplary embodiment.

3 illustrates a method of generating an output image, according to an exemplary embodiment.

4 illustrates projection of a plurality of output images into a predetermined space, according to an embodiment.

5 illustrates projection of a plurality of output images into a plurality of predetermined spaces according to an example.

6 illustrates a relationship between a plurality of output images projected into a predetermined space and a viewer according to an example.

7 illustrates projection of a plurality of output images into a plurality of predetermined spaces by a lens according to an example.

8 illustrates a view difference between each of a plurality of output images projected into a predetermined space, according to an example.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements.

1 illustrates an imaging device according to an exemplary embodiment.

In FIG. 1, an imaging apparatus 100 that performs image processing based on an input image is illustrated.

The imaging apparatus 100 may include a communicator 110, an image processor 120, an image outputter 130, and a storage 140.

The imaging apparatus 100 may be an apparatus that processes an input image and outputs an output image that may be recognized as a 3D stereoscopic image to a viewer of the imaging apparatus 100. As each of the output images having different viewpoints of the imaging apparatus 100 are projected to the left and right eyes of the viewer of the imaging apparatus 100, respectively, the viewer may display an image output from the imaging apparatus 100 without 3D glasses. It can be recognized as a stereoscopic image. For example, the imaging apparatus 100 may be an autostereoscopic 3D display apparatus, and may be a 3D television, a mobile phone and a 3D monitor, or a 3D stereoscopic image generating apparatus included in a 3D TV, a mobile phone and a 3D monitor. .

The imaging apparatus 100 may be a multi-view 3D display apparatus. The multi-view 3D display device may have a wider viewing range in which a viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image, as compared to the 2 view 3D display device. In other words, the viewer of the multiview 3D display device may recognize a 3D stereoscopic image in a wider range than the viewer of the two view 3D display device.

The viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image at a position away from the imaging apparatus 100 by a predetermined appropriate viewing distance. The predetermined viewing distance may be a value that may be determined according to the performance and / or characteristics of the components included in the imaging apparatus 100. The predetermined optimal viewing distance is the distance between the viewer of the imaging apparatus 100 and the viewer of the imaging apparatus 100, in which the output image of the imaging apparatus 100 can be most clearly recognized as a three-dimensional stereoscopic image by the viewer of the imaging apparatus 100. Can be. In other words, when the distance between the imaging apparatus 100 and the viewers of the imaging apparatus 100 is a predetermined proper viewing distance, each of the output images having different viewpoints of the imaging apparatus 100 is different from the left eye of the viewer of the imaging apparatus 100. And can be projected accurately into each of the right eye.

The communication unit 190 may be a device separate from the imaging device 100. The communication unit 190 may be a device that transmits an input image to the imaging apparatus 100 through a wired and / or wireless communication line.

As the input image input from the communication unit 190 is processed and output by the imaging apparatus 100, the viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image. The input image input to the communication unit 190 may be an image processed by the imaging apparatus 100. The input image may be an image used for generating an output image. The communication unit 190 may be a device that transmits a two-view image captured by the stereo camera to the imaging device 100 as part of the stereo camera system.

The input image may be a two-view image captured by a stereo camera. The input image may be data in which a two view image is compressed and / or encoded.

The communication unit 110 may receive an input image from the communication unit 190 through a wired and / or wireless communication line. The communication unit 110 may transmit the received input image to the image processing unit 120. Alternatively, the communication unit 110 may store the received input image in the storage 140.

The communicator 110 may be a hardware module such as a network interface card, a network interface chip, and a networking interface port.

The image processor 120 may be a device that processes operations required to generate a plurality of output images having different viewpoints based on the input image. For example, when the input image is encoded data, the image processor 120 may decode the input image. The image processor 120 may include at least one processor for processing operations required to generate a plurality of output images. For example, the image processor 120 may include a graphics processing unit.

The image processor 120 may generate a plurality of output images of a multiview. The input image may be an image received from the communication unit 190 or an image stored in the storage unit 140 to be described later. The input image may be a two-view image or a two-dimensional image including depth information.

A method of generating output images having different viewpoints from the input image will be described in more detail with reference to FIGS. 2 and 3 below.

The storage unit 140 may store information related to the setting and operation of the imaging apparatus 100 and / or the input image. The input image stored in the storage 140 may be an image transmitted from the communication unit 190 or an image transmitted from an external electronic device or an electronic medium other than the communication unit 190. For example, the input image stored in the storage 140 may include depth information.

The storage 140 may be a hardware module for storing information such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, and the like.

The image output unit 130 may be an apparatus that outputs an output image generated by the image processor 120 and projects the output image into a space where a viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image. have.

Although not shown, the image output unit 130 may include a display panel. The display panel of the image output unit 130 may include a plurality of pixels. For example, the image output unit 130 may be a liquid crystal display (LCD), a plasma display panel (PDP) device, or an organic light emitting diode (OLED) display device.

The output image generated by the image processor 120 may be output by being assigned to the pixel of the image output unit 130. The output image output by the image output unit 130 may be projected to the left and right eyes of the viewer of the imaging apparatus 100 located at a predetermined viewing distance from the image output unit 130.

For a method of outputting the output image generated by the image processor 120 on the image processor 130 and the output image generated by the image processor 120 to the space, see FIGS. 2 to 8 below. This is explained in more detail.

2 illustrates an image processing method according to an exemplary embodiment.

According to the description above with reference to FIG. 1, the image processor 120 may generate a plurality of output images of a multiview based on an input image, and the image output unit 130 is generated by the image processor 120. The plurality of output images may be output.

In operation 210, the communicator 110 may receive a plurality of input images. The plurality of input images may be a plurality of images at different viewpoints. The plurality of input images may include a left input image that is an image of a left view and a right input image that is an image of a right view as compared to a left input image.

The left input image may be an image further including leftmost information not included in the right input image. In other words, the information included in the left input image may correspond to visual information that a person recognizes through the left eye. The right input image may be an image further including the rightmost information not included in the left input image. In other words, the information included in the right input image may correspond to visual information that a person recognizes through the right eye. For example, the left input image and the right input image may be images captured by each lens of the stereo camera.

In operation 220, the image processor 120 may generate a plurality of output images of a multiview based on the plurality of input images. In other words, each of the plurality of output images may be generated from a plurality of input images including a left input image and a right input image.

The viewpoints of the plurality of output images generated by the image processor 120 may be different from each other. For example, the plurality of output images may be output to a left input view image corresponding to the left input image including the leftmost information among the plurality of input images and a right input image of the input image including the rightmost information among the input images. It may include a corresponding rightmost view output image.

The number of output images generated by the image processor 120 may be different from the number of input images. For example, the number of output images generated by the image processor 120 may be equal to or greater than the number of input images. Each of the plurality of input images may be an image photographed through each lens of the stereo camera. When the plurality of input images are two, the plurality of output images may include one or more intermediate view output images of one or more different views in addition to the left view output image and the right view output image.

Each of the intermediate view output images may be an image further including information on the right side than the leftmost view output image, and further including information on the left side than the rightmost view output image. For example, the plurality of output images generated by the image processor 120 may include at least three output images. When there are three or more output images, two of the plurality of output images may be a leftmost viewpoint output image and a rightmost viewpoint output image.

A method of generating a plurality of output images of a multiview from a plurality of input images will be described in more detail with reference to FIG. 3 below.

In operation 230, the image outputter 130 may output a plurality of output images generated by the image processor 120. The plurality of output images generated by the image processor 120 may be output at different positions on the display panel of the image output unit 130. The position on the display panel where each of the plurality of output images generated by the image processor 120 is output may be a pixel of the display panel to which each output image is assigned. Each of the plurality of output images generated by the image processor 120 may be output to each of adjacent pixels of the display panel. The corresponding pixels of the plurality of output images generated by the image processor 120 may be adjacent pixels among the plurality of pixels of the display panel. The corresponding pixels of the plurality of output images generated by the image processor 120 may be pixels from which each of the plurality of output images is output from among the plurality of pixels of the image output unit 130. For example, each output image generated by the image processor 120 may be output from each of adjacent pixels of the display panel of the image output unit 130 in the order of viewpoints from the leftmost viewpoint output image to the rightmost viewpoint output image. have.

Each output image output to each pixel of the image output unit 130 may be light including information related to each output image. For example, the information of each output image included in the light output from each pixel may be color information (eg, an RGB value) of each output image.

The plurality of output images generated by the image processor 120 may be projected in a continuous space. An area in which the output image is not projected may not exist between regions in which each output image is projected in a predetermined space where a plurality of output images are projected. In other words, each of the plurality of output images may be projected so as to have no gaps in the predetermined space or enough gaps that the viewer cannot feel.

Regions in a predetermined space where each of the plurality of output images generated by the image processor 120 are projected may be different from each other. In other words, an area in which two or more output images are repeatedly projected in a predetermined space where a plurality of output images are projected may not exist.

When a plurality of output images generated by the image processing unit 120 are projected into a predetermined space existing on a plane spaced apart from the imaging apparatus 100 by a predetermined proper viewing distance, the plurality of output images are arranged in a continuous space. The areas within a given space in which each of the plurality of output images are projected may be different from each other. In this case, the viewer of the imaging apparatus 100 located in the predetermined space where the plurality of output images are projected may recognize the output image in all areas of the predetermined space, and the left and right eyes of the viewer of the imaging apparatus 100 may be Only one output image of each of the plurality of output images may be projected.

A method of projecting each of the plurality of output images generated by the image processor 120 into the space will be described in more detail with reference to FIGS. 4 to 8 below.

Since the technical contents described above with reference to FIGS. 1 and 2 may be applied as it is, a more detailed description will be omitted below.

3 illustrates a case where a plurality of output images are generated from a plurality of input images including a left input image 310 and a right input image 320 described above with reference to FIG. 2. The difference in the position of the circle displayed in each of the images 310 to 345 may represent a viewpoint difference between the images 310 to 345.

The left input image 310 and the right input image 320 are images captured by a stereo camera, input images stored in the storage 140 of FIG. 1, or inputs transmitted from the communication unit 190 to the imaging apparatus 100. May be images.

The

input images

310 and 320 may include predetermined depth information representing a depth of an object expressed in the

input images

310 and 320. For example, when only the left input image 310 is projected to the left eye and the right eye of the viewer of the imaging apparatus 100, the viewer of the imaging apparatus 100 may determine predetermined depth information of the left input image 310 based on empirical recognition. Through this, the user may feel a 3D effect on the object represented in the left input image 310. The predetermined depth information representing the depth of the object represented in the

input images

310 and 320 may include shape information, size information, perspective information, and occlusion of the object of the object represented in the

images

310 and 320. The information may be one or more of occlusion information and lighting information.

The difference between the viewpoints of the left input image 310 and the right input image 320 may be one viewpoint. The difference between the viewpoints of the left input image 310 and the right input image 320 may be determined based on an interpupillary distance (IPD) of a viewer of the imaging apparatus 100 that is expected with respect to the imaging apparatus 100. For example, when the difference between the viewpoints of the images is one viewpoint, one viewpoint may be equal to one IPD. 1 The IPD may be a value determined based on a distance between pupils of a general person. In other words, in general, a viewpoint difference between visual information recognized through a left eye and visual information recognized through a right eye may be 1 IPD. 1 IPD may be a value corresponding to a distance between two lenses of a stereo camera.

The viewer of the imaging apparatus 100 is that the left input image 310 is projected to the left eye, the right input image 320 is projected to the right eye, and the difference between the viewpoints of the left input image 310 and the right input image 320 is 1 IPD. In this case, the images projected by both eyes of the viewer of the imaging apparatus 100 may be recognized as stereoscopic stereoscopic images.

The images 330 to 350 are output images generated from the left input image 310 and the right input image 320, and include the leftmost view output image 330, the first intermediate view output image 335, and the second intermediate view. The output image 340 and the rightmost view output image 345 may be provided.

The leftmost view output image 330 may correspond to the left input image 310. In other words, the information included in the leftmost view output image 330 may correspond to visual information that a person recognizes through the left eye. The rightmost view output image 345 may correspond to the right input image 320. In other words, the information included in the rightmost viewpoint output image 345 may correspond to visual information that a person recognizes through the right eye.

Each of the intermediate view output images may be an image further including information on the right side than the leftmost view output image, and further including information on the left side than the rightmost view output image. In other words, each of the intermediate view output images may be an output image of a view existing between the leftmost view and the right view among the plurality of output images.

The plurality of output images 330 to 345 generated from the plurality of

input images

310 and 320 may include the leftmost view output image corresponding to the left input image 310 of the plurality of

input images

310 and 320. 330, the rightmost view output image 345 corresponding to the right input image 320 among the plurality of

input images

310 and 320, and an interpolation image of the plurality of

input images

310 and 320. . The interpolation image of the plurality of

input images

310 and 320 may be an image generated based on the interpolation between the left input image 310 and the right input image 320. The interpolated image may be generated as one or more images of different viewpoints. For example, the first intermediate view output image 335 and the second intermediate view output image 340 may be interpolation images generated based on interpolation of the left input image 310 and the right input image 320.

The interpolation of the left input image 310 and the right input image 320 is performed by using the information included in the left input image 310 and the right input image 320, and the viewpoint and the right input image of the left input image 310 ( It may be a process of generating an image of an intermediate view existing between the viewpoints of 320. The image processor 120 may generate an interpolated image by applying an image interpolation technique to the left input image 310 and the right input image 320. One or more known algorithms that may be used for interpolation of an image may be used in an image interpolation technique applied by the image processor 120 to the left input image 310 and the right input image 320.

The plurality of output images 330 to 345 may include predetermined depth information included in the plurality of

input images

310 and 320 described above.

Each of the plurality of output images 330 to 345 may be projected by the image output unit 130 to different regions of the predetermined space described above with reference to FIG. 2. For example, each of the plurality of output images 330 to 345 in the order of viewpoints from the rightmost viewpoint output image 340 to the leftmost viewpoint output image 330 is the leftmost of the regions where a predetermined space is divided into four sections. From the region of to the region of the rightmost side, the predetermined space can be projected into each of the regions which are divided into quarters.

A method in which each of the plurality of output images 330 to 345 is projected into a space will be described in more detail with reference to FIGS. 4 to 8 below.

In FIG. 4, only four output images are generated from two input images, but the above description may be applied to a case in which five or more output images are generated from one or more input images. When a plurality of output images are generated from one input image, the input image may include additional information (eg, depth information) for generating the plurality of output images.

When there are N output images generated by the image processor 120, the imaging apparatus 100 may be an N-view autostereoscopic 3D display apparatus. For example, when four output images 330 to 345 are generated by the image processor 120, the imaging apparatus 100 may be a four-view autostereoscopic 3D display apparatus.

In FIG. 4, a case in which each of the plurality of output images 330 to 345 output to the image output unit 130 is projected onto different areas of the continuous space 410 is illustrated. The plurality of output images 330 to 345 may be projected in the continuous space 410.

The continuous space 410 may be a space that exists on a plane separated by a predetermined viewing distance from the imaging apparatus 100 described above with reference to FIG. 2.

Values 1 to 2 displayed in the plurality of output images 330 to 345 may be relative values displayed to distinguish a viewpoint difference between the plurality of output images 330 to 345. For example, a value 1 displayed in the leftmost view output image 330 may represent an image of the leftmost view. The larger the number displayed on each output image, the more the output image may be an output image of the right view. The value displayed on the rightmost viewpoint output image 345 may be 2. The difference in viewpoints between the leftmost viewpoint output image 330 and the rightmost viewpoint output image 345 is 1, indicating that the viewpoint difference between the leftmost viewpoint output image 330 and the rightmost viewpoint output image 345 is 1 IPD. have.

Accordingly, the difference between the viewpoint of the leftmost output image 330 and the viewpoint of the rightmost output image 345 among the plurality of output images 330 through 345 may be one viewpoint, and the plurality of output images 330 through 345. The difference between the viewpoints of two adjacent output images may be within one viewpoint. In the viewpoints of the plurality of output images 330 to 345, the difference between the viewpoints of two adjacent output images may be constant. For example, as illustrated, a difference in viewpoints of adjacent images among the plurality of output images 330 to 345 may be constant (however, in FIG. 4, values of two decimal places or less are discarded for convenience). Alternatively, the image processor 120 may adjust a viewpoint difference between two adjacent output images among the plurality of output images 330 to 345. For example, the image processor 120 adjusts variables used in an image interpolation technique applied to the left input image 310 and the right input image 320 when generating the interpolated images, and thus the difference between the viewpoints of two adjacent output images. Can be adjusted.

The order from the left to the right of the areas in which the plurality of output images 330 to 345 are projected in the continuous space 410 in which the plurality of output images 330 to 345 are projected is the plurality of output images 330 to 345. The order from the left to the right of the time points of the 345 may be reversed. For example, as illustrated, each of the plurality of output images 330 to 345 is each of pixels adjacent to each other of the image output unit 130 in the order from the left to the right of the viewpoints of the plurality of output images 330 to 345. Can be output in order from left to right, and each of the outputted output images 330 to 345 is the right side of the viewpoints of the plurality of output images 330 to 345 in the continuous space 410. From the leftmost region among the quadrant regions of the continuous space 410 in the order from the left to the left may be projected in the right direction. The size of each of the quadrant regions of the continuous space 410 may be the same size, and the width of each region may be 1 IPD or less. In other words, the width of each region in the continuous space 410 that each of the plurality of output images 330 to 345 reaches may be 1 IPD or less.

For example, the rightmost viewpoint output image 345 may be projected to the leftmost region in the continuous space 410 of the plurality of output images 330 to 345. In addition, the leftmost view output image 330 may be projected to the rightmost region in the continuous space 410 of the plurality of output images 330 to 345. As illustrated, the second intermediate view output 340, which is the right image among the remaining first intermediate view output image 335 and the second intermediate view output 340 image, is the rightmost view output image (in the continuous space 410). 345 may be projected in a region to the right adjacent to the projected area, and the first intermediate view output image 335 is adjacent to the left to the region where the leftmost view output image 330 in the continuous space 410 is projected. Can be projected onto the area.

Since the technical contents described above with reference to FIGS. 1 to 3 may be applied as it is, a more detailed description will be omitted below.

There may be a plurality of continuous spaces 410 in which the plurality of output images 330 to 345 are projected. The plurality of output images 330 to 345 may be projected into each of the plurality of continuous spaces in the same order as projected in the continuous space 410.

The set of pixels 510, which are adjacent pixels of the image output unit 130 from which the plurality of output images 330 to 345 are output, may be plural. The plurality of output images 330 to 345 are sets of respective pixels of the plurality of sets of pixels 510, 510-1 and 510-2 in the same order as output from each pixel of the set of pixels 510. It can be output at each pixel of. Each set of pixels may be in a continuous position on the image output unit 130.

Although not described in FIG. 5 for convenience, the plurality of output images 330 to 345 output from each of the plurality of sets of pixels 510, 510-1, and 510-2 are projected onto the continuous space 410. Can be projected into each of a plurality of contiguous spaces in the same order.

As the number of sets of pixels 510 increases, the number of consecutive spaces 410 may also increase. The continuous space 410 may be a viewing range in which a viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image. In other words, as the number of sets of pixels 510 increases, a viewing range in which a viewer of the imaging apparatus 100 may recognize a 3D stereoscopic image may be widened.

In FIG. 5, for convenience, the sets of pixels are shown only in the lateral direction of the image output unit 130, but the sets of pixels may also exist in the longitudinal direction of the image output unit 130.

Recognition of the viewer's three-dimensional stereoscopic image of the plurality of output images 330 to 345 projected into a plurality of consecutive spaces in the continuous space 410 is described below. This is described in more detail with reference to 6.

Since the technical contents described above with reference to FIGS. 1 to 4 may be applied as it is, a more detailed description will be omitted below.

The left and right eyes of the viewer of the imaging apparatus 100 may be located in the plurality of continuous spaces described above with reference to FIG. 5. For example, the

contiguous spaces

410 and 610 may correspond to the plurality of contiguous spaces described above with reference to FIG. 5. The continuous space 610 is a space equivalent to the continuous space 410 and may be a space adjacent to the continuous space 410.

Each of the left and right eyes of the viewer of the imaging apparatus 100 is located in a plurality of spaces existing on a plane separated by a predetermined appropriate viewing distance from the imaging apparatus 100, respectively. Different output images may be projected from among the plurality of output images 330 to 345, respectively. In other words, the

consecutive spaces

410 and 610 may be a viewing range of the viewer of the imaging apparatus 100.

In FIG. 6, for convenience, a plurality of output images are respectively displayed as a whole to each pixel of the set of pixels 510, but information corresponding to only a part of the displayed image is output to each pixel of the set of pixels 510. Can be.

When the leftmost viewpoint output image 330 is projected to the left eye of the viewer of the imaging apparatus 100 and the rightmost viewpoint output image 345 is projected to the right eye of the viewer of the imaging apparatus 100, the stereoscopic is displayed to the viewer. An image is provided. The viewer can recognize the stereoscopic output images projected by both eyes as a three-dimensional stereoscopic image. On the contrary, when the rightmost viewpoint output image 345 is projected to the left eye of the viewer of the imaging apparatus 100 and the leftmost viewpoint output image 345 is projected to the right eye of the viewer of the imaging apparatus 100, Pseudoscopic images are provided. The viewer of the imaging apparatus 100 may not recognize the output images of Pseudoscopic projected by both eyes as a 3D stereoscopic image. In the case of pseudoscopic, the difference in viewpoint between the output images projected by both eyes of the viewer is completely reversed compared to stereoscopic. Accordingly, a stereoscopic sense may be formed that is opposite to the empirical recognition of the viewer of the imaging apparatus 100. The viewer of the imaging apparatus 100 may not recognize the output images of the pseudoscopic projections with both eyes as a 3D stereoscopic image, and may feel fatigue when viewing the output images of the pseudoscopic.

When the right view output image is projected to the left eye of the viewer of the imaging device 100 and the left view output image is projected to the right eye of the viewer of the imaging device 100, the view point of the left view output image projected to the right eye is The viewer is provided with a semi-pseudoscopic image when the viewpoint of the left view is more than the viewpoint of the right view output image projected by and the view difference between the left view output image and the left view output image is less than one view point. The viewer can recognize the output image of the semi- pseudoscopic projected by both eyes as a three-dimensional stereoscopic image. The viewer of the imaging apparatus 100 may recognize the semi-pseudoscopic output images projected by both eyes as three-dimensional stereoscopic images through predetermined depth information included in the output images described above with reference to FIG. 3. The output images of the semi- pseudoscopic may be similar to the output images of the pseudoscopic. Since the viewpoint difference between the output images projected by both viewers is less than one viewpoint, the viewer of the imaging apparatus 100 outputs the images through predetermined depth information included in the semi- pseudoscopic output images based on empirical recognition. You can feel the three-dimensional effect on the object expressed inside. For example, a viewer of the imaging apparatus 100 protrudes a bright portion of a part of an object represented in output images projected by both eyes based on occlusion information and lighting information included in the output images. A dark portion of a part of the object expressed in the output images projected by both eyes may be recognized as a depressed portion.

As shown in the case of (1) to (3), the left eye and the right eye of the viewer of the imaging apparatus 100 of the imaging apparatus 100 have a continuous space (one of the plurality of consecutive spaces 410 and 610). 410, the right view output image of the plurality of output images 330 to 345 is projected to the left eye of the viewer of the imaging apparatus 100, and the plurality of the right eye to the viewer of the imaging apparatus 100. The left view output image of the output images 330 to 345 may be projected. The left viewpoint output image and the right viewpoint output image are images of viewpoints adjacent to each other among the plurality of output images 330 to 345, and the viewpoint of the left viewpoint output image may be a view on the left side more than the viewpoint of the right viewpoint output image. have. The viewpoint difference between the left view output image and the right view output image projected by both eyes of the viewer of the imaging apparatus 100 may be less than one view point. The viewpoint difference between the left view output image and the right view output image projected by the viewer's both eyes may be semi- pseudoscopic. That is, when the viewer is located in one continuous space 410 of the plurality of

consecutive spaces

410 and 610, the viewer may be provided with a semi- pseudoscopic image. The viewpoint difference between the left view output image and the left view output image projected by both eyes of the viewer of the imaging apparatus 100 may be constant.

As shown in the case of (4), when the viewer of the imaging apparatus 100 is located in two

consecutive spaces

410 and 610 of the plurality of consecutive spaces, the viewer enters the left eye of the viewer of the imaging apparatus 100. The leftmost viewpoint output image 330 is projected, and the rightmost viewpoint output image 345 is projected into the right eye of the viewer of the imaging apparatus 100. The viewpoint difference between the leftmost viewpoint output image 330 and the rightmost viewpoint output image 345 may be one viewpoint. The difference in viewpoint between the leftmost viewpoint output image 330 and the rightmost viewpoint output image 345 may be stereoscopic. In other words, when the viewer is located in two consecutive spaces of the plurality of

consecutive spaces

410 and 610, the viewer may be provided with a stereoscopic image.

The viewer of the imaging apparatus 100 is a semi- pseudoscopic or stereoscopic because the viewpoint difference between the images projected into both eyes of the viewer of the imaging apparatus at all possible positions in the

contiguous spaces

410 and 610 is semi- pseudoscopic or stereoscopic. The output images projected by both eyes at all possible positions in the

continuous spaces

410 and 610 may be recognized as three-dimensional stereoscopic images. In other words, the stereoscopic image or the semi- pseudoscopic image of one point of view may be provided to the viewer in all viewing ranges of the imaging apparatus 100.

In the case of (1) to (4) illustrated, each three-dimensional stereoscopic image recognized by the viewer is an image based on two adjacent output images among the plurality of images 330 to 345, or the plurality of three-dimensional images. The image is based on the leftmost view output image 330 and the right view output image 345 among the images 330 to 345. Therefore, the 3D stereoscopic images may be different images.

The imaging apparatus 100 may include a plurality of lenses 720, 720-1, and 720-2. The plurality of lenses may be part of the image output unit 130. The plurality of lenses may be a collection of lenses arranged in the lateral direction of the image output unit 130. In other words, the plurality of lenses may be a lens array. The lens array may also be disposed in the longitudinal direction of the image output unit 130.

In FIG. 7, it is shown that the plurality of output images 330 to 345 output from the set of pixels 510 described above with reference to FIG. 5 are projected by the lens 720 into a plurality of contiguous spaces. In FIG. 7, for convenience, the plurality of output images 330 to 345 are shown to be emitted from the lens 720 and projected into the continuous space 410. Light is emitted from each pixel of the set 510, and the emitted light can be refracted by passing through the lens 720 to be projected into a plurality of contiguous spaces.

In addition, although the number of the plurality of consecutive spaces is illustrated as four in FIG. 7, the number of the plurality of consecutive spaces may increase according to the number of sets of the plurality of pixels 510, 510-1 and 510-2. Can be.

The number of sets of pixels 510, 510-1, and 510-2 and the number of lenses of the plurality of pixels may be the same.

Each of the plurality of output images 330 through 345 output from the set of pixels 510 may be light including information associated with each

output image

330, 335, 340 or 345. For example, the information of each

output image

330, 335, 340, or 345 included in the light output from each pixel in the set of pixels 510 may include the color information of each

output image

330, 335, 340, or 345. For example, an RGB value).

Light emitted by adjacent pixels of the set of pixels 510 may cause the lens 720 located in front of the adjacent pixels of the set of pixels 510 of the plurality of lenses 720, 720-1, and 720-2 to pass through the lens 720. Can be projected through. In other words, each of the plurality of lenses 720, 720-1, and 720-2 can refract light emitted from a set of pixels corresponding to each lens, and light emitted from a set of pixels corresponding to each lens. Can be projected into each of a plurality of contiguous spaces.

Lens 720 may project light of adjacent pixels of set of pixels 510 in a direction opposite to the order from left to right of adjacent pixels of set of pixels 510. Lens 720 may project light of adjacent pixels of set of pixels 510 into each of a plurality of contiguous spaces.

Each of the plurality of lenses may be made of plastic.

For example, each of the plurality of lenses 720, 720-1, and 720-2 may be a lenticular lens. The lens 720 may have properties similar to those of the convex lens.

Unlike the illustrated example, each of the plurality of lenses may correspond to each pixel of the image output unit 130. In other words, the number of pixels of the image output unit 130 and the number of lenses may be the same. When each of the plurality of lenses corresponds to each pixel of the image output unit 130, each shape of the plurality of lenses may be hemispherical.

Alternatively, each of the plurality of lenses may have a semi-cylindrical shape extending in the longitudinal direction of the image output unit 130. The number of the plurality of lenses is equal to the number of sets of pixels 510, 510-1 and 510-2 in the lateral direction of the image output unit 130 or the number of pixels in the lateral direction of the image output unit 130. May be the same. The plurality of semi-cylindrical lenses may be disposed diagonally rather than in the longitudinal direction of the image output unit 130.

The plurality of lenses may be manufactured in the form of a sheet or film having a size corresponding to the size of the image output unit 130. The sheet or film including the plurality of lenses may be attached to the front surface of the display panel of the image output unit 130.

Each of the plurality of lenses may be an electro-active lenticular lens. The electronic active lenticular lens is an electronic liquid crystal lens, and may be a lens whose refractive index changes as a voltage is applied to molecules of the electronic liquid crystal of the electronic active lenticular lens. When each of the plurality of lenses included in the imaging apparatus 100 is an electronic active lenticular lens, a viewer of the imaging apparatus 100 may view a three-dimensional stereoscopic image and a voltage according to a voltage applied to molecules of an electronic liquid crystal of the electronic active lenticular lens. You can watch both two-dimensional images.

Since the technical contents described above with reference to FIGS. 1 to 6 may be applied as they are, a detailed description thereof will be omitted below.

As described above with reference to FIG. 4, values of 1 to 2 corresponding to the plurality of output images 330 to 345 may be relative values displayed to distinguish a viewpoint difference between the plurality of output images 330 to 345. .

Each of the plurality of output images 330-345 output from the set of pixels 510 may be projected by the lens 720 into each of the quadrant regions of the contiguous space 410.

In the continuous space 410 in which the plurality of output images 330 to 345 are projected, the order from the left to the right of the areas where the plurality of output images 330 to 345 are projected is a plurality of output images ( 330 to 345 may be in reverse order from left to right.

As shown, each of the plurality of output images 330 to 345 is a direction from the leftmost region to the right side of the quadrant regions of the continuous space 410, from the rightmost viewpoint output image 345 to the most. Each of the views up to the left view output image 330 may be projected.

A viewpoint difference between adjacent output images among the plurality of output images 330 to 345 may be constant. For example, the viewpoint difference may be 1/3 IPD. The viewpoint difference between the rightmost viewpoint output image 345 and the leftmost viewpoint output image 330 may be 1 IPD.

Since the technical contents described above with reference to FIGS. 1 to 7 may be applied as it is, a more detailed description will be omitted below.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims

Performed by the imaging device,

Generating a plurality of multi-view output images based on the plurality of input images; And

Outputting the plurality of output images

Including,

The viewpoints of the plurality of output images are different from each other,

The order from the left to the right of the areas in which the plurality of output images are projected in the space in which the plurality of output images are projected is opposite to the order from the left to the right of the viewpoints of the plurality of output images.
The method of claim 1,

And the plurality of output images are projected in a continuous space.
The method of claim 2,

The continuous space is plural,

And the plurality of output images are projected into each of the plurality of contiguous spaces in the same order.
The method of claim 3,

The plurality of input images include a left input image and a right input image,

The plurality of output images includes a leftmost viewpoint output image corresponding to the left input image and a rightmost viewpoint output image corresponding to the right input image,

The rightmost viewpoint output image is projected to the leftmost of the plurality of output images, the leftmost viewpoint output image is projected to the rightmost of the plurality of output images,

When the viewer of the imaging apparatus is located in one continuous space of the plurality of continuous spaces, the right view output image of the plurality of output images is projected to the left eye of the viewer, and to the right eye of the viewer. A left viewpoint output image is projected from among the plurality of output images, and the left viewpoint output image and the right viewpoint output image are images of viewpoints adjacent to each other among the plurality of output images, and the viewpoint of the left viewpoint output image is Right view is more left view than the view of the output image,

When the viewer is located in two consecutive spaces of the plurality of consecutive spaces, the leftmost view output image is projected to the left eye of the viewer, and the rightmost view output image is to the right eye of the viewer. Projected image processing method.
The method of claim 1,

The difference between viewpoints of two adjacent output images among the plurality of output images is within 1 viewpoint,

The difference between the viewpoint of the leftmost output image and the viewpoint of the rightmost output image among the plurality of output images is one viewpoint.
The method of claim 1,

And the plurality of output images comprises at least three output images.
The method of claim 1,

The plurality of input images include a left input image and a right input image,

The plurality of output images includes a leftmost viewpoint output image corresponding to the left input image, a rightmost viewpoint output image and an interpolated image corresponding to the right input image,

The interpolation image is an image generated based on an interpolation between the left input image and the right input image.
The method of claim 1,

The plurality of input images include a left input image and a right input image,

The plurality of output images includes a leftmost viewpoint output image corresponding to the left input image, a rightmost viewpoint output image and an interpolated image corresponding to the right input image,

And the rightmost viewpoint output image is projected to the leftmost side of the plurality of output images, and the leftmost viewpoint output image is projected to the rightmost side of the plurality of output images.
The method of claim 1,

The plurality of input images include a left input image and a right input image,

And a difference between a viewpoint of the left input image and a viewpoint of the right input image is one viewpoint.
The method of claim 9,

And the first time point is determined based on an interpupillary distance (IPD) of a viewer expected for the imaging device.
The method of claim 1,

And a difference between viewpoints of two adjacent output images in the viewpoints of the plurality of output images is constant.
The method of claim 1,

The imaging device includes a plurality of pixels,

Corresponding pixels of the plurality of output images are adjacent pixels of the plurality of pixels.
The method of claim 12,

The imaging device includes a plurality of lenses,

The light emitted by the adjacent pixels is projected through a lens located in front of the adjacent pixels of the plurality of lenses,

And the lens projects the light of the adjacent pixels in a direction opposite to the order from the left to the right of the adjacent pixels.
The method of claim 13,

The plurality of output images are projected into each of the plurality of consecutive spaces in the same order,

And the lens projects the light of the adjacent pixels into each of the plurality of contiguous spaces.
A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 14.
An image processor; And

Video output section

Including,

The image processor generates a plurality of multi-view output images based on a plurality of input images,

The image output unit outputs the plurality of output images,

The viewpoints of the plurality of output images are different from each other,

The order from the left to the right of the areas in which the plurality of output images are projected in the space in which the plurality of output images are projected is opposite to the order from the left to the right of the viewpoints of the plurality of output images.
The method of claim 16,

The plurality of output images are projected in a continuous space,

The continuous space is plural,

And the plurality of output images are projected into each of the plurality of contiguous spaces in the same order.
The method of claim 17,

The plurality of input images include a left input image and a right input image,

The plurality of output images includes a leftmost viewpoint output image corresponding to the left input image and a rightmost viewpoint output image corresponding to the right input image,

The rightmost viewpoint output image is projected to the leftmost of the plurality of output images, the leftmost viewpoint output image is projected to the rightmost of the plurality of output images,

When the viewer of the imaging apparatus is located in one continuous space of the plurality of continuous spaces, the right view output image of the plurality of output images is projected to the left eye of the viewer, and to the right eye of the viewer. A left viewpoint output image is projected from among the plurality of output images, and the left viewpoint output image and the right viewpoint output image are images of viewpoints adjacent to each other among the plurality of output images, and the viewpoint of the left viewpoint output image is Right view is more left view than the view of the output image,

When the viewer is located in two consecutive spaces of the plurality of consecutive spaces, the leftmost view output image is projected to the left eye of the viewer, and the rightmost view output image is to the right eye of the viewer. Projected image device.
The method of claim 16,

The image output unit,

Multiple pixels

Including,

And corresponding pixels of the plurality of output images are adjacent pixels of the plurality of pixels.
The method of claim 19,

The image output unit,

Multiple lenses

More,

The light emitted by the adjacent pixels is projected through a lens located in front of the adjacent pixels of the plurality of lenses,

And the lens projects the light of the adjacent pixels in a direction opposite to the order from the left to the right of the adjacent pixels.