WO2012096065A1

WO2012096065A1 - Parallax image display device and parallax image display method

Info

Publication number: WO2012096065A1
Application number: PCT/JP2011/077716
Authority: WO
Inventors: 矢作　宏一; 敏中村; 雅子末廣; 三沢　岳志; 友和中村
Original assignee: 富士フイルム株式会社
Priority date: 2011-01-13
Filing date: 2011-11-30
Publication date: 2012-07-19

Abstract

Provided is a parallax image display method whereby: a plurality of images captured from at least two different viewpoints is acquired; a parallax map is generated, said parallax map associating parallax values represented by the differences in the positions of corresponding regions between a first image and a second image captured from a different viewpoint than the first image, which are included in the plurality of acquired images, with the positions of the regions of the first image; the colors of the corresponding regions are determined according to the parallax value for each region; a parallax color image, obtained by changing the color of each region of the first image to the determined color, is created; and the parallax color image and the first image are synthesized.

Description

Parallax image display device and parallax image display method

The present invention relates to a parallax image display device and a parallax image display method, and in particular, parallax image display in which the stereoscopic effect of a generated stereoscopic image can be confirmed on a two-dimensional display monitor before generating the stereoscopic image. The present invention relates to a device and a parallax image display method.

Conventionally, when trying to grasp the stereoscopic effect of a 3D image on a 2D monitor, the parallax value is a parallax value of two or more images that are the basis of a stereoscopic image and are captured from two or more different viewpoints. A method of determining the stereoscopic effect of the generated stereoscopic image from the parallax value has been considered. However, the parallax value indicating the degree of parallax is merely a numerical value, and it is difficult to sensuously grasp the actual stereoscopic effect from the numerical value of the parallax value.

Therefore, a parallax image in which a parallax map in which the parallax value is associated with the pixel position of the first image of two or more images is generated, and the degree of the parallax value on the parallax map is expressed by a luminance difference or a color transition. In general, a technique for confirming the stereoscopic effect of an image for stereoscopic viewing by generating the image is known.

However, the parallax image in which the parallax value is expressed by a luminance difference or the like may not show the characteristics of the original image (left in FIG. 1) at all as shown on the right in FIG.

In the parallax image shown on the right in FIG. 1, the subject existing in the foreground in the left original image in FIG. 1 has the largest parallax value, and the subject with the largest parallax value is displayed in a dark color close to black. ing.

In the original image on the left in FIG. 1, the disparity value is smaller as the subject is farther, and in the right disparity image in FIG. 1, the subject is displayed such that the luminance is larger as the subject has a smaller disparity value.

As a result, in the case of FIG. 1, a subject far away to some extent exceeds the range in which the difference in parallax value can be expressed by a luminance difference, and the far subject can only be displayed in a parallax image with a white color having the maximum luminance.

In such a case as shown in FIG. 1, although the stereoscopic effect of the stereoscopic image generated from the original image can be roughly grasped from the parallax image, the dark color portion having the large parallax value or the parallax value in the parallax image is small. There was a problem that it was difficult to grasp how the whitish part corresponds to the original image.

Therefore, for example, in Japanese Patent Application Laid-Open No. 2003-047027, based on the created parallax map, the right image and the left image are segmented into a plurality of striped image pieces, and these striped striped image pieces are alternately arranged. A so-called lenticular image is created, and a stereoscopic image forming system, a stereoscopic image forming method, a program, and a storage medium according to which a stereoscopic effect is confirmed with the created lenticular image are disclosed.

Japanese Patent Laid-Open No. 2008-103820 discloses a stereoscopic image processing apparatus that generates a lenticular image for a moving image using parallax data between a right image and a left image, and confirms a stereoscopic effect using the generated lenticular image. The invention is disclosed.

However, the technique disclosed in Japanese Patent Application Laid-Open No. 2003-047027 is to check the effect of stereoscopic vision of a created lenticular image by means such as a lenticular lens. Hereinafter, there was a problem that the stereoscopic effect of the stereoscopic image could not be confirmed on the “2D monitor”.

In addition, a stereoscopic image processing apparatus disclosed in Japanese Patent Application Laid-Open No. 2008-103820 also uses a dedicated stereoscopic image reproduction apparatus (3D monitor) corresponding to a lenticular image in order to confirm the stereoscopic effect of a moving image converted into a lenticular image. ), There is a problem that the moving image must be reproduced.

The present invention has been made to solve the above problem, and a parallax image display device and a parallax image in which the stereoscopic effect of a stereoscopic image can be confirmed in advance on a 2D monitor before the stereoscopic image is created. It is an object to provide a display method and a parallax image display program.

To achieve the above object, a parallax image display device according to a first aspect includes an acquisition unit that acquires a plurality of images taken from two or more different viewpoints, and a plurality of images acquired by the acquisition unit. The disparity value represented by the difference in position for each corresponding region between the first image included and the second image different from the first image is made to correspond to the position of the region of the first image. A parallax map generating unit that generates a parallax map, a color determining unit that determines a color of a corresponding region according to a parallax value for each region, and a color determining unit that determines a color of each region of the first image A parallax image creation unit that creates a parallax color image that has been changed to the color, and an image synthesis unit that synthesizes the parallax color image and the first image.

According to the parallax image display device according to the first aspect, the parallax value indicating the stereoscopic effect of the stereoscopic image generated from a plurality of images taken from two or more different viewpoints can be recognized by color. A parallax color image is created, and the image synthesis unit synthesizes the parallax image and the first image, so that the pixels of the first image are mixed into the parallax color image by synthesis, and the parallax value is recognized by the luminance difference or the like In addition to being able to do so, an image is generated in which an outline of the subject can be grasped.

The image generated by the image composition unit does not require a special display device such as a display device having a lenticular lens, and is a bitmap image that can be displayed on a normal 2D monitor. The effect of the stereoscopic effect of the stereoscopic image created from the second image can be confirmed in advance on the 2D monitor before creating the stereoscopic image.

The parallax image display method according to the second aspect acquires a plurality of images taken from two or more different viewpoints, and the first image included in the acquired plurality of images is different from the first image. A disparity map in which a disparity value represented by a position difference for each corresponding region with an image is associated with a position of the region of the first image is generated, and a region corresponding to the disparity value for each region Is determined, a parallax color image in which the color of each region of the first image is changed to the determined color is generated, and the parallax color image and the first image are synthesized.

The parallax image display program according to the third aspect is an acquisition unit that acquires a plurality of images taken from two or more different viewpoints, and a first image included in the plurality of images acquired by the acquisition unit. And a second image different from the first image generate a parallax map in which a parallax value represented by a position difference for each corresponding region corresponds to a position of the region of the first image A parallax map generation unit, a color determination unit that determines a color of a corresponding region according to a parallax value for each region, and a parallax color image obtained by changing the color of each region of the first image to a color determined by the color determination unit Is a program for functioning as a parallax image creation unit that creates the image, and an image synthesis unit that synthesizes the parallax color image and the first image.

Note that the region includes not only one pixel but also a region in which several pixels are gathered in a square shape, such as 2 × 2 to 3 × 3.

As described above, according to the present invention, the parallax values between the first image and the second image are associated with each region of the first image, and the degree of the associated parallax value is expressed in color. By combining the first image with the parallax color image, the stereoscopic effect of the stereoscopic image created from the first image and the second image can be reduced to 2D before creating the stereoscopic image. The effect that it can confirm beforehand with a monitor is acquired.

It is the figure which compared the original image and the parallax image. 1 is a block diagram of a parallax image display device according to a first embodiment of the present invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of two or more images image | photographed from two or more different viewpoints. It is a figure which shows an example of the parallax image which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the image which synthesize | combined the 1st image and the parallax image in the 1st Embodiment of this invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the image which synthesize | combined the color difference component and parallax image of the 1st image in the 2nd Embodiment of this invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 3rd Embodiment of this invention. It is a figure which shows an example of the image which synthesize | combined the 1st image and the parallax color image in the 3rd Embodiment of this invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 4th Embodiment of this invention. It is a figure which shows an example of the image which synthesize | combined the parallax resolution image which changed the resolution of the 1st image according to the color difference component and parallax value of a 1st image in the 4th Embodiment of this invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 5th Embodiment of this invention. It is a crosspoint image which displayed the crosspoint of the 1st image in a 5th embodiment of the present invention. It is a crosspoint image which displayed the crosspoint of the 1st image in a 5th embodiment of the present invention. It is a flowchart which shows the process of the parallax image display apparatus which concerns on the 6th Embodiment of this invention. It is a parallax absolute value maximum image which displayed the area | region of the pixel with the largest parallax value in the pop-out direction in the 6th Embodiment of this invention. It is the parallax absolute value maximum image which displayed the area | region with the largest parallax value in the depth direction in the 6th Embodiment of this invention. 6 is an operation display unit of an apparatus capable of processing in the first to sixth embodiments of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 2 is a block diagram of the parallax image display device according to the first embodiment of the present invention.

As shown in FIG. 2, the parallax image display device 200 according to the first exemplary embodiment of the present invention is input with an image input unit 201 to which two or more images taken from two or more different viewpoints are input. Based on two or more images, the first image (left image in the first to seventh embodiments) and the second image (first to seventh embodiments) taken from different viewpoints of the left image. The right image), an image processing unit 202 for creating an image for confirming the stereoscopic effect of the stereoscopic image created from the image processing unit 202, an operation unit 203 for operating the parallax image display device 200, and image processing A display unit 204 that displays a processing result of the unit 202; and a storage unit 205 that stores an input image or the like.

The image input unit 201 receives two or more images taken from two or more different viewpoints taken by a camera or the like that takes a stereoscopic image. The image input unit 201 may be directly connected to a camera that captures a stereoscopic image, but an information storage medium (magnetic disk, optical disk, magneto-optical disk, memory card, IC card) that stores the image captured by the camera. Etc.) may be a data reading device that reads and inputs an image captured by the camera.

The operation unit 203 is a device for a user or the like to input an instruction to the parallax image display device 200, and input devices such as a touch panel, a keyboard, a mouse, and a pen tablet are used.

The display device 204 is a device such as an LCD that displays a processing process or a processing result of the image processing unit 202, and is a 2D monitor in the present embodiment.

The storage unit 205 is a storage device such as a RAM (Random Access Memory), a HDD (Hard Disk Drive), or a flash memory.

Further, the image processing unit 202 calculates a parallax represented by a difference in position for each corresponding pixel between a left image included in two or more input images and a right image taken from a different viewpoint from the left image. A parallax map creation unit 2021 that creates a parallax map corresponding to the pixel position of the left image, a parallax image creation unit 2022 that creates a parallax image indicating the magnitude of the parallax value of the created parallax map in luminance, An image synthesis unit 2023 that synthesizes the created parallax image with the left image.

The image processing unit 202 may be a computer having a CPU and a memory. In this case, the image processing unit 202 operates according to a program for causing the computer stored in the storage unit 205 to function as an image processing apparatus.

The image obtained by combining the parallax image and the left image by the image combining unit 2023 is output to the display device 204 as the processing result of the image processing unit 202.

Further, the input image, the parallax map, the parallax image, the image synthesized by the image synthesis unit 2023, and the like are stored in the storage unit 205.

Next, processing of the parallax image display device according to the first embodiment of the present invention will be described with reference to FIG. Here, FIG. 3 is a flowchart showing processing of the parallax image display device according to the first embodiment of the present invention.

First, it is determined in step 301 whether or not two or more images taken from two or more different viewpoints are input to the image input unit.

Here, FIG. 4 is an example of two or more images captured from two or more different viewpoints input to the image input unit, and a right image (right in FIG. 2) and a left image captured from the right viewpoint. It is a figure which shows the left image (left of FIG. 2) image | photographed from the viewpoint. In step 302, a parallax map is created from two or more images photographed from two or more different viewpoints shown in FIG. In the left image and the right image in FIG. 4, it is assumed that the parallax occurs in the horizontal direction (horizontal direction) of the image and no shift occurs in the vertical direction (vertical direction).

In step 302, the parallax map creation unit 2021 shown in FIG. 2 performs stereo matching on the left image and the right image to generate a parallax map.

Stereo matching uses a set of two images taken from different viewpoints, specifies which region of the left image corresponds to which region of the right image, and the three-dimensional position of each region It is a method to guess.

Specifically, as shown in FIG. 4, for example, with the left image as a reference, corresponding pixels (x2, y2) on the right image corresponding to the pixels (x1, y1) on the left image are extracted. The parallax d between the pixel (x1, y1) on the left image and the corresponding pixel (x2, y2) on the right image can be calculated as d = x2-x1, and the pixel position of the left image with this parallax d as a reference Store in (x1, y1).

The parallax value stored at the pixel position of the left image with the parallax d as a reference is taken as a parallax map. In the present embodiment, the parallax value is calculated for each pixel of the image. However, the parallax is calculated for a region in which several pixels are gathered in a square shape, such as 2 × 2 to 3 × 3. A value may be calculated, and when the parallax value is calculated for a region where pixels are gathered in this way, it is considered that the processing speed can be improved.

This parallax map shows the parallax value for each pixel, but since it is just numerical data, what kind of stereoscopic effect the parallax value can produce in an actual stereoscopic image even when read by a user or the like? I can't recognize that.

Therefore, in Step 303, the parallax image creation unit 2022 shown in FIG. 2 converts the parallax value for each pixel of the parallax map into a luminance value so that the user or the like can recognize the stereoscopic effect of the stereoscopic image, A parallax image, which is an image representing the parallax value corresponding to each pixel of the image in luminance, is created.

Various methods for converting the parallax value into the luminance value are conceivable. However, it is common to convert the parallax value according to a correspondence table in which a one-to-one correspondence between the parallax value and the luminance value is described.

In step 304, it is determined whether or not the conversion process from the parallax value to the luminance value has been performed for all the pixels. If all the pixels have been converted, the parallax image file created by the conversion in step 305 is determined. Is stored in the storage unit 205 shown in FIG.

FIG. 5 is a diagram showing an example of a parallax image according to the first embodiment of the present invention. In FIG. 5, a subject with a large parallax value is represented with low luminance, and a subject with a small parallax value is represented with high luminance.

In FIG. 5, all the subjects on the image are expressed in monochrome shades. However, the outline of each subject is unclear, and portions with the same parallax value are expressed with the same brightness, so that details of the subject such as trees in a distant view cannot be grasped at all.

It is also conceivable to grasp the correspondence between the original image and the parallax image by alternately comparing FIG. 5 with the left image of FIG. 4 which is the original image. However, comparing the two images is cumbersome, and after comparing the images to understand the correspondence with the original image, the stereoscopic effect of the stereoscopic image that is finally generated is changed to the subject of the original image. It is difficult to recognize it in contrast to

Therefore, in step 306, the image composition unit 2023 shown in FIG. 2 synthesizes the left image and the parallax image in FIG. 4 for each pixel.

In this step 306, the image composition unit 2023 performs weighting of w1: w2, for example, 1: 9, on the RGB values of the respective image pixels between the left image and the parallax image, and synthesizes both images.

In step 306, the left image and the parallax image are synthesized by making the weight w2 for the parallax image heavier than the weight w1 for the left image and calculating the weighted average value according to the following equation (1). . a is the pixel value of the left image, and b is the luminance value of the pixel of the parallax image corresponding to the pixel of the pixel value a. This weighted average value is obtained for all the pixels of the parallax image and the left image.
(A · w1 + b · w2) / (w1 + w2) (1)
However, it is preferable that w1 <w2, and w1 = 1 and w2 = 9.

In the case of a color image, each pixel of the left image has RGB values aR, aG, and aB, respectively. Therefore, a weighted average is given for each of the RGB values of each pixel as in the following equation (2). Ask for.
(AR · w1 + b · w2) / (w1 + w2)
(AG · w1 + b · w2) / (w1 + w2) (2)
(AB · w1 + b · w2) / (w1 + w2)

In the present embodiment, the weighting is performed on the RGB value of each pixel of the left image. However, when the parallax value of the parallax image is calculated for a region where several pixels are gathered in a square shape, such as 2 × 2 to 3 × 3, the weighting of the RGB values of the left image is also the parallax image The RGB values are weighted for the area corresponding to this area.

When weighting RGB values for a region where several pixels are gathered in a square shape such as 2 × 2 to 3 × 3, the average value of the RGB values of each pixel in the region is calculated and calculated. The average value of the obtained RGB values is weighted by the above equation (2).

In the subsequent step 307, the image composition unit 2023 determines whether or not the composition processing for all pixels has been completed. If the composition processing for all pixels has been completed, in step 308, the left image and the parallax image are obtained. The synthesized image is displayed on the display device 204 shown in FIG. 2, and the processing in steps 301 to 308 is completed.

FIG. 6 shows an example of an image obtained by synthesizing the left image and the parallax image in the first embodiment of the present invention. In the image of FIG. 6, in addition to recognizing the parallax value by the luminance difference, the pixels of the left image are mixed by synthesis, and the color difference component of the left image is also displayed lightly. Therefore, an outline of a subject such as a tree in a distant view can be grasped from the image of FIG.

Also, the image obtained by combining the left image and the parallax image is a bitmap image that can be displayed on a normal 2D monitor, without requiring a special display device such as a display device having a lenticular lens.

The user or the like grasps the stereoscopic effect of the finally obtained stereoscopic image by visually recognizing an image obtained by combining the left image and the parallax image displayed on the screen of the 2D monitor.

As described above, according to the first embodiment of the present invention, by creating an image obtained by synthesizing the parallax image obtained by converting the parallax value for each pixel of the left image into luminance and the left image, Even in the 2D monitor, the stereoscopic effect of the stereoscopic image generated from the captured image can be grasped.

[Modification 1 of the first embodiment]
In the first embodiment, in the synthesis of the left image and the parallax image in step 306 in FIG. 3, the weight for the RGB value of the pixel of the left image is w1, the weight for the luminance value of the parallax image is w2, : W2 = 1: 9 is applied to the RGB value and the luminance value of the pixels of each image to synthesize both images.

The weights in the synthesis of the left image and the parallax image are changed according to the parallax value in the first modification and the second modification of the first embodiment described below.

Among these, in the first modification of the first embodiment, the weight w1 of the left image is changed to the weight w2 of the parallax image in the pixel related to the subject near the cross point where the optical axes of the left and right lenses of the stereoscopic camera intersect. Make it heavier.

Furthermore, the pixel related to the subject in front of or behind the cross point increases the weight w2 of the parallax image and decreases the weight w1 of the left image as the distance from the cross point increases.

The absolute value of the parallax value is the smallest “0” in the pixel of the subject related to the cross point, and the absolute value of the parallax value increases as the distance from the cross point increases in the pixel of the subject far from the cross point.

For this reason, in the first modification of the first embodiment, as the absolute value of the parallax value increases, the weight w1 for the value obtained from the RGB values of the pixels included in the left image area corresponding to the parallax image area , And the weight w2 for the value obtained from the luminance value of the pixel included in the parallax image area is increased.

Various methods for changing the weight according to the parallax value are conceivable. When the parallax value is d, the ratio of the weight w1 of the left image and the weight w2 of the parallax image uses the following formula (3). Here, m is a predetermined coefficient statistically determined through an experiment, and is a positive real number. A and B are positive real numbers that have a relationship of A ≧ B and A + B = 10 and can take values of 0 to 10, respectively.
Left image weight w1: Parallax image weight w2 = A−m | d |: B + m | d | (3)

As described above, since the parallax value of the pixel related to the subject existing at the cross point is “0” and the absolute value of the parallax value is the minimum, using the above formula (3), the subject existing at the cross point The weight w1 to the RGB value for the pixel of the left image at “A” is “A”, and the weight w2 to the luminance value for the pixel of the parallax image at the subject existing at the cross point is “B”.

As described above, A and B are positive real numbers that have a relationship of A ≧ B and A + B = 10 and can take values of 0 to 10, respectively. Therefore, as an example, if A = 9 and B = 1, w1 = 9 and w2 = 1 for the subject present at the cross point, and a weight of 9: 1 is assigned to each image pixel between the left image and the parallax image. The two values are synthesized by being applied to the RGB value and the luminance value.

Also, as an example, if A = 10 and B = 0, the subject existing at the cross point is w1 = 10 and w2 = 0, and only the pixels of the left image are used for image synthesis, and the luminance value of the parallax image is This is not used for image synthesis.

The absolute value of the parallax value of the parallax value in the pixel related to the subject existing at a position away from the cross point increases as the distance from the cross point increases. Therefore, according to Expression (3), the weight w1 of the left image is relatively light, and the weight w2 of the parallax image is relatively heavy.

Further, a correspondence table in which disparity values are associated with weights may be prepared in advance, and the weights may be changed according to the disparity values according to the correspondence table (lookup table), regardless of the above equation (3).

According to the first modification of the first embodiment, in the subject near the cross point, the weight w1 of the pixel of the left image, which is the original image, is relatively heavier than the weight w2 of the pixel of the parallax image. Since the left image and the parallax image are combined, the state of the subject near the cross point in the original image can be confirmed. In addition, the farther away from the cross point the subject located at the position away from the cross point, the higher the pixel weight w2 of the parallax image compared to the left image pixel weight w1, which is the original image, and the left. Since the image and the parallax image are combined, it is easy to grasp the stereoscopic effect of the stereoscopic image generated from the captured image.

[Modification 2 of the first embodiment]
In the second modification of the first embodiment, in the pixel related to the subject close to the camera, the weight w1 of the left image is heavier than the pixel weight w2 of the parallax image, and the pixel related to the subject farther from the camera is parallax. The weight w2 of the image is increased and the weight w1 of the left image is decreased.

The parallax value is the maximum at the pixel of the subject closest to the camera, and the parallax value becomes smaller at the pixel of the subject far from the camera as the distance from the camera is longer.

For this reason, in the second modification of the first embodiment, as the parallax value becomes smaller, the weight w1 for the value obtained from the RGB value of the pixel included in the region of the left image corresponding to the region of the parallax image is reduced. At the same time, the weight w2 for the value obtained from the luminance value of the pixel included in the parallax image area is increased.

There are various methods for changing the weight according to the parallax value. When the parallax value is d, the ratio of the weight w1 of the left image and the weight w2 of the parallax image uses the following formula (4). Here, p is a predetermined coefficient that is statistically determined through experiments, and is a positive real number. dn is the parallax value of the pixel of the subject closest to the camera.

A and B are positive real numbers that have a relationship of A ≧ B and A + B = 10 and can take values of 0 to 10, respectively.
Left image weight w1: parallax image weight w2
= A−p | d−dn |: B + p | d−dn | (4)

Since the parallax value of the subject closest to the camera is dn, | d−dn | in Expression (4) is “0”. Therefore, the weight w1 to the RGB value for the pixel of the left image in the subject closest to the camera is “A”, and the weight w2 to the luminance value for the pixel of the parallax image in the subject closest to the camera is “B”.

As described above, A and B are positive real numbers that can take values of 0 to 10 each having a relationship of A ≧ B and A + B = 10. Therefore, as an example, when A = 9 and B = 1, w1 = 9 and w2 = 1 for the subject closest to the camera, and a 9: 1 weighting is applied to the RGB of each image pixel between the left image and the parallax image. The two images are combined by applying the value and the luminance value.

As an example, if A = 10 and B = 0, w1 = 10 and w2 = 0 for the subject closest to the camera, and only the pixels of the left image are used for image synthesis, and the luminance value of the parallax image is It will not be used for image composition.

In a subject that is located away from the camera, | d−dn | in the above equation (4) increases, so the weight w1 of the left image is relatively light, and the weight w2 of the parallax image is relatively small. It gets heavier.

Further, a correspondence table (lookup table) in which disparity values and weights are associated with each other may be prepared in advance, and the weights may be changed according to the disparity values according to the correspondence table, regardless of the above equation (4).

According to the second modification of the first embodiment, in the subject closest to the camera, the pixel weight w1 of the left image, which is the original image, is relatively heavier than the pixel weight w2 of the parallax image. Thus, since the left image and the parallax image are combined, the state of the original image of the subject closest to the camera can be confirmed. In addition, as the subject that is located farther from the camera is farther away from the camera, the pixel weight w2 of the parallax image is relatively heavier than the pixel weight w1 of the left image that is the original image. Since the parallax image is synthesized, it is easy to grasp the stereoscopic effect of the stereoscopic image generated from the captured image.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The parallax image display device according to the second embodiment of the present invention is different from the parallax image display device according to the first embodiment of the present invention shown in FIG. The configuration is the same.

Next, processing of the parallax image display device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 7 is a flowchart showing processing of the parallax image display device according to the second embodiment of the present invention.

Of these, the processing from step 701 to 705 is the same as the processing from step 301 to 305 in FIG. 3 in the first embodiment of the present invention, and thus the description thereof is omitted.

In step 706, which is a process after the parallax image is created, the image synthesis unit 2023 synthesizes the color difference component of the left image with the parallax image. In the first embodiment of the present invention, the left image and the parallax image are weighted 1: 9 to the RGB values of the pixels of both images, and the two images are combined. In this embodiment, only the color difference component is extracted from the left image, and the extracted color difference component is combined with the parallax image.

The color difference in the present embodiment is RY and BY obtained by subtracting the luminance value Y from the R and B pixel values among the RGB pixel values originally including the luminance value Y.

For example, in the YCrCb system, the relationship between the color differences RY and BY and the luminance value Y is expressed by the following equation (5), where the color difference BY is Cb and the color difference RY is Cr.
Y = 0.29900R + 0.58700G + 0.11400B
Cr = 0.50,000R-0.48869G-0.81131B (5)
Cb = −0.16874R−0.33126G + 0.50000B

From the above equation (5), the following equation (6) relating to conversion from YCrCb to RGB is calculated.
R = Y + 1.40200Cr
G = Y−0.34414Cb−0.71414Cr (6)
B = Y + 1.77200 Cb

On the other hand, the parallax image is a monochrome image expressed only by luminance values. A pixel in the parallax image has a luminance value Y ′, and the color difference between the pixels of the left image corresponding to the pixel of the parallax image having the luminance value Y ′ is Cr and Cb in Equation (6). .

In step 706, the image composition unit 2023 substitutes Y ′, which is the luminance value of the parallax image, for the luminance value Y of the above equation (6) for each pixel, thereby extracting the color difference component and the parallax image extracted from the left image. Is synthesized.

In the subsequent step 707, the image composition unit 2023 determines whether or not the composition processing has been completed for all pixels. If the composition processing has been completed for all pixels, in step 708, the color difference component and the parallax of the left image are determined. The image combined with the image is displayed on the display device 204 shown in FIG. 2, and the processing in steps 701 to 708 is completed.

FIG. 8 shows an example of an image obtained by synthesizing the color difference component of the left image and the parallax image in the second embodiment of the present invention. In addition to recognizing the parallax value from the difference in luminance of each pixel, the color difference component of the left image is displayed more clearly than in the case of the first embodiment, so that the difference in color of each subject can be recognized, As a result, the outline of each subject can be grasped.

In this embodiment, the color difference is extracted for the RGB value of each pixel of the left image. However, the number of pixels of the parallax image is 2 × 2 to 3 × 3. In the case where the calculation is performed for a region gathered in a shape, the color difference extracted from the left image is also extracted for a region corresponding to the region of the parallax image.

When extracting a color difference for a region where several pixels are gathered in a square shape such as 2 × 2 to 3 × 3, an average value of RGB values of each pixel in the region is calculated, and the calculated RGB Color difference is extracted from the average value.

The image obtained by synthesizing the color difference component and the parallax image of the left image does not require a special display device such as a display device including a lenticular lens, and is a bitmap image that can be displayed on a normal 2D monitor.

The user or the like grasps the stereoscopic effect of the finally obtained stereoscopic image by visually recognizing an image obtained by synthesizing the color difference component of the left image displayed on the screen of the 2D monitor and the parallax image.

As described above, according to the second embodiment of the present invention, by creating an image obtained by combining the parallax image obtained by converting the parallax value for each pixel of the left image into luminance and the left image color difference component. Even in a normal 2D monitor, the stereoscopic effect of the stereoscopic image generated from the captured image can be grasped.

[Third Embodiment]
Subsequently, a third embodiment of the present invention will be described. The parallax image display device according to the third embodiment of the present invention differs from the parallax image display device according to the first embodiment of the present invention shown in FIG. Other configurations are the same.

Next, processing of the parallax image display device according to the third embodiment of the present invention will be described with reference to FIG. FIG. 9 is a flowchart showing processing of the parallax image display device according to the third embodiment of the present invention.

First, it is determined in step 901 whether or not two or more images taken from two or more different viewpoints are input to the image input unit. Similar to the first embodiment, the input images are two or more images taken from two or more different viewpoints shown in FIG.

Subsequently, in step 902, the parallax map creation unit 2021 shown in FIG. 2 creates a parallax map from the left image and the right image by stereo matching, as in the first embodiment. In the left image and the right image in FIG. 4, it is assumed that the parallax occurs in the horizontal direction (horizontal direction) of the image and no shift occurs in the vertical direction (vertical direction).

In step 903, the parallax value corresponding to each pixel of the left image, for example, as the left image so that the user or the like can recognize the stereoscopic effect of the stereoscopic image from the parallax map by the parallax image creation unit 2022 shown in FIG. Is represented by a color, for example, a color difference value, and an image in which the color difference value for each pixel is shown at the position of the corresponding pixel on the left image is generated, and this is used as a parallax color image.

In addition to the parallax image creation unit 2022, a separate color determination unit (not shown) that determines the color of the corresponding left image region according to the parallax value for each region of the parallax map is provided, and the parallax image creation unit 2022 is provided. May create a parallax color image in which the resolution of each region of the left image is changed to the color determined by the color determination unit.

There are various methods for converting the parallax value into the color difference value. However, the conversion is performed according to the correspondence table describing the correspondence between the parallax value and the color difference value. In this embodiment, the color difference changes from red to blue as the parallax value is small when the parallax value is large, and the correspondence table associates the large parallax value with the red color difference, A smaller parallax value corresponds to a color difference in which red to blue are emphasized.

In this case, the disparity value is classified for each determined range, and a correspondence table in which the color is associated with the disparity value of the classified range is provided, and the color corresponding to the disparity value of the region of the left image is supported. You may make it determine based on a table | surface.

In step 904, the parallax image creation unit 2022 determines whether or not the conversion process from the parallax value to the color difference value has been performed for all the pixels. If all the pixels have been converted, the conversion is performed in step 905. The parallax color image file created by the above is stored in the storage unit 205 shown in FIG.

Even in a parallax color image in which the parallax value is expressed by a color difference, there are cases where the characteristics of the left image can be observed to some extent. However, in order to grasp the details of a subject such as a grove of a distant view, it is desirable to synthesize a left image and a parallax color image in which parallax is expressed by a color difference, as in the first embodiment.

In step 906, as in the first embodiment, the image composition unit 2023 shown in FIG. 2 applies 1: 9 weighting to the RGB values for each pixel of the left image and the parallax image. The two images are combined, but the weight ratio may be other than 1: 9.

In the present embodiment, the above-described weighting is performed on the RGB value of each pixel of the left image. However, some pixels have a parallax value of 2 × 2 to 3 × 3 of the parallax image. When the calculation is performed for a region gathered in a square shape, the RGB value is weighted for the region corresponding to the region of the parallax image.

In the subsequent step 907, the image composition unit 2023 determines whether or not the composition processing has been completed for all pixels. If the composition processing has been completed for all pixels, in step 908, the left image and the parallax color image are determined. 2 is displayed on the display device 204 shown in FIG. 2, and the processing of steps 901 to 908 is completed.

FIG. 10 shows an example of an image obtained by synthesizing the left image and the parallax color image in the third embodiment of the present invention. In the actual image, the image is indicated by the front dot that is the region having the largest parallax value. The part with the small parallax value is represented in blue.

Further, in addition to recognizing the parallax value by the color difference, the pixels of the left image are mixed by synthesis and the pixels of the left image are also displayed, so that an outline of a subject such as a grove of a distant view can be grasped.

Also, the image obtained by combining the left image and the parallax color image is a bitmap image that can be displayed on a normal 2D monitor, without requiring a special display device such as a display device having a lenticular lens.

The user or the like grasps the stereoscopic effect of the finally obtained stereoscopic image by visually recognizing an image obtained by synthesizing the left image displayed on the screen of the 2D monitor and the parallax color image.

As described above, according to the third embodiment of the present invention, by creating an image obtained by combining the parallax color image obtained by converting the parallax value for each pixel of the left image into a color difference and the left image, Even in a normal 2D monitor, the stereoscopic effect of the stereoscopic image generated from the captured image can be grasped.

In the third embodiment, the following various modifications are possible.

First, the image composition unit 2023 in FIG. 2 extracts the color difference components of the pixels included in each region of the left image, and the color difference between the region of the left image corresponding to each region of the parallax color image and each region of the parallax color image. The ingredients may be synthesized.

Also, the image composition unit 2023 in FIG. 2 performs pixel conversion of the pixels of the left image that is the original image according to the parallax value, as in Modification 1 of the first embodiment or Modification 2 of the first embodiment. You may make it change the weight w1 and the weight w2 of the pixel of a parallax color image.

Also, a parallax luminance image in which the parallax value of the parallax map area is expressed by luminance may be created, and the image synthesis unit 2023 in FIG. 2 may synthesize the parallax luminance image and the parallax color image.

Also, a resolution determining unit that determines the resolution of the corresponding region of the left image according to the parallax value for each region of the parallax map, and the parallax resolution image in which the resolution of each region of the left image is changed to the resolution determined by the resolution determining unit And a parallax resolution image creating unit that creates a parallax resolution image, and the image synthesis unit 2023 in FIG. 2 may synthesize the parallax resolution image and the parallax color image.

The resolution determination unit may reduce the resolution of the region of the left image as the parallax value corresponding to the region in the parallax map becomes smaller, or the absolute value of the parallax value corresponding to the region in the parallax map The resolution of the area of the left image may be lowered as the value increases.

In addition, the resolution determination unit classifies the parallax value for each determined range, has a correspondence table in which the resolution is associated with the parallax value of the classified range, and corresponds to the position of the region of the left image in the parallax map The disparity value being used may be the disparity value of the region of the left image, and the resolution corresponding to the disparity value of the region of the left image may be determined based on the correspondence table.

Although various descriptions can be considered in the correspondence table, a range in which the included parallax value is small may correspond to a lower resolution as the included parallax value is small, or a range in which the absolute value of the included parallax value is large Alternatively, a lower resolution may be associated with an increase in the absolute value of the included parallax value.

Furthermore, the resolution determination unit may be a sharpness determination unit that determines the sharpness of the corresponding left image region in accordance with the parallax value for each region of the parallax map.

In addition, the disparity value in the disparity map area is converted into a luminance value, the position of the disparity map in which the absolute value of the disparity value is within a threshold is specified, and the specified area is colored with a predetermined color. A cross point image creating unit that creates a point image may be further included, and the image composition unit 2023 in FIG. 2 may compose the cross point image and the parallax color image.

Furthermore, the parallax value of the area where the absolute value of the parallax value is maximum is determined by converting the parallax value of the area of the parallax map into a luminance value, specifying the position of the area where the absolute value of the parallax value is maximum in the parallax map. 2 further includes a parallax absolute value maximum image creating unit that creates a parallax absolute value maximum image in which the region is colored with different colors depending on whether the area is positive or negative. The image composition unit 2023 in FIG. The maximum image and the parallax color image may be combined.

[Fourth Embodiment]
Subsequently, a fourth embodiment of the present invention will be described. The parallax image display device according to the fourth embodiment of the present invention is different from the parallax image display device according to the first embodiment of the present invention shown in FIG. Although the functions are different, other configurations are the same.

Next, processing of the parallax image display device according to the fourth embodiment of the present invention will be described with reference to FIG. FIG. 11 is a flowchart showing processing of the parallax image display device according to the fourth embodiment of the present invention.

First, it is determined in step 1101 whether a right image and a left image taken from two or more different viewpoints are input to the image input unit. Similar to the first embodiment, the input images are two or more images taken from two or more different viewpoints shown in FIG.

Subsequently, in step 1102, the parallax map creation unit 2021 shown in FIG. 2 creates a parallax map from the left image and the right image by stereo matching, as in the first embodiment. In the left image and the right image in FIG. 4, it is assumed that the parallax occurs in the horizontal direction (horizontal direction) of the image and no shift occurs in the vertical direction (vertical direction).

In step 1103, the parallax image creation unit 2022 shown in FIG. 2 converts the resolution of the left image for each pixel according to the parallax value of the parallax map so that the user or the like can recognize the stereoscopic effect of the stereoscopic image. . Specifically, an image having a large parallax value, that is, a high resolution of a pixel related to a subject close to the camera and a low parallax value, that is, a low resolution of a pixel related to a subject far from the camera is generated. A resolution image is used.

In addition to the parallax image creation unit 2022, a resolution determination unit (not shown) that determines the resolution of the corresponding region of the left image according to the parallax value for each region of the parallax map is separately provided, and the parallax image creation unit 2022 Alternatively, a parallax resolution image in which the resolution of each region of the left image is changed to the resolution determined by the resolution determination unit may be created.

As a method of changing the resolution, specifically, the image is divided into rectangular blocks such as 3 × 3, 5 × 5, and 7 × 7, and the pixel values in each block are changed to the pixel values in the block. Mosaic filling with average values is used. In a region where the parallax value for which the resolution is desired to be increased is large, the square block is reduced, and in a region where the parallax value is desired to be reduced for the resolution, the square block is increased.

When mosaicking pixels according to the parallax value, start mosaicing from the area where you want to increase the resolution, that is, the area where the blocks to be mosaicked are small, and then sequentially mosaic the areas where the blocks to be mosaicked are large Preferably it is done. This is because if the mosaic is started from an area where the block to be mosaicked is large, the pixels are suddenly smoothed in a wide area.

The correspondence between the parallax value and the size of the block to be mosaicked is based on, for example, a correspondence table in which the two are matched.

In addition, by increasing the resolution of an area centered on a pixel having a small absolute parallax value and decreasing the resolution of an area centering on a pixel having a large absolute parallax value, pixels near the crosspoint are clarified. Can also be displayed.

In step 1104, the parallax image creation unit 2022 determines whether or not processing has been performed on all pixels. If processing has been performed on all pixels, in step 1105, the resolution of the left image is changed according to the parallax value. The parallax resolution image file is stored in the storage unit 205 shown in FIG.

Even in a parallax resolution image in which the parallax value is expressed by a difference in resolution, the characteristics of the left image may be observed to some extent. However, in order to grasp the details of the subject such as leaves, it is desirable to synthesize the color difference component of the left image and the parallax resolution image expressing the parallax value with the difference in resolution, as in the second embodiment. .

In step 1106, which is a step after the parallax resolution image is created, the image synthesis unit 2023 synthesizes the color difference component of the left image with the parallax resolution image for each pixel.

In this embodiment, the color difference is extracted for the RGB value of each pixel of the left image. However, some pixels have a parallax value of 2 × 2 to 3 × 3. In the case where the calculation is performed for a region gathered in a shape, the color difference extracted from the left image is also extracted for a region corresponding to the region of the parallax image.

In step 1106, the color difference component of the left image is synthesized with the parallax resolution image. However, as in step 306 described in the first embodiment, the first modification, or the second modification, the pixel of the left image and the parallax resolution image. These pixels may be combined.

In the subsequent step 1107, the image composition unit 2023 determines whether or not the composition process for all pixels has been completed. If the composition process for all pixels has been completed, in step 1108, the image composition unit 2023 An image obtained by synthesizing the color difference component and the parallax resolution image is displayed on the display device 204 shown in FIG. 2, and the processing of steps 1101 to 1108 is completed.

FIG. 12 shows an example of an image obtained by combining the color difference component of the left image and the parallax resolution image obtained by changing the resolution of the left image according to the parallax value in the fourth embodiment of the present invention. In addition to recognizing the parallax value due to the difference in resolution, the color difference component of the original left image is also displayed, so the difference in color of each subject can be recognized, and this allows the outline of each subject to be grasped. ing.

An image obtained by combining the color difference component of the original left image and the parallax resolution image does not require a special display device such as a display device having a lenticular lens, and can be displayed on a normal 2D monitor. It is a bitmap image.

The user or the like grasps the stereoscopic effect of the finally obtained stereoscopic image by visually recognizing an image obtained by combining the color difference component of the original left image displayed on the screen of the 2D monitor and the parallax resolution image. To do.

In the fourth embodiment of the present invention, the image resolution is changed, but the sharpness of the image may be changed. In this case, in addition to the parallax image creation unit 2022, a separate sharpness determination unit (not shown) that determines the sharpness of the corresponding left image region according to the parallax value for each region of the parallax map is provided. 2022 may create a parallax sharpness image in which the resolution of each region of the left image is changed to the sharpness determined by the sharpness determination unit.

For example, Gaussian blur that smoothes an image using a Gaussian function is used to change the sharpness of the image.

In a region where the parallax value for which sharpness is to be increased is large, the region is the center pixel, and the range of the surrounding pixels when calculating the weighted average of the pixel values of the center pixel and the surrounding pixels by a Gaussian function is used. Make it smaller.

On the other hand, in a region with a small parallax value for which sharpness is desired to be reduced, the region is the central pixel, and the surrounding pixels when calculating the weighted average of the pixel values of the central pixel and the pixels around the central pixel by a Gaussian function Increase the range.

It is also possible to clearly display pixels in the vicinity of the cross point by increasing the sharpness of the pixel area having a small absolute value of the parallax value and decreasing the sharpness of the pixel area having the large absolute value of the parallax value. .

As described above, according to the fourth embodiment of the present invention, an image in which a parallax resolution image obtained by converting a parallax value for each pixel of a left image into a difference in resolution and a left image color difference component is created. By doing so, it is possible to grasp the stereoscopic effect of the stereoscopic image generated from the left image even with a normal 2D monitor.

[Modification of Fourth Embodiment]
In the fourth embodiment, in step 1103 of FIG. 11, the parallax image creation unit 2022 shown in FIG. 2 has the resolution of the left image for each pixel so that the user can recognize the stereoscopic effect of the stereoscopic image. Was converted according to the parallax value of the parallax map. Specifically, an image having a large parallax value, that is, a high resolution of a pixel related to a subject close to the camera and a low parallax value, that is, a low resolution of a pixel related to a subject far from the camera is generated. A resolution image was obtained.

In the modification of the fourth embodiment, the parallax image creation unit 2022 in FIG. 2 classifies the parallax values for each determined range, and has a correspondence table in which resolutions corresponding to the classified ranges are set in advance. .

Also, the parallax image creation unit 2022 identifies the parallax value corresponding to the pixel of the left image from the parallax map, identifies the resolution corresponding to the identified parallax value with reference to the correspondence table, and determines the pixel of the left image Is changed to the specified resolution.

As a method for changing the resolution, specifically, as in the fourth embodiment, the image is divided into rectangular blocks of 3 × 3, 5 × 5, 7 × 7, etc. Mosaicing is used in which the pixel values are filled with the average value of the pixel values in the block. In a region where the parallax value for which the resolution is desired to be increased is large, the square block is reduced, and in a region where the parallax value is desired to be reduced for the resolution, the square block is increased.

However, in the modification of the fourth embodiment, for example, the pixel of the left image whose parallax value is 0 to 2 has a 5 × 5 mosaic size and the parallax value is 3 to 5 The pixel of the left image has a resolution of the pixel of the left image corresponding to the range to which the parallax value of the pixel belongs, such as the size of the block to be mosaicized is 3 × 3. Apply and change the size.

Note that when mosaicking pixels according to a certain range of parallax values, start mosaicing from the area where you want to increase the resolution, i.e., the area where the block to be mosaicked is small, and the area where the block to be mosaicated is sequentially large It is preferable to perform mosaic. This is because if the mosaic is started from an area where the block to be mosaicked is large, the pixels are suddenly smoothed in a wide area.

Correspondence between a certain range of parallax values and the size of the block to be mosaicked according to the resolution is based on the correspondence table associating both as described above.

In this correspondence table, among the parallax value ranges, the larger the parallax value included in this range, the smaller the size of the block to be mosaicked, and the corresponding parallax value range. The smaller the included parallax value, the higher the resolution of the pixels related to the subject close to the camera, and the higher the resolution of the pixels related to the subject far from the camera. It is possible to generate and display an image that is intermittently lowered according to the distance from the image.

In the above correspondence table, among the parallax value ranges, the smaller the absolute value of the parallax values included in this range, the smaller the size of the block to be mosaicked, and the parallax value range. Among the ranges, the larger the absolute value of the disparity value included in this range, the larger the size of the block to be mosaicked is associated, thereby clearly displaying the pixels near the crosspoint, and the crosspoint It is possible to generate and display an image in which the resolution of the subject away from the screen is intermittently lowered according to the distance from the cross point.

Also, in the modification of the fourth embodiment, it is assumed that the parallax value corresponding to each pixel of the left image corresponds to the parallax map, that is, the case where the parallax value of the parallax image is calculated for each pixel. However, the parallax value of the parallax image may be calculated for a region where several pixels are gathered in a square shape, such as 2 × 2 or 3 × 3.

Even when the parallax value is calculated for a region where several pixels are gathered in a square shape such as 2 × 2 to 3 × 3, the parallax image creation unit 2022 in FIG. 2 can determine the parallax value. And a correspondence table in which resolutions corresponding to the classified ranges are set in advance.

Also, the parallax image creation unit 2022 identifies the parallax value corresponding to the area of the left image from the parallax map, identifies the resolution corresponding to the identified parallax value by referring to the correspondence table, and determines the area of the left image Is changed to the specified resolution.

The processing after step 1103 in FIG. 11 is the same as that of the fourth embodiment in the modification of the fourth embodiment, and the resolution is changed according to the parallax value as described in step 1106 in FIG. The color difference component or pixel of the left image may be combined with the processed image.

In the modification of the fourth embodiment, the parallax image creation unit 2022 in FIG. 2 classifies the parallax values for each determined range, and presets sharpness corresponding to the parallax values in the classified range. The disparity value corresponding to the left image area is identified from the disparity map, the sharpness corresponding to the identified disparity value is identified with reference to the correspondence table, and the sharpness of the left image area is determined. May be changed to the specified sharpness.

In this correspondence table, the higher the parallax value included in this range, the higher the sharpness, and the smaller the parallax value included in this range of the parallax value range. The lower the sharpness of the range, the higher the sharpness of the pixels related to the subject closer to the camera, and the lower the sharpness of the pixels related to the subject far from the camera is generated and displayed according to the distance from the camera. can do.

In the above correspondence table, among the parallax value ranges, the smaller the absolute value of the parallax values included in this range, the higher the sharpness is associated, and the parallax value ranges included in this range. The higher the absolute value of the parallax value, the lower the sharpness, so that the pixels near the crosspoint are clearly displayed, and the sharpness of the subject away from the crosspoint is intermittent according to the distance from the crosspoint. It is possible to generate and display an image with a low height.

As in the fourth embodiment, for example, Gaussian blur that smoothes an image using a Gaussian function is used to change the sharpness of the image.

In an area where sharpness is desired to be increased, the area is set as the central pixel, and the range of the surrounding pixels when the weighted average of the pixel values of the central pixel and the surrounding pixels is calculated by a Gaussian function is reduced.

On the other hand, in an area where sharpness is to be lowered, the area is set as the central pixel, and the range of the surrounding pixels when calculating the weighted average of the pixel values of the central pixel and the surrounding pixels by a Gaussian function is increased. To do.

According to the modification of the fourth embodiment, the stereoscopic effect of the stereoscopic image generated from the left image is changed by stepwise changing the resolution of the image for each predetermined range of parallax values. It becomes easy to grasp.

[Fifth Embodiment]
Subsequently, a fifth embodiment of the present invention will be described. The parallax image display device according to the fifth embodiment of the present invention differs from the parallax image display device according to the first embodiment of the present invention shown in FIG. Other configurations are the same.

Next, processing of the parallax image display device according to the fifth embodiment of the present invention will be described with reference to FIG. FIG. 13 is a flowchart showing processing of the parallax image display device according to the fifth embodiment of the present invention.

Of these, the processing from Steps 1301 to 1304 is the same as the processing from Steps 301 to 304 in FIG. 3 in the first embodiment of the present invention, and a description thereof will be omitted.

In step 1305, the parallax image creation unit 2022 identifies pixels whose absolute value of the parallax value in the parallax map is equal to or smaller than a predetermined threshold on the parallax image created in step 1303, and colors the identified pixels.

For example, if the threshold value is “2”, pixels having an actual parallax value of −2 or more and 2 or less are colored in a predetermined color. Although colored green in this embodiment, the color may be red or blue if it can be easily distinguished from other pixels.

In the following step 1306, the parallax image creation unit 2022 determines whether or not the parallax value is not less than −2 and not more than 2 for all pixels, and when the confirmation process is completed, the parallax value 2 is displayed on the display device 204 in FIG. 2 by coloring the pixels of −2 to 2 and displaying the cross points.

14 and 15 are cross-point images displaying cross-points in the fifth embodiment of the present invention.

FIG. 14 shows a case where the cross point is in the foreground, and FIG. 15 shows a case where the cross point is in the foreground.

In FIGS. 14 and 15, a subject with a large parallax value is represented with low luminance, and a subject with a small parallax value is represented with high luminance. The pixels other than the cross-point pixels are expressed in monochrome shades, but the outline of each subject is unclear, and the parts with the same parallax value are expressed with the same brightness. I can't figure out the details at all.

It is also conceivable to grasp the correspondence between the original image and the cross-point image by alternately comparing these FIG. 14 or FIG. 15 with the left of FIG. 4 which is the original image. However, comparing the two images is cumbersome, and after comparing the images to understand the correspondence with the original image, the stereoscopic effect of the stereoscopic image that is finally generated is changed to the subject of the original image. It is difficult to recognize it in contrast to

Therefore, in step 1308, the image composition unit 2023 shown in FIG. 2 synthesizes the left image and the cross point image for each pixel.

In step 1308, as in the first embodiment, the image composition unit 2023 applies a 1: 9 weighting to the RGB values of the pixels of each image in the left image and the crosspoint image, and outputs both images. Is being synthesized.

In subsequent step 1309, the image composition unit 2023 determines whether or not the composition processing for all pixels has been completed. If the composition processing for all pixels has been completed, in step 1310, the left image, the crosspoint image, 2 is displayed on the display device 204 shown in FIG. 2, and the processing of steps 1301 to 1310 is completed.

The image obtained by synthesizing the left image and the cross-point image is a bitmap image that can be displayed on a normal 2D monitor, without requiring a special display device such as a display device having a lenticular lens.

The user or the like grasps the stereoscopic effect of the finally obtained stereoscopic image by visually recognizing an image obtained by synthesizing the left image and the crosspoint image displayed on the screen of the 2D monitor.

As described above, according to the fifth embodiment of the present invention, a crosspoint image is created from a parallax image obtained by converting the parallax value for each pixel of the left image into luminance, and the created crosspoint image and the left By creating an image synthesized with the image, the stereoscopic effect of the stereoscopic image generated from the captured image can be grasped even with a normal 2D monitor.

[Sixth Embodiment]
Subsequently, a sixth embodiment of the present invention will be described. The parallax image display device according to the sixth embodiment of the present invention differs from the parallax image display device according to the first embodiment of the present invention shown in FIG. Other configurations are the same.

Next, processing of the parallax image display device according to the sixth embodiment of the present invention will be described with reference to FIG. FIG. 16 is a flowchart showing processing of the parallax image display device according to the sixth embodiment of the present invention.

In FIG. 16, the processing from Steps 1601 to 1604 is the same as the processing from Steps 301 to 304 in FIG. 3 in the first embodiment of the present invention, and the description thereof will be omitted.

In step 1605, the parallax image creation unit 2022 identifies the region where the absolute value of the parallax value in the left image is the maximum on the parallax image created in step 1603 based on the parallax map created in step 1602, It is determined whether or not the specified region has the maximum parallax value in the pop-out direction, that is, whether or not it is the foreground portion of the left image.

The parallax value is 0 at the cross point where the optical axes of the left and right lenses of the stereoscopic image shooting camera intersect, and the farther away from the cross point, that is, the closer to the camera, or the farther from the camera across the cross point, the more. The absolute value of the parallax value increases.

In the sixth embodiment of the present invention, the position where the parallax value is positive is assumed to be a position close to the camera from the cross point, that is, the foreground, and the position where the parallax value is negative is set to the camera rather than the cross point. It is assumed that the position is far from the camera, that is, a distant view.

If the parallax value of a pixel having a large absolute value of the parallax value is positive in step 1605, the parallax image creation unit 2022 determines that the pixel has the maximum parallax value in the pop-out direction. In step 1606, the parallax image The creation unit 2022 creates a parallax absolute value maximum image in which the pixel is colored with a predetermined color, red.

Note that the color is not limited to red, and it is sufficient that the area can be distinguished from other areas.

In subsequent step 1607, the parallax image creation unit 2022 determines whether or not processing has been performed for all pixels. If processing has been performed for all pixels, in step 1608, the region where the parallax value is maximum in the pop-out direction. Is displayed on the display device 204 shown in FIG.

FIG. 17 is a parallax absolute value maximum image displaying a pixel region having the maximum parallax value in the pop-out direction in the sixth embodiment of the present invention.

In FIG. 17, a subject with a large parallax value is represented with low luminance, and a subject with a small parallax value is represented with high luminance. Except for the pixel area where the parallax value is the largest in the pop-out direction, it is expressed in monochrome shades, but the outline of each subject is unclear, and the parts with the same parallax value are expressed with the same brightness. I cannot grasp details of subjects such as trees in a distant view.

By comparing FIG. 17 with the original left image of FIG. 4 alternately, the correspondence between the original image and the parallax absolute value maximum image displaying the region of the pixel having the maximum parallax value in the pop-out direction is grasped. It is also possible to do. However, comparing the two images is cumbersome, and after comparing the images to understand the correspondence with the original image, the stereoscopic effect of the stereoscopic image that is finally generated is changed to the subject of the original image. It is difficult to recognize it in contrast to

Therefore, in step 1609, the image composition unit 2023 shown in FIG. 2 synthesizes the left image in FIG. 4 that is the left image and the parallax absolute value maximum image that displays the region of the pixel having the maximum parallax value in the pop-out direction. .

In step 1609, as in the first embodiment, the image composition unit 2023 uses a left image and a parallax absolute value maximum image displaying a region of a pixel having the maximum parallax value in the pop-out direction. Is applied to the RGB values of the pixels of the respective images to synthesize both images.

In the following step 1610, the image composition unit 2023 determines whether or not the composition process for all pixels is completed. If the composition process for all pixels is completed, in step 1611, the image composition unit 2023 performs a parallax with the left image in the pop-out direction. An image obtained by combining the parallax absolute value maximum image displaying the pixel region having the maximum value is displayed on the display device 204 shown in FIG. 2, and the processing of steps 1601 to 1611 is completed.

If it is determined in step 1605 that the parallax value has the maximum absolute value in the depth direction, the processing in steps 1627 to 1631 is performed. The processing in steps 1627 to 1631 is the same as the processing in steps 1607 to 1611 described above except that the region having the maximum parallax value in the depth direction in step 1626 is colored with blue, which is a different color from step 1606. The description is omitted.

FIG. 18 shows a parallax absolute value maximum image displaying a region having the maximum parallax value in the depth direction in the sixth embodiment of the present invention. The region where the parallax value in the depth direction is the maximum is displayed in blue.

In the sixth embodiment of the present invention, it is determined whether the region where the absolute value of the parallax value is the maximum is in the pop-out direction (before the cross point) or in the depth direction (behind the cross point). it can. Furthermore, as in the fifth embodiment of the present invention described above, the cross point area can be displayed together with the area where the absolute value of the parallax value is maximum.

[Seventh Embodiment]
The seventh embodiment is an operation display unit of an apparatus capable of processing in the above first to sixth embodiments.

The configuration of the seventh embodiment is basically the same as that shown in the block diagram of the parallax image display device according to the first embodiment of the present invention in FIG. The difference is that 204 is an operation display unit 500 integrated in a touch panel format.

Note that the operation display unit 500 according to the present embodiment may use an LCD or the like as the display unit and a pointing device such as a mouse or a pen tablet as the operation unit, in addition to the touch panel.

The operation display unit 500 illustrated in FIG. 19 displays the left image 501 that is the original image on the left of the top row, and the parallax created based on the parallax value calculated from the left image and the right image on the right side An image 502 is displayed.

In the middle left of FIG. 19, when performing image composition, “pixel mixing” that combines the left image and the parallax image by weighting the RGB values as in the first embodiment of the present invention. A pixel mixing button 503 for selecting and a color difference mixing button 504 for selecting “color difference mixing” for synthesizing the color difference of the left image with the parallax image as in the second embodiment of the present invention are provided. ing.

In addition, a slider 505 for changing the weighting of RGB values when “Pixel mixture” is selected is provided on the middle right of FIG. 19. The user moves the slider 505 on the touch panel left and right. By performing the operation, the weighting ratio of the RGB values of the left image and the parallax image can be arbitrarily changed within the range of 0:10 to 10: 0.

At the left of the second row from the bottom of FIG. 19 is a crosspoint adjustment interface 506 that is an interface that can change the crosspoint of the stereoscopic image. The user can display the crosspoint according to the fifth embodiment of the present invention. The position of the cross point can be changed in the front-rear direction from the near side to the depth while confirming on the parallax image displayed on the upper right of FIG.

In the center of the second row from the bottom of FIG. 19, the region having the maximum parallax value according to the sixth embodiment of the present invention can be reflected in the parallax image displayed on the upper right of FIG. There is a maximum parallax button 507, and on the right of the second row from the bottom in FIG. 19, there is a trimming button 508 that is an interface for trimming an image, and in the bottom row in FIG. A size designation button 509 for designating is provided.

As described above, according to the parallax image display device of the seventh embodiment, the user can determine what kind of stereoscopic effect from two or more images read from the image input unit 201 in FIG. Whether the image is created is confirmed by the processing according to the first to sixth embodiments of the present invention described above, and in some cases, a desired stereoscopic effect can be obtained by changing the cross point. it can.

Although various methods for changing the cross point are conceivable, the parallax value is adjusted for the entire parallax image, and the parallax image is created again based on the adjusted parallax value and displayed.

In the first to seventh embodiments, the case where the left image and the right image are stereo-matched using the left image as a reference to generate a parallax map is described, but the right image may be used as a reference.

In addition, when three or more images are acquired, a parallax map may be generated for each of the images as the first image (reference image) and the other image as the second image.

If the image processing unit 202 in FIG. 2 is a computer having a CPU and a memory, the processing routines of the first to sixth embodiments are programmed, and the program is executed by the CPU. May be.

The entire disclosure of Japanese Patent Application No. 2011-005011 is incorporated herein by reference. All documents, patent applications, and technical standards mentioned in this specification are to the same extent as if each individual document, patent application, and technical standard were specifically and individually described to be incorporated by reference, Incorporated herein by reference.

Claims

An acquisition unit for acquiring a plurality of images taken from two or more different viewpoints;
The parallax value represented by the difference in position for each corresponding region between the first image included in the plurality of images acquired by the acquisition unit and the second image different from the first image, A parallax map generating unit that generates a parallax map corresponding to the position of the region of the first image;
A color determining unit that determines a color of a corresponding region according to a parallax value for each region;
A parallax image creation unit that creates a parallax color image in which the color of each region of the first image is changed to the color determined in the color determination unit;
An image synthesis unit that synthesizes the parallax color image and the first image;
A parallax image display device including:
The color determination unit includes a correspondence table that classifies the parallax values for each determined range, and associates colors with the parallax values of the classified range, and the color determination unit includes the correspondence table in which the color of the first image is included in the parallax map. The parallax value corresponding to the position of the area is set as the parallax value of the area of the first image, and a color corresponding to the parallax value of the area of the first image is determined based on the correspondence table. The parallax image display device according to 1.
The image composition unit extracts a color difference component of a pixel included in each region of the first image, and extracts the color difference component of the first image corresponding to each region of the parallax color image and each region of the parallax color image. The parallax image display device according to claim 1, wherein the parallax image display device combines the color difference components of the regions.
The image synthesizing unit assigns a weight to a value obtained from an RGB value of a pixel included in each region of the parallax color image when each of the parallax color image and the first image is divided into a plurality of regions. The first image corresponding to each of the regions of the parallax color image and each of the regions of the parallax color image is made heavier than the weight obtained from the RGB values of the pixels included in each region of the first image. The parallax image display device according to claim 1, wherein the parallax image display device combines the region.
The image synthesizing unit is configured to use the parallax color rather than a weight for a value obtained from an RGB value of a pixel included in a region of the parallax color image corresponding to a region where the absolute value of the parallax value is a minimum value in the parallax map. Weight is increased with respect to values obtained from RGB values of pixels included in the first image region corresponding to the image region, and the absolute value of the parallax value is larger than the minimum value in the parallax map The parallax color image region and the first image region corresponding to the region of the parallax color image correspond to the region of the parallax color image as the absolute value of the parallax value increases. The weight for the value obtained from the RGB value of the pixel included in the region of the first image is reduced, and the weight for the value obtained from the RGB value of the pixel included in the region of the parallax color image is reduced. The it was heavy, the parallax image display device according to claim 1 or claim 2 is combined with the region of the first image corresponding to each of the regions of the parallax-color image and the areas of the parallax-color image.
The parallax color image area is more than a weight for a value obtained from an RGB value of a pixel included in the parallax color image area corresponding to the area where the parallax value is the maximum value in the parallax map. Of the parallax color image corresponding to the region where the parallax value is smaller than the maximum value in the parallax map. The region and the region of the first image corresponding to the region of the parallax color image are included in the region of the first image corresponding to the region of the parallax color image as the parallax value decreases. The parallax color is reduced by reducing the weight for the value obtained from the RGB value of the pixel and increasing the weight for the value obtained from the RGB value of the pixel included in the region of the parallax color image. Parallax image displaying device according to claim 1 or claim 2 is combined with the region of the first image corresponding to each of the regions of the parallax-color image and the areas of the image.
A parallax luminance image creating unit that creates a parallax luminance image in which the parallax value of the region of the parallax map is expressed by luminance;
The parallax image display device according to claim 1, wherein the image synthesis unit synthesizes the parallax luminance image and the parallax color image.
A resolution determining unit that determines the resolution of the region of the first image corresponding to the disparity value for each region of the disparity map;
A parallax resolution image creation unit that creates a parallax resolution image in which the resolution of each region of the first image is changed to the resolution determined in the resolution determination unit;
The parallax image display device according to claim 1, wherein the image synthesis unit synthesizes the parallax resolution image and the parallax color image.
The parallax image display device according to claim 8, wherein the resolution determination unit decreases the resolution of the region of the first image as the parallax value corresponding to the region in the parallax map decreases. .
The parallax according to claim 8, wherein the resolution determination unit decreases the resolution of the region of the first image as the absolute value of the parallax value corresponding to the region in the parallax map increases. Image display device.
The resolution determination unit includes a correspondence table that classifies the parallax values for each determined range, and associates resolutions with the parallax values of the classified range, and the resolution determination unit includes the correspondence table that corresponds to the resolution of the first image in the parallax map. The parallax value corresponding to the position of the area is set as the parallax value of the area of the first image, and the resolution corresponding to the parallax value of the area of the first image is determined based on the correspondence table. The parallax image display device according to 8.
The parallax image display according to claim 11, wherein the resolution determination unit associates a lower resolution with a smaller parallax value included in the range where the parallax value included in the range is smaller in the correspondence table. apparatus.
The resolution determination unit associates a lower resolution with a larger absolute value of the included parallax value in the range in which the absolute value of the parallax value included in the range is larger in the correspondence table. The parallax image display device described in 1.
A sharpness determination unit that determines the sharpness of the region of the first image corresponding to the disparity value for each region of the parallax map;
A parallax sharpness image creation unit that creates a parallax sharpness image in which the resolution of each region of the first image is changed to the sharpness determined in the sharpness determination unit;
The parallax image display device according to claim 1, wherein the image synthesis unit synthesizes the parallax sharpness image and the parallax color image.
The parallax image display device according to claim 14, wherein the sharpness determination unit reduces the sharpness of the region of the first image as the parallax value corresponding to the region in the parallax map decreases. .
The parallax according to claim 14, wherein the sharpness determination unit reduces the sharpness of the region of the first image as the absolute value of the parallax value corresponding to the region in the parallax map increases. Image display device.
The sharpness determination unit has a correspondence table that classifies the parallax values for each determined range, and associates the parallax values of the classified ranges with sharpness, and the parallax map includes the correspondence table The parallax value corresponding to the position of the area is set as the parallax value of the area of the first image, and the sharpness corresponding to the parallax value of the area of the first image is determined based on the correspondence table. 14. The parallax image display device according to 14.
The parallax image display according to claim 17, wherein the sharpness determination unit associates a lower sharpness with a smaller parallax value included in the range where the parallax value included in the range is smaller in the correspondence table. apparatus.
The sharpness determination unit associates, in the correspondence table, the range having a large absolute value of the parallax value included in the range with a lower sharpness as the absolute value of the parallax value included is larger. The parallax image display device described in 1.
A cross formed by converting the parallax value of the area of the parallax map into a luminance value, specifying the position of the area where the absolute value of the parallax value is within a threshold in the parallax map, and coloring the specified area with a predetermined color It further has a cross point image creation unit for creating a point image,
The parallax image display device according to claim 1, wherein the image synthesis unit synthesizes the cross point image and the parallax color image.
The parallax value of the area where the parallax value of the parallax map is converted into a luminance value, the position of the area where the absolute value of the parallax value is maximum is specified in the parallax map, and the parallax value of the area where the absolute value of the parallax value is maximum A parallax absolute value maximum image creating unit that creates a parallax absolute value maximum image in which the region is colored with a different color in a case where is positive and a negative case,
The parallax image display device according to claim 1, wherein the image synthesis unit synthesizes the parallax absolute value maximum image and the parallax color image.
The parallax map generation unit, the color determination unit, the parallax luminance image generation unit, the resolution determination unit, the parallax resolution image generation unit, the sharpness determination unit, the parallax sharpness image generation unit, the parallax image generation unit, and the image An operation unit for inputting an instruction to the synthesis unit;
The parallax map generation unit, the color determination unit, the parallax luminance image generation unit, the resolution determination unit, the parallax resolution image generation unit, the sharpness determination unit, the parallax sharpness image generation unit, the parallax image generation unit, and the image A display unit for displaying a processing result by the synthesis unit;
The parallax image display device according to any one of claims 1 to 19, further comprising:
The parallax image display device according to any one of claims 1 to 22, wherein the region is a region including one pixel.
Acquire multiple images taken from two or more different viewpoints,
The disparity value represented by the difference in position for each corresponding region between the first image included in the acquired plurality of images and an image different from the first image is expressed in the region of the first image. Generate a disparity map corresponding to the position,
Determine the color of the corresponding area according to the parallax value for each area,
Creating a parallax color image in which the color of each region of the first image is changed to the determined color;
A parallax image display method for synthesizing the parallax color image and the first image.
When the parallax color image and the first image are combined, the pixels included in each region of the parallax color image when the parallax color image and the first image are each divided into a plurality of regions. Weights for values obtained from RGB values are heavier than weights for values obtained from RGB values of pixels included in each region of the first image, and each region of the parallax color image and the region of the parallax color image 25. The parallax image display method according to claim 24, wherein the first image region corresponding to each of the first image region and the first image region is combined.
The parallax image display method according to claim 24 or 25, wherein the area is an area including one pixel.
Computer
An acquisition unit for acquiring a plurality of images taken from two or more different viewpoints;
The parallax value represented by the difference in position for each corresponding region between the first image included in the plurality of images acquired by the acquisition unit and the second image different from the first image, A parallax map generating unit that generates a parallax map corresponding to the position of the region of the first image;
A color determining unit that determines a color of a corresponding region according to a parallax value for each region;
A parallax image creation unit that creates a parallax color image in which the color of each region of the first image is changed to the color determined in the color determination unit;
And a parallax image display program for causing the parallax color image and the first image to function as an image synthesis unit.
The image synthesizing unit assigns a weight to a value obtained from an RGB value of a pixel included in each region of the parallax color image when each of the parallax color image and the first image is divided into a plurality of regions. The first image corresponding to each of the regions of the parallax color image and each of the regions of the parallax color image is made heavier than the weight obtained from the RGB values of the pixels included in each region of the first image. The parallax image display program according to claim 27, wherein the parallax image display program combines the areas.
29. The parallax image display program according to claim 27 or 28, wherein the area is an area including one pixel.