WO2019167571A1

WO2019167571A1 - Image processing device and image processing method

Info

Publication number: WO2019167571A1
Application number: PCT/JP2019/004143
Authority: WO
Inventors: 一輝大橋; 泰史佐藤; 建行藤井; 松原　義明
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2018-02-27
Filing date: 2019-02-06
Publication date: 2019-09-06

Abstract

The purpose of the present invention is to acquire a high-quality image in which a problem resulting from a disparity between two images acquired by two image capturing units is solved. Provided is an image processing device equipped with: a first color image capturing element in which a color filter covering a light receiving surface includes neither a white region nor an unfiltered region; a second color image capturing element in which the color filter covering the light receiving surface includes the white region or the unfiltered region; a disparity information acquisition unit which acquires information relating to a disparity between a first color image captured by the first color image capturing element and a second color image captured by the second color image capturing element; a composition information generation unit which, on the basis of the information relating to the disparity and acquired by the disparity information acquisition unit, generates information relating to a way of compositing both the images; and an image composition unit which, on the basis of the composition information generated by the composition information generation unit, composites the first color image with the second color image.

Description

Image processing apparatus and image processing method

The present technology relates to an image processing apparatus and an image processing method. More specifically, the present technology relates to an image processing apparatus and an image processing method for synthesizing two images and obtaining one image.

In recent years, smartphones equipped with dual cameras have attracted attention. For example, the smartphone is provided with two cameras on the back thereof. One image can be acquired using two image data captured by each of the two cameras.

For example, in the following Patent Document 1, a first imaging unit that captures a first image, a second imaging unit that captures a second image, the first image, and the second image An image processing apparatus comprising: a parallax determination unit that determines the presence or absence of parallax; and a synthesis unit that synthesizes each pixel of the first image and the second image according to a determination result of the parallax determination unit. Has been. Since the image processing apparatus can appropriately combine two images according to the parallax, an appropriate image quality can be improved by combining a plurality of imaging units to capture and overlap the same range. .

JP 2017-69926 A

The dual camera has a configuration in which, for example, two imaging units are arranged side by side on one surface such as the back surface of a smartphone. Since the viewpoints of the two image capturing units are different, there is a parallax between the two images captured by the two image capturing units. Therefore, when the two images are combined, image quality degradation caused by the parallax can occur. Therefore, it is required to combine the two images more appropriately to obtain a higher quality image.

This technology aims to provide a new image processing technology.

The present inventors have found that an image processing apparatus having a specific configuration can solve the problem caused by the parallax and can acquire a high-quality image.

That is, in the present technology, the color filter covering the light receiving surface does not include either the white region or the no-filter region, and the color filter covering the light receiving surface includes the white region or the no-filter region. A second color image pick-up device, a first color image picked up by the first color image pick-up device, and a second color image picked up by the second color image pick-up device. A parallax information acquisition unit that acquires information on parallax, a synthesis information generation unit that generates information on how to combine both images based on the information on parallax acquired by the parallax information acquisition unit, and the synthesis information generation unit An image processing apparatus comprising: an image combining unit configured to combine the first color image with the second color image based on the generated combination information; Subjected to.
According to one embodiment of the present technology, the color filter included in the first color image pickup device may be a Bayer array color filter.
According to one embodiment of the present technology, the color filter included in the second color image pickup device may include a white region.
According to one embodiment of the present technology, the color filter included in the second color image pickup device may include a region of a WRGB array or a WRG array.
According to one embodiment of the present technology, the parallax information acquisition unit may acquire parallax for each pixel or each pixel block between the first color image and the second color image.
According to one embodiment of the present technology, the image processing apparatus may further include a parallax compensation unit that performs parallax compensation on the first color image based on the information on the parallax.
According to one embodiment of the present technology, the synthesis information generation unit includes a search range outside region detection unit that detects a region having a disparity value larger than a predetermined search range in a disparity map created based on information on disparity. sell.
According to one embodiment of the present technology, the composite information generation unit may include an occlusion area detection unit that detects an occlusion area by comparing the first color image and the second color image.
According to one embodiment of the present technology, the composite information generation unit calculates a value obtained by dividing a color difference variance value by a luminance variance value for each predetermined region in the first color image or the second color image. A color difference / luminance determination unit that detects an area in which the divided value is equal to or greater than a predetermined value may be included.
According to one embodiment of the present technology, the synthesis information generation unit detects an out-of-search-range area in which a pixel or a pixel block having a disparity value larger than a predetermined search range is detected in a disparity map created based on information on disparity An occlusion area detection unit that detects an occlusion area by comparing the first color image and the second color image, and for each predetermined area in the first color image or the second color image. Includes at least one component selected from a color difference / brightness determination unit that calculates a value obtained by dividing the chrominance dispersion value by the luminance dispersion value and detects an area in which the divided value is equal to or greater than a predetermined value. For the area detected in the component, the combination information that the first color image should not be combined with the second color image can be generated.
According to one embodiment of the present technology, the combination information that the combination information generation unit should or should not combine the first color image for each region that forms the second color image Can be generated.
In accordance with one embodiment of the present technology, when the ratio of the area in the second color image that the first color image should not be synthesized exceeds a predetermined value, according to one embodiment of the present technology In addition, it is possible to generate synthesis information indicating that the first color image should not be synthesized for all regions of the second color image.
According to one embodiment of the present technology, the synthesis information generation unit may give a synthesis rate for synthesizing the first color image for each region constituting the second color image.

Further, the present technology is imaged by the first color image imaging device, the second color image imaging device having higher sensitivity than the first color image imaging device, and the first color image imaging device. A parallax information acquisition unit that acquires information about parallax between the first color image and the second color image captured by the second color image imaging device, and the parallax acquired by the parallax information acquisition unit Based on the information, the first color image is added to the second color image based on the combination information generated by the combination information generation unit that generates information on how to combine the two images. An image processing apparatus including an image composition unit for composition is also provided.

In addition, the present technology provides a first image acquisition step of acquiring a first color image by a first color image pickup device, in which the color filter covering the light receiving surface includes neither a white area nor an unfiltered area; A color filter covering the surface includes a white area or a non-filter area; a second image acquisition step of acquiring a second color image by a second color image pickup device; the first color image and the second color image; A parallax information acquisition step for acquiring information on parallax between color images, and a synthesis information generation step for generating information on how to combine both images based on the information on parallax acquired in the parallax information acquisition step; An image synthesis step of synthesizing the first color image with the second color image based on the synthesis information generated in the synthesis information generation step; To provide an image processing method comprising.

The present technology also includes a first image acquisition step of acquiring a first color image by the first color image imaging device, and a second color image imaging device having higher sensitivity than the first color image imaging device. A second image acquisition step of acquiring a second color image, a parallax information acquisition step of acquiring information on the parallax between the first color image and the second color image, and the parallax information acquisition step Based on the information on the parallax acquired in step S2, a combination information generation step for generating information on how to combine both images, and on the second color image based on the combination information generated in the combination information generation step, An image processing method including an image combining step of combining the first color image is also provided.

This technology can acquire a high-quality image in which the problem caused by the parallax between the two images acquired by the two imaging units is solved. In addition, the effect show | played by this technique is not necessarily limited to the effect described here, and may be the any effect described in this specification.

It is a figure which shows the example of the image obtained by the image obtained by the image processing according to this technique, and another image composition method. It is a figure which shows the structural example of two color image pick-up elements with which the image processing apparatus according to this technique is equipped. It is an example of the block diagram of the image processing apparatus according to this technique. It is a figure shown with the example of the color filter which covers the light-receiving surface of the 1st color image pick-up element and the 2nd color image pick-up element with which the image processing device according to this art is equipped. It is a figure which shows as an example of the color filter which covers the light-receiving surface of the 2nd color image imaging element with which the image processing apparatus according to this technique is equipped. It is a figure which shows the structural example of the block diagram of the synthetic | combination information generation part contained in the image processing apparatus according to this technique. It is a figure for demonstrating an occlusion area | region. It is a figure for demonstrating an example of the detection method of an occlusion area | region. It is a figure which shows the example of the area | region where image quality degradation may arise. It is a figure which shows an example of the flow of the image processing method according to this technique. It is a figure which shows an example of the flow of a synthetic | combination information generation process. It is a figure which shows an example of the flow of a search range outside area | region detection process. It is a figure which shows an example of the flow of an occlusion area | region detection process. It is a figure which shows an example of the flow of a color difference / luminance determination process. It is a figure which shows an example of the flow of a synthetic | combination process with a 1st color image and a 2nd color image. It is a figure which shows an example of the flow of the post-process of a synthesized image. It is a figure which shows an example of the image pick-up element which has a three-layer structure. It is a figure which shows the structural example of the image processing apparatus according to this technique.

Hereinafter, preferred embodiments for carrying out the present technology will be described. Note that the embodiments described below are representative embodiments of the present technology, and the scope of the present technology is not limited to these embodiments. In addition, description of this technique is performed in the following order.
1. First embodiment (image processing apparatus)
(1) Description of the first embodiment (2) First example of the first embodiment (image processing apparatus)
2. Second embodiment (image processing apparatus)
3. Third Embodiment (Image Processing Method)
(1) Description of the third embodiment (2) First example of the third embodiment (image processing method)
4). Fourth embodiment (image processing method)
5. Device configuration example

1. First embodiment (image processing apparatus)

(1) Description of the first embodiment

In the image processing device according to the present technology, the color filter covering the light receiving surface includes the first color image pickup element that does not include either the white region or the no filter region, and the color filter covering the light receiving surface includes the white region or the no filter region. A second color image pickup device may be included. Thus, the image processing device according to the present technology can include two different color image pickup devices.
Furthermore, the image processing apparatus according to the present technology provides a parallax between a first color image captured by the first color image imaging device and a second color image captured by the second color image imaging device. Generated by the disparity information acquisition unit that acquires information on the information, the combination information generation unit that generates information on how to combine both images based on the information on the disparity acquired by the disparity information acquisition unit, and the combination information generation unit And an image composition unit for compositing the first color image with the second color image based on the synthesized information. As described above, the image processing device according to the present technology acquires disparity information between two color images captured by the two color image capturing elements, generates information on a synthesis method based on the disparity information, Then, based on the information relating to the generated synthesis method, the two images are synthesized.

The color filter that covers the light receiving surface of the first color image pickup device does not include either the white region or the no-filter region, but the color filter that covers the light receiving surface of the second color image pickup device is the white region or filter. Includes no area. Therefore, the sensitivity of the second color image pickup device is higher than that of the first color image pickup device.
Further, the image synthesizing unit captures the first color image captured by the first color image imaging device by the second color image imaging device having higher sensitivity than the first color image imaging device. Composite to the second color image. That is, the first color image is combined with the second color image with reference to the second color image having higher image quality than the first color image. As a result, for example, even if a part of the second color image is not synthesized with the first color image, the part has high image quality.
As described above, according to the present technology, even if there is a region in which the first color image is not combined in the second color image serving as a reference, the combined image has an effect of high image quality.
The effect will be further described below while comparing the present technique with other methods conceivable for the synthesis of two images. For the description, reference is made below to FIG.

When combining two images, for example, it is conceivable to combine a color image with the black and white captured image on the basis of the black and white captured image in order to increase the sensitivity of the reference image. In such composition, if there is an area in the black-and-white captured image where it is determined that the image quality is deteriorated by compositing two images, it is conceivable that image composition is not performed for the area. However, when the color image is adopted for the area where image synthesis is not performed, the viewpoint of the color image is different from the viewpoint of the monochrome black-and-white captured image as a reference, and thus the viewpoint moves only in the area. Occurs. In order to prevent the viewpoint from moving, it may be possible to adopt a black-and-white captured image for the area where image synthesis is not performed. However, there is a problem that only the area of the synthesized image is black and white. Such an image is not allowed. In addition, when images are not synthesized to solve these problems, a color image with low sensitivity must be selected and output.
For example, as shown in FIG. 1A, a case is assumed in which a human part in both images cannot be synthesized when the image a1 that is a monochrome image and the image a2 that is a color image are synthesized on the basis of the image a1. In this case, if the color image a <b> 2 is adopted for the human part that cannot be synthesized, the viewpoint movement occurs only for the human part in the synthesized image. In addition, it is not allowed to adopt the black and white image a1 for the person portion that cannot be synthesized. When images are not combined, a color image a2 with low sensitivity must be selected and output.
In the present technology, a color image is used as a reference image for image composition. For this reason, for the region where image synthesis is not performed, the above-described problem of viewpoint movement does not occur by adopting a reference color image. In addition, since the reference image is a color image, the area where image composition is not performed is not black and white but color.
For example, as shown in FIG. 1C, when the color image b2 and the color image c1 captured with higher sensitivity than the color image b2 are combined based on the color image c1, human portions in both images cannot be combined. Assume a case. An image obtained by the synthesis is c3. In this case, since the color image c1 is adopted for the person portion c10 that cannot be synthesized, the above-described problem of viewpoint movement does not occur. In addition, since the image serving as a reference for synthesis is the color image c1, the region where the image synthesis is not performed is color.

Further, when two images are combined, for example, it is conceivable to combine color images having the same sensitivity without adopting a black and white captured image. In such a composition, when an area that is determined to deteriorate image quality due to the composition of two images is in a reference color image, the image composition may not be performed for the area. . For a region where image synthesis is not performed, either a reference color image or the other color image is adopted. However, the image quality is deteriorated in the area where the composition is not performed as compared with other areas where the composition is performed.
For example, as shown in FIG. 1B, a case is assumed in which a human part in both images cannot be synthesized when the color image b1 and the color image b2 having the same sensitivity as the color image b1 are synthesized on the basis of the color image b1. . An image obtained by the synthesis is b3. In this case, by adopting the color image a1 for the person portion that cannot be combined, the problem of viewpoint movement does not occur, but the image quality of the person portion b10 that does not perform combining is deteriorated compared to other regions where combining is performed. .
In the present technology, of the two images to be combined, the reference color image is an image captured by an image sensor having higher sensitivity than the image sensor that captured the other color image as described above. In addition, a higher-sensitivity color image is adopted for a region where no synthesis is performed. As a result, in the image synthesis according to the present technology, the degree of image quality degradation in the area where the synthesis is not performed is smaller than in the case of synthesizing color images having the same sensitivity.
For example, as shown in FIG. 1C, when two color images are synthesized as in the case of FIG. 1B, a color image c1 captured with higher sensitivity than the color image b2 instead of the color image b1. Is used for synthesis. That is, it is assumed that the color image b2 and the color image c1 having higher sensitivity than the color image b2 are combined with the color image c1 as a reference. In this case, by adopting the color image c1 for the person portion that cannot be combined, the problem of viewpoint movement does not occur. In addition, since the reference color image c1 is an image picked up by an image sensor having a higher sensitivity than the color image b1, image quality deterioration in a region where no synthesis is performed, compared to the case of FIG. The degree of is smaller.

Further, in the present technology, the parallax information acquisition unit acquires information about the parallax between the first color image and the second color image, and the synthesis information generation unit uses the information about the parallax to calculate Information relating to the composition method is generated, and the image composition unit synthesizes the first color image with the second color image based on the information related to the composition method. As a result, it is possible to perform image composition for an image area where image quality is deteriorated by image composition and to perform composition for other areas. Furthermore, since the image used as the reference for the combination is a second color image with higher image quality, the second color image with higher image quality is adopted for the region where image combination is not performed. For this reason, the degree of image quality degradation in a region where image synthesis is not performed is low.

The image processing apparatus according to the present technology includes at least two color image pickup elements. Of the two color image pickup devices, the color filter covering the light receiving surface of one image pickup device does not include either the white region or the no-filter region, and the color filter covering the light receiving surface of the other image pickup device is the white region. Or it includes the unfiltered area. The former is called a first color image pickup device, and the latter is called a second color image pickup device. Due to the difference in the color filters, the second color image pickup device is more sensitive than the first color image pickup device.

In the present technology, the first color image imaging element and the second color image imaging element are preferably arranged side by side on one surface of the image processing apparatus. Preferably, the first color image pickup device and the second color image pickup device may be arranged so as to be able to pick up images in the same direction.
For example, as illustrated in FIG. 2, when the image processing apparatus according to the present technology is the smartphone 10, the first color image pickup device 11-1 and the second color image pickup device 11-1 are provided on the surface opposite to the display surface 12 of the smartphone 10. The color image pickup elements 11-2 are arranged side by side so that the same direction can be picked up. The first color image pickup device and the second color image pickup device may be provided not only on the opposite surface but also on the display surface.
When two image sensors are arranged in this manner, the viewpoints of both image sensors are different from each other, so that parallax occurs between color images captured by both image sensors. The image processing apparatus according to the present technology acquires information on the parallax using the parallax information acquisition unit. Based on the information on the parallax, the combination information generation unit generates information on how to combine both images. Then, based on the information related to the composition method, the image composition unit synthesizes two images.

The image processing apparatus according to the present technology may be a smartphone as described above, for example. The image processing apparatus according to the present technology may be an information processing apparatus other than a smartphone. For example, the image processing device according to the present technology is a mobile phone other than a smartphone, a video processing device (particularly a portable video processing device), a game device (particularly a portable game device), a notebook PC (Personal Computer), or a tablet type. It may be a PC. That is, an information processing apparatus provided with the first color image pickup element and the second color image pickup element and configured to be able to synthesize images according to the present technology is included in the image processing apparatus according to the present technology. Is done.

(2) First example of first embodiment (image processing apparatus)

Hereinafter, an example of an image processing apparatus according to the present technology will be described with reference to FIG. FIG. 3 is a block diagram of an example of an image processing apparatus according to the present technology.

As shown in FIG. 3, the image processing apparatus 300 includes a first color image imaging element 301-1, a second color image imaging element 301-2, a preprocessing unit 302-1, a preprocessing unit 302-2, a parallax. An information acquisition unit 303, a parallax compensation unit 304, a synthesis information generation unit 305, an image synthesis unit 306, and a post-processing unit 307 are provided.

The color filter that covers the light receiving surface of the first color image pickup element 301-1 includes neither a white area nor an unfiltered area. The first color image pickup element 301-1 can be composed of, for example, a red (R) pixel, a blue (B) pixel, and a green (G) pixel. That is, the color filter that covers the light receiving surface of the first color image pickup element 301-1 includes a red (R) filter, a blue (B) filter, and a green (G) filter. In a more preferred embodiment, the first color image pickup device 301-1 can be composed of only R pixels, B pixels, and G pixels. That is, the color filter that covers the light receiving surface of the first color image pickup element 301-1 includes only a red (R) filter, a blue (B) filter, and a green (G) filter.
The number of G pixels in the first color image pickup element 301-1 is preferably larger than the number of R pixels and B pixels. Further, the number of R pixels and B pixels may be equal. More preferably, a Bayer array pixel array can be employed for the first color image pickup element 301-1.
That is, in the present technology, the color filter that covers the light receiving surface of the first color image pickup element 301-1 can be a Bayer array color filter. An example of the Bayer array is shown in FIG. As shown in FIG. 4B, the number of R pixels, B pixels, and G pixels in the Bayer array may be R: B: G = 1: 1: 2.
In the present technology, the first color image pickup device 301-1 is not limited to the one having a Bayer array, and may be an image pickup device having a three-layer structure as shown in FIG.
As components other than the color filter constituting the first color image pickup device 301-1, components used in an image pickup device known in this technical field may be employed. For example, a known CMOS image sensor or CCD image sensor is used. The components used in the above may be employed.

The color filter covering the light receiving surface of the second color image pickup element 301-2 includes a white region or a non-filter region, and preferably includes a white region. That is, the second color image pickup element 301-2 includes white (W) pixels or unfiltered (C) pixels, and preferably includes W pixels. The W pixel is also called a transparent pixel.
The total number of pixels corresponding to the white area and the non-filter area may occupy, for example, 1/8 or more, preferably ¼ or more, or ½ or more of all pixels.
For example, as illustrated in FIG. 4A, an image sensor in which W pixels, R pixels, G pixels, and B pixels are provided at 1: 1: 1: 1 is a second color image image sensor 301. -2 can be used. That is, a WRGB sensor can be used as the second color image pickup element 301-2. In this case, the color filter included in the second color image pickup element may include a WRGB array region.
Alternatively, as shown in FIG. 5A, an image sensor in which W pixels, R pixels, and G pixels are provided at 2: 1: 1 is a second color image image sensor 301-. 2 can be used. That is, the WRG sensor can be used as the second color image pickup element 301-2. In this case, the color filter included in the second color image pickup device may include a WRG array region.
Alternatively, as shown in FIG. 5B, an image pickup device in which yellow (Y) pixels, C pixels, magenta (M) pixels, and G pixels are provided at 1: 1: 1: 1. The second color image pickup element 301-2 can be used. That is, a YCMG sensor can be used as the second color image pickup element 301-2. In this case, the color filter included in the second color image pickup device may include a YCMG array region.
Alternatively, as shown in FIG. 5C, an image sensor in which W pixels, Y pixels, and cyan (Cy) pixels are provided at 2: 1: 1 is a second color image image sensor. It can be used as 301-2. That is, the WYCy sensor can be used as the second color image pickup element 301-2. In this case, the color filter included in the second color image pickup device may include a WYCy array region.
Alternatively, as shown in FIG. 5D, an image sensor composed of W pixels, R pixels, and B pixels can be used as the second color image image sensor 301-2. In the pixel configuration shown in FIG. 5D, W pixels are incorporated in the clear bit array. That is, the W clear bit sensor can be used as the second color image pickup element 301-2. In the clear bit arrangement, the pixel arrangement direction is rotated 45 degrees with respect to the normal arrangement direction.
According to a preferred embodiment of the present technology, the color filter included in the second color image pickup device includes a region of a WRGB array or a WRG array. That is, the second color image pickup element can be preferably a WRGB sensor or a WRG sensor.
As a component other than the color filter constituting the second color image pickup device 301-2, a component used in an image pickup device known in this technical field may be employed. For example, a known CMOS image sensor or CCD image sensor is used. The components used in the above may be employed.

In accordance with a preferred embodiment of the present technology, a first color image imaged by the first color image imaging element 301-1 and a second color image imaged by the second color image imaging element 301-2 Are taken at the same time. By combining two images captured at the same time, a more appropriate image can be combined.

The preprocessing units 302-1 and 302-2 can perform preprocessing performed in a normal imaging apparatus. Examples of the preprocessing include gain adjustment, white balance correction, noise reduction, demosaic processing, scaling, and module shift correction processing. The pre-processing units 302-1 and 302-2 can perform one or more of these processes.

The parallax information acquisition unit 303 is between the first color image captured by the first color image imaging element 301-1 and the second color image captured by the second color image imaging element 301-2. Obtain information on parallax. The information regarding the parallax can include parallax of each region in the image, for example, parallax for each pixel and / or parallax for each pixel block. A pixel block refers to a block composed of a plurality of pixels. Information on the parallax can be acquired by corresponding point detection processing such as block matching, for example.
As described above, the image processing apparatus according to the present technology can be synthesized based on the second color image captured by the second color image imaging element 301-2 having higher sensitivity. Therefore, it is preferable that the parallax information acquisition unit 303 acquires information related to the parallax of the first color image with the second color image as a reference.
Further, the parallax information acquisition unit 303 may generate a parallax map. The parallax map is a map showing the distribution of parallax between two color images. The parallax map can be used, for example, by a synthesis information generation unit 305 described below, in particular, an out-search range region detection unit 601 and / or an occlusion region detection unit 602 included in the synthesis information generation unit 305.
The parallax information acquisition unit 303 can acquire a parallax map indicating the parallax distribution of the first color image with respect to the second color image serving as a reference for synthesis, for example. The disparity information acquisition unit 303 distributes the disparity of the second color image with respect to the first color image in addition to the disparity map indicating the disparity distribution of the first color image with respect to the second color image that is a reference for combining. You may acquire the parallax map which shows. As described above, the occlusion area detection unit 602 described below can detect an occlusion area by using two parallax maps based on images from two viewpoints.
The parallax information acquisition unit 303 outputs information on parallax to the parallax compensation unit 304 and / or the synthesis information generation unit 305.

The parallax compensation unit 304 can perform parallax compensation of the first color image based on the information on the parallax acquired by the parallax information acquisition unit 303. For example, the parallax compensation unit 304 can move the position of the pixel or the pixel block on the image data of the first color image based on the information on the parallax. The movement of the position of the pixel or the pixel block may be performed so that the parallax between the first color image and the second color image is eliminated, for example. In particular, the parallax compensation unit 304 can move the pixel or pixel block of the first color image to the position of the pixel or pixel block in the second color image corresponding to the pixel or pixel block. The parallax compensation unit 304 generates a first color image that is parallax-compensated by parallax compensation. By the parallax compensation, each region of the first color image is moved to each corresponding region in the second color image, and as a result, appropriate image composition is possible.
The parallax compensation unit 304 outputs the first color image subjected to parallax compensation to the image synthesis unit 306.

The combination information generation unit 305 generates information about how to combine the first color image and the second color image. More preferably, the synthesis information generation unit 305 generates information on how to combine the first color image and the second color image that have been subjected to parallax compensation. For example, the synthesis information generation unit 305 should synthesize or synthesize the first color image (particularly the parallax-compensated first color image) for each region constituting the second color image. Compositing information that should not be generated can be generated. The synthesis information can be generated for each pixel or each pixel block. As a result of the determination, the synthesis information generation unit 305 may generate information that should be combined or should not be combined for each pixel or each pixel block.
The synthesis information generation unit 305 outputs information related to the synthesis method to the image synthesis unit 306.

A block diagram of a more detailed configuration example of the synthesis information generation unit 305 is shown in FIG. As illustrated in FIG. 6, the composite information generation unit 305 may include an out-of-search area detection unit 601, an occlusion area detection unit 602, and a color difference / luminance determination unit 603. The composite information generation unit 305 may include any one, two, or three of the search range outside region detection unit 601, the occlusion region detection unit 602, and the color difference / luminance determination unit 603. Further, the composite information generation unit 305 may include a comprehensive determination unit 604 as illustrated in FIG.
Hereinafter, the above-described components that can be included in the composite information generation unit 305 will be described.

The search area outside area detection unit 601 can detect an area having a parallax value larger than a predetermined search range in a parallax map created based on information on parallax. When image composition is performed for a region outside the search range, the possibility of image quality deterioration increases. Therefore, it is preferable not to perform image composition in the region outside the search range. The area may be a pixel or a pixel block having a parallax value larger than a predetermined search range. For example, the region outside the search range detection unit 601 may detect a region having a disparity value larger than a predetermined search range in the disparity map generated by the disparity information acquisition unit 303.
The information related to the parallax used by the search range outside area detection unit 601 is not limited to the parallax map generated by the parallax information acquisition unit 303. For example, the out-of-search range area detection unit 601 may detect pixels or pixel blocks having a parallax value larger than a predetermined search range using, for example, information on parallax obtained by a distance measuring sensor (particularly, parallax). Good. As an example of the distance measuring sensor, for example, a ToF (Time of Flight) type distance measuring sensor can be cited.
The out-of-search-range area detection unit 601 can output an area having a parallax value larger than the predetermined search range, for example, to the comprehensive determination unit 604.

The occlusion area detection unit 602 can detect the occlusion area by comparing the first color image and the second color image. The occlusion area is an area that is imaged by one of the two image sensors of the first color image sensor and the second color image sensor, but is not imaged by the other image sensor. Say. For example, as shown in FIG. 7, when an image is picked up from the image sensor 701 in the direction of the dotted line, the area A of the background B is not blocked by the foreground F, and is picked up. On the other hand, when the image is picked up from the image pickup element 702 in the direction of the one-dot chain line, the area A of the background B is blocked by the foreground F and is not picked up. Area A is an occlusion area. When image synthesis is performed for the occlusion area, the possibility of image quality deterioration increases. Therefore, it is preferable not to perform image composition in the occlusion area.
The detection of the occlusion area uses a bi-directional parallax map to move the left image to the position of the right image, and then detect the area that does not return to the same position when moved to the left image again. May be done by: Alternatively, adjacent parallax values are compared from a parallax map of one viewpoint, and an area where the difference exceeds a predetermined range can be detected as an occlusion area. As a result, the occlusion area can be detected more easily.
The occlusion area detection unit 602 can output the detected occlusion area to the comprehensive determination unit 604, for example.

More specifically, the detection of the occlusion area is performed by, for example, two parallaxes of a first parallax map of the second color image with respect to the first color image and a second parallax map of the first color image with respect to the second color image. This may be done using a map. In this case, the occlusion area detection unit 602 moves the first color image based on the first parallax map, and then moves the first color image after the movement based on the second parallax map. Pixels or pixel blocks that do not return to the position of the first color image can be detected as the occlusion area. Alternatively, when the occlusion area detection unit 602 moves the second color image based on the second parallax map and then moves the second color image after the movement based on the first parallax map, Pixels or pixel blocks that do not return to the position of the second color image can be detected as occlusion areas.

An example of an occlusion area detection method using two parallax maps will be described with reference to FIG. FIG. 8 shows an example in which an occlusion region is detected using the left image L captured by the left image sensor and the right image R captured by the right image sensor. As shown in FIG. 8A, in the region that is not the occlusion region, the left image L is moved based on the parallax map of the right image R with respect to the left image L, and then the moved left image L is changed to the right image. When moved based on the parallax map of the left image L with respect to R, the position returns to the position before the movement. On the other hand, as shown in FIG. 8B, when the two movements are performed, the occlusion area does not return to the position before performing these movements. Thus, the occlusion area detection unit 602 can detect the occlusion area based on the two parallax maps.

The color difference / luminance determination unit 603 calculates a value obtained by dividing the color difference dispersion value by the luminance dispersion value for each predetermined region in the first color image or the second color image, and the divided value is a predetermined value. A region that is greater than or equal to the value can be detected. For example, the color difference / luminance determination unit 603 can detect an area where there is no luminance difference but there is a color difference.
For example, a region 901 in which a red region Ar and a blue region Ab are adjacent as shown in FIG. 9 is assumed. The red area Ar and the blue area Ab have the same luminance signal but different color difference signals. When the parallax is obtained by luminance-based corresponding point detection, the parallax value between the red area Ar and the blue area Ab cannot be obtained accurately, and as a result, the color edge may not be accurately identified. Therefore, the boundary area 902 between the red area Ar and the blue area Ab may deteriorate image quality when the first color image and the second color image are combined. The chrominance variance value / luminance variance value of the area 901 is larger than that of the area consisting only of the red area Ar. Therefore, the color difference / luminance determination unit 603 can obtain a color difference variance value / luminance variance value for each region, and can detect a region where this value is larger than a predetermined value. The color difference / luminance determination unit 603 can output the detected area to the comprehensive determination unit.

The comprehensive determination unit 604 can determine whether to synthesize each region in the image based on information output from a component that detects a region that should or should not be image-synthesized. Based on the result of the determination, the comprehensive determination unit 604 can generate combination information regarding how to combine the first color image and the second color image. The component may be included in the composite information generation unit 305. For example, any one of the out-of-search range region detection unit 601, the occlusion region detection unit 602, and the color difference / luminance determination unit 603 described above. There can be one, two, or three. The overall determination unit 604 can determine that the region detected by at least one of these components is a region where the first color image should not be combined with the second color image. Then, with respect to the region, it is possible to generate synthesis information indicating that the first color image is not synthesized with the second color image.

Preferably, when the synthesis information generation unit 305 includes all of the out-of-search range region detection unit 601, the occlusion region detection unit 602, and the color difference / luminance determination unit 603, the region detected by any of these components The composition information that the first color image should not be composed with the second color image can be generated.

The composition information generation unit 305, when the ratio of the areas in the second color image that the first color image should not be synthesized exceeds a predetermined value, all of the second color images Compositing information that the first color image should not be synthesized for the region may be generated. When the combination information is generated, the first color image is not combined with the second color image. As a result, the second color image itself is output. In this technique, even in such a case, since the second color image itself has high image quality, the degree of image quality deterioration can be suppressed. For example, when the first color image is completely different from the second color image due to an abnormality in the first color image pickup device, image quality deterioration is avoided by generating such composite information. be able to.

According to another embodiment of the present technology, the synthesis information generation unit 305 may determine the synthesis rate α [0 ≦ α ≦ 1] for each region in the second color image. When the synthesis rate α = 0, the first color image is not synthesized with the second color image. When the synthesis rate α = 1, for example, the first color image with respect to the second color image can be synthesized at the maximum ratio. The maximum ratio may be appropriately set by a person skilled in the art or by factors such as the types of two images to be combined. The maximum proportion may be, for example, 30% to 70%, in particular 40% to 60%. For example, when α = 1, the first color image and the second color image may be combined at a ratio of 1: 1 (that is, 50%). The composition information generation unit 305 can output the composition rate α determined for each region to the image composition unit 306 as composition information.

The image synthesis unit 306 can synthesize the first color image with the second color image based on the synthesis information generated by the synthesis information generation unit 305 using the second color image as a reference. For example, among the areas constituting the second color image, for the area where the first color image is to be synthesized, the first color image is synthesized with the second color image, and For the region where the first color image should not be synthesized, the second color image itself is adopted without synthesizing the first color image.
When the synthesis information includes the synthesis rate α, the image synthesis unit 306 can perform image synthesis for each region in the image according to the synthesis rate.

The post-processing unit 307 can perform processing for reducing a difference between the first color image imaging device and the second color image imaging device. The post-processing performed by the post-processing unit 307 is performed on the area where the first color image is combined with the second color image, and the post-processing performed by the post-processing unit 307 is performed on the area where the combination is not performed. On the other hand, the post-processing performed by the post-processing unit 307 may be different.
For example, when there is a color difference between the first color image and the second color image due to the spectral difference between the two color image pickup devices, the post-processing unit 307 determines the color difference. To compensate, Matrix conversion processing and / or multi-axis color conversion processing can be performed for each region. When there is an SN difference between the first color image and the second color image, the post-processing unit 307 can perform NR with different NR intensity for each region in order to compensate for the SN difference.
When the synthesis information includes the synthesis rate α, the post-processing unit 307 can perform post-processing according to the synthesis rate for each region.

Specific image processing performed by the image processing apparatus according to the present technology will be described below in “3. Third embodiment (image processing method)”.

2. Second embodiment (image processing apparatus)

The image processing device according to the present technology may include a first color image imaging device and a second color image imaging device having higher sensitivity than the first color image imaging device. As described above, the image processing device according to the present technology may include two color image pickup elements having different sensitivities.
Furthermore, the image processing device according to the present technology relates to a parallax between a first color image captured by the first color image imaging device and a second color image captured by the second color image imaging device. Generated by a disparity information acquisition unit that acquires information, a combination information generation unit that generates information on how to combine both images based on information on the disparity acquired by the disparity information acquisition unit, and the combination information generation unit And an image composition unit for compositing the first color image with the second color image based on the composition information. As described above, the image processing device according to the present technology acquires disparity information between two color images captured by the two color image capturing elements, generates information on a synthesis method based on the disparity information, Then, based on the information relating to the generated synthesis method, the two images are synthesized.

The effect described in “1. First embodiment (image processing apparatus)” is achieved by the image processing apparatus.
Further, the difference in sensitivity between the two color image pickup devices can be introduced, for example, by selecting the type of color filter that covers the light receiving surface of each image pickup device. The color filter of each image sensor may be, for example, as described in “1. First embodiment (image processing apparatus)”. Therefore, as a specific example of the color filter, the color filter described in “1. First embodiment (image processing apparatus)” may be employed in this embodiment. For example, the color filter that covers the light receiving surface of the first color image pickup device does not include either the white region or the no-filter region, and the color filter that covers the light receiving surface of the second color image pickup device has the white region. Or a non-filtered area, preferably a white area.
Arrangement of the two color image pickup devices, a specific example of the image processing device, each component constituting the image processing device (for example, an image pickup device, a preprocessing unit, a parallax information acquisition unit, a parallax compensation unit, a composite information generation unit, an image Regarding the specific processing of the image processing apparatus and the composition unit and the post-processing unit, the description given in “1. First embodiment (image processing apparatus)” applies to the present embodiment. Therefore, the description about these is abbreviate | omitted.

3. Third Embodiment (Image Processing Method)

(1) Description of the third embodiment

An image processing method according to the present technology includes: a first image acquisition step of acquiring a first color image by a first color image pickup device, wherein the color filter covering the light receiving surface includes neither a white area nor an unfiltered area; The color filter covering the light receiving surface includes a second image acquisition step of acquiring a second color image by a second color image pickup element including a white region or a non-filter region. As described above, the image processing method according to the present technology includes a color image imaging process using two different color image imaging elements.
Furthermore, the image processing method according to the present technology relates to a parallax information acquisition step of acquiring information about a parallax between the first color image and the second color image, and a parallax acquired in the parallax information acquisition step. Based on the information, the first color image is added to the second color image based on the combination information generation step for generating information on how to combine the two images and the combination information generated in the combination information generation step. And an image composition step for composition. As described above, the image processing method according to the present technology obtains disparity information between the two color images captured in the two color image capturing steps, generates information on how to combine based on the disparity information, Then, based on the information relating to the generated synthesis method, the two images are synthesized.

The effect described in “1. First embodiment (image processing apparatus)” is achieved by the image processing method according to the present technology. As an image processing apparatus for performing an image processing method according to the present technology, for example, described in “1. First embodiment (image processing apparatus)” or “2. Second embodiment (image processing apparatus)”. The following image processing apparatus may be used.

(2) First example of third embodiment (image processing method)

Hereinafter, an image processing method according to the present technology will be described with reference to FIGS. 3 and 10. FIG. "(2) First example of first embodiment (image processing apparatus)". FIG. 10 is a diagram illustrating an example of a flow of an image processing method according to the present technology.

In step S101, the image processing apparatus 300 starts image processing according to the present technology.

In step S102, the image processing apparatus 300 performs imaging with the first color image imaging element 301-1 and the second color image imaging element 301-2. Imaging with these imaging elements is preferably performed simultaneously. As a result of imaging by these imaging elements, a first color image (more particularly, first color image data) and a second color image (more particularly, second color image data) are acquired.

In step S103, the image processing apparatus 300 may perform preprocessing on each of the first color image and the second color image as necessary. More specifically, preprocessing can be performed by each of the preprocessing units 302-1 and 302-2. The pretreatment is performed in the above 1. 1 or two or more of the pre-processing examples described regarding the pre-processing units 302-1 and 302-2 in “(2) First example (image processing apparatus) of the first embodiment” Good. The preprocessing performed on the first color image and the second color image may be different processing or the same.

In step S104, the image processing apparatus 300 includes the first color image captured by the first color image imaging element 301-1 and the second color image captured by the second color image imaging element 301-2. Information on the parallax between. More specifically, acquisition of information related to parallax can be performed by the parallax information acquisition unit 303. For example, the above 1. As described in “(2) First example of first embodiment (image processing apparatus)”, the parallax information can be acquired by the parallax information acquisition unit 303. The information regarding the parallax can include parallax of each region in the image, for example, parallax for each pixel and / or parallax for each pixel block.
The information regarding the parallax may include a parallax map, for example. The parallax information acquisition unit 303 can acquire a parallax map indicating the parallax distribution of the first color image with respect to the second color image serving as a reference for synthesis, for example. The disparity information acquisition unit 303 distributes the disparity of the second color image with respect to the first color image in addition to the disparity map indicating the disparity distribution of the first color image with respect to the second color image that is a reference for combining. You may acquire the parallax map which shows.

In step S105, the image processing apparatus 300 can perform the parallax compensation of the first color image based on the information regarding the parallax acquired in step S104. The parallax compensation is performed, for example, in the above 1. As described in “(2) First example (image processing apparatus) of the first embodiment”, the parallax compensation unit 304 may perform this.
In step S105, the position of the pixel or the pixel block can be moved based on the information on the parallax with respect to the image data of the first color image. The movement of the position may be performed so that the parallax between the first color image and the second color image is eliminated, for example. In particular, each region (especially a pixel and / or pixel block) of the first color image can be moved to the position of each region in the second color image corresponding to the region. By the parallax compensation, a first color image with parallax compensation is generated. By the parallax compensation, each region of the first color image is moved to each corresponding region in the second color image, and as a result, appropriate image composition is possible.

In step S106, the image processing apparatus 300 generates information on how to synthesize the first color image and the second color image. More preferably, in step S106, the image processing apparatus 300 generates information regarding how to combine the first color image and the second color image that have been subjected to parallax compensation. For example, the above-described 1. As described in “(2) First example of first embodiment (image processing apparatus)” in FIG. More specific processing in step S106 will be separately described below with reference to FIG.

In step S107, the image processing apparatus 300 synthesizes the parallax-compensated first color image with the reference second color image based on the information regarding the synthesis method generated in step S106. The synthesis can be performed by, for example, the above 1. As described in “(2) First example of first embodiment (image processing apparatus)” in FIG. More specific processing of step S107 will be separately described below with reference to FIG.

In step S108, the image processing apparatus 300 performs post-processing on the image obtained as a result of the synthesis in step S107. The post-processing is, for example, the above-described 1. As described in “(2) First example of first embodiment (image processing apparatus)”, the post-processing unit 307 may perform the processing. More specific processing in step S108 will be separately described below with reference to FIG.

In step S109, the image processing apparatus 300 ends the image processing according to the present technology.

An example of the synthesis information generation process by the synthesis information generation unit in step S106 will be described below with reference to FIG. FIG. 11 is an example of the flow of the composite information generation process.

In step S201, the composite information generation unit 305 starts a composite information generation process.

In step S202, the composite information generation unit 305 detects a region outside the search range. The detection can be performed by the search area outside detection unit 601 in particular. In step S202, for example, information on the parallax acquired in step S104 can be used. More specific processing of the out-of-search area detection processing by the out-of-search area detection unit 601 will be separately described below with reference to FIG.

In step S203, the composite information generation unit 305 detects an occlusion area. This detection can be performed by the occlusion area detection unit 602 in particular. In step S203, for example, information on the parallax acquired in step S104 can be used. More specific processing of the occlusion region detection processing by the occlusion region detection unit 602 will be separately described below with reference to FIG.

In step S204, the composite information generation unit 305 calculates a value obtained by dividing the color difference dispersion value by the luminance dispersion value for each predetermined region in the first color image or the second color image, and the divided value. A region where is equal to or greater than a predetermined value is detected. The detection can be performed by the color difference / luminance determination unit 603 in particular. More specific processing of the color difference / luminance determination processing by the color difference / luminance determination unit 603 will be separately described below with reference to FIG.

In step S205, the synthesis information generation unit 305 determines whether each area in the image should be synthesized based on the processing results in steps S202 to S204. For example, for each region in the image, it is determined that the region detected in at least one of steps S202 to S204 should not be image-synthesized. The composite information generation unit 305 outputs the determination result as information regarding how to combine both images.

In step S206, the synthesis information generation unit 305 ends the synthesis information generation process.

<Out-search area detection processing>

An example of the out-of-search area detection processing by the out-of-search area detection unit 601 in step S202 will be described below with reference to FIG. FIG. 12 is an example of the flow of the search area outside detection process.

In step S301 in FIG. 12, the out-of-search area detection unit 601 starts out-of-search area detection processing.

In step S302, the out-of-search-range area detection unit 601 determines an attention area in the parallax map generated by the parallax information acquisition unit 303, for example.

In step S303, the out-of-search area detection unit 601 acquires the parallax value of the attention area.

In step S304, the out-of-search area detection unit 601 determines whether the acquired parallax value is equal to or greater than a predetermined threshold. When the acquired parallax value is equal to or greater than the predetermined threshold value, the search area outside detection unit 601 advances the process to step S305. When the acquired parallax value is less than the predetermined threshold value, the out-of-search-range area detection unit 601 advances the process to step S306.

In step S304, the out-of-search-range area detection unit 601 outputs an area in which the obtained parallax value is equal to or greater than a predetermined threshold to the comprehensive determination unit 604 as an out-of-search area area.

In step S306, the out-of-search area detection unit 601 determines whether there is an area in the disparity map where the out-of-search area detection processing is not performed. If there is an area where the process has not been performed, the search-outside area detection unit 601 returns the process to step S302, and performs a search-outside area detection process on the area. When there is no area where the process is not performed, the out-of-search area detection unit 601 advances the process to step S307.

In step S <b> 307, the out-of-search area detection unit 601 ends the out-of-search area detection.

An example of the occlusion area detection process performed by the occlusion area detection unit 602 in step S203 will be described below with reference to FIG. FIG. 13 is an example of the flow of the occlusion area detection process.

In step S401 in FIG. 13, the occlusion area detection unit 602 starts an occlusion area detection process.

In step S402, the occlusion area detection unit 602 moves each area of the first color image based on the first parallax map of the second color image with respect to the first color image.

In step S403, the occlusion area detection unit 602 moves each area of the first color image after the movement in step S402 again based on the second parallax map of the first color image with respect to the second color image.

In step S404, the occlusion area detection unit 602 detects an area that does not return to the position of the original first color image as an occlusion area. The occlusion area detection unit 602 outputs the detected occlusion area to the comprehensive determination unit 604.

In step S405, the occlusion area detection unit 602 ends the occlusion area detection process.

An example of color difference / luminance determination processing by the color difference / luminance determination unit 603 in step S204 will be described below with reference to FIG. FIG. 14 is an example of the flow of color difference / luminance determination processing.

In step S501 in FIG. 14, the color difference / luminance determination unit 603 starts the color difference / luminance determination process.

In step S502, the color difference / luminance determination unit 603 determines a region of interest in the second color image, for example.

In step S503, the color difference / luminance determination unit 603 calculates the value of the color difference variance / luminance variance of the region of interest.

In step S504, the color difference / luminance determination unit 603 determines whether the calculated color difference variance / luminance variance value is equal to or greater than a predetermined threshold. If the calculated value is equal to or greater than the predetermined threshold, the color difference / luminance determination unit 603 advances the process to step S505. If the calculated value is less than the predetermined threshold, the color difference / luminance determination unit 603 advances the process to step S506.

In step S505, the color difference / luminance determination unit 603 outputs an area where the calculated value is equal to or greater than a predetermined threshold to the general determination unit 604 as an area where image quality may be degraded.

In step S506, the color difference / luminance determination unit 603 determines whether there is an area in the second color image where the processing in steps S502 to S505 has not been performed. When there is an area where the process is not performed, the color difference / luminance determination unit 603 returns the process to step S502 and performs the color difference / brightness determination process on the area. If there is no area where the process is not performed, the color difference / luminance determination unit 603 advances the process to step S507.

In step S507, the color difference / luminance determination unit 603 ends the color difference / luminance determination processing.

An example of image composition processing by the image composition unit 306 in step S107 will be described below with reference to FIG. FIG. 15 is an example of the flow of image composition processing.

In step S601 in FIG. 15, the image composition unit 306 starts composition processing of the first color image and the second color image that have been subjected to parallax compensation.

In step S602, the image composition unit 306 determines a region of interest (for example, a pixel of interest or a pixel block of interest) in the second color image, for example.

In step S603, the image synthesis unit 306 refers to the synthesis information generated by the synthesis information generation unit 305.

In step S604, as a result of referring to the synthesis information generated by the synthesis information generation unit 305, the image synthesis unit 306 synthesizes the parallax-compensated first color image with the second color image in the region of interest. If so, the process advances to step S605. If not, the process proceeds to step S606.

In step S605, the image combining unit 306 combines the parallax-compensated first color image with the second color image in the region of interest. Regarding the region of interest, the image composition unit 306 outputs an image obtained by the composition.

In step S606, the image composition unit 306 outputs the second color image regarding the attention area.

In step S607, the image composition unit 306 determines whether there is an area in the second color image where the processing in steps S602 to S606 has not been performed. If there is an area where the process is not performed, the image composition unit 306 returns the process to step S602 and performs the image composition process on the area. If there is no area where the process is not performed, the image composition unit 306 advances the process to step S608.

In step S608, the image composition unit 306 ends the image composition process.

<Post-processing>

An example of post-processing by the post-processing unit 307 in step S108 will be described below with reference to FIG. FIG. 16 is an example of the post-processing flow.

In step S701 in FIG. 16, the post-processing unit 307 starts post-processing of the composite image obtained as a result of the synthesis by the image synthesis unit 306.

In step S702, the post-processing unit 307 determines a region of interest (for example, a pixel of interest or a pixel block of interest) in the composite image, for example.

In step S703, the post-processing unit 307 refers to the synthesis information generated by the synthesis information generation unit 305.

In step S <b> 704, the post-processing unit 307 combines the parallax-compensated first color image with the second color image as a result of referring to the synthesis information generated by the synthesis information generation unit 305. If so, the process advances to step S705. If not, the process proceeds to step S706.

In step S705, the post-processing unit 307 performs signal processing suitable for the synthesized image on the region of interest.

In step S706, the post-processing unit 307 performs signal processing suitable for an unsynthesized image on the region of interest.

In step S <b> 707, the post-processing unit 307 determines whether there is an area that has not been post-processed in the composite image. If there is an area that has not been post-processed, the post-processing unit 307 returns the process to step S702 and performs post-processing on the area. If there is no area where post-processing is not performed, the post-processing unit 307 advances the processing to step S708.

In step S708, the post-processing unit 307 ends the post-processing.

4). Fourth embodiment (image processing method)

In the image processing method according to the present technology, two color images are acquired using the first color image pickup device and the second color image pickup device having higher sensitivity than the first color image pickup device. . As described above, in the image processing method according to the present technology, two color images are acquired by using two color image pickup elements having different sensitivities.
Furthermore, the image processing method according to the present technology relates to a parallax between a first color image captured by the first color image imaging device and a second color image captured by the second color image imaging device. Generated in the disparity information acquisition step for acquiring information, the composite information generation step for generating information on how to combine both images based on the information on the disparity acquired in the disparity information acquisition step, and the composite information generation step And an image combining step of combining the first color image with the second color image based on the combined information. As described above, in the image processing method according to the present technology, parallax information between two color images captured by the two color image imaging elements is acquired, and information on how to combine is generated based on the parallax information. Then, two images are synthesized based on the information on the generated synthesis method.

By the image processing method, the effects described in “1. First embodiment (image processing apparatus)” are produced.
In addition, specific processing in each step in the image processing method may be, for example, as described in “3. Third embodiment (image processing method)”. Therefore, the description about the specific process in each process is abbreviate | omitted.

5. Device configuration example

Hereinafter, an example of the configuration of the image processing apparatus according to the present technology will be described with reference to FIG. FIG. 18 is a diagram illustrating an example of a schematic configuration of an image processing device according to the present technology.

An image processing apparatus 1001 shown in FIG. 18 includes a CPU (Central Processing Unit) 1002 and a RAM 1003. The CPU 1002 and the RAM 1003 are connected to each other via a bus 1005 and are also connected to other components of the image processing apparatus 1001 via the bus 1005. The image processing apparatus 1001 includes an imaging apparatus 1011. Similarly, the imaging device 1011 may be connected to other components. The image pickup apparatus 1011 includes the first color image pickup element and the second color image described in “1. First embodiment (image processing apparatus)” or “2. Second embodiment (image processing apparatus)”. Includes an image pickup device. The image processing method according to the present technology may be performed by the imaging device 1011.
A CPU 1002 performs control and calculation of the image processing apparatus 1001. An arbitrary processor can be used as the CPU 1002, and examples thereof include a Xeon (registered trademark) series, a Core (trademark) series, and an Atom (trademark) series processor. Image processing according to the present technology may be realized by the CPU 1002, for example.
The RAM 1003 includes, for example, a cache memory and a main memory, and can temporarily store a program used by the CPU 1002, for example, a program for causing the apparatus to execute an image processing method according to the present technology.

The image processing apparatus 1001 may include a disk 1004, a communication apparatus 1006, an output apparatus 1007, and a drive 1009. Any of these components can be connected to the bus 1005.
The disk 1004 includes an operating system (for example, WINDOWS (registered trademark), UNIX (registered trademark), or LINUX (registered trademark)), a program for executing an image processing method according to the present technology, and various data (for example, Image data) can be stored.
The communication device 1006 connects the image processing device 1001 to the network 1010 by wire or wireless. The communication apparatus 1006 can acquire various data (for example, image data) from the image processing apparatus 1001 via the network 1010. The acquired data can be stored in the disk 1004, for example. The type of the communication device 1006 may be appropriately selected by those skilled in the art.
The output device 1007 can output an image obtained by image processing according to the present technology. The output device 1007 may be a display, for example.
The input device 1008 is a device for the user to operate the image processing device 1001. For example, when the image processing apparatus 1001 is a smartphone, the display may function as the input apparatus 1008.
The drive 1009 can read information recorded on the recording medium and output the information to the RAM 1003. The recording medium is, for example, an SD memory card or a flash memory, but is not limited thereto.

With respect to the present technology described above, those skilled in the art will recognize that various modifications, combinations, sub-combinations, or alternatives may occur due to design requirements or other factors within the scope of the present technology or equivalents thereof. Understand what is possible.

In addition, this technique can also take the following structures.
[1] a first color image pickup device in which the color filter covering the light receiving surface does not include any of the white area and the non-filter area;
A color filter that covers the light receiving surface includes a white area or a non-filter area;
A parallax information acquisition unit that acquires information about parallax between the first color image captured by the first color image imaging device and the second color image captured by the second color image imaging device; ,
Based on information on parallax acquired by the parallax information acquisition unit, a synthesis information generation unit that generates information on how to combine both images;
An image combining unit that combines the first color image with the second color image based on the combination information generated by the combination information generation unit;
An image processing apparatus.
[2] The image processing apparatus according to [1], wherein the color filter included in the first color image pickup device is a Bayer array color filter.
[3] The image processing device according to [1] or [2], wherein the color filter included in the second color image pickup device includes a white region.
[4] The image processing apparatus according to any one of [1] to [3], wherein the color filter included in the second color image pickup device includes a region of a WRGB array or a WRG array.
[5] The parallax information acquisition unit acquires parallax for each pixel or each pixel block between the first color image and the second color image. Any one of [1] to [4] The image processing apparatus according to one.
[6] The image processing according to any one of [1] to [5], further including a parallax compensation unit that performs parallax compensation on the first color image based on the information on the parallax. apparatus.
[7] The composite information generation unit includes a search range outside region detection unit that detects a region having a disparity value larger than a predetermined search range in a disparity map created based on information on disparity. 6]. The image processing apparatus according to any one of [6].
[8] Any one of [1] to [7], wherein the composite information generation unit includes an occlusion area detection unit that detects an occlusion area by comparing the first color image and the second color image. The image processing apparatus according to one.
[9] The composite information generation unit calculates a value obtained by dividing the color difference dispersion value by the luminance dispersion value for each predetermined region in the first color image or the second color image, and the divided value is The image processing apparatus according to any one of [1] to [8], further including a color difference / luminance determination unit that detects an area that is equal to or greater than a predetermined value.
[10] The composite information generating unit
A region outside the search range that detects a pixel or a pixel block having a parallax value larger than a predetermined search range in a parallax map created based on information on parallax;
An occlusion area detecting unit for detecting an occlusion area by comparing the first color image and the second color image; and a color difference for each predetermined area in the first color image or the second color image. A color difference / luminance determination unit that calculates a value obtained by dividing the variance value by the luminance variance value and detects an area in which the divided value is equal to or greater than a predetermined value;
Including at least one component selected from
Generating combined information that the first color image should not be combined with the second color image for the area detected in any of the components;
[1] The image processing apparatus according to any one of [9].
[11] The composition information generation unit generates composition information indicating that the first color image should be synthesized or should not be synthesized for each region constituting the second color image. ] The image processing apparatus according to any one of [10] to [10].
[12] When the ratio of the area in the second color image that the first color image should not be synthesized exceeds a predetermined value, the synthesis information generation unit The image processing apparatus according to any one of [1] to [11], which generates synthesis information that the first color image should not be synthesized for all regions of the color image.
[13] Any one of [1] to [12], wherein the synthesis information generation unit assigns a synthesis rate for synthesizing the first color image for each region constituting the second color image. An image processing apparatus according to 1.
[14] a first color image pickup device;
A second color image pickup device having higher sensitivity than the first color image pickup device;
A parallax information acquisition unit that acquires information about parallax between the first color image captured by the first color image imaging device and the second color image captured by the second color image imaging device; ,
Based on information on parallax acquired by the parallax information acquisition unit, a synthesis information generation unit that generates information on how to combine both images;
An image combining unit that combines the first color image with the second color image based on the combination information generated by the combination information generation unit;
An image processing apparatus.
[15] a first image acquisition step of acquiring a first color image by a first color image pickup device, wherein the color filter covering the light receiving surface does not include any of the white region and the non-filter region;
A second image acquisition step of acquiring a second color image by a second color image pickup device, wherein the color filter covering the light receiving surface includes a white region or a non-filter region;
A parallax information obtaining step of obtaining information on parallax between the first color image and the second color image;
Based on information on parallax acquired in the parallax information acquisition step, a synthesis information generation step of generating information on how to combine both images;
An image synthesis step of synthesizing the first color image with the second color image based on the synthesis information generated in the synthesis information generation step;
An image processing method including:
[16] a first image acquisition step of acquiring a first color image by the first color image pickup device;
A second image acquisition step of acquiring a second color image by means of a second color image pickup device having higher sensitivity than the first color image pickup device;
A parallax information obtaining step of obtaining information on parallax between the first color image and the second color image;
Based on information on parallax acquired in the parallax information acquisition step, a synthesis information generation step of generating information on how to combine both images;
An image synthesis step of synthesizing the first color image with the second color image based on the synthesis information generated in the synthesis information generation step;
An image processing method including:

300 Image Processing Device 301-1 First Color Image Imaging Element 301-2 Second Color Image Imaging Element 302-1 Preprocessing Unit 302-2 Preprocessing Unit 303 Parallax Information Acquisition Unit 304 Parallax Compensation Unit 305 Composite Information Generation Unit 306 Image composition unit 307 Post-processing unit

Claims

A color filter covering the light-receiving surface, the first color image pickup element including neither a white area nor an area without a filter;
A color filter that covers the light receiving surface includes a white area or a non-filter area;
A parallax information acquisition unit that acquires information about parallax between the first color image captured by the first color image imaging device and the second color image captured by the second color image imaging device; ,
Based on information on parallax acquired by the parallax information acquisition unit, a synthesis information generation unit that generates information on how to combine both images;
An image combining unit that combines the first color image with the second color image based on the combination information generated by the combination information generation unit;
An image processing apparatus.
The image processing apparatus according to claim 1, wherein the color filter included in the first color image pickup device is a Bayer array color filter.
The image processing apparatus according to claim 1, wherein the color filter included in the second color image pickup device includes a white region.
The image processing apparatus according to claim 1, wherein the color filter included in the second color image pickup device includes a region of a WRGB array or a WRG array.
The image processing apparatus according to claim 1, wherein the parallax information acquisition unit acquires parallax for each pixel or each pixel block between the first color image and the second color image.
The image processing apparatus according to claim 1, further comprising a parallax compensation unit that performs parallax compensation on the first color image based on information on the parallax.
2. The image processing according to claim 1, wherein the composite information generation unit includes a search range outside region detection unit that detects a region having a parallax value larger than a predetermined search range in a parallax map created based on information on parallax. apparatus.
The image processing apparatus according to claim 1, wherein the composite information generation unit includes an occlusion region detection unit that detects an occlusion region by comparing the first color image and the second color image.
The composite information generation unit calculates a value obtained by dividing a color difference variance value by a luminance variance value for each predetermined region in the first color image or the second color image, and the divided value is a predetermined value. The image processing apparatus according to claim 1, further comprising a color difference / luminance determination unit that detects the region as described above.
The composite information generation unit
A region outside the search range that detects a pixel or a pixel block having a parallax value larger than a predetermined search range in a parallax map created based on information on parallax;
An occlusion area detecting unit for detecting an occlusion area by comparing the first color image and the second color image; and a color difference for each predetermined area in the first color image or the second color image. A color difference / luminance determination unit that calculates a value obtained by dividing the variance value by the luminance variance value and detects an area in which the divided value is equal to or greater than a predetermined value;
Including at least one component selected from
Generating combined information that the first color image should not be combined with the second color image for the area detected in any of the components;
The image processing apparatus according to claim 1.
The composition information generation unit generates composition information indicating that the first color image should be synthesized or should not be synthesized for each region constituting the second color image. Image processing apparatus.
When the ratio of the area in the second color image that should not be combined with the first color image exceeds a predetermined value, the combination information generation unit The image processing apparatus according to claim 1, wherein synthesis information is generated that the first color image should not be synthesized for all regions.
The image processing apparatus according to claim 1, wherein the synthesis information generation unit assigns a synthesis rate for synthesizing the first color image for each region constituting the second color image.
A first color image pickup device;
A second color image pickup device having higher sensitivity than the first color image pickup device;
A parallax information acquisition unit that acquires information about parallax between the first color image captured by the first color image imaging device and the second color image captured by the second color image imaging device; ,
Based on information on parallax acquired by the parallax information acquisition unit, a synthesis information generation unit that generates information on how to combine both images;
An image combining unit that combines the first color image with the second color image based on the combination information generated by the combination information generation unit;
An image processing apparatus.
A first image acquisition step of acquiring a first color image by a first color image pickup device, wherein the color filter covering the light receiving surface does not include any of the white region and the non-filter region;
A second image acquisition step of acquiring a second color image by a second color image pickup device, wherein the color filter covering the light receiving surface includes a white region or a non-filter region;
A parallax information obtaining step of obtaining information on parallax between the first color image and the second color image;
Based on information on parallax acquired in the parallax information acquisition step, a synthesis information generation step of generating information on how to combine both images;
An image synthesis step of synthesizing the first color image with the second color image based on the synthesis information generated in the synthesis information generation step;
An image processing method including:
A first image acquisition step of acquiring a first color image by the first color image pickup device;
A second image acquisition step of acquiring a second color image by means of a second color image pickup device having higher sensitivity than the first color image pickup device;
A parallax information obtaining step of obtaining information on parallax between the first color image and the second color image;
Based on information on parallax acquired in the parallax information acquisition step, a synthesis information generation step of generating information on how to combine both images;
An image synthesis step of synthesizing the first color image with the second color image based on the synthesis information generated in the synthesis information generation step;
An image processing method including: