JP2017103695A - Image processing apparatus, image processing method, and program of them - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus for forming composite image data by using a plurality of image data of which a field angle including a common subject is difference.SOLUTION: An image processing method comprises: selection means 402 of selecting second image data of which a field angle is narrower than first image data as a correction target image data from a plurality of image data when the first image data is as reference image data; division means of dividing the correction target image data from a plurality of regions; color matching means of matching a color to a corresponding region in the reference image data in each region obtained by dividing the correction target image data; and composite means 404 of composing the correction target image data to be matched the color to the reference image data, and forming composite image data. The color matching means includes: correction value calculation means 407 of calculating a correction value in each region of the correction target image data so that an average pixel value of the region becomes the average region value of the target region corresponding to the region; and correction means 409 of correcting the correction target image data on the basis of the correction value.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program thereof. Specifically, the present invention relates to an image processing apparatus, an image processing method, and a program thereof that generate image data having a desired angle of view based on captured image data captured by a plurality of imaging apparatuses having different angle of view.

  Conventionally, as a method for enlarging a desired area of a captured image, a technique (electronic zoom) that realizes zoom by image processing is known. In the electronic zoom, as the zoom magnification is increased, the enlargement magnification is also increased, and there is a problem that the resolution is gradually deteriorated. Thus, there is a technology that realizes zooming by image processing without using a plurality of cameras having different angles of view without impairing the resolution (see Patent Document 1). In Patent Document 1, a subject is imaged using an imaging device including a camera with a short focal length and a wide angle of view and a camera with a long focal length and a narrow angle of view. By synthesizing a wide-angle image with a wide angle of view and a telephoto image with a narrow angle of view at a predetermined ratio and combining them, image data with a desired virtual angle of view can be generated without impairing the resolution.

  However, in Patent Document 1, the color and brightness between captured images differ due to variations in camera characteristics and the influence of shading, and a pseudo contour (a step in color or brightness at the boundary) is added to the synthesized virtual field angle image. There is a problem that occurs.

  There is a method of making the pseudo contour inconspicuous by correcting the boundary portion of the synthesis with respect to the above-described problem of Patent Document 1 (see Patent Document 2). In Patent Document 2, the boundary portion is corrected by replacing the image data at the boundary portion of the synthesis with correction data that gradually approaches the telephoto image data from the wide-angle image data. That is, the level difference is made inconspicuous by smoothing the image value at the boundary.

JP 2014-225843 A Japanese Patent No. 4299561

  However, Patent Document 2 has a problem that if a complex texture is included in the boundary of synthesis, the correction process becomes complicated and the correction accuracy also decreases. Further, the correction process of Patent Document 2 is a process that makes the step inconspicuous by smoothing the image value at the boundary, but is not a process that substantially eliminates the step at the boundary. When the difference in color and brightness between captured images is large, there is a problem that it is not sufficient to correct the boundary as in Patent Document 2.

  Therefore, it is conceivable to perform color matching not only on the boundary but also on the entire image. However, in the generation of a virtual angle of view image, a telephoto image with a narrow angle of view is reduced and synthesized at a predetermined ratio in the center of a wide angle image with a wide angle of view. In addition to the step, a step due to shading also occurs. This is because when a high image height portion of a telephoto image that is greatly affected by shading is combined with a middle image height portion of a wide-angle image that is relatively less affected by shading, shading becomes more noticeable at the boundary of the composition. Such steps due to shading cannot be eliminated by color matching in the entire image.

  On the other hand, steps due to shading can be dealt with by performing shading correction so that the brightness of the entire image is uniform before color matching, but when such shading correction is performed, the correction amount increases as the image height increases. Increases and noise is amplified. As a result, even if the step due to shedding is eliminated, a step due to noise is newly generated.

  The present invention has been made in view of the above-described problems, and provides a pseudo contour of a synthesis boundary portion that is generated when a plurality of captured image data having different angles of view including a common subject are synthesized. The purpose is to suppress.

  An image processing apparatus according to the present invention is an image processing apparatus for generating composite image data using a plurality of pieces of image data having different angles of view including a common subject. And selecting means for selecting second image data having a smaller angle of view than the first image data as correction target image data, and dividing the correction target image data into a plurality of regions. A dividing unit; a color matching unit that performs color matching with a corresponding region in the reference image data for each region obtained by dividing the correction target image data; and the correction target image data after the color matching. Synthesizing with reference image data to generate the synthesized image data, and the color matching unit calculates a correction value for each area of the correction target image data as an average of the area. A correction value calculation unit that calculates an elementary value to be an average pixel value of the corresponding region corresponding to the region; and a correction unit that corrects the correction target image data based on the correction value. To do.

  The present invention has an effect of suppressing a pseudo contour at a boundary portion of synthesis that is generated when a plurality of captured image data having different angles of view including a common subject is synthesized.

1 is a diagram illustrating an example of a multi-lens imaging device including a plurality of imaging units according to Embodiment 1. FIG. 1 is a block diagram illustrating an example of an internal configuration of a multi-lens imaging device. The figure which shows an example of the internal structure of an imaging part. FIG. 2 is a block diagram illustrating a functional configuration according to the first embodiment. FIG. 6 is a diagram simply showing image composition according to the first embodiment. 5 is a flowchart of image processing according to the first embodiment. The figure for demonstrating the amount of parallax between image data. FIG. 6 is a diagram for explaining corresponding area calculation in the first embodiment. FIG. 10 is a diagram simply illustrating image composition according to the second embodiment. FIG. 10 is a diagram illustrating variations in the order of synthesis processing in the second embodiment.

Example 1
In this embodiment, a method for suppressing a pseudo contour at a boundary portion that occurs when a telephoto image is reduced and combined with a wide-angle image will be described.

  FIG. 1 is a diagram illustrating an example of a multi-eye imaging apparatus including a plurality of imaging units having different angles of view. As illustrated in FIG. 1A, the imaging device 100 includes two imaging units 101 and 102 that acquire color image data and an imaging button 103. As shown in FIG. 1B, the imaging units 101 and 102 are arranged so that their optical axes 104 and 105 are parallel to each other. In the present embodiment, the horizontal interval between the imaging units 101 and 102 is R, the horizontal field angle of the imaging unit 101 is θ1, and the horizontal field angle of the imaging unit 102 is θ2. Note that θ1> θ2 and that the imaging unit 101 has a wider angle of view than the imaging unit 102. When the user presses the imaging button 103, the imaging units 101 to 102 receive light information of the subject with a sensor (imaging device), the received signal is A / D converted, and a plurality of digital data (captured image data) is simultaneously obtained. can get. It is assumed that the distance L from the imaging device 100 to the subject derived based on the focus setting at the time of imaging is given to the captured image data as tag information.

  With such a multi-lens imaging device, it is possible to obtain a group of captured images obtained by imaging a common subject from a plurality of viewpoint positions. Although the number of imaging units is two here, the number of imaging units is not limited to two, and this embodiment can be applied regardless of the number of imaging units as long as the imaging device has a plurality of imaging units. is there.

  FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 100. A central processing unit (CPU) 201 generally controls each unit described below. The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The ROM 203 stores a control program executed by the CPU 201 and the like.

  The bus 204 serves as a transfer path for various data. For example, captured image data acquired by the imaging units 101 to 102 is sent to a predetermined processing unit via the bus 204. The operation unit 205 receives a user instruction. Specifically, a button, a mode dial, and the like are included, and an imaging instruction and a zoom instruction can be received. A display unit 206 displays captured images and characters. For example, a liquid crystal display is used. Further, the display unit 206 may have a touch screen function. In that case, a user instruction such as an imaging instruction or a zoom instruction using a touch screen can be handled as an input of the operation unit 205.

  A display control unit 207 performs display control of images and characters displayed on the display unit 206. The imaging control unit 208 controls the imaging unit based on an instruction from the CPU 201 such as focusing, opening / closing a shutter, and adjusting an aperture. The digital signal processing unit 209 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the captured image data received via the bus 204.

  The encoding unit 210 performs processing for converting captured image data into a file format such as JPEG or MPEG. The external memory control unit 211 is an interface for coupling the imaging apparatus 100 to an external memory 213 (for example, a PC, a hard disk, a memory card, a CF card, an SD card, a USB memory).

  The image processing unit 212 performs image processing such as image synthesis using the captured image data group acquired by the imaging units 101 and 102 or the captured image data group output from the digital signal processing unit 209.

  Although there are other components of the image pickup apparatus other than those described above, the description thereof is omitted because it is not the main point of the present embodiment.

  FIG. 3 is a diagram illustrating an internal configuration of the imaging unit 101. The imaging unit 101 includes lenses 301 to 302, a diaphragm 303, a shutter 304, an optical low-pass filter 305, an iR cut filter 306, a color filter 307, a sensor 308, and an A / D conversion unit 309. The lenses 301 to 302 are a focus lens 301 and a shake correction lens 302, respectively. The sensor 308 is, for example, a sensor such as a CMOS or a CCD, and detects the amount of light of the subject focused by each lens. The detected light amount is output from the sensor 308 as an analog value, converted into a digital value by the A / D conversion unit 309, and output to the bus 204 as digital data. Note that the imaging unit 102 can have the same configuration.

  FIG. 4 is a functional block diagram illustrating functional units provided in the image processing unit 212. As illustrated in FIG. 4, the image processing unit 212 includes a captured image data acquisition unit 400, a reference image data selection unit 401, a correction target image data selection unit 402, a parallax amount calculation unit 403, an image registration unit 404, and an area division unit. 405 and a corresponding area calculation unit 406. Further, a correction value calculation unit 407, a correction value shaping unit 408, a pixel value correction unit 409, and an image composition unit 410 are provided. Each functional unit is realized by the CPU 201 executing a control program.

  The captured image data acquisition unit 400 acquires a plurality of captured image data captured by the imaging units 101 and 102. Alternatively, the captured image data acquisition unit 400 may acquire captured image data recorded in advance in the RAM 202, the external memory 213, or the like. The captured image data acquisition unit includes an angle of view θ1 (or focal length f1) of the imaging unit 101, an angle of view θ2 (or focal length f2) of the imaging unit 102, and an interval R between the imaging unit 101 and the imaging unit 102. Imaging information is acquired from a storage device such as the ROM 203 or the external memory 213.

  FIG. 5 is a diagram for describing processing for reducing and combining a telephoto image with a wide-angle image for an image captured using a two-lens imaging device. A captured image 501 indicates a wide-angle image captured by the image capturing unit 101, and a captured image 502 indicates a telephoto image captured by the image capturing unit 102. An image 503 is a virtual angle-of-view image generated by pasting an image obtained by reducing the captured image 502 at a predetermined ratio in the center of the captured image 501, and the image 504 is subjected to the color matching process of the present embodiment. It is an image.

  In this embodiment, an example in which an image 503 having the same angle of view as the captured image 501 is described will be described. However, the present invention is not limited to this, and an image having a different angle of view may be generated. An example of performing image composition by pasting a captured image 502 reduced at a predetermined ratio in the center of the captured image 501 when the captured image 502 is combined with the captured image 501 will be described. That is, image synthesis is performed using 0 as the weight value of the captured image 501 and 1 as the weight value of the captured image 502. However, the present invention is not limited to this, and image synthesis may be performed using another weight.

  The reference image data selection unit 401 selects captured image data (hereinafter referred to as reference image data) serving as a color matching reference from among a plurality of captured image data acquired by the captured image data acquisition unit 400. Here, the data of the captured image 501 on the side where the image is pasted is selected as the reference image data. The correction target image data selection unit 402 selects captured image data (hereinafter, correction target image data) that is actually color-matched from among the plurality of captured image data acquired by the captured image data acquisition unit 400. Here, the data of the captured image 502 on the side where the image is to be pasted is selected as the correction target image data. The parallax amount calculation unit 403 calculates the parallax amount of the correction target image data with respect to the reference image data for each pixel. The image alignment unit 404 aligns the reference image data and the correction target image data, and determines a region for combining (pasting) the correction target image. The area dividing unit 405 divides the correction target image data into areas. The corresponding area calculation unit 406 calculates the area of the reference image data corresponding to each area of the divided correction target image data based on the parallax amount calculated by the parallax amount calculation unit 403. The correction value calculation unit 407 calculates a color correction value for each region of the correction target image data so that the average value of the color of the region of the correction target image data becomes the average value of the color of the corresponding reference image data region. To do. The correction value shaping unit 408 performs shaping so as to reduce the difference between the correction values between adjacent regions with respect to the correction value calculated by the correction value calculation unit 407. The pixel value correction unit 409 corrects the color of each area of the correction target image data using the correction value shaped by the correction value shaping unit 408. The image synthesis unit 410 synthesizes the correction target image data corrected by the pixel value correction unit 409 with the reference image data based on the region where the correction target image data determined by the image alignment unit 404 is synthesized.

  In the present embodiment, each functional unit described above is provided in the image processing unit 212 of the imaging device 100, but may be provided in an image processing device that can communicate with the imaging device 100.

  FIG. 6 is a flowchart of image processing according to the present embodiment.

  In step S601, the captured image data acquisition unit 400 acquires a plurality of captured image data. Also, imaging information including the angle of view θ1 (or focal length f1) of the imaging unit 101, the angle of view θ2 (or focal length f2) of the imaging unit 102, and the interval R between the imaging unit 101 and the imaging unit 102 is acquired.

  In step S602, the reference image data selection unit 401 selects the wide-angle image data on the side where the image is to be pasted as reference image data, and the correction target image data selection unit 402 corrects the telephoto image data on the side where the image is to be pasted. Select as target image data. Here, the captured image 501 is a reference image, and the captured image 502 is a correction target image.

  In step S603, the parallax amount calculation unit 403 calculates the parallax amount of the correction target image data with respect to the reference image data. The amount of parallax can be obtained by searching for corresponding points between the correction target image data and the reference image data.

  Specifically, first, the parallax amount calculation unit 403 sets the correction target image data to the same imaging magnification as that of the reference image data based on imaging information such as the angle of view (or focal length) of the imaging unit acquired in step S601. Reduce so that. For example, if the horizontal view angles θ1 and θ2 of the imaging units 101 and 102 shown in FIG. 1B are used, the horizontal reduction ratio is tan (θ2 / 2) / tan (θ1 / 2). Next, the parallax amount of the reduced correction target image data with respect to the reference image data is calculated for each pixel.

  FIG. 7 is a diagram for explaining the amount of parallax between image data.

  An object in the three-dimensional space can be simply represented by the camera model shown in FIG. The point P in the three-dimensional space observed from the viewpoint O is observed as a point p where the straight line connecting the viewpoint O and the point P intersects the projection plane I by perspective projection. f represents the distance from the lens center to the projection plane, that is, the focal length.

  Next, consider observing the same point P in the three-dimensional space with the two cameras CL and CR installed at the viewpoints OL and OR. As shown in FIG. 7B, the points observed from the two viewpoints OL and OR are projected as points pL and pR on the respective projection planes IL and IR by the camera model of FIG. 7A. At this time, straight lines connecting the respective viewpoints and the points projected on the projection plane intersect at a point P. The difference between the position of the point pL with respect to the point at which the viewpoint OL is projected onto the projection plane IL and the position of the point pR with respect to the point at which the viewpoint OR is projected onto the projection plane IR is the amount of parallax at the point P. If the positions of pL and pR can be obtained by corresponding point search, the amount of parallax can be calculated.

  In step S604, the image alignment unit 404 aligns the reference image data and the correction target image data. That is, an area for combining (pasting) the correction target image with the reference image is determined. Note that the corresponding point search is executed by the parallax amount calculation unit 403, and alignment can be performed using the result. In addition to this, a known technique such as SIFT (Scale Invariant Feature Transform) that calculates and aligns image feature points may be used. SIFT is described, for example, in US Pat. No. 6,711,293.

  In step S605, the area dividing unit 405 divides the correction target image data into areas. For the correction target image data, a color correction value is obtained for each divided area, and color matching with the reference image data is performed. The correction value is a value that matches the average value of the pixels in the region with the average value of the pixels in the corresponding region of the reference image data. If the area is too large, the difference in correction value from the adjacent area becomes large, and a new pseudo contour is generated between the areas. Therefore, it is desirable to divide the area with a small size.

  On the other hand, the corresponding region of the reference image data is calculated based on the parallax amount calculated by the parallax amount calculation unit 403, and the parallax amount calculated by the parallax amount calculation unit 403 often includes an error. A value obtained by converting this error into the number of pixels on the projection plane I is E. If the area is divided into a size smaller than E, the corresponding area error generated in the corresponding area calculation unit 406 is large. The error in the pixel value becomes large. Therefore, it is desirable to determine an appropriate region size after estimating the maximum parallax amount error Emax generated in the parallax amount calculation unit 403 in advance.

  In step S606, the corresponding area calculation unit 406 calculates the corresponding area of the reference image data corresponding to each area of the correction target image data divided in step S605 based on the parallax amount calculated in step S603.

  FIG. 8 is a diagram for explaining the corresponding area calculation in the present embodiment. The captured image 501 is a reference image, and the captured image 502 is a correction target image. A region 801 indicates a region where the captured image 502 is pasted as a result of the alignment performed by the image alignment unit 404. The parallax map 802 is obtained by mapping the parallax amount for each pixel between the captured image 501 and the captured image 502 calculated in step S603. In the parallax map 802, the parallax amount is expressed such that the whiter the pixel, the smaller the parallax amount, and the blacker the pixel, the larger the parallax amount. Hereinafter, a procedure for obtaining the coordinate values of the areas 807 and 808 of the captured image 501 corresponding to the areas 803 and 804 of the captured image 502 will be described.

  An area 805 is an area on the parallax map 802 corresponding to the area 803. The average amount of parallax in the area 805 is obtained, and the value is set to D1. The positional relationship of the imaging device is the positional relationship between the imaging unit 101 and the imaging unit 102 in FIG. That is, the imaging unit 101 and the imaging unit 102 are located on the same horizontal line. For this reason, the coordinate value of the area 807 is obtained as a place shifted by D1 in the −x direction from the area 803.

  Similarly, the region 806 is a region on the parallax map 802 corresponding to the region 804. When the average amount of parallax in the region 806 is obtained and the value is set to D2, the coordinate value of the region 808 is obtained as a location shifted by D2 in the −x direction from the region 804.

  Note that the amount of parallax indicated by each pixel on the parallax map 802 is determined by the distance from the point in the three-dimensional space corresponding to that pixel to the projection plane of the imaging apparatus. That is, the closer the distance, the greater the amount of parallax. A region 806 that is closer than the region 805 has a relatively large average amount of parallax. Therefore, D1 is greater than D2.

  As described above, in step S606, the corresponding area of the reference image data corresponding to each area of the correction target image data is calculated based on the average parallax amount in the area, thereby efficiently matching the accurate corresponding area. Can be obtained.

  In step S607, the correction value calculation unit 407 calculates a color correction value for each area from the area of the correction target image data divided in step S605 and the corresponding area of the reference image data calculated in step S606.

  The correction value can be obtained as a difference value between the average value of the pixels in the region divided by the region dividing unit 405 and the average value of the pixels in the corresponding region obtained by the corresponding region calculating unit 406. For example, it is assumed that the average pixel value (average luminance value) of the region 803 is R = 113, G = 78, and B = 52. Further, it is assumed that the average pixel value (average luminance value) of the region 807 is R = 104, G = 67, and B = 45. In that case, the correction values are R = −9, G = −11, and B = −7.

  Further, the correction value can be obtained not as a difference value but as a ratio. In that case, the correction values are R = 104/113, G = 67/78, and B = 45/52.

  In step S608, it is determined whether correction value calculation has been completed in all regions of the correction target image data divided in step S805. If completed, the process proceeds to step S809. If not completed, the process returns to step S806. Through the processes in steps S606 to S608, correction values for all the regions divided in step S605 are calculated.

  In step S609, the correction value shaping unit 408 shapes the correction value calculated in step S608. As for the correction value, if the size of the divided region is large and the difference in the correction value between adjacent regions is large, a pseudo contour is generated at the boundary of the region. In order to suppress this, it is desirable to reduce the size of the divided region. However, if the error is smaller than the error Emax in the parallax amount calculation unit 403, the error of the corrected pixel value becomes large. In order to solve this problem, the correction value shaping unit 408 applies a Gaussian filter to the entire correction value, and performs shaping so as to reduce the difference in correction values between adjacent regions. However, if the filter is applied too strongly, the correction values are averaged and the error of the corrected pixel value becomes large. Therefore, it is desirable to set an appropriate Gaussian filter according to the size of the region.

  In step S610, the pixel value correction unit 409 corrects the pixel value of the correction target image data for each divided area based on the correction value shaped in step S609. When the correction value is obtained as the difference value in step S607, the correction is performed by adding the correction value of the corresponding area to the pixel value of the correction target image data. When the correction value is obtained as a ratio in step S607, the correction is performed by multiplying the correction value.

  In step S611, the image composition unit 410 synthesizes the correction target image data corrected in step S610 with the reference image data based on the region where the correction target image data determined by the image alignment unit 404 is synthesized (pasted). To generate composite image data.

  As described above, the telephoto image data picked up by the image pickup unit 102 is divided into a plurality of regions, and color matching with the corresponding region of the wide-angle image data picked up by the image pickup unit 101 is performed for each divided region. That is, the high image height area of the telephoto image data that is color-matched between the captured images and that is greatly affected by shading is colored in the corresponding area of the medium image height area of the wide-angle image data that is relatively less affected by shading. It is corrected to match. As a result, it is possible to correct a color and brightness level difference at the boundary of synthesis including a level difference due to shading that occurs during image synthesis. As a result, it is possible to suppress a pseudo contour at the boundary of synthesis.

(Example 2)
In the present embodiment, a method for suppressing a step of color and brightness at a boundary when imaging is performed using an imaging apparatus having three or more eyes and a plurality of images whose field angles are sequentially narrowed is described.

  FIG. 9 is a diagram for explaining a process of synthesizing three images whose angles of view are sequentially narrowed for an image captured using a three-lens imaging device. An image 901 represents an image captured by an imaging device having a wide-angle focal length, an image 902 represents an image captured by an imaging device having a standard focal length, and an image 903 represents an image captured by an imaging device having a telephoto focal length. An image 904 is a virtual angle-of-view image obtained by combining the images 901 to 903, and an image 905 is an image that has been subjected to the color matching process of the present embodiment.

  FIG. 10 is a diagram illustrating variations in the composition processing order when the images 901 to 903 are composited.

  In FIG. 10A, first, an image 1001 is generated by performing image synthesis using an image 902 having a standard angle of view as a reference image and a telephoto image 903 as a correction target image. Next, an image 1002 is obtained by performing image composition using the wide-angle image 901 as a reference image and the image 1001 as a correction target image. In this procedure, the telephoto image 903 is corrected twice when the image 1001 is generated and when the image 1002 is generated. As the number of corrections increases, the distance from the original image increases. Therefore, it is not desirable to perform correction according to the procedure of FIG.

  In FIG. 10B, first, an image 1003 is obtained by performing image composition using the wide-angle image 901 as a reference image and the telephoto image 903 as a correction target image. Next, an image 1004 is obtained by performing image composition using the image 1003 as a reference image and the image 902 having a standard angle of view as a correction target image. In this procedure, when the image 1003 is generated, an error is likely to occur in the parallax amount calculation unit 403 because the resolution of the reference image and the correction target image are greatly different. As a result, the maximum error Emax of the parallax amount increases, and the area size in the area dividing unit 405 has to be increased. Therefore, it is not desirable to perform the correction in the procedure of FIG.

  FIG. 10C is a diagram showing the order of synthesis processing in this embodiment. In FIG. 10C, first, an image 1005 is obtained by performing image composition using the wide-angle image 901 as a reference image and the image 902 having a standard angle of view as a correction target image. Next, an image 1006 is obtained by performing image composition (second image composition) using the image 1005 as a reference image and the telephoto image 903 as a correction target image. In the procedure of this embodiment shown in FIG. 10C, there is no problem that the same image is corrected a plurality of times in the case of the procedure of FIG. Further, the problem of greatly different resolution does not occur in the procedure of FIG. As described above, a high-quality composite image can be obtained while suppressing the step of color and brightness at the boundary of composition according to the composition processing order of the present embodiment.

  In the present embodiment, the synthesis processing order in the three-lens configuration has been described, but the present invention can also be applied to a configuration with three or more eyes. Specifically, the reference image data selection unit 401 selects the focal length in order from the image captured by the imaging unit on the wide angle side, and the correction target image data selection unit 402 selects the image on the telephoto side next to the reference image data. The composition processing order may be set so as to select.

  As described above, even when an imaging apparatus having three or more eyes is used, a pseudo contour at the boundary can be suppressed.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (13)

An image processing apparatus for generating composite image data using a plurality of image data having different angles of view including a common subject,
Selection means for selecting, from the plurality of image data, first image data as reference image data, and second image data having a smaller angle of view than the first image data as correction target image data;
Dividing means for dividing the correction target image data into a plurality of regions;
Color matching means for performing color matching with a corresponding area in the reference image data for each area obtained by dividing the correction target image data;
Combining the correction target image data after the color matching with the reference image data, and generating the composite image data, and
The color matching means includes
Correction value calculation means for calculating a correction value for each area of the correction target image data so that an average pixel value of the area becomes an average pixel value of the corresponding area corresponding to the area;
An image processing apparatus comprising: correction means for correcting the correction target image data based on the correction value.   The image processing apparatus according to claim 1, wherein the correction value is a difference between an average pixel value of the area and an average pixel value of the corresponding area.   The image processing apparatus according to claim 1, wherein the correction value is a ratio of an average pixel value of the corresponding area to an average pixel value of the area. The color matching unit further includes a shaping unit that shapes the correction value calculated by the correction value calculation unit so that a difference in correction values between adjacent regions is small,
The image processing apparatus according to claim 1, wherein the correction unit corrects the correction target image data based on a correction value shaped by the shaping unit. Parallax amount calculating means for calculating a parallax amount of the correction target image data with respect to the reference image data for each pixel;
The image processing apparatus according to claim 1, further comprising: a corresponding area calculation unit that calculates the corresponding area based on an average value of the parallax amounts in the area.   The image processing apparatus according to claim 5, wherein the dividing unit divides the correction target image data by a size determined based on an error in the parallax amount calculation of the parallax amount calculating unit.   The image processing apparatus according to claim 1, wherein the first image data is wide-angle image data, and the second image data is telephoto image data.   The selection means uses the composite image data as reference image data in the second image composition, and uses third image data having a narrower angle of view than the second image data as correction target image data in the second image composition. The image processing apparatus according to claim 1, wherein the image processing apparatus is selected.   9. The first image data is wide-angle image data, the second image data is image data with a standard angle of view, and the third image data is telephoto image data. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the pixel value is a luminance value.   11. The image processing apparatus according to claim 1, wherein the plurality of pieces of image data are captured image data obtained by simultaneously capturing the common subject using image capturing units having different angles of view. An image processing method for generating composite image data using a plurality of image data having different angles of view including a common subject,
A selection step of selecting, from the plurality of image data, first image data as reference image data, and second image data having a smaller angle of view than the first image data as correction target image data;
A division step of dividing the correction target image data into a plurality of regions;
For each region obtained by dividing the correction target image data, a color matching step for performing color matching with a corresponding region in the reference image data;
Combining the correction target image data after the color matching with the reference image data to generate the composite image data, and
The color matching step includes
A correction value calculating step for calculating a correction value for each area of the correction target image data so that an average pixel value of the area becomes an average pixel value of the corresponding area corresponding to the area;
A correction step of correcting the correction target image data based on the correction value.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 11.