JP6732440B2 - Image processing apparatus, image processing method, and program thereof - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program thereof. Specifically, the present invention relates to an image processing device, an image processing method, and a program thereof that generate image data of a desired angle of view based on imaged image data taken by a plurality of image pickup devices having different angles of view.

Conventionally, as a method of enlarging a desired region of a captured image, a technique (electronic zoom) for realizing zoom by image processing is known. In the electronic zoom, as the zoom magnification is increased, the enlargement magnification is also increased, and there is a problem that the sense of resolution is gradually deteriorated. Therefore, there is a technique that realizes zooming by image processing without impairing the sense of resolution by using a plurality of cameras having different angles of view (see Patent Document 1). In Patent Document 1, a subject is imaged using an imaging device including a camera having a short focal length and a wide angle of view and a camera having a long focal length and a narrow angle of view. By compressing a wide-angle image having a wide angle of view and a telephoto image having a narrow angle of view at a predetermined ratio and combining them, image data having a desired virtual angle of view can be generated without impairing the sense of resolution.

However, in Patent Document 1, the color and brightness between captured images differ due to variations in camera characteristics and the effect of shading, and a pseudo contour (step of color and brightness at the boundary) in the combined virtual angle-of-view image. There is a problem that occurs.

In order to solve the problem of Patent Document 1 described above, there is a method of making a pseudo contour inconspicuous by correcting the boundary portion of synthesis (see Patent Document 2). In Patent Document 2, the boundary portion is corrected by replacing the image data of the combined boundary portion with correction data that gradually approaches the wide-angle image data to the telephoto image data. That is, the step is made inconspicuous by smoothing the image value at the boundary portion.

JP, 2014-225843, A Japanese Patent No. 4299561

However, Patent Document 2 has a problem that the correction process becomes complicated and the correction accuracy also deteriorates when a complicated boundary includes a complicated texture. Further, the correction process of Patent Document 2 is a process of making the image value at the boundary portion smooth so as to make the step inconspicuous, but is not a process of substantially eliminating the step at the boundary portion. When the difference in color or brightness between captured images is large, there is also a problem that it is not sufficient to correct the boundary portion as in Patent Document 2.

Therefore, it is conceivable to perform color matching not only on the boundary but on the entire image. However, in the generation of the virtual angle-of-view image, the telephoto image with a narrow angle of view is reduced and combined at the center of the wide-angle image with a wide angle of view at a predetermined ratio. In addition to steps, steps due to shading also occur. This is because if the high image height portion of a telephoto image that is greatly affected by shading is combined with the middle image height portion of a wide-angle image that is relatively unaffected by shading, the shading is more noticeable at the boundary portion of the combination. Such a step due to shading cannot be eliminated by color matching of the entire image.

On the other hand, a step due to shading can be dealt with by performing shading correction so that the brightness of the entire image becomes uniform before color matching. However, when such shading correction is performed, the correction amount increases as the image height becomes higher. Becomes larger and the noise is amplified. As a result, even if the step due to shedding is eliminated, a step due to noise is newly generated.

The present invention has been made in view of the above-described problems, and a pseudo contour of a boundary portion of synthesis, which is generated when synthesizing a plurality of captured image data having a different field angle including a common subject, is generated. The purpose is to suppress.

An image processing apparatus according to the present invention is an image processing apparatus for generating composite image data by using a plurality of image data including a common subject and having different angles of view. Selecting the second image data having a narrower angle of view as the correction target image data than the first image data as reference image data, and dividing the correction target image data into a plurality of regions. Dividing means, deriving means for deriving the parallax amount of the correction target image data with respect to the reference image data, and referring to the parallax amount for each area obtained by dividing the correction target image data, to obtain the parallax amount. a color matching means for performing color matching with the corresponding region in the reference image data corresponding, the correction target image data after the color matching in the reference image data to determine a combined area for synthesizing the reference image data, And a combining unit that generates the combined image data by combining the correction target image data with the combination region in the reference image data. The color matching unit corrects each region of the correction target image data. Correction value calculating means for calculating a value so that the average pixel value of the area becomes the average pixel value of the corresponding area corresponding to the area, and correction means for correcting the correction target image data based on the correction value And having.

INDUSTRIAL APPLICABILITY The present invention has an effect of suppressing a pseudo contour of a boundary portion of combining, which occurs when combining a plurality of pieces of captured image data including a common subject and having different angles of view.

FIG. 3 is a diagram illustrating an example of a multi-lens imaging device including a plurality of imaging units according to the first embodiment. FIG. 3 is a block diagram showing an example of an internal configuration of a multi-lens imaging device. The figure which shows an example of an internal structure of an imaging part. 3 is a block diagram showing a functional configuration according to the first embodiment. FIG. FIG. 3 is a diagram simply showing image composition according to the first embodiment. 3 is a flowchart of image processing according to the first embodiment. The figure for demonstrating the amount of parallax between image data. FIG. 6 is a diagram for explaining a corresponding area calculation in the first embodiment. FIG. 9 is a diagram simply showing image synthesis according to the second embodiment. FIG. 9 is a diagram showing a variation of a combination processing order according to the second embodiment.

(Example 1)
In the present embodiment, a method of suppressing a pseudo contour of a boundary portion that occurs when a telescopic image is reduced and combined with a wide-angle image will be described.

FIG. 1 is a diagram showing an example of a multi-lens system imaging device including a plurality of imaging units having different angles of view. As shown in FIG. 1A, the image capturing apparatus 100 includes two image capturing units 101 and 102 that acquire color image data and an image capturing button 103. As shown in FIG. 1B, the image pickup units 101 and 102 are arranged such that their optical axes 104 and 105 are parallel to each other. In this embodiment, the horizontal interval between the image pickup units 101 and 102 is R, the horizontal angle of view of the image pickup unit 101 is θ1, and the horizontal angle of view of the image pickup unit 102 is θ2. Note that θ1>θ2, and the image pickup unit 101 has a wider field angle than the image pickup unit 102. When the user presses the image capturing button 103, the image capturing units 101 to 102 receive the optical information of the subject by the sensor (image capturing element), the received signal is A/D converted, and a plurality of digital data (captured image data) are simultaneously obtained. can get. It is assumed that the captured image data is provided with the distance L from the image capturing apparatus 100 to the subject, which is derived based on the focus setting at the time of capturing, as tag information.

With such a multi-lens type imaging device, it is possible to obtain a captured image group in which a common subject is captured from a plurality of viewpoint positions. Although the number of image capturing units is two here, the number of image capturing units is not limited to two, and as long as the image capturing apparatus has a plurality of image capturing units, this embodiment is applicable regardless of the number. is there.

FIG. 2 is a block diagram showing an internal configuration of the image pickup apparatus 100. A central processing unit (CPU) 201 centrally controls each unit described below. The RAM 202 functions as a main memory, a work area, etc. of the CPU 201. The ROM 203 stores a control program executed by the CPU 201 and the like.

The bus 204 serves as a transfer path for various data. For example, the captured image data acquired by the image capturing units 101 to 102 is sent to a predetermined processing unit via the bus 204. The operation unit 205 receives a user instruction. Specifically, it includes a button, a mode dial, and the like, and can receive an imaging instruction and a zoom instruction. The display unit 206 displays a captured image and characters. For example, a liquid crystal display is used. Further, the display unit 206 may have a touch screen function. In that case, a user instruction such as an imaging instruction or a zoom instruction using the touch screen can be treated as an input of the operation unit 205.

The display control unit 207 controls the display of images and characters displayed on the display unit 206. The imaging control unit 208 controls the imaging unit based on an instruction from the CPU 201, such as focusing, opening/closing a shutter, and adjusting a diaphragm. The digital signal processing unit 209 performs various types of processing such as white balance processing, gamma processing, and noise reduction processing on the captured image data received via the bus 204.

The encoding unit 210 performs processing of converting captured image data into a file format such as JPEG or MPEG. The external memory control unit 211 is an interface for connecting the imaging device 100 to an external memory 213 (for example, a PC, a hard disk, a memory card, a CF card, an SD card, a USB memory).

The image processing unit 212 performs image processing such as image composition using the captured image data group acquired by the image capturing units 101 and 102 or the captured image data group output from the digital signal processing unit 209.

It should be noted that although there are components other than those described above in the image pickup apparatus, they are not the main subject of the present embodiment, and a description thereof will be omitted.

FIG. 3 is a diagram showing an internal configuration of the image pickup unit 101. The imaging unit 101 includes lenses 301 to 302, a diaphragm 303, a shutter 304, an optical low pass filter 305, an iR cut filter 306, a color filter 307, a sensor 308, and an A/D conversion unit 309. The lenses 301 to 302 are a focus lens 301 and a blur correction lens 302, respectively. The sensor 308 is, for example, a sensor such as a CMOS or CCD, and detects the light amount of the subject focused by each of the above lenses. The detected light amount is output from the sensor 308 as an analog value, converted into a digital value by the A/D conversion unit 309, and output as digital data to the bus 204. The image pickup unit 102 may have the same configuration.

FIG. 4 is a functional block diagram showing functional units included in the image processing unit 212. As shown in FIG. 4, the image processing unit 212 includes a captured image data acquisition unit 400, a reference image data selection unit 401, a correction target image data selection unit 402, a parallax amount calculation unit 403, an image registration unit 404, and a region division unit. 405 and the corresponding area|region calculation part 406 are provided. Further, it includes a correction value calculation unit 407, a correction value shaping unit 408, a pixel value correction unit 409, and an image synthesis unit 410. Each functional unit is realized by the CPU 201 executing a control program.

The captured image data acquisition unit 400 acquires a plurality of captured image data captured by the image capturing units 101 and 102. Alternatively, the captured image data acquisition unit 400 may acquire captured image data previously recorded in the RAM 202, the external memory 213, or the like. Further, the captured image data acquisition unit includes an angle of view θ1 (or focal length f1) of the image capturing unit 101, an angle of view θ2 (or focal length f2) of the image capturing unit 102, and a distance R between the image capturing units 101 and 102. Imaging information is acquired from a storage device such as the ROM 203 or the external memory 213.

FIG. 5 is a diagram for explaining a process of reducing and synthesizing a wide-angle image with a telephoto image of an image captured by using a twin-lens imaging device. A captured image 501 is a wide-angle image captured by the image capturing unit 101, and a captured image 502 is a telephoto image captured by the image capturing unit 102. The image 503 is a virtual angle-of-view image generated by pasting an image obtained by reducing the captured image 502 at a predetermined ratio in the center of the captured image 501, and the image 504 is subjected to the color matching processing of this embodiment. It is an image.

In the present embodiment, an example in which the image 503 having the same angle of view as the angle of view of the captured image 501 is generated will be described, but the present invention is not limited to this, and an image with another angle of view may be generated. Further, an example will be described in which, when the captured image 502 is synthesized with the captured image 501, the captured image 502 reduced in a predetermined ratio is pasted to the center of the captured image 501 to perform image synthesis. That is, image composition is performed using 0 as the weight value of the captured image 501 and 1 as the weight value of the captured image 502. However, the present invention is not limited to this, and image combining may be performed using another weight.

The reference image data selection unit 401 selects the captured image data (hereinafter referred to as reference image data) that serves as a reference for color matching from the plurality of captured image data acquired by the captured image data acquisition unit 400. Here, the data of the captured image 501 on the side where the image is pasted is selected as the reference image data. The correction target image data selection unit 402 selects, from among the plurality of captured image data acquired by the captured image data acquisition unit 400, captured image data for which color matching is actually performed (hereinafter, correction target image data). Here, the data of the captured image 502 on the side where the image is pasted is selected as the correction target image data. The parallax amount calculation unit 403 calculates the parallax amount of the correction target image data with respect to the reference image data for each pixel. The image alignment unit 404 aligns the reference image data and the correction target image data, and determines a region where the correction target image is combined (pasted). The area dividing unit 405 divides the correction target image data into areas. The corresponding area calculation unit 406 calculates the area of the reference image data corresponding to each area of the divided correction target image data, based on the parallax amount calculated by the parallax amount calculation unit 403. The correction value calculation unit 407 calculates the correction value of the color for each area of the correction target image data so that the average value of the colors of the area of the correction target image data becomes the average value of the colors of the corresponding reference image data. To do. The correction value shaping unit 408 performs shaping to reduce the difference between the correction values calculated by the correction value calculation unit 407 and the correction values between adjacent regions. The pixel value correction unit 409 corrects the color of each region of the correction target image data using the correction value shaped by the correction value shaping unit 408. The image combining unit 410 combines the correction target image data corrected by the pixel value correction unit 409 with the reference image data, based on the area for combining the correction target image data determined by the image alignment unit 404.

In addition, in the present embodiment, each of the functional units described above is included in the image processing unit 212 of the image capturing apparatus 100, but may be included in an image processing apparatus capable of communicating with the image capturing apparatus 100.

FIG. 6 is a flowchart of the image processing according to this embodiment.

In step S601, the captured image data acquisition unit 400 acquires a plurality of captured image data. In addition, imaging information including the angle of view θ1 (or the focal length f1) of the imaging unit 101, the angle of view θ2 (or the focal length f2) of the imaging unit 102, and the interval R between the imaging units 101 and 102 is also acquired.

In step S602, the reference image data selection unit 401 selects the wide-angle image data on the image pasting side as the reference image data, and the correction target image data selection unit 402 corrects the telephoto image data on the image pasting side. Select as target image data. Here, the captured image 501 is a reference image, and the captured image 502 is a correction target image.

In step S603, the parallax amount calculation unit 403 calculates the parallax amount of the correction target image data with respect to the reference image data. The parallax amount can be obtained by performing a corresponding point search between the correction target image data and the reference image data.

Specifically, first, the parallax amount calculation unit 403 sets the correction target image data to the same image pickup magnification as the reference image data based on the image pickup information such as the angle of view (or focal length) of the image pickup unit acquired in step S601. To reduce. For example, when represented using the horizontal field angles θ1 and θ2 of the imaging units 101 and 102 shown in FIG. 1B, the reduction ratio in the horizontal direction is tan(θ2/2)/tan(θ1/2). Next, the parallax amount of the reduced correction target image data with respect to the reference image data is calculated for each pixel.

FIG. 7 is a diagram for explaining the parallax amount between image data.

An object in the three-dimensional space can be simply represented by the camera model shown in FIG. The point P in the three-dimensional space observed from the viewpoint O is observed by perspective projection as a point p where the straight line connecting the viewpoint O and the point P intersects the projection plane I. f represents the distance from the lens center to the projection surface, that is, the focal length.

Next, consider observing the same point P in the three-dimensional space by the two cameras CL and CR installed at the viewpoints OL and OR. As shown in FIG. 7B, the points observed from the two viewpoints OL and OR are projected as points pL and pR on the respective projection planes IL and IR by the camera model of FIG. 7A. At this time, a straight line connecting each viewpoint and the point projected on the projection plane intersects at a point P. The difference between the position of the point pL with respect to the point where the viewpoint OL is projected on the projection surface IL and the position of the point pR with respect to the point where the viewpoint OR is projected on the projection surface IR is the parallax amount of the point P. If the positions of pL and pR can be obtained by the corresponding point search, the parallax amount can be calculated.

In step S604, the image alignment unit 404 aligns the reference image data with the correction target image data. That is, the area in which the correction target image is combined (pasted) with the reference image is determined. It should be noted that the parallax amount calculation unit 403 executes the corresponding point search, and the result can be used to perform the alignment. Other than that, a known technique such as SIFT (Scale lnvariant Feature Transform) for calculating a feature point of an image and performing alignment may be used. SIFT is described in, for example, US Pat. No. 6,711,293.

In step S605, the area dividing unit 405 divides the correction target image data into areas. For the correction target image data, a color correction value is obtained for each divided area, and color matching with the reference image data is performed. The correction value is a value that matches the average value of the pixels in the area with the average value of the pixels in the corresponding area of the reference image data. If the areas are too large, the difference in correction value between the adjacent areas becomes large, and a new pseudo contour is generated between the areas. Therefore, it is desirable to divide the region into smaller sizes.

On the other hand, the corresponding area of the reference image data is calculated based on the parallax amount calculated by the parallax amount calculation unit 403, and the parallax amount calculated by the parallax amount calculation unit 403 often includes an error. If a value obtained by converting this error into the number of pixels on the projection plane I is E and the area is divided into smaller areas than E, the error in the corresponding area generated by the corresponding area calculation unit 406 is large. The error in the pixel value becomes large. Therefore, it is desirable to determine an appropriate area size after estimating the maximum error Emax of the parallax amount generated in the parallax amount calculation unit 403 in advance.

In step S606, the corresponding area calculation unit 406 calculates the corresponding area of the reference image data corresponding to each area of the correction target image data divided in step S605, based on the parallax amount calculated in step S603.

FIG. 8 is a diagram for explaining the calculation of the corresponding area in this embodiment. The captured image 501 is a reference image, and the captured image 502 is a correction target image. A region 801 indicates a region where the captured image 502 is pasted, which is the result of the alignment performed by the image alignment unit 404. The parallax map 802 is a map of the parallax amount for each pixel between the captured image 501 and the captured image 502 calculated in step S603. In the parallax map 802, the parallax amount is expressed such that the whiter the pixel, the smaller the parallax amount, and the darker the pixel, the greater the parallax amount. Hereinafter, a procedure for obtaining the coordinate values of the regions 807 and 808 of the captured image 501 corresponding to the regions 803 and 804 of the captured image 502 will be described.

The area 805 is an area on the parallax map 802 corresponding to the area 803. The average parallax amount in the area 805 is calculated, and the value is set as D1. The positional relationship between the image pickup devices is the positional relationship between the image pickup unit 101 and the image pickup unit 102 in FIG. That is, the image capturing unit 101 and the image capturing unit 102 are located on the same horizontal line. Therefore, the coordinate value of the area 807 is obtained as a location displaced from the area 803 by D1 in the −x direction.

Similarly, the area 806 is an area on the parallax map 802 corresponding to the area 804. When the average parallax amount in the area 806 is obtained and the value is set to D2, the coordinate value of the area 808 is obtained as a location shifted from the area 804 by D2 in the −x direction.

The amount of parallax indicated by each pixel on the parallax map 802 is determined by the distance from the point in the three-dimensional space corresponding to that pixel to the projection surface of the imaging device. That is, the smaller the distance, the larger the parallax amount. A region 806 that is closer than the region 805 has a relatively large average parallax amount. Therefore, D1 is greater than D2.

As described above, in step S606, the corresponding area of the reference image data corresponding to each area of the correction target image data is calculated based on the average parallax amount in the area, and thus the accurate corresponding area is efficiently generated. Obtainable.

In step S607, the correction value calculation unit 407 calculates a color correction value for each area from the area of the correction target image data divided in step S605 and the corresponding area of the reference image data calculated in step S606.

The correction value can be obtained as a difference value between the average value of the pixels in the area divided by the area dividing unit 405 and the average value of the pixels in the corresponding area obtained by the corresponding area calculating unit 406. For example, assume that the average pixel value (average brightness value) of the area 803 is R=113, G=78, and B=52. Further, it is assumed that the average pixel value (average luminance value) of the area 807 is R=104, G=67, B=45. In that case, the correction values are R=-9, G=-11, and B=-7.

Further, the correction value can be obtained not as a difference value but as a ratio. In that case, the correction values are R=104/113, G=67/78, and B=45/52.

In step S608, it is determined whether the calculation of the correction value is completed in all the regions of the correction target image data divided in step S805. If it has ended, the process proceeds to step S809, and if it has not ended, the process returns to step S806. By the processing of steps S606 to S608, the correction values of all the areas divided in step S605 are calculated.

In step S609, the correction value shaping unit 408 shapes the correction value calculated in step S608. As for the correction value, when the size of the divided area is large and the difference between the correction values of the adjacent areas is large, a pseudo contour occurs at the boundary of the areas. In order to suppress this, it is desirable to reduce the size of the divided area, but if it is smaller than the error Emax in the parallax amount calculation unit 403, the error in the corrected pixel value will increase. To solve this problem, the correction value shaping unit 408 applies a Gaussian filter to the entire correction value to perform shaping so as to reduce the difference in correction value between adjacent regions. However, if the filter is applied too strongly, the correction values will be averaged and the error in the corrected pixel value will increase. Therefore, it is desirable to set an appropriate Gaussian filter according to the size of the area.

In step S610, the pixel value correction unit 409 corrects the pixel value of the correction target image data for each divided area based on the correction value shaped in step S609. When the correction value is obtained as the difference value in step S607, the correction is performed by adding the correction value of the corresponding area to the pixel value of the correction target image data. When the correction value is obtained as a ratio in step S607, the correction is performed by multiplying the correction value.

In step S611, the image combining unit 410 combines the correction target image data corrected in step S610 with the reference image data based on the area for combining (pasting) the correction target image data determined by the image alignment unit 404. To generate composite image data.

As described above, the telephoto image data captured by the image capturing unit 102 is divided into a plurality of regions, and the color matching with the corresponding region of the wide-angle image data captured by the image capturing unit 101 is performed for each divided region. In other words, the high image height area of the telephoto image data in which the color matching between the captured images is performed and the influence of shading is large is the same as the area corresponding to the middle image height portion of the wide-angle image data in which the influence of shading is relatively small. Corrected to match. As a result, it is possible to correct the difference in color and brightness at the boundary portion of the combination including the step due to shading, which occurs when the images are combined. As a result, it is possible to suppress the pseudo contour of the boundary portion of the combination.

(Example 2)
In the present embodiment, a method for suppressing a difference in color or brightness of a boundary portion when an image is captured by using an image pickup apparatus having three or more eyes and a plurality of images whose view angles are sequentially narrowed is combined will be described.

FIG. 9 is a diagram for explaining a process of synthesizing three images whose view angles are sequentially narrowed with respect to an image picked up by a three-eye image pickup apparatus. An image 901 represents an image captured by an imaging device having a wide-angle focal length, an image 902 represents an image captured by an imaging device having a standard focal length, and an image 903 represents an image captured by an imaging device having a telephoto focal length. An image 904 is a virtual view angle image obtained by combining the images 901 to 903, and an image 905 is an image subjected to the color matching processing of the present embodiment.

FIG. 10 is a diagram showing variations in the order of combination processing when combining the images 901 to 903.

In FIG. 10A, an image 1001 is generated by first performing image synthesis by using the image 902 having the standard angle of view as the reference image and the telephoto image 903 as the correction target image. Next, an image 1002 is obtained by performing image composition by using the wide-angle image 901 as a reference image and the image 1001 as a correction target image. In this procedure, the telephoto image 903 is corrected twice when the image 1001 is generated and when the image 1002 is generated. The larger the number of corrections, the farther from the original image it is, so it is not desirable to perform the correction in the procedure of FIG.

In FIG. 10B, an image 1003 is obtained by first performing image composition by using the wide-angle image 901 as the reference image and the telephoto image 903 as the correction target image. Next, an image 1004 is obtained by performing image synthesis by using the image 1003 as a reference image and the image 902 having the standard angle of view as the correction target image. In this procedure, when the image 1003 is generated, the resolutions of the reference image and the correction target image are significantly different, so that an error is likely to occur in the parallax amount calculation unit 403. As a result, the maximum error Emax of the parallax amount increases, and the area size in the area dividing unit 405 must be increased. Therefore, it is not desirable to perform the correction in the procedure of FIG.

FIG. 10C is a diagram showing a compositing processing order in this embodiment. In FIG. 10C, an image 1005 is obtained by first performing image composition by using the wide-angle image 901 as a reference image and the image 902 with a standard angle of view as a correction target image. Next, an image 1006 is obtained by performing image combination (second image combination) using the image 1005 as the reference image and the telephoto image 903 as the correction target image. In the procedure of this embodiment shown in FIG. 10C, the problem that the same image is corrected a plurality of times does not occur in the case of the procedure of FIG. Further, in the case of the procedure of FIG. 10(b), the problem that the resolution greatly differs does not occur. As described above, according to the combination processing order of the present embodiment, it is possible to obtain a high-quality combined image while suppressing the difference in color and brightness of the boundary portion of combination.

In the present embodiment, the order of combination processing in the trinocular configuration has been described, but the present invention is also applicable to a trinocular configuration or more. Specifically, the reference image data selection unit 401 sequentially selects from the images captured by the imaging unit with the wide-angle side focal length, and the correction target image data selection unit 402 selects the image on the telephoto side next to the reference image data. The combination processing order may be set so as to select.

As described above, it is possible to suppress the pseudo contour of the boundary portion even when the image pickup apparatus having three or more eyes is used.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (16)

An image processing apparatus for generating composite image data using a plurality of image data having different angles of view including a common subject,
Selecting means for selecting, from the plurality of image data, the first image data as reference image data and the second image data having a narrower angle of view than the first image data as correction target image data;
Dividing means for dividing the correction target image data into a plurality of areas,
Derivation means for deriving a parallax amount of the correction target image data with respect to the reference image data,
A color matching unit that refers to the parallax amount for each region obtained by dividing the correction target image data and performs color matching with a corresponding region in the reference image data according to the parallax amount,
By determining a combination area in the reference image data in which the correction target image data after the color matching is combined and combining the correction target image data in the combination area in the reference image data, And a synthesizing means for generating,
The color matching means,
Correction value calculating means for calculating the correction value for each area of the correction target image data so that the average of the pixel values of the area is equal to the average of the pixel values of the corresponding area corresponding to the area; An image processing apparatus comprising: a correction unit that corrects the correction target image data based on the above. The image processing apparatus according to claim 1, wherein the correction value is a difference between an average of pixel values of the area and an average of pixel values of the corresponding area. The image processing apparatus according to claim 1, wherein the correction value is a ratio of an average of pixel values of the corresponding area to an average of pixel values of the area. The color matching unit further includes a shaping unit that shapes the correction value calculated by the correction value calculation unit so that a difference between the correction values between adjacent regions becomes small.
The image processing apparatus according to any one of claims 1 to 3, wherein the correction unit corrects the correction target image data based on the correction value shaped by the shaping unit. A parallax amount calculation unit that calculates a parallax amount of the correction target image data with respect to the reference image data for each pixel,
The image processing apparatus according to claim 1, further comprising: a corresponding area calculation unit that calculates the corresponding area based on an average value of the parallax amounts in the area. The image processing apparatus according to claim 5, wherein the division unit divides the correction target image data into a size determined based on an error in the parallax amount calculation unit. The image processing apparatus according to claim 1, wherein the first image data is wide-angle image data, and the second image data is telephoto image data. The selecting means sets the combined image data as reference image data in the second image combination, and sets the third image data having a narrower angle of view than the second image data in the correction target image data in the second image combination. 7. The method according to any one of claims 1 to 6, characterized in that
The image processing device according to item. The first image data is wide-angle image data, the second image data is image data of a standard angle of view, and the third image data is telephoto image data. The image processing device described. The image processing apparatus according to claim 1, wherein the pixel value is a brightness value. 11. The image processing apparatus according to claim 1, wherein the plurality of image data are captured image data obtained by simultaneously capturing the common subject by image capturing units having different angles of view. An image processing apparatus for generating composite image data using a plurality of image data having different angles of view including a common subject,
Selecting means for selecting, from the plurality of image data, the first image data as reference image data and the second image data having a narrower angle of view than the first image data as correction target image data;
Dividing means for dividing the correction target image data into a plurality of areas,
Color matching means for performing color matching with the corresponding area in the reference image data for each area obtained by dividing the correction target image data;
Combining the correction target image data after the color matching with the reference image data to generate the combined image data,
The color matching means,
Correction value calculating means for calculating the correction value for each area of the correction target image data so that the average of the pixel values of the area is equal to the average of the pixel values of the corresponding area corresponding to the area; A correction unit that corrects the correction target image data based on, and a parallax amount calculation unit that calculates a parallax amount of the correction target image data with respect to the reference image data for each pixel,
An image processing apparatus comprising: a corresponding area calculation unit that calculates the corresponding area based on an average value of the parallax amounts in the area. An image processing apparatus for generating composite image data using a plurality of image data having different angles of view including a common subject,
Selecting means for selecting, from the plurality of image data, the first image data as reference image data and the second image data having a narrower angle of view than the first image data as correction target image data;
Dividing means for dividing the correction target image data into a plurality of areas,
Color matching means for performing color matching with the corresponding area in the reference image data for each area obtained by dividing the correction target image data;
Combining the correction target image data after the color matching with the reference image data to generate the combined image data,
The color matching means,
Correction value calculating means for calculating the correction value for each area of the correction target image data so that the average of the pixel values of the area is equal to the average of the pixel values of the corresponding area corresponding to the area; A correction means for correcting the correction target image data based on
The selecting means sets the combined image data as reference image data in the second image combination, and sets the third image data having a narrower angle of view than the second image data in the correction target image data in the second image combination. An image processing apparatus characterized by selecting as. An image processing apparatus for synthesizing reference image data and a correction target image data including a subject common to the reference image data and having a narrower angle of view than the reference image data,
Derivation means for deriving a parallax amount of the correction target image data with respect to the reference image data,
Calculation means for calculating the corresponding area in the reference image data by referring to the parallax amount for each area obtained by dividing the correction target image data,
Color matching means for performing color matching on each area in the correction target image data based on the corresponding area in the reference image data;
In the reference image data, to determine the combining region for synthesizing the correction target image data after the color matching, to synthesize the correction target image data to the combining region of the reference image data, synthetic image data An image processing apparatus having a synthesizing unit for generating An image processing method for generating composite image data using a plurality of image data having different angles of view including a common subject,
A selection step of selecting, from the plurality of image data, the first image data as reference image data, and the second image data having a narrower angle of view than the first image data as correction target image data;
A dividing step of dividing the correction target image data into a plurality of areas,
A deriving step of deriving a parallax amount of the correction target image data with respect to the reference image data;
A color matching step of performing color matching with a corresponding area in the reference image data according to the parallax amount by referring to the parallax amount for each area obtained by dividing the correction target image data,
By determining a combination area in the reference image data in which the correction target image data after the color matching is combined and combining the correction target image data in the combination area in the reference image data, And a synthesis step to generate,
The color matching step is
A correction value calculating step for calculating a correction value for each area of the correction target image data so that the average of the pixel values of the area is equal to the average of the pixel values of the corresponding area corresponding to the area; And a correction step of correcting the correction target image data based on the image processing method. A program for causing a computer to function as the image processing device according to claim 1.