JP2015073185A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain highly accurate distance information in a scene having a wide dynamic range.SOLUTION: An image processing device includes: acquisition means for acquiring a first image group obtained by imaging the same object from a plurality of different view point with a first exposure condition and a second image group obtained by imaging the object from a plurality of different view point with a second exposure condition different from the first exposure condition; first generation means for generating a first and a second distance map from the first and second image groups; and second generation means for generating a third distance map from the first and second distance maps.

Description

  The present invention relates to a technique for obtaining subject distance information.

  Conventionally, a technique for estimating a distance to a subject using images taken from a plurality of viewpoints (hereinafter, multi-viewpoint images) is known. As a method of estimating the distance to the subject, there is a method of searching for a corresponding point corresponding to the same subject from a multi-viewpoint image using a block matching method and obtaining the distance to the subject by triangulation. When this method is used, there is a problem that distance information of an overexposed highlight portion or a shadow portion that is crushed in black cannot be obtained well in a scene with a wide dynamic range.

  In Patent Document 1, a stereo camera captures images twice with different exposure times, generates two high dynamic range composite images corresponding to two viewpoints, and calculates the distance to the subject using the high dynamic range composite images. The technology is described.

  Further, Patent Document 2 describes a technique for obtaining a distance from two images obtained by one imaging in a stereo camera and having different viewpoints and exposure times. In the technique described in Patent Literature 2, two virtual exposure images in which the brightness is adjusted to the other captured image are generated from the captured image, and matching information obtained by matching the captured image and the virtual exposure image is integrated, The distance to the subject is obtained based on the integrated matching information.

JP 2003-18617 A JP 2011-254170 A

  However, in the method described in Patent Document 1, it is necessary to perform matching between images taken twice with different exposures. Therefore, when the camera position is different between the two shots, the distance between only the images with the same exposure is set. There is a problem that the accuracy of the distance information is reduced as compared with the case where it is output.

  In addition, in the method described in Patent Document 2, in order to perform matching by matching the brightness of images taken at different exposure times, sufficient matching is performed for a region that is overexposed or underexposed. There is a problem that accuracy cannot be obtained.

  In view of the above problems, an object of the present invention is to obtain highly accurate distance information even in a scene with a wide dynamic range.

A first image group in which the same subject is imaged from a plurality of different viewpoints in the first exposure condition, and a second exposure condition different from the first exposure condition, the same subject from a plurality of different viewpoints. Acquisition means for acquiring a captured second image group;
A generating unit configured to generate a third distance map based on the first distance map generated from the first image group and the second distance map generated from the second image group; An image processing apparatus.

  According to the present invention, the distance to the subject can be obtained with high accuracy even in a scene with a wide dynamic range.

1 is a diagram illustrating an appearance of a multi-view camera according to Embodiment 1. FIG. 1 is a block diagram illustrating a configuration of a multi-view camera of Embodiment 1. FIG. FIG. 3 is a diagram illustrating an internal configuration of an imaging unit according to the first embodiment. FIG. 2 is a block diagram illustrating a configuration of an image processing unit according to the first embodiment. 3 is a flowchart illustrating processing in an image processing unit according to the first exemplary embodiment. 5 is a flowchart illustrating distance map generation processing according to the first embodiment. FIG. 3 is a diagram illustrating a distance calculation method according to the first embodiment. FIG. 3 is a diagram illustrating an outline of a distance map synthesis process according to the first embodiment. 5 is a flowchart illustrating distance map synthesis processing according to the first embodiment.

[Example 1]
In this embodiment, a multi-lens camera including four imaging units is used as an imaging device that acquires a multi-viewpoint image. A method for obtaining subject distance information from an image acquired by the multi-lens camera will be described.

  FIG. 1 is a diagram of a multi-view camera 100 (hereinafter referred to as a camera 100) according to the present embodiment as viewed from the front. The multi-lens camera 100 includes four imaging units 101 to 104 that acquire color images by imaging and a shooting button 105. All the imaging units are arranged on the same plane of the casing, and their optical axes are all parallel and perpendicular to the arranged plane. When the user presses the shooting button 105, the imaging units 101 to 104 receive light information of the subject with a sensor (imaging device), the received signal is A / D converted, and a plurality of color image data (digital data) are simultaneously received. Taken.

  With such a multi-lens imaging device, it is possible to obtain a color image group obtained by simultaneously photographing the same subject from a plurality of viewpoint positions. Although the number of imaging units is four here, the number of imaging units is not limited to four, and the imaging device only needs to have a plurality of imaging units.

  FIG. 2 is a block diagram showing the internal configuration of the camera 100. The CPU 201 is a processor that comprehensively controls each component described below. The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The ROM 203 stores a control program executed by the CPU 201 and the like. The CPU 201 executes a program stored in the ROM 203 using the RAM 202 as a work memory, and controls each component of the multiview camera 100 via the bus 204.

  The bus 204 serves as a transfer path for various data. For example, the image data acquired by the imaging units 101 to 104 is sent to a predetermined configuration unit via the bus 204.

  An operation unit 205 is an operation unit such as a button or a mode dial, and receives a user instruction and outputs it to the CPU 201.

  A display unit 206 is a liquid crystal display and displays captured images and characters. The display unit 206 may have a touch screen function. In that case, a user instruction using the touch screen can be handled as an input of the operation unit 205.

  A display control unit 207 is a control circuit that performs display control of a captured image and characters displayed on the display unit 206.

  The imaging control unit 208 is a control circuit that controls the imaging unit based on instructions from the CPU 201 such as focusing, opening / closing a shutter, and adjusting an aperture. In the present embodiment, the imaging control unit 208 performs imaging with the first exposure time among the imaging units 101 to 104, and the imaging units 103 and 104 are longer than the first exposure time. The imaging units 101 to 104 are controlled so as to perform imaging with the second exposure time.

  The digital signal processing unit 209 is a processing circuit that performs various processing such as white balance processing, gamma processing, and noise reduction processing on image data received via the bus 204.

  The encoding unit 210 performs processing for converting image data received via the bus 204 into a file format such as JPEG or MPEG.

  The external memory control unit 211 is an interface for connecting the camera 100 to a medium 213 (for example, a hard disk, a memory card, a CF card, an SD card, a USB memory).

  The image processing unit 212 is a processing circuit that processes the image data acquired by the imaging units 101 to 104 or the image data output from the digital signal processing unit 209. In this embodiment, the distance is calculated using image data photographed under a plurality of exposure conditions, details of which will be described later.

  The description of the components other than those described above is omitted, but various configurations can be adopted as long as the configuration achieves the effects of the present invention.

  FIG. 3 is a diagram illustrating an internal configuration of the imaging units 101 to 104. The imaging units 101 to 104 include lenses 301 to 303, a diaphragm 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit 310. The lenses 301 to 303 are a zoom lens 301, a focus lens 302, and a shake correction lens 303, respectively. The sensor 309 is a sensor such as a CMOS or a CCD, for example, and detects the amount of light of the subject focused by each of the lenses. The detected light amount is output from the sensor 309 as an analog value, converted into a digital value by the A / D conversion unit 310, and output as digital image data to the bus 204.

(Image processing unit)
FIG. 4 is a block diagram illustrating a configuration of the image processing unit 212. Hereinafter, processing performed in the image processing unit 212 will be described with reference to a block diagram shown in FIG. 4 and a flowchart shown in FIG. In the image processing unit 212, a computer-executable program describing the procedure shown in the flowchart of FIG. 5 is read from the ROM 203 onto the RAM 202, and then the CPU 201 executes the program to execute the process. Although the image processing unit 212 is a processing circuit, a part or all of each functional block of the image processing unit 212 may be replaced with processing by the CPU 201 that executes the program.

  First, in step S <b> 501, the image acquisition unit 401 acquires image data input from the imaging units 101 to 104 or the digital signal processing unit 209 via the bus 204.

  In step S <b> 502, image data corresponding to two images captured under the same exposure condition is selected from the image data acquired by the image acquisition unit 401, and is output to the distance map generation unit 410. The exposure condition of each image data may be recorded in the RAM 202 at the time of shooting, or may be given to the image data using an Exif format or the like. In this embodiment, the image group captured by the image capturing units 101 and 102 and the image group captured by the image capturing units 103 and 104 are selected as images captured under the same exposure conditions.

  Next, in step S <b> 503, the distance map generation unit 410 generates a distance map corresponding to the image group indicated by the image data using the image data input from the image acquisition unit 401, and sends it to the distance map synthesis unit 420. Output. Details of the distance map generation processing will be described later. Here, the distance map is data indicating the distance from the imaging device to the subject at each pixel position in the image as two-dimensional information. In the present embodiment, the distance map is a monochrome image of 256 gradations, and a pixel that is close to the imaging device is represented by white (255) and a pixel that is far from the image is represented by black (0). The format of the distance map is not limited to this format, and may be a table storing subject distances at each pixel position.

  Next, in step S504, the distance map synthesis unit 420 determines whether distance maps corresponding to all image groups indicated by the input image data have been generated. If it is determined that all the distance maps have been generated, the process proceeds to step S505. If it is determined that all distance maps have not been generated, the process returns to step S502, and an image group corresponding to another exposure condition is selected.

  Finally, in step S505, the distance map synthesis unit 420 synthesizes the plurality of distance maps input from the distance map generation unit 410 to generate a final distance map. Details of the distance map composition processing will be described later.

(Distance map generation processing)
Next, details of the distance map generation processing performed in step S503 of the flowchart shown in FIG. 5 will be described. In the distance map generation process, the distance map is generated by estimating the distance information of the subject at each pixel position of the captured scene based on the images acquired from a plurality of different viewpoints. A known method can be used for calculating the distance information. For example, a stereo method or a multi-baseline stereo method can be applied. In this embodiment, the distance information of the subject is estimated using the stereo method. Hereinafter, the details of the distance map generation processing will be described using the flowchart shown in FIG.

  First, in step S601, the corresponding point searching unit 402 acquires image data indicating an image group captured under the same exposure conditions. The image group indicated by the image data acquired here is an image group corresponding to the imaging units 101 and 102 captured under the same exposure conditions. Hereinafter, an image corresponding to the former is referred to as a reference image, and an image corresponding to the latter is referred to as a target image.

  In step S602, the corresponding point search unit 402 selects a region including a target pixel for calculating distance information and surrounding pixels in the reference image, and performs pattern matching with the target image using a block that is the selected region. Here, the target pixel is selected by reading information indicating the coordinates of the target pixel stored in the RAM 202. By this pattern matching, the corresponding point search unit 402 searches the target image for a pixel (corresponding pixel) corresponding to the target pixel of the reference image. The corresponding point search unit 402 outputs the obtained target pixel and the coordinate information of the corresponding pixel to the distance calculation unit 403.

  In step S603, the distance calculation unit 403 calculates the distance information d of the subject at the pixel position of the target pixel from the arrangement information of each imaging unit and the coordinate information of the target pixel and the corresponding pixel input from the corresponding point search unit 402. Is calculated. The distance information d is expressed by the following formula using the parameters α, β, and s shown in FIG. Note that the arrangement information of each imaging unit is stored in the ROM 203 in advance.

  Here, α is calculated from the horizontal angle of view of the imaging unit 102, the position of the imaging unit 102, and the coordinates of the target pixel. β is calculated from the horizontal angle of view of the imaging unit 101, the position of the imaging unit 101, and the coordinates of the pixel of interest. s is the horizontal distance between the imaging units 101 and 102, and is calculated from the arrangement information of the imaging units 101 and 102. The distance calculation unit 403 calculates each parameter of the above formula based on the arrangement information of each imaging unit stored in the ROM 203 and the coordinate information of the target pixel and the corresponding pixel input from the corresponding point search unit 402 and is calculated. Substituting the obtained value into the above equation, d is calculated. The calculated d is stored in the RAM 202 together with the coordinate information of the target pixel.

  In step S604, the distance calculation unit 403 determines whether distance information has been calculated for all pixel positions. If it is determined that distance information has been calculated for all pixel positions, the process proceeds to step S606. If it is determined that distance information has not been calculated for all pixel positions, the process advances to step S605.

  In step S <b> 605, the distance calculation unit 403 updates information indicating the coordinates of the target pixel stored in the RAM 202. For example, it is rewritten to the coordinates of a pixel for which distance calculation has not yet been performed, such as a pixel adjacent to the target pixel used in the immediately preceding process. When the update of the coordinate information is completed, the process returns to step S602, and the corresponding point search unit 402 performs pattern matching on the new target pixel.

  In step S606, the distance calculation unit 403 generates a distance map in which the distance from the camera 100 to the subject at each pixel position is two-dimensionally arranged, and outputs the distance map to the distance map synthesis unit 420.

  In this embodiment, the image capturing unit 101 and the image capturing unit 102 are selected as image capturing units having the same exposure condition in step S501. However, the present invention is not limited to this. That is, for example, the imaging units 101 and 104 may be selected as imaging units that capture images under the same exposure conditions.

(Distance map composition processing)
Here, the details of the distance map synthesis process performed in step S505 of the flowchart shown in FIG. 5 will be described. In the distance map synthesis process, a plurality of distance maps generated in step S503 are synthesized according to the luminance value at each pixel position to generate a new distance map. FIG. 8 shows an outline thereof.

  Images 801 to 804 in FIG. 8 are images captured by the imaging units 101 to 104, respectively. In this embodiment, since the imaging units 101 and 102 capture images with a short exposure time, the background portions of the images 801 and 802 are blackened. On the other hand, since the imaging units 103 and 104 capture images with a long exposure time, the images 803 and 804 are overexposed in the foreground portion. Since the pattern matching is not correctly performed in the blacked out area or the overexposed area, the distance information is not accurately calculated.

  For this reason, in the distance map 805 generated from the images 801 and 802 and the distance map 806 generated from the images 803 and 804, the distance is not estimated well in the blackened area and the overexposed area, respectively. Therefore, in this embodiment, the distance maps 805 and 806 are synthesized according to the luminance value, thereby generating a distance map in which the distances are accurately calculated in all regions. The details of the processing will be described below with reference to the flowchart shown in FIG.

  First, in step S901, the shift image generation unit 404 outputs a plurality of distance maps output from the distance calculation unit 403 and image data corresponding to the distance map output from the image acquisition unit 401 (hereinafter, corresponding image data). Called). Note that the corresponding image data acquired here is image data indicating an image used as a reference image when generating each distance map.

  Next, in step S902, each component of the distance map generation unit 420 selects one from the distance maps generated in step S503.

  Next, in step S903, the shift amount calculation unit 404 calculates a shift amount for aligning the distance map selected in step S902 with the reference viewpoint. Here, the reference viewpoint is a viewpoint at which a combined distance map is finally generated, and is a viewpoint corresponding to the imaging unit 101 in the present embodiment. Note that this viewpoint is not limited to the viewpoint corresponding to the imaging unit 101, and may be a viewpoint corresponding to another imaging unit or a virtual viewpoint positioned at the center of all imaging units. Here, the vertical shift amount Δx and the horizontal shift amount Δy of each pixel of the shifted distance map can be calculated by the following equations.

  Here, W and H are the horizontal and vertical image sizes of the corresponding image data. Θx and θy are the horizontal viewing angle and the vertical viewing angle of the imaging unit that captured the corresponding image data, and Sx and Sy are the horizontal distance and vertical distance between the viewpoint corresponding to the selected distance map and the reference viewpoint. is there. D is the distance from the camera 100 to the subject at that pixel.

  The shift amount calculation unit shifts the distance map and the corresponding image data based on the shift amount calculated here, and generates a shift distance map and shift image data. It should be noted that a value indicating an abnormal value is set for the pixel of the shift image data that does not correspond to the pixel of the corresponding image data. For example, if zero, which is the minimum value of a pixel, is set as the abnormal value, a weighting coefficient described later can be set to zero. The shift amount calculation unit 404 outputs the generated shift distance map and shift image data to the weight calculation unit 405. Further, the shift amount calculation unit 404 outputs the generated shift distance map to the shift distance map composition unit 406.

  Next, in step S904, the weight calculation unit 405 calculates a weighting factor used in the synthesis of the distance map. Here, the weight calculation unit 405 calculates the weighting coefficient for each pixel of the shift distance map input from the shift amount calculation unit 404 according to the luminance value of the pixel of the shift image data, and stores it in the RAM 202. The weighting factor wi (x, y) in each pixel is determined based on, for example, the following equation according to the luminance value of the shift image data.

  Here, Ii (x, y) is the brightness value of the shift image data in each pixel, and Imax is the maximum brightness value of the shift image data. Α and β are threshold values indicating the shadow portion and the highlight portion. A pixel having a luminance value smaller than α is determined as a shadow portion, and an image having a luminance value larger than β is determined as a highlight portion. As described above, the accuracy of distance estimation deteriorates in the highlight portion and the shadow portion. Therefore, in the above formula, the weighting coefficient of the highlight part and the shadow part is set to be small. This makes it possible to synthesize the distance map by increasing the weight of the appropriate exposure area with high distance estimation accuracy. Note that the values of α and β may be stored in advance in the ROM 203 according to the characteristics of the image sensor, or may be set by the user through the operation unit 205. In this embodiment, the weighting coefficients of the shadow part and the highlight part are linear, but they may be expressed in a non-linear manner or a table reference type. Further, the weighting factors of the shadow part and the highlight part may be zero.

  In step S905, the weight calculation unit 405 determines whether weight calculation has been performed for all pixels. If it is determined that the weight has been calculated for all the pixels, the process proceeds to step S907. If it is determined that the weight is not calculated for all the pixels, the process proceeds to step S906.

  In step S <b> 906, the weight calculation unit 405 updates information indicating the coordinates of the pixel of interest stored in the RAM 202. When the information indicating the coordinates of the target pixel is updated, the process returns to step S904.

  In step S907, the shift distance map composition unit 406 determines whether the shift amount and the weight are calculated in all the distance maps. If it is determined that the shift amount and the weight are calculated in all the distance maps, the process proceeds to step S908. If it is determined that the shift amount and the weight are not calculated in all the distance maps, the process proceeds to step S902.

  In step S908, the shift distance map combining unit 406 combines distance maps. In this embodiment, the distance D (x, y) at each pixel of the composite distance map is obtained by the product of the distance Di (x, y) and the weighting factor wi (x, y) at each pixel of the shift distance map. it can. The distance D (x, y) at each pixel of the combined distance map can be calculated using the following equation.

  Here, n is the number of distance maps to be combined. The shift distance map synthesis unit 406 substitutes the distance at each pixel of the shift distance map output from the shift amount calculation unit 404 and the weight value calculated in step S904 into the above formula to obtain a synthesized distance map.

  With the above processing, a highly accurate distance map can be obtained even in a scene with a wide dynamic range.

  Note that the weighting synthesis in this embodiment includes processing for selecting a distance value of one distance map from distance values of a plurality of distance maps. That is, the case where the weight other than the selected distance value is 0 is included.

  In this embodiment, the image pickup units are arranged two-dimensionally evenly. However, this is not the only case. As described above, the distance can be estimated if there is image data with a different viewpoint, and may be arranged at an arbitrary position. In the present embodiment, the configuration and processing of each unit have been described on the assumption that all images captured by the imaging unit are color images. However, part or all of the image captured by the imaging unit may be changed to a monochrome image. In that case, the color filter 308 of FIG. 3 is omitted.

  In this embodiment, the weight value is obtained for each pixel. However, if the weight value is different for each pixel position, the weight value may be determined in different units. That is, the weight value may be determined for each image block including a plurality of pixels. In that case, since the boundary of each image block may become unnatural, a smoothing filter or the like may be applied.

  In this embodiment, two distance maps corresponding to two types of image groups with different exposure times were synthesized to generate a new distance map. Any combination of image groups that can guarantee a skipped portion may be used. In general, the exposure condition can be changed by changing the shutter speed, F value, ISO sensitivity, or switching the ND filter.

  In the present embodiment, the image acquisition unit 401 functions as an acquisition unit that acquires the first image group and the second image group. Here, the first image group includes images obtained by capturing the same subject from a plurality of different viewpoints under the first exposure condition, and the second image group includes a second image different from the first exposure condition. In the exposure condition, it includes images obtained by imaging the same subject from a plurality of different viewpoints.

  The distance map generator 410 functions as a first generator that generates first and second distance maps from the first and second image groups.

Further, the distance map synthesis unit 420 functions as a second generation unit that generates a third distance map from the first and second distance maps.
In addition, the imaging units 101 to 104 function as an imaging unit that acquires the first and second image groups by imaging.

[Example 2]
In the first embodiment, the method for determining the weight of weighted synthesis based on the luminance value of each pixel has been described. In the second embodiment, a method of performing weighted synthesis based on the focus state of each subject will be described.

  When an image is taken with a camera having an autofocus function, the focus position is controlled so that the proper exposure area is in focus. In other words, a high-accuracy distance map can be generated by performing synthesis so as to increase the weight of the focused area without using the luminance value in each pixel. In addition, according to this method, it is possible to reduce the weight of a subject that is blurred in one image group due to a difference in the focus position, even though both the image groups under the two exposure conditions have appropriate exposure. That is, it is possible to generate a distance map with high accuracy even if subjects with different exposure states are included while being properly exposed under a plurality of different exposure conditions.

  The configuration and processing contents of the present embodiment are almost the same as those of the first embodiment. In the present embodiment, the weight calculation processing method in step S904 is different.

The calculation of the weight value in step S904 of the present embodiment is performed for each of a plurality of image blocks obtained by dividing the distance map in units of several pixels. When the distance map is divided into N × M blocks, the weight value corresponding to the block (s, t) (s = 1, 2,... N, t = 1, 2,. , T). At that time, if the contrast ratio of the image block in the captured image corresponding to the block (s, t) is ci (s, t), the weight value of each block is determined as follows.
Wbi (s, t) = ci (s, t)
Thereby, the weight value of each image block is determined according to the ratio of contrast ratios in a plurality of exposure conditions. The weight value may be set so as to be inclined according to the absolute value or relative value of the contrast ratio, instead of using the contrast ratio ratio as it is.

  Using the weight value wbi (s, t) set by the above formula and the distance Di (x, y) at each pixel of the shift distance map, the distance D (x, y) at each pixel of the combined distance map Is calculated by the following equation.

  Here, n is the number of distance maps to be combined. In the above equation, each Di (x, y) is multiplied by the weight value wbi (s, t) of the image block (s, t) corresponding to the pixel coordinate (x, y).

  With the above calculation method, a high-precision distance map can be obtained even in a distance map and a scene with a wide dynamic range. In addition, a distance map with high accuracy can be generated even if subjects with different exposure states are included while being properly exposed under a plurality of different exposure conditions.

  Note that the determination of the focus state of each subject is not limited to only based on the contrast ratio of each image block. For example, the weight value may be determined based on the magnitude of a high frequency component when each image block is Fourier transformed.

[Other Embodiments]
Embodiments of the present invention are not limited to the above examples. For example, the above embodiments may be combined. That is, the weight value may be determined in consideration of both the luminance value of each pixel and the focus state of each image block.

  In the above embodiment, the distance map is synthesized based on an image captured by a multi-lens camera including a plurality of imaging units. However, an image captured in another form may be used. For example, images captured by a plurality of independent cameras may be used. Alternatively, an image captured by an image including a Plenoptic imaging system that can extract images from a plurality of viewpoints from the output of one sensor may be used. In that case, when imaging is performed with a camera having two or more Plenoptic imaging systems, a plurality of image groups with different exposure conditions can be acquired by simultaneous imaging.

  Further, the images do not have to be taken at the same time, and may be any images as long as the same subject can be captured. That is, the case where an image captured at different time points by an existing monocular camera is used without using a multi-lens camera, and the case where images captured by a plurality of cameras are used are also included.

  In the above embodiment, the final distance map is obtained by weighting and synthesizing a plurality of distance maps generated from a group of images captured under different exposure conditions, but similar results can be obtained. For example, the final distance map may be obtained using other methods. For example, the distance map may be generated by reading out and arranging values stored in advance in a format such as a table based on a plurality of distance maps and the luminance value of the captured image corresponding to the distance map.

  In addition, the embodiment is not limited to the camera including the imaging unit and the processing unit, and may be an information processing system in which a part of the processing performed in the above-described embodiment is performed by another computer. For example, it may be an information processing system in which image data acquired by a camera is output to a computer, and the computer synthesizes a distance map based on the input image.

  Note that the present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Further, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of one device.

  Further, the present invention achieves the functions of the above-described embodiments by supplying a software program directly or remotely to a system or apparatus, and the computer of the system or apparatus reads and executes the supplied program code. Including cases. In this case, the supplied program is a computer program corresponding to the flowchart shown in the drawings in the embodiment. Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention. In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Examples of the computer-readable storage medium for supplying the computer program include a hard disk, an optical disk, and a magneto-optical disk. Moreover, MO, CD-ROM, CD-R, CD-RW, a magnetic tape, a non-volatile memory card, ROM, DVD (DVD-ROM, DVD-R), etc. may be sufficient.

  As another program supply method, a client computer browser is used to connect to a homepage on the Internet, and the computer program of the present invention is downloaded from the homepage to a storage medium such as a hard disk. In this case, the downloaded program may be a compressed file including an automatic installation function. A WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

DESCRIPTION OF SYMBOLS 100 Multiview camera 101-104 Image pick-up part 212 Image processing part 401 Image acquisition part 410 Distance map production | generation part 420 Distance map synthetic | combination part

Claims (12)

A first image group in which the same subject is imaged from a plurality of different viewpoints in the first exposure condition, and a first image group in which the subject is imaged from a plurality of different viewpoints in a second exposure condition different from the first exposure condition. Acquisition means for acquiring a second image group;
First generation means for generating first and second distance maps from the first and second image groups;
An image processing apparatus comprising: a second generation unit configured to generate a third distance map from the first and second distance maps.   The first image group is an image group of images that are simultaneously captured, and the second image group is also an image group of images that are simultaneously captured. The image processing apparatus described.   The image processing apparatus according to claim 2, wherein the first and second image groups are captured simultaneously.   The second generation means weights and synthesizes the third distance map with the first distance map and the second distance map based on the first and second image groups. The image processing apparatus according to claim 1, wherein the image processing apparatus generates the image processing apparatus.   5. The image processing apparatus according to claim 4, wherein the second generation unit determines a weight in the weighted synthesis based on luminance values of images included in the first and second image groups. .   6. The image according to claim 4, wherein the second generation unit determines a weight in the weighted synthesis based on a focus state of images included in the first and second image groups. Processing equipment.   The image processing apparatus according to claim 6, wherein the focus state is indicated by a contrast ratio for each image block in images included in the first and second image groups.   The image processing apparatus according to claim 4, wherein a weight in the weighting synthesis is determined for each pixel position in the first and second distance maps.   The image processing apparatus according to claim 1, further comprising an imaging unit that acquires the first and second image groups by imaging.   The image processing apparatus according to claim 2, wherein the first and second exposure conditions are exposure times. A first image group in which the same subject is imaged from a plurality of different viewpoints in the first exposure condition, and a first image group in which the subject is imaged from a plurality of different viewpoints in a second exposure condition different from the first exposure condition. Obtaining a second image group;
Generating first and second distance maps from the first and second image groups;
And a step of generating a third distance map from the first and second distance maps.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 10.