JP2020115683A - Image processing system, imaging apparatus, image processing method, and program - Google Patents

Abstract

To provide a system and method capable of reducing differences in luminance and color differences of a plurality of combined images.SOLUTION: An image processing system that performs image processing on a plurality of images including overlapping image areas as overlapping areas includes an operation unit 40 that calculates an evaluation value for evaluating each image by using pixel values of one or more pixels in each of the overlapping areas, a determination unit 41 that determines whether there is a correction target image from among the plurality of images on the basis of the calculated evaluation value, an image determination unit 42 that determines a correction reference image to be a reference at the time of correction from among the plurality of images on the basis of the calculated evaluation value when it is determined that there is a correction target image, and an image correction unit 43 that corrects the correction target image on the basis of the determined correction reference image.SELECTED DRAWING: Figure 10

Description

The present invention relates to an image processing system that performs image processing on a plurality of images including overlapping image areas as overlapping areas, an imaging apparatus, an image processing method, and a program for causing a computer to execute the method.

A spherical imaging camera is known as an imaging device capable of imaging any direction of 360°, and is used as a surveillance camera or the like. The omnidirectional imaging camera uses a plurality of wide-angle lenses or fish-eye lenses, images with a plurality of imaging elements, performs distortion correction, projective transformation, and the like on the obtained images, and combines them into a single image. Generate a spherical image. An image obtained by picking up images by the adjacent image pickup devices has an image overlapping area in which a part of the image overlaps, and in the omnidirectional imaging camera, the images are combined in the image overlapping area.

Since the omnidirectional camera captures images using multiple image sensors with different image capturing directions, the brightness and color of the image overlap area of the images obtained by each image sensor are different, resulting in poor visibility when combined. There was a problem of becoming. Therefore, a technique is proposed in which an initial gain for uniformizing the color tones of the image overlapping area and a correction gain for reducing the difference from other image pickup elements are used to correct the image overlapping area and reduce the difference in the color tones of the image overlapping area. (For example, see Patent Document 1).

Since the omnidirectional imaging camera images a wide range using a wide-angle lens or a fisheye lens, it is easy for a light source such as the sun or illumination to enter the imaging range. Therefore, it is known that there is a high possibility that a part of the image is blurred in white and flare appears as if the light is blurred. Since flare does not occur uniformly in each image, not only a difference in luminance but also a color difference occurs between an image with flare and an image without flare. Therefore, when these images are combined, there is a problem that the joints are conspicuous.

In the above conventional technique, it is possible to reduce the difference in color tones in the image overlapping areas and make the joints inconspicuous, but since the other areas are not corrected, the combined image is an image having a difference in luminance or a color difference. Becomes

Therefore, it has been desired to provide a system and method that can reduce the difference in brightness and the difference in color between a plurality of images to be combined.

The present invention has been made in view of the above problems, and is an image processing system that performs image processing on a plurality of images that include overlapping image areas as overlapping areas, and includes one or more pixels in each overlapping area. A calculation unit that calculates an evaluation value for evaluating each image using pixel values, and a determination that determines whether or not there is a correction target image among a plurality of images based on the evaluation value calculated by the calculation unit Image that determines a correction reference image to be a reference when performing correction from a plurality of images based on the evaluation value calculated by the calculation unit when the correction unit determines that the correction target image exists An image processing system including a determination unit and an image correction unit that corrects an image to be corrected based on the correction reference image determined by the image determination unit is provided.

According to the present invention, it is possible to reduce the difference in brightness and the difference in color between a plurality of images to be combined.

The external view of an imaging device. The figure which showed the hardware constitutions of a captured image storage. The figure explaining the fish-eye lens used for an imaging device. The figure explaining the overlap area in the some image imaged with the imaging device. The figure explaining the format of a spherical image. The figure explaining the conversion table which converts into a spherical image from a fish-eye image. The flowchart which showed the flow of the process which produces|generates a spherical image. The figure which showed the result of performing distortion correction. The figure explaining the method of detecting a connection position. The functional block diagram of an image processing system. 3 is a flowchart showing the flow of overall processing executed by the image processing system. The figure explaining acquisition of the evaluation value in step 1110 shown in FIG. 12 is a flowchart showing a detailed processing flow of determination in step 1120 shown in FIG. 11. 12 is a flowchart showing the flow of detailed processing for creating a correction map in step 1130 shown in FIG. 11. FIG. 6 is a diagram illustrating creation of a correction exclusion map. The figure explaining the method of calculating the correction amount of an overlap area. FIG. 6 is a diagram illustrating a method of calculating a correction amount of an entire image from a correction amount of an overlapping area. FIG. 16 is a diagram illustrating a method of correcting the correction map created in FIG. 14 with the correction exclusion map created in FIG. 15. 14 is a flowchart showing the flow of one process for calculating the average value of overlapping regions in step 1310 shown in FIG. 13. 14 is a flowchart showing the flow of another process for calculating the average value of overlapping regions in step 1310 shown in FIG. 13. The figure explaining the method of calculating the matching degree of step 2030 shown in FIG. The figure explaining the interpolation processing of the correction amount of step 1430 shown in FIG. The figure explaining the interpolation processing of the correction amount of step 1430 shown in FIG. The figure explaining the interpolation processing of the correction amount of step 1430 shown in FIG. The figure explaining the interpolation processing of the correction amount of step 1430 shown in FIG. 6 is a flowchart showing the flow of processing for correcting the correction amount of the overlapping area. 6 is a flowchart showing the flow of processing for limiting the correction amount of the overlapping area. The figure which illustrated the threshold value table used by the process shown in FIG.

FIG. 1 is a diagram showing an appearance of an image pickup apparatus including an image processing system. Here, the imaging device is a spherical imaging camera, but the imaging device is not limited to this. The image capturing apparatus may be an image capturing apparatus capable of capturing overlapping image regions (overlapping regions) with a plurality of image capturing elements and connecting the captured plurality of images in the overlapping regions. The process of joining a plurality of captured images in the overlapping area can be executed by using an image processing IC or software.

The omnidirectional imaging camera 10 includes two fisheye lenses 11 and 12 having an angle of view of more than 180° and two image pickup devices 13 and 14 corresponding to the fisheye lenses 11 and 12 so that images can be taken in all directions from the imaging position. Equipped with. Here, a configuration including two fish-eye lenses 11 and 12 and two imaging elements 13 and 14 is illustrated, but the configuration is not limited to this, and a configuration including three or more may be used. It should be noted that the angle of view is a range in which an image can be captured using the fisheye lenses 11 and 12 in terms of angles.

The fisheye lenses 11 and 12 can adopt an equidistant projection method in which the distance from the center of the captured image is proportional to the incident angle of light. As the image pickup devices 13 and 14, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like that converts incident light into an electric signal can be used. The two image pickup devices 13 and 14 pick up images in all directions so that the picked-up images include overlapping regions as overlapping image regions.

The image capturing is performed by the photographer pressing the image capturing SW 15, and the image capturing elements 13 and 14 are exposed at the same time by using the image capturing SW 15 as a trigger. The image pickup devices 13 and 14 acquire an image by converting the received light into an electric signal. The acquired image is referred to as a fisheye image because it is an image obtained by using the fisheye lenses 11 and 12. The two fisheye images acquired by the two image pickup devices 13 and 14 are subjected to image conversion in the subsequent image processing and are combined in the overlapping region to generate a spherical image.

The omnidirectional imaging camera 10 can store data of the generated omnidirectional image, receive a request, output to a device including a display device such as a PC (not shown), and display the data on the display device. Further, the omnidirectional imaging camera 10 can also output the generated omnidirectional image to a printer (not shown), an MFP (Multi-Function Peripheral), or the like for printout. Further, it is also possible to output to an MFP or a PC for FAX transmission or mail transmission.

FIG. 2 is a diagram showing a hardware configuration of the omnidirectional imaging camera 10. In FIG. 2, the fisheye lenses 11 and 12 are omitted. The celestial sphere imaging camera 10 includes two image pickup devices 13 and 14 and an image pickup SW 15, a controller 20, an SDRAM 21, and an external storage device 22. The SDRAM 21 is used together with the controller 20 and stores a program for realizing predetermined image processing. The external storage device 22 stores image-processed data, that is, the above-described spherical image data.

The controller 20 includes a CPU 23, a ROM 24, an SRAM 25, an image processing block 26, an SDRAM I/F 27, and an external storage I/F 28, each connected to a bus 29. The CPU 23 controls the entire celestial sphere imaging camera 10. The ROM 24 stores a program for starting the omnidirectional imaging camera 10, a conversion table described later, and the like. The SRAM 25 provides a work area to the CPU 23, and the image processing block 26 executes the above predetermined image processing together with the CPU 23, the SRAM 25, the SDRAM 21 and the like. As the image processing block 26, an ASIC (Application Specific Integrated Circuit) which is an integrated circuit for a specific purpose can be used.

The omnidirectional imaging camera 10 acquires two fish-eye images by the two imaging elements 13 and 14. At this time, the image pickup devices 13 and 14 include an A/D converter, and the converted electric signal is converted into digital data by the A/D converter. The image pickup devices 13 and 14 output the digital data as fisheye image data to the image processing block 26 included in the controller 20. The image processing block 26, together with the CPU 23 and the like, performs the above-described image conversion as a predetermined image processing and a process of connecting the images to generate a spherical image, and stores the generated spherical image data in an external storage I/O. The data is stored in the external storage device 22 via F28.

The celestial sphere imaging camera 10 is also provided with a communication I/F, communicates with a PC, an MFP or the like (not shown) in a wired or wireless manner, and transmits celestial sphere image data to those devices for display or print output. Can be made. When wireless communication is performed, a wireless LAN such as Wi-Fi, Bluetooth (registered trademark), infrared communication, or the like can be used.

The fisheye lens 11 will be described in detail with reference to FIG. The fisheye lens 12 is the same as the fisheye lens 11, so only the fisheye lens 11 will be described here. A fish-eye image captured by the image sensor 13 having the fish-eye lens 11 having an angle of view of more than 180° is an image of a subject that is approximately a hemisphere centered on the image capturing position.

Here, as shown in FIG. 3A, when the incident angle of light on the fisheye lens 11 is φ, the distance between the center of the image and the image point is the image height h, and the projection function is f, these relationships are as follows: It can be expressed as Equation 1.

The projection function f differs depending on the property of the fish-eye lens 11, and for example, when an equidistant projection type fish-eye lens is adopted, as the incident angle φ becomes larger as shown by the arrow, as shown in FIG. It is represented by the proportional relationship that the height h increases. In addition, in FIG. 3B, the blackened region outside the circle is a region where no light is incident.

With reference to FIG. 4, an overlapping region of two fisheye images captured by the image pickup devices 13 and 14 will be described. In FIG. 4, the image sensor 13 is referred to as “image sensor 1”, and the image sensor 14 is referred to as “image sensor 2”. Since the fisheye lenses 11 and 12 have an angle of view exceeding 180°, the fisheye images picked up by the image pickup devices 13 and 14 include overlapping regions. FIG. 4A is a fish-eye image captured by the image sensors 13 and 14. The black-filled region is a region in which no light is incident, the white region is a region up to an incident angle of 90°, and a region indicated by diagonal lines. Indicates an area where the incident angle exceeds 90°.

The hatched area in FIG. 4A indicates an overlapping image area of two fisheye images, and thus can be defined as an overlapping area. However, in the fisheye lenses 11 and 12, the image height h increases, and distortion and aberrations are more likely to occur as the image point moves away from the center of the image. In addition, the outer frame around the fisheye lenses 11 and 12 may be reflected. Images such as regions or outer frames in which distortion or aberration has occurred cannot be used to join images.

Therefore, as shown in FIG. 4B, the overlapping area 30 can be limited to a ring-shaped area indicated by vertical stripes having a predetermined width inside thereof. Incidentally, in FIG. 4B, the two fish-eye images are obtained by simultaneously exposing and imaging the image pickup devices 13 and 14, so that the overlapping region 30 is basically an image of the same subject.

Next, the omnidirectional image will be described with reference to FIG. The fisheye image has a format in which a substantially half spherical surface shown in FIG. 5A is represented by a circle, and the longitude of the globe corresponds to the horizontal angle and the latitude corresponds to the vertical angle. The horizontal angle is in the range of 0 to 360°, and the vertical angle is in the range of 0 to 180°.

The spherical image has a format represented by a rectangle shown in FIG. 5B, and is an image generated by combining two hemispherical images in which the horizontal direction is the horizontal angle and the vertical direction is the vertical angle. is there. In reality, the image is larger than the hemisphere image due to the overlapping area, but here, it is referred to as a hemisphere image.

In the two hemisphere images, the same pixel value as the pixel value of the pixel corresponding to the horizontal angle and the vertical angle of the fisheye image is set to the same horizontal angle and vertical angle in the format represented by the rectangle shown in FIG. It is generated as an image of the corresponding pixel. This hemisphere image can be generated by projectively transforming a fish-eye image, and by combining the two generated hemisphere images, a hemisphere representing 360° in all directions in the horizontal and vertical directions. Images can be generated.

FIG. 6 illustrates a conversion table used in projective conversion of a fisheye image. As shown in FIG. 6A, the conversion table associates the coordinate values that are the values of the horizontal angle and the vertical angle of the fish-eye image that is the pre-change image with the coordinate values of the hemisphere image that is the post-change image. It is the attached table. The coordinate value of the pre-change image is represented by (x, y), and the coordinate value of the post-change image is represented by (θ, φ). As shown in FIG. 6B, with respect to the coordinates (0, 0) of the upper left corner of each image, the pixel in the pre-change image and the corresponding pixel in the post-change image are determined, and those pixels are The combination of coordinate values is held as data in the conversion table. The correspondence relationship can be obtained from the projective relationship between the pre-conversion image and the post-conversion image.

The conversion table can be created in advance based on the lens design data and the like for each of the two fisheye lenses 11 and 12 and the image pickup devices 13 and 14, and is stored in the ROM 24 shown in FIG. It can be read and used accordingly. By using this conversion table, the fisheye image can be projectively converted to correct the distortion of the fisheye image. A celestial sphere image can be generated by combining the corrected images.

The flow of processing for generating a spherical image will be described with reference to FIG. 7. Two fish-eye images are picked up by the two image pickup devices 13 and 14, and the two fish-eye images are input to start the processing from step 700. In step 710, distortion correction is performed by projectively converting each fisheye image using the conversion table as shown in FIG. 6A stored in the ROM 24. By this distortion correction, two hemisphere images shown in FIG. 5A are obtained.

In step 720, a joining position is detected in order to join the two obtained hemisphere images in the overlapping region. Details of the detection of the connection position will be described later. In step 730, the conversion table used in step 710 is corrected based on the detection result. The purpose of the correction and the details of the specific processing will be described later. In step 740, rotation conversion is performed on the corrected conversion table to create a conversion table for image generation. The purpose of the rotation conversion is to match the vertical direction of the image with the zenith direction of the omnidirectional imaging camera 10 in the conversion table for image generation.

In step 750, projective transformation is performed on the two fish-eye images using a transformation table for image generation to correct image distortion. In step 760, a blending process for combining the two images whose distortions have been corrected is executed. The two images are combined in the overlapping area, but when the data exists only in the overlapping area of one image, the data is used as it is to perform the combining. After the blending process is completed, the process proceeds to step 770 and this process is completed.

The distortion correction in step 710 of FIG. 7 will be described in detail with reference to FIG. When two fisheye images picked up by the two image pickup devices 13 and 14 are projectively converted using the respective conversion tables and distortion correction is performed, each of them has a rectangular semi-sphere as shown in FIG. 8A. Converted to an image. The image 31 captured and converted by the image sensor 13 is the upper image in FIG. 8A, and the image 32 captured and converted by the image sensor 14 is the same as the lower image in FIG. 8A. Become. The image area in the central portion where the upper image and the lower image overlap becomes the overlapping area 30, and the overlapping area 30 can be easily detected.

In the image thus converted, the vertical direction of the image is the zenith direction shown by the arrow A of the omnidirectional imaging camera 10 including the image pickup devices 13 and 14 having the fisheye lenses 11 and 12 shown in FIG. 8B. It is rotated by about 90° with respect to.

The process of detecting the connection position will be described with reference to FIG. 9. Template detection, which is generally well known, can be used to detect the connection position. In the template matching, after detecting the overlapping area as shown in FIG. 8, as shown in FIG. 9A, an image of a predetermined size in the overlapping area 30 is obtained from one of the two hemisphere images. The template image 34 is taken out. A plurality of template images 34 having a size w in the horizontal direction and a size h in the vertical direction are taken out at equal intervals at a predetermined interval step. In FIG. 9A, the template image 34 is a rectangular image indicated by numbers 1-6. In the template image 34, the coordinates of the upper left corner of the image are set as the extraction coordinates (sx1,sy1) of the template image 34 and the like. The extracted template image 34 is, for example, the image shown in FIG.

As shown in FIG. 9C, the connecting position is the search position at the coordinates (kx,ky) of the upper left corner of the template image 34, and the search position is set in the other overlapping area of the two semi-sphere images. While shifting, the position with the highest evaluation value is searched for and detected. The search range can be within the overlap region. The difference between the coordinate (kx, ky) detected by the template matching and the coordinate value (sx1, sy1) of the extracted coordinate of the template image 34 is output as the connection position detection result. This detection result is discrete data because a plurality of template images are taken out at a predetermined interval step. Therefore, it is possible to obtain the data between the data by performing linear interpolation or the like.

The overlapping areas are images of the same subject, but parallax occurs because the image pickup devices 13 and 14 are directed differently with respect to the subject. When parallax occurs, the subject looks like a double image, so it is necessary to match it with one of the images. The detection of the connection position is performed to correct such parallax. Since the parallax varies depending on the position of the subject, it is necessary to detect the optimum connection position for each area within the overlapping area each time an image is captured.

The correction of the conversion table performed in step 730 in FIG. 7 is performed using the coordinates detected by the template matching. Specifically, the difference as the detection result of the connection position is added to the coordinate value (x, y) of the conversion table used for one fish-eye image, and the coordinate value is corrected. Since this correction is a correction for joining one fish-eye image to the other image to be joined, the correction of the conversion table used for the other fish-eye image is not performed.

So far, the process of converting the fish-eye images captured by using the two image pickup devices 13 and 14 and connecting them to generate a spherical image has been described. Hereinafter, a process of determining whether flare has occurred in the fisheye image, correcting the flare in the fisheye image appropriately, and reducing the difference in luminance and the color difference caused by the flare will be described. Here, flare is taken as an example in which a difference in brightness and a color difference are generated, but the example is not limited to flare as long as the difference results in an unnatural image.

FIG. 10 is a functional block diagram of an image processing system for realizing this processing. The image processing system is an image processing system that performs image processing on a plurality of images input from the plurality of image pickup devices 13 and 14. Image processing system
The calculation unit 40, the determination unit 41, the image determination unit 42, and the image correction unit 43 are included. These functional units are realized, for example, by the CPU 23 shown in FIG. 2 executing a program stored in the SDRAM 21 or by the image processing block 26.

The calculation unit 40 calculates an evaluation value for evaluating each image by using the pixel value of one or more pixels in each overlapping area. The evaluation value can be, for example, an average value or a variance value of pixel values of a plurality of pixels in each overlapping area. The pixel value can include the signal amount of each color of RGB when the RGB color space is adopted in the color image. When the YCbCr color space is adopted, it is possible to include a luminance value, a hue and saturation value of a blue color, and a hue and saturation value of a red color.

The determination unit 41 determines whether or not there is a correction target image among the plurality of images based on the evaluation value calculated by the calculation unit 40. For example, the determination unit 41 has a threshold value and can determine the presence or absence of the correction target image by comparing with the threshold value. Specifically, the determination unit 41 can determine that an image whose average value or variance exceeds the threshold value is a correction target image, and if there is such an image, determine that there is a correction target image.

The image determination unit 42, when the determination unit 41 determines that there is an image to be corrected, based on the evaluation value calculated by the calculation unit 40, a correction reference that is a reference when performing correction from a plurality of images. Determine the image. The correction reference image can be selected from the images excluding the correction target images of the plurality of images. For example, an image that is combined with the correction target image and is not the correction target image is used as the correction reference image. You can decide. In the case of the omnidirectional imaging camera 10 described above, if one is the correction target image, the other image can be determined as the correction reference image.

The image correction unit 43 corrects the correction target image based on the correction reference image determined by the image determination unit 42. The image correction unit 43 corrects the brightness and color of the correction target image so as to approach the brightness and color of the correction reference image. By this correction, it is possible to reduce the difference in brightness and the difference in color between the images to be combined.

Processing executed by the image processing system will be described with reference to FIG. 11. The image processing system starts the process from step 1100 by accepting the inputs of the two fish-eye images captured by the two image sensors 13 and 14. In step 1110, the calculation unit 40 calculates an evaluation value for evaluating each image using the pixel value of one or more pixels in each overlapping region of the two fisheye images.

In step 1120, the determination unit 41 determines whether or not there is an image with flare (flare image) in the two fisheye images based on the evaluation value calculated by the calculation unit 40. In step 1120, if there is a flare image, the process proceeds to step 1130, and if it does not exist, the process proceeds to step 1150, and this process ends.

In step 1130, the image determination unit 42 determines the image in which flare has not occurred as a correction reference image, and the image correction unit 43 uses the evaluation value calculated in step 1110 to perform correction for correcting the flare image. Create a correction map that maps the values. In step 1140, the image correction unit 43 corrects the flare image using the created correction map. After the correction is completed, the process proceeds to step 1150 and this process is completed.

Calculation of the evaluation value in step 1110 shown in FIG. 11 will be described with reference to FIG. Since the evaluation value is calculated using the pixel values of one or more pixels in the overlapping area, it is necessary to acquire the pixel values of one or more pixels in the overlapping area 30 shown by the vertical stripes in FIG. is there. One method will be described below. Since this method is an example, any method can be used as long as the evaluation value can be calculated.

First, as shown in FIG. 12A, each of the two input fish-eye images is divided into a plurality of evaluation areas each having a predetermined number of rectangles each having the same size. For example, when the size of each fish-eye image is 1952 (pix)×1932 (pix), it can be divided into 48×48. Since the number of divisions is an example, it is possible to determine the optimum number of divisions by a test or the like and adopt the number of divisions. The image processing system may include a region dividing unit that divides the image into a plurality of evaluation regions as a functional unit for performing such division.

Next, as shown in FIG. 12B, an evaluation area including the entire area within the overlapping area 30 indicated by vertical stripes is detected as an evaluation area corresponding to the overlapping area 30. In FIG. 12B, the detected evaluation area 35 is shown in gray. The image processing system can include an area detection unit as a functional unit for performing this detection.

The detected evaluation area 35 is composed of a plurality of pixels. Each pixel has a pixel value. For the evaluation value, for each evaluation area 35, the pixel values are summed and divided by the number to obtain an average value, and the average values obtained for each evaluation area are summed and divided by the number to obtain an average value. Can be calculated by Further, the evaluation value can be calculated by using the average value obtained for each evaluation area and obtaining the variance value thereof. Therefore, as the evaluation value, the above average value or variance value can be used. The image processing system can include an average value calculation unit for calculating the average value of the evaluation area as a functional unit.

When the omnidirectional imaging camera 10 having the structure shown in FIG. 1 is used to capture an image of 360°, the photographer's finger pressing the image capturing SW 15 may be greatly reflected. In this case, in the overlapping area 30, one has a finger image and the other has no finger image. Since the average value and the variance value calculated depending on the presence or absence of the image of the finger greatly change, it is desirable not to use the image of that portion in the calculation of the evaluation value.

Therefore, the lower 1/4 area of the overlapping area 30 where the finger is expected to be reflected is set as an unnecessary area, and only the evaluation areas included in the other overlapping areas 30 are evaluated as the evaluation areas corresponding to the overlapping area 30. It can be detected as a region 35. Here, the lower 1/4 region of the overlapping region is set as the unnecessary region, but the present invention is not limited to this, and a region of ⅕ or less or 1/3 or more may be the unnecessary region. The area may be added to the unnecessary area.

By setting unnecessary areas in this way and excluding obstacles such as fingers from the evaluation target, it is possible to obtain the average value or variance value using only the evaluation area with high correlation between the flare image and the correction reference image. it can.

The determination of the presence or absence of the flare image in step 1120 shown in FIG. 11 will be described with reference to FIG. This process starts from step 1300 in response to the calculation of the evaluation value. In step 1310, all the average values of the evaluation areas corresponding to the overlapping areas calculated as the evaluation values are averaged to calculate the average value. This process is performed for each image to be joined. Therefore, when joining two images, an average value is calculated for each of the two images.

In step 1320, the average values calculated between the stitched images are compared. Here, the absolute value of the difference between the average values is calculated, and the image with the minimum average value is extracted. Then, these pieces of information are output as comparison results. For example, in the case where image 1 and image 2 are connected, if the average values are AVE_1 and AVE_2, the comparison results are |AVE_1-AVE_2|, AVE_1>AVE_2, image 2 and AVE_1<AVE_2, image 1 Become.

In step 1330, it is determined whether or not the absolute value of the difference of the comparison result is larger than a preset threshold T. If the difference is large, the difference in the brightness value or the like is large, so the process proceeds to step 1340, flare occurs, and it is determined that the image is a flare image. On the other hand, if the difference is small, the difference is small, and therefore the process proceeds to step 1350, and it is determined that flare has not occurred and the image is not a flare image. When this determination is completed, the process proceeds to step 1360 and this process is completed.

The image having the smallest average value extracted and compared in step 1320 can be determined as a correction reference image that is an image in which flare has not occurred. Basically, when images are taken by simultaneous exposure, the images of the same subject in the overlapping area should be taken with the same brightness. However, when flare occurs in one of the images, the average value of the overlapping area is high. Therefore, the threshold value is set as described above, and when it is larger than the threshold value, it can be determined that the image is a flare image. Further, since it is highly possible that flare does not occur in the image having the smallest average value, it can be used as the correction reference image.

The creation of the correction map in step 1130 shown in FIG. 11 will be described with reference to FIG. In response to the determination that there is a flare image, this process is started from step 1400. In step 1410, a correction exclusion map is created that specifies the area where the correction is excluded. The correction exclusion map is a correction exclusion map for a block at a position corresponding to an evaluation region included in a correction exclusion region in the correction target image determined based on each pixel value of a plurality of pixels forming the correction target image. The value to be stored is stored. In addition, values not excluded from correction are stored in the other blocks. The image processing system can include, as a functional unit, an area determination unit for determining a correction exclusion area in order to create this correction exclusion map.

When correcting an image in which flare has occurred based on an image in which flare has not occurred and which has the smallest average value, the entire image in which flare has occurred is corrected. When the correction is performed on the entire image, the brightness is lowered as a whole and the color is changed. Then, when the images are combined, the resulting image may look unnatural. For example, although the actual brightness or color of the light source is the brightness or color of the image in which flare is occurring, the brightness is reduced by the correction, and the color also becomes dark. Therefore, a correction exclusion map is created so that the image of the subject that should not be corrected like the light source is not corrected.

In step 1420, the correction amount of the overlapping area is calculated from the evaluation value such as the average value or the variance value of the evaluation areas corresponding to the overlapping area. The correction amount is a numerical value indicating how much the luminance value should be reduced and how much the color should be changed. In step 1430, based on the correction amount calculated in step 1420, the correction amount of the correction target image, that is, the entire image in which flare has occurred is calculated by interpolation processing, and a correction map is created using the calculated correction amount. To do. The correction map stores each calculated correction amount in a block at a position corresponding to each evaluation region of the correction target image.

In step 1440, the correction exclusion map created in step 1410 is applied to the correction map created in step 1430 to modify the correction map. Then, in step 1450, if extremely high or low values exist in the correction values in the correction map, leveling processing for leveling those values, that is, low-pass filter (LPF) processing is performed. A Gaussian filter can be used as the LPF.

The LPF process is not limited to one time and may be performed a plurality of times. However, if the number of times is large, the correction value for the image that has been corrected is greatly changed, and the meaning of the correction is lost. Therefore, it is preferable that the number of times is small. In the embodiment shown in FIG. 14, twice as shown in step 1460. The image processing system may further include a leveling processing unit as a functional unit for performing this leveling processing.

In step 1460, it is determined whether the LPF process has been performed twice. If it has not been performed twice, the process returns to step 1440. If it has been performed twice, the process proceeds to step 1470. In step 1470, the correction map is resized. The resizing process is a process of changing the number of evaluation regions included in the correction map to the number of pixels of the correction target image. Therefore, the image processing system can include the size change processing unit as a functional unit.

As shown in FIG. 12, the correction map generated up to step 1440 has the size of the number of divisions obtained by dividing the fisheye image in the horizontal direction and the vertical direction. That is, if it is divided into 48×48, the size is 48×48. Since the actual fish-eye image has a size of 1952 (pix)×1932 (pix) in the above example, the resizing process changes the size from 48×48 to 1952×1932. When this resizing process is completed, the process proceeds to step 1480, and this process is completed.

As the resizing method, any method known so far can be used. For example, bilinear, nearest neighbor, bicubic, and other methods can be used to resize.

A method of creating the correction exclusion map in step 1410 shown in FIG. 14 will be described with reference to FIG. In FIG. 15, as shown in FIG. 15A, in the image in which the light source 50 is reflected, the image of the light source 50 is set as the image of the subject whose correction is excluded. First, as shown in FIG. 15B, the image is divided into a plurality of evaluation areas, the average value and the variance value of each evaluation area are calculated, and it is determined for each evaluation area whether or not it is a correction exclusion area. It should be noted that this determination can be performed by the area determination unit.

The condition of the correction exclusion area can be “high brightness and achromatic color” or “high brightness and the maximum value of brightness is greater than or equal to a threshold”. When “high brightness and achromatic color” is set as the condition 1, high brightness can be set to 200 or more when the brightness value is 0 to 255, and achromatic color is set to when the color difference value is −127 to 127, for example. It can be -1 to 1. When the condition 2 is “high brightness and the maximum value of brightness is greater than or equal to a threshold”, the high brightness can be set to, for example, 230 or more when the brightness value is 0 to 255, and the brightness value is 0 to 255 as a threshold. In this case, it can be set to 250, for example. These numerical values are examples and the present invention is not limited to these numerical values.

It is desirable to determine the condition of the correction exclusion area based on both the condition 1 and the condition 2. When one evaluation area as a whole has uniform brightness and color, it can be judged only by the condition 1. However, when a high-luminance subject such as the light source 50 and a low-luminance subject such as a tree branch exist in one evaluation area, Condition 1 is not satisfied, so the image of the light source 50 is corrected, and the light source 50 is corrected. Is a dark and unnatural image. By including this condition 2, an image including such a light source 50 and a tree branch can be appropriately extracted as a correction exclusion area.

In FIG. 15B, eight regions 51 are extracted as regions that meet these conditions. The correction exclusion map has the same format as that when the correction target image is divided into a plurality of evaluation areas. As shown in FIG. 15C, the correction exclusion map stores “1”, which is the value of the correction exclusion, in the block at the position corresponding to the evaluation area that matches the condition. A value "0" that is not excluded from correction is stored in the block at the position corresponding to the evaluation area that does not meet the conditions.

A method of calculating the correction amount in step 1420 shown in FIG. 14 will be described with reference to FIG. 16A is a diagram showing a flare image and a reference image, and FIG. 16B is an enlarged view of the evaluation region 35 included in the overlapping region 30 shown by a broken line in FIG. 16A. FIG. The evaluation area 35 corresponding to the connecting position is an area indicated by the same reference numeral. That is, an area in which the evaluation area indicated by reference symbol a in the left figure extracted from the flare image in FIG. 16B and the evaluation area indicated by reference symbol a in the right figure extracted from the reference image are connected to each other. Become. This is because the omnidirectional camera 10 images the front and back sides at the same time, and the same image in the overlapping area is rotated by 180°.

At the time of joining, the values of the two evaluation areas indicated by the symbol a show a high value because one flare has occurred, and the other shows a lower value because no flare has occurred. For this reason, it is necessary to perform correction so that the same value is obtained. The correction amount c used for the correction can be calculated by the following Expression 2 or Expression 3. In Expression 2, E _b represents the evaluation value of the reference image as the correction reference image, and E _t is the evaluation value of the correction target image, that is, the flare image.

The correction amount c may be calculated using either of the above equations 2 and 3. The calculated correction amount c is stored as the value of the corresponding evaluation area of the flare image. Therefore, when the correction amount c for the code a shown in FIG. 16B is calculated, the correction amount c is stored as the value of the code a.

The correction amount interpolation processing of step 1430 shown in FIG. 14 will be described with reference to FIG. The correction amount c of the overlapping area is calculated by using the above Expression 2 or Expression 3. For the inside of the overlapping area, the correction amounts calculated for the plurality of evaluation areas 35 in the overlapping area are used, and the correction amount is interpolated toward the center of the image as shown by the arrow in FIG. The amount of correction for each evaluation area is determined. The detailed method will be described later.

As for the outside of the overlapping area, the LPF processing is performed in the subsequent step 1450. Therefore, it is necessary to set a correction amount that does not significantly change the correction amount of the overlapping area even if the LPF processing is performed. Thus, for example, as shown in FIG. 17B, the area indicated by the evaluation area x has the same correction amount as that of the evaluation area one level above, and the area indicated by the evaluation area y has the same correction amount. The correction amount is the same as the correction amount of the evaluation area on the left side. Note that which evaluation area to use for the correction amount can be determined in advance by setting.

The correction of the correction map in step 1440 shown in FIG. 14 will be described with reference to FIG. 18A is a diagram showing a part of the correction map created by the processing shown in FIG. 14, and FIG. 18B is a diagram showing a part of the correction exclusion map shown in FIG. 15C. Is. In the correction map shown in FIG. 18A, the ratio of the evaluation values of the above formula 2 is set as the correction amount, and the correction amount is stored as the correction amount of each evaluation area.

To correct the correction map shown in FIG. 18A, refer to the correction exclusion map shown in FIG. 18B, and when the correction amount of the correction exclusion map in the corresponding evaluation area is “0”, Do not modify. On the other hand, when the correction amount of the correction exclusion map is "1", the correction amount is changed to a value such that the correction amount is not corrected. Referring to FIG. 18B, since the value of the correction amount 1 is stored in the evaluation areas of the coordinates (x3,y2), (x3,y3), (x4,y2), it is at the same position on the correction map. Replace the block value with "1.00" and correct the correction map. As a result, the correction map can be corrected for the correction exclusion area so as to weaken the correction effect.

The correction of the flare image is performed using the correction map after the resizing process is performed in the process shown in FIG. The correction map has an image size of the same number of pixels as the flare image due to the resizing process. When the correction map stores the correction amount of each pixel determined using the ratio of the evaluation values calculated by Equation 2, the correction amount of each pixel corresponding to the pixel value of each pixel in the flare image. To correct the pixel value of each pixel in the flare image. When the correction map stores the correction amount of each pixel determined using the difference between the evaluation values calculated by Equation 3, the correction amount of each corresponding pixel is calculated from the pixel value of each pixel in the flare image. Is added to correct the pixel value of each pixel in the flare image.

In the case of a monochrome image, there is one correction map for the flare image, but in the case of a color image, the flare image is composed of three planes, and therefore there are three correction maps. That is, if the color image is an RGB image, the flare image is composed of planes of three colors of red, blue, and green, three correction maps for correcting each plane are created, and the flare image is corrected. Used for.

Since the flare image can be corrected in this way, it is possible to reduce the difference in luminance and the color difference when they are combined. Further, since it is possible to exclude the correction of the light source or the like and not perform the correction for the light source or the like, it is possible to prevent the light source from becoming dark and prevent the color shift of the saturated portion of the color. Furthermore, by applying an LPF to the correction map and performing a leveling process, a sharp change in the correction amount is eliminated, and a change in the pixel value between the light source excluding the correction and its surroundings is mitigated, resulting in an unnatural correction. It can be prevented.

Another example of the process of calculating the average value of the overlapping areas in step 1310 shown in FIG. 13 will be described with reference to FIG. In FIG. 13, the average value is calculated using all the evaluation areas in the overlapping area, but in the example shown in FIG. 19, conditions are set in advance and the evaluation areas are classified to calculate the average value. Specifically, before starting the processing, thresholds (upper limit T _up , lower limit T _dw ) are set in advance for the average value of the evaluation area, and the values are set as the upper limit value and the lower limit value.

This process is started from step 1900, and in step 1910, one of the evaluation areas is arbitrarily selected, and it is determined whether or not the evaluation area is within the overlapping area. The evaluation area can be selected in order from the evaluation area in the upper left corner of the image, for example. Since this is an example, it is possible to select by other methods.

If the evaluation area is not within the overlapping area, the process proceeds to step 1920, and the average value of the evaluation area is excluded from integration. That is, the average value is not integrated. On the other hand, if it is in the overlapping area, the process proceeds to step 1930, and it is determined whether the average value of the evaluation area is larger than the upper limit T _up . If it is larger, the process proceeds to step 1920, and the average value of the evaluation area is excluded from integration. If they are the same or smaller, the process proceeds to step 1940, and it is determined whether the average value of the evaluation area is smaller than the lower limit T _dw . If it is smaller, the process proceeds to step 1920, and the average value of the evaluation area is excluded from integration. If they are the same or larger, the process proceeds to step 1950 and the average value of the evaluation area is integrated.

If the integration is excluded in step 1920, or if the integration is performed in step 1950, the process proceeds to step 1960, and it is determined whether or not all evaluation areas have been completed. If it has not been completed yet, the processing returns to step 1910, and the same processing is performed on the next evaluation area. When the processing is completed, the process proceeds to step 1970, the value obtained by integrating the average values of the evaluation areas in the overlapping area is divided by the number of the evaluation areas to calculate the average value of the overlapping area, and this processing is performed in step 1980. finish.

The upper limit T _up and the lower limit T _dw can be determined in consideration of the influence when determining the presence/absence of a flare image so that an evaluation region in which integration should be excluded can be appropriately excluded.

Still another example of the process of calculating the average value of the overlapping areas in step 1310 shown in FIG. 13 will be described with reference to FIG. In FIG. 19, the upper limit T _up and the lower limit T _dw are provided as the threshold values, but in the example shown in FIG. 20, the threshold value of the index called the matching degree (matching degree) is set, and the evaluation area is calculated based on the matching degree. , It is determined whether the evaluation area is excluded from the integration. The matching degree is an index indicating the degree of matching between the image of each evaluation area included in the overlapping area of the correction reference image and the image of each evaluation area included in the overlapping area of the flare image that is the correction target image. .. The image processing system may include a matching degree calculation unit for calculating the matching degree as a functional unit.

This processing is started from step 2000, and in step 2010, it is determined whether or not the evaluation area is within the overlapping area. If it is not in the overlapping area, the process proceeds to step 2020, and the average value of the evaluation area is excluded from integration. If it is within the overlapping area, the process proceeds to step 2030, and the matching degree of the evaluation area is calculated. The matching degree and its calculation method will be described later. In step 2040, it is determined whether the calculated matching degree is smaller than a matching degree threshold value T _m which is a preset matching degree threshold value.

If it is larger, the process proceeds to step 2020, and the average value of the evaluation area is excluded from integration. If they are the same or smaller, the process proceeds to step 2050 and the average value of the evaluation area is integrated. If the integration is excluded in step 2020, or if the integration is performed in step 2050, the process proceeds to step 2060, and it is determined whether or not all evaluation areas have been completed. If not finished yet, the process returns to step 2010, and the same processing is performed on the next evaluation area. When the processing is completed, the process proceeds to step 2070, the value obtained by integrating the average values of the evaluation areas in the overlapping area is divided by the number of the evaluation areas to calculate the average value of the overlapping area, and this processing is performed in step 2080. finish.

With reference to FIG. 21, the matching degree and the method of calculating the matching degree used in FIG. 20 will be described. The relationship between the connection position and the evaluation area is as shown in FIG. 16, but since the evaluation area divides the image evenly in the horizontal direction and the vertical direction, the connection position and the evaluation area completely match each other. Not not.

For example, as shown in FIG. 21, the evaluation area 35 is a gray area, and the coordinates of the center of the area are O ₁ and O ₂ . Then, the connection positions detected by template matching and the like are defined as P ₁ and P _2, and a region indicated by a broken line having the same size as the evaluation region centering on the connection position is defined as a connection region 36.

The image of the subject included in the connection area 36 is the same in the flare image and the reference image. When the center O ₁ of the evaluation area 35 and the center P _{1 of} the connection area 36 and the center O _{2 of} the evaluation area 35 and the center P ₂ of the connection area 36 match, the evaluation area 35 and the connection area 36 match, so The images of the subject in the region 35 match.

However, if any of the center coordinates do not match, the images of the subject in the evaluation area 35 do not match. As long as the images of the subject such as the sky or the flat wall are small, even if the images of the subject do not completely match, the evaluation value is not significantly affected. On the other hand, in the case of an image of a subject whose change is large, the brightness and tint differ with a slight shift, and the influence on the evaluation value is large. Therefore, it is determined whether or not the images of the subject in the evaluation area 35 are matched by using the index of the matching degree.

The matching degree can be calculated by the following Expression 4 or Expression 5 using the variance value calculated for each evaluation area 35. In Expression 4, m is the degree of matching, σ ₁ is the dispersion value of the flare image, and σ ₂ is the dispersion value of the reference image. Further, in Expression 5, v ₁ is the brightness value of the flare image, and v ₂ is the brightness value of the reference image.

The degree of matching m in the above equation 4 is defined by the absolute value of the difference in the variance value between the flare image and the reference image. Therefore, the higher the degree of matching is, the more the images of the subject generally match, but in this case, the smaller the degree of matching is, the more the images of the subject match. The variance value tends to increase when the change of the subject image is large, and the variance value tends to decrease when the change of the subject image is small. From this, an image of a subject having a large change has a large influence on the matching degree even if it is slightly displaced, and conversely, an image of a subject having a small change has a small influence on the matching degree.

Note that the pixel value used to calculate the matching degree may be the pixel value of all pixels, but using all the pixel values has a calculation cost, which affects the brightness of the image. It is preferable to use only pixel values. As such a pixel value, a G value can be used for an RGB image, and a Y value can be used for a YCbCr image.

In the above formula 5, it is shown that the variance value is not used and only the brightness value is calculated. Since the matching degree m in Expression 5 is defined by the absolute value of the difference between the brightness values, the smaller the matching degree is, the more the images of the objects match, as in Expression 4.

One example of the correction amount interpolation method shown in FIG. 17 will be described with reference to FIG. The average value of the plurality of evaluation areas 35 in the overlapping area 30 shown in gray in FIG. 22A is averaged to calculate the average value of the overlapping area. The calculated average value is used as the correction amount of the portion indicated by the diagonal line in the center inside the overlapping region 30.

As shown in FIG. 22B, a center C, an overlapping area O of intersections extending from the center C in the horizontal direction and the vertical direction toward the overlapping area, and a calculation target area T therebetween are set as the correction amount r. Let _c , r _o , and r _t . Further, the distance between the center C and the area T is d ₁ , and the distance between the area T and the overlapping area O is d ₂ . Then, the correction amount r _t of the region T can be expressed by the following formula 6 calculated by a weighted average using the distance as a weight.

The correction amount of each area extending in the vertical direction and the horizontal direction from the center C of FIG. 22B toward the overlapping area is calculated by using the above Expression 6. The correction amount of the white region between the calculation region shown in gray and the overlapping region in which the correction amount of FIG. 22C is calculated can be calculated by a weighted average according to the distance between them. it can. Let T be the area to be calculated, and H be the intersection with the calculation area when extending from the area T in the vertical direction. Further, the intersection point with the overlapping area when extended downward in the vertical direction is O ₁ , the intersection point with the calculation area when extending from the area T to the right side in the horizontal direction is V, and the overlapping area when extended to the left side in the horizontal direction with The intersection of is set to O ₂ . The distance between the region T and the region V is d _h1 , the distance between the region T and the region O ₂ is d _h2 , the distance between the region T and the region H is d _v1 , and the distance between the region T and the region O ₁ is d _v2 . To do. Further, the respective correction amounts of the regions T, H, V, O ₁ and O ₂ are defined as r _t , r _h , r _v , r _o1 and r _o2 . Then, the correction amount r _t of the region T can be calculated by the following Expression 7.

The correction amounts of all the evaluation areas inside the overlapping area can be calculated using the above Expressions 6 and 7.

In the correction amount interpolation method shown in FIG. 22, interpolation is performed uniformly regardless of a specific direction. If there are images of the same subject in the overlapping areas in the images to be joined, no problem will occur, but if there are obstacles and the images of the subjects are different, problems will arise with such uniform interpolation. For example, there is a case where the above finger is shown in only one image as a subject. In such a case, since the interpolation processing cannot be performed uniformly, the interpolation processing is performed sequentially from a specific direction as in this example. This method will be described with reference to FIG.

As shown in FIG. 23A, for example, three evaluation areas O ₁ , O ₂ , and O ₃ in the overlapping area are used, and the areas one step below the uppermost portion in the vertical direction of the overlapping area are sequentially moved downward. To calculate the correction amount. First, the area T ₁ inside the overlapping area, which is in contact with the uppermost evaluation area in the overlapping area shown in FIG. 23A, is set as the area for which the correction amount is calculated. The correction amount is evaluated area _{O 1} in contact with the overlapping region inside of one level upper region _{T 1,} the correction amount _{r 1, r} 2 of the evaluation region _O 2, _{O 3} in contact with the overlapping region inside the left and right regions _{T 1,} and r _3, the distance _{d 1} of the region _{T 1} and area _{O 2,} is calculated using the distance _{d 2} in the region _{T 1} and region _{O 3.} This correction amount r _t1 can be calculated using the following equation 8.

The above equation 8 is used to calculate the correction amount for all areas inside the overlapping area in the horizontal direction on the left and right of the area T ₁ shown in FIG. Next, the correction amount r _t2 of the area T ₂ inside the overlapping area that is one step lower than that is calculated. The correction amount r _t2 is the correction amount r _p of the region P one step above the region T ₂ , and the correction amounts r ₄ and r ₅ of the evaluation regions O ₄ and O _{5 that} are in contact with the inner side of the horizontal overlapping region. T ₂ and the region _{O 4} of the distance _{d 3,} is calculated using the distance _{d 4} areas _{T 2} and the region _{O 5.} This correction amount r _t2 can be calculated by the following equation 9. By repeatedly calculating this downward in the vertical direction, it is possible to calculate the correction amounts of all the areas inside the overlapping area.

If the evaluation area in the overlapping area has an extremely large or small correction amount, the correction amount may be spread over the entire area by interpolation, resulting in an unnatural image.

In the creation of the correction exclusion map in step 1410 shown in FIG. 14, when the average coordinate value of the block that has become the correction exclusion area is calculated, the average coordinate value can be obtained as the center of the correction exclusion area. If a correction amount is set so as not to perform correction on a predetermined region (center region) including the obtained center, and interpolation processing is performed toward that center region, an extremely large or small correction amount is obtained. Ripple can be minimized.

As shown in FIG. 24, when the central area of the correction exclusion area is an area indicated by diagonal lines, the area is set as an interpolation reference area, and the evaluation area inside the overlapping area from the surrounding overlapping area toward the interpolation reference area. The correction amount of is calculated by interpolation. The correction amount can be calculated using the weighted average shown in the above equations 6 and 7. It should be noted that the calculation may be performed using Equations 8 and 9.

In FIG. 24, the correction amount of the evaluation area inside the overlapping area is calculated by interpolation from four directions toward the interpolation reference area, but it may be calculated by interpolation specialized in a specific direction. The cases where the calculation should be performed in a specific direction are, for example, when there are obstacles in the overlapping area or when there is a problem with uniform interpolation. The shaded area shown in FIG. 25 stores a value such that the interpolation reference area 60 extends in the vertical direction and the interpolation reference area 60 is not corrected. Therefore, the inner area of the overlapping area is divided into two by the interpolation reference area 60, and the correction amount can be calculated by the weighted average shown in the above equations 6 and 7 for each of the divided areas. It should be noted that the calculation may be performed using Equations 8 and 9.

The overlapping regions having a high degree of matching include images of the same subject, and thus have high reliability in joining. On the other hand, the overlapping area having a low degree of matching includes images of different subjects, and thus has low reliability in joining. Therefore, in the overlapping area having a low degree of matching, an incorrect value is likely to be calculated even if the correction amount is calculated.

With reference to FIG. 26, a process of correcting the correction amount of the overlapping area using the matching degree will be described. This correction can be performed after calculating the correction amount of the overlapping area in step 1420 shown in FIG. Before starting the processing, a threshold value T _m is provided for the matching degree and the value is set.

This process is started from step 2600, and in step 2610, it is determined whether or not the evaluation area is within the overlapping area. If it is not within the overlap area, the correction amount is not corrected for the evaluation area, and the process proceeds to step 2650. If it is within the overlapping area, the process proceeds to step 2620 and the matching degree of the evaluation area is referred to. If the matching degree is not calculated, the matching degree is calculated by the above method.

In step 2630, it is determined whether or not the referred matching degree is larger than a preset matching degree threshold value T _m . If it is smaller, the correction amount is not corrected for the evaluation area, and the process proceeds to step 2650. On the other hand, if it is larger, the process proceeds to step 2640 and the correction amount of the evaluation area is corrected. In the correction, the correction amount of the evaluation area is corrected to a value such that the correction is not performed. Specifically, when the correction amount is a value represented by the ratio of the above evaluation values, the correction amount is corrected to “1.0”, and when the correction amount is a value represented by the difference between the above evaluation values, the correction amount Is corrected to "0".

In step 2650, it is determined whether the processing has been completed for all the evaluation areas. If it has not been completed yet, the processing returns to step 2610, and the same processing is performed on the next evaluation area. When the processing is completed, the process proceeds to step 2660 and the processing is completed.

In the calculation of the correction amount of the overlapping area in step 1420 shown in FIG. 14, an unintended correction amount may be calculated. For example, there are cases where there are no images of the same subject in the overlapping areas to be joined, or where the brightness is extremely different. Even in such a case, the correction amount can be limited so that the correction amount falls within a predetermined range so that the correction amount does not become extremely large or small.

The restriction process will be described with reference to FIG. This limiting process can be executed after the correction amount of the overlapping region is calculated in step 1420 shown in FIG. 14 or after the correction amount is calculated by the interpolation process in step 1430. Before starting the limiting process, a threshold table showing the relationship between the pixel value of the flare image and the threshold of the limiting process is created.

The threshold value table can be, for example, a graph as shown in FIGS. 28A and 28B, or a table holding these as numerical values. FIG. 28A shows a table when the correction amount is the ratio of the evaluation values, and FIG. 28B shows a table when the correction amount is the difference of the evaluation values. In the threshold table, thresholds of the upper limit T _up and the lower limit T _dw corresponding to the signal amount as the pixel value of the flare image are defined.

The upper limit T _up and the lower limit T _dw in FIG. 28A can be expressed by the following equations 10 and 11 when the signal amount s of the flare image and the signal amount a that allows correction are set.

The upper limit T _up and the lower limit T _dw in FIG. _28B can be expressed by the following equations 12 and 13 when the signal amount s of the flare image and the signal amount a that allows correction are set.

This process is started from step 2700, and in step 2710, it is determined whether the evaluation area is within the overlap area. If it is not within the overlap area, the correction amount is not corrected for the evaluation area, and the process proceeds to step 2750. If it is within the overlap region, the process proceeds to step 2720, and the threshold value table shown in FIG. 28 is referred to, and the correction amount threshold values T _up and T _dw are acquired. T _up and T _dw can be calculated using Equations 10 and 11 above or Equations 12 and 13 above.

In step 2730, it is determined whether the correction amount of the evaluation area is larger than the upper limit T _{up of the} threshold table or smaller than the lower limit T _dw . If it is larger than the upper limit T _up or smaller than the lower limit T _dw , the limiting process is not executed for the correction amount of the evaluation area, and the process proceeds to step 2750. On the other hand, if it is equal to or smaller than the upper limit T _up and smaller than or equal to the lower limit T _dw , the process proceeds to step 2740, and the limiting process is executed for the correction amount of the evaluation area.

The limiting process corrects the correction amount of the evaluation area to a value such that the correction is not performed. Specifically, when the correction amount is a value represented by the ratio of the above evaluation values, the correction amount is corrected to “1.0”, and when the correction amount is a value represented by the difference between the above evaluation values, the correction amount Is corrected to "0".

In step 2750, it is determined whether the processing has been completed for all the evaluation areas. If it has not been completed yet, the processing returns to step 2710, and the same processing is performed on the next evaluation area. When the processing is completed, the process proceeds to step 2760 and the processing is completed.

From the above, the present invention can appropriately perform correction even if flare occurs in an image. At this time, the image with the lowest flare occurrence probability can be used as the correction reference image and other images can be corrected. Further, since the average value with a low degree of matching in the normal case (the average value with a high degree of matching calculated by the above equations 4 and 5) is not integrated, only the average value with a high correlation between the flare image and the correction reference image is obtained. It can be used to calculate the average value of the overlap region. Even if an extremely bright or extremely dark subject such as a light source enters, the average value of the overlapping areas is calculated by excluding the average value of them, so that the influence on the average value calculation of the overlapping area can be reduced. it can.

By performing the limiting process, it is possible to prevent the correction amount from being corrected when the correction amount becomes extremely large, and by performing the leveling process, it is possible to prevent a partial extreme correction. be able to. Further, by performing the resizing process on the correction map, the correction can be performed by the calculation in pixel units, and the complicated conversion of the correction map is not necessary. By calculating the correction amount for the entire evaluation region using the weighted average, the correction does not cause a signal step such as a difference in luminance or a color difference in the image, and a natural correction can be performed. Since the correction amount is calculated based on the above ratio and difference, a signal step does not occur when the correction is performed.

The present invention has been described so far by the above-described embodiments as an image processing system and an image processing method, but the present invention is not limited to the above-described embodiments. The present invention can be modified within a range that can be conceived by those skilled in the art, such as other embodiments, additions, changes, deletions, etc. It is included in the scope of the invention. Therefore, the present invention can also provide a program for causing a computer to execute the image processing method, a recording medium on which the program is recorded, a server device that provides the program via a network, and the like.

Reference numeral 10... Spherical imaging camera, 11, 12... Fisheye lens, 13, 14... Imaging element, 15... Imaging SW, 20... Controller, 21... SDRAM, 22... External storage device, 23... CPU, 24... ROM, 25... SRAM, 26... Image processing block, 27... SDRAM I/F, 28... External storage I/F, 29... Bus, 30... Overlap area, 31, 32... Image, 34... Template image, 35... Evaluation area, 36... Connection region, 40... Calculation unit, 41... Judgment unit, 42... Image determination unit, 43... Image correction unit, 50... Light source, 51... Region, 60... Interpolation reference region

Japanese Patent No. 4739122

Claims (6)

An image processing system that performs image processing on a plurality of images including overlapping image areas as overlapping areas, and generates a spherical image from the plurality of images,
The image processing system,
An image processing system, comprising: determining whether or not there is an image to be corrected among the plurality of images based on the images in the overlapping area of each image. Based on each pixel value of a plurality of pixels of the image to be corrected, to determine the correction exclusion area in the image to be corrected is not corrected,
The image processing system according to claim 1, wherein an image region excluding the correction exclusion region is corrected. Each of the plurality of images is divided into a plurality of evaluation areas,
The image processing system according to claim 1, wherein for each of the divided evaluation areas, it is determined whether or not there is an image to be corrected in the plurality of images. An image pickup apparatus comprising: a plurality of image pickup devices; and an image processing system that performs image processing on a plurality of images including overlapping image regions picked up by the plurality of image pickup devices as overlapping regions, the image processing system But,
An imaging apparatus, which determines whether or not there is an image to be corrected among the plurality of images based on the images in the overlapping region of each image, and generates a spherical image. An image processing method executed by an image processing system that performs image processing on a plurality of images including overlapping image regions as overlapping regions, and generates a spherical image from the plurality of images,
An image processing method, comprising: determining whether or not there is an image to be corrected among the plurality of images based on the images in the overlapping region of each image. A program for causing a computer to execute the image processing method according to claim 5.