JP2021150831A - Imaging apparatus, correction method, and program - Google Patents

Abstract

To provide an imaging apparatus which can reduce difference in brightness or color at seams.SOLUTION: An imaging apparatus 10 creates an image by synthesizing a plurality of images captured by image pickup devices 22. The apparatus includes: a correction amount calculation section 43 for calculating a duplicate area which is overlapped when the images are synthesized and the correction amount inside the duplicate area; an outside area correction value calculation section 44 which calculates the correction value of pixels outside the duplicate area of a second image where the same object appears as in pixels inside the duplicate area of a first image on the basis of the pixel value of the pixels inside the duplicate area; a correction section 46 which corrects the first image and the second image using the correction amount and the correction value; and a synthetic image creation section 47 which duplicates and synthesizes the duplicate area of the first image and the second image which have been corrected by the correction section.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device, a correction method, and a program.

An omnidirectional imaging camera is known as an imaging device capable of imaging in any direction of 360 °. The omnidirectional imaging camera uses a plurality of wide-angle lenses or fisheye lenses, images are imaged by a plurality of image pickup elements, distortion correction, projection conversion, etc. are performed on the obtained plurality of images, and these are combined into one image. Generate a spherical image. An image obtained by imaging with an adjacent image sensor has an image overlapping region in which a part of the image overlaps, and in the spherical imaging camera, each image is combined in the image overlapping region.

A technique for correcting the brightness of two images is known (see, for example, Patent Document 1). In Patent Document 1, a technique of using the brightness values of overlapping regions of a plurality of image sensors to bring the brightness values between the image sensors closer to each other by making corrections to make the bright one darker and the darker one brighter to eliminate discomfort at the joints. Is disclosed.

However, the conventional technique has a problem that the brightness or the color of the joint is likely to be different due to the difference in the amount of light.

Since a spherical image camera captures a wide range using a wide-angle lens or a fisheye lens, it is easy for a light source such as the sun or illumination to enter the imaging range. For this reason, it is known that there is a high possibility that a part of the image becomes white and blurry, and flare that appears as if the light is blurred occurs. Since flare does not occur uniformly in each pixel, the image in which flare occurs is corrected to be dark. As a result, the blurred image becomes darker than the actual brightness and color, resulting in a strange image.

In view of the above problems, an object of the present invention is to provide an imaging device capable of reducing the difference in brightness or color of joints.

In view of the above problems, the present invention is an imaging device that synthesizes images captured by a plurality of image pickup elements to create one image, and the overlapping region overlapped at the time of compositing and the correction amount inside the overlapping region. Based on the correction amount calculation unit that calculates An out-of-area correction value calculation unit that calculates the correction value of the above, a correction unit that corrects the first image and the second image using the correction amount and the correction value, and the first correction unit corrected by the correction unit. It is characterized by having a composite image generation unit that overlaps and synthesizes the overlapping region of one image and the second image.

It is possible to provide an imaging device capable of reducing the difference in brightness or color of the joint.

It is an example of the cross-sectional view which shows the image pickup apparatus. It is a figure which shows the configuration example of the image pickup apparatus. It is a figure explaining the function which the image pickup apparatus has. It is a figure explaining the fisheye lens. It is a figure explaining the spherical image. It is a figure explaining the conversion table used at the time of projective transformation of a fisheye image. This is an example of a diagram illustrating an overlapping region of two fisheye images captured by an image sensor. This is an example of a diagram showing an overlapping area when two rectangular image data are combined. This is an example of a flowchart for explaining the overall flow of processing performed by the image pickup apparatus. This is an example of image data used for determining a flare image. This is an example of an evaluation area in which information necessary for determining a flare image is calculated. This is an example of a flowchart for explaining a process of determining whether or not the flare image determination unit is a flare image. This is an example of a diagram showing a correction exclusion map in the initial state when it is determined that there is a flare image. This is an example of a diagram schematically showing a correction exclusion map. This is an example of a flowchart showing a procedure in which the correction exclusion map creation unit creates a correction exclusion map. It is an example of the figure explaining the calculation method of the correction amount. This is an example of a diagram showing a method of creating a correction map for the inside of the overlapping area. As a prior art, this is an example of a diagram illustrating a correction map outside the overlapping region. It is an example of the figure which shows the overlap area. It is an example of the figure explaining the synthesis of the overlap region and the vignetting region outside the overlap region. It is an example of the figure which shows the calculation method of the correction value outside the overlap area. It is an example of a figure explaining another method of obtaining h'. It is a figure which shows an example of the correction map, the correction exclusion target map, and the correction map which were tentatively created. This is an example of a diagram illustrating a method of converting the resolution of the correction map. This is an example of a flowchart for explaining the flow of processing in which the composite image generation unit generates a composite image.

Hereinafter, as an example of the embodiment for carrying out the present invention, the image pickup apparatus and the correction method performed by the image pickup apparatus will be described.

<Terminology>
Synthesis means combining two or more things into one.

The overlapping area is an area in which the same subject appears in two images. Of these, a region with less distortion and the like may be used. The overlapping area may be constant or variable for each image.

The inside of the overlapping region is a direction in which the image height h becomes smaller when the image is captured by a fisheye lens and projected by an equidistant projection method or the like. The outside of the overlapping region is the direction in which the image height h increases.

The correction amount or correction value is a value used to bring the appearance of images closer to each other, such as the brightness and color of two images.

<Configuration example>
FIG. 1 is a cross-sectional view showing the image pickup apparatus 10 according to the present embodiment. The image pickup apparatus 10 shown in FIG. 1 includes an image pickup body 12, a housing 14 for holding the image pickup body 12, a controller, a battery, and other parts, and a shutter button 18 provided in the housing 14.

The image sensor 12 shown in FIG. 1 includes two lens optical systems 20A and 20B (also referred to as lens optical system 20) and two image sensor 22A such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. , 22B (also referred to as an image sensor 22). In this embodiment, a combination of the lens optical system 20 and the image sensor 22 one by one is referred to as an image pickup optical system. The lens optical system 20 can be configured as a fisheye lens, for example, with 7 elements in 6 groups. In the example shown in FIG. 1, the fisheye lens has a total angle of view larger than 180 degrees (= 360 degrees / n; n = 2), preferably has an angle of view of 185 degrees or more, and is more preferable. Has an angle of view of 190 degrees or more.

The optical elements (lens, prism, filter, aperture diaphragm) of the two lens optical systems 20A and 20B are positioned so that their optical axes are orthogonal to the central portion of the light receiving region of the corresponding image pickup element 22 and. The positional relationship with respect to the image pickup elements 22A and 22B is determined so that the light receiving region becomes the image plane of the corresponding fisheye lens. Each of the image pickup elements 22 is a two-dimensional image pickup element in which the light receiving region forms an area area, and converts the light collected by the combined lens optical system 20 into an image signal.

In the embodiment shown in FIG. 1, the lens optical systems 20A and 20B have the same specifications, and are combined in opposite directions so that their optical axes match. The image pickup devices 22A and 22B convert the received light distribution into an image signal and output it to the image processing means on the controller. The image processing means joins and synthesizes the captured images input from the imaging elements 22A and 22B, respectively, to generate an image having a solid angle of 4π radians (hereinafter referred to as “all-sky image”). The spherical image is an image taken in all directions that can be seen from the imaging point.

As described above, since the fisheye lens has a total angle of view exceeding 180 degrees, when composing a spherical image, overlapping image portions represent the same image in the captured images captured by each imaging optical system. It is used as a reference for joining images as reference data. The generated spherical image can be, for example, a display device, a printing device, an SD (registered trademark) card, a compact flash (registered trademark), or the like provided in the image pickup body 12 or connected to the image pickup body 12. It is output to an external storage medium or the like.

<Configuration to perform the following image processing>
FIG. 2 shows a configuration example of the image pickup apparatus 10 that executes the process described below. The image pickup device 10 includes image pickup elements 1 and 2 (reference numerals 22A and 22B), receivers 1 and 2 (reference numerals 23A and 23B), ISP (Image Signal Processing) 1 and 2 (reference numerals 24A and 24B), SRAM 25, and the like. It has a brightness measurement unit 26, an image processing unit 1 and 2 (reference numerals 27A and 27B), a MEMC28, a DRAM 29, a brightness correction circuit 30, an arbitrary deformation circuit 31, an image processing unit 2 (reference numeral 32), and a transmission unit 33. doing.

The image sensors 1 and 2 correspond to the image sensors 22A and 22B of FIG. The receiving units 1 and 2 are channels that receive image data from the image sensors 1 and 2. ISPs 1 and 2 perform white balance, gamma correction, Bayer complementation, etc. on RAW data (raw data). The image processing unit 1 performs correction processing of an optical system such as a lens, black level correction, variation correction of an image sensor, scratch correction, and the like.

The luminance measurement unit 26 determines a flare image, creates a correction exclusion map, calculates a correction value, and the like. The MEMC 28 is a memory controller and uses the DRAM 29 to control the flow of image data. The brightness correction circuit 30 corrects the brightness of the spherical image according to the measurement result of the brightness measuring unit 26. The arbitrary transformation circuit 31 transforms the image data. The image processing unit 2 combines two fisheye images to create an equirectangular image. The transmission unit 33 transmits image data to the outside or stores it in an external memory.

<About the functions of the imaging device>
FIG. 3 is a diagram illustrating a function of the image pickup apparatus 10. As shown in FIG. 3, the luminance measurement unit 26 includes a flare image determination unit 41, a correction exclusion map creation unit 42, a correction amount calculation unit 43, and an out-of-region correction value calculation unit 44. The luminance correction circuit 30 has a correction exclusion map application unit 45 and a correction unit 46. The image processing unit 2 has a composite image generation unit 47. Each function shown in FIG. 3 is realized by hardware, but the CPU executes a program stored in a storage unit such as an SSD (Solid State Drive) included in the image pickup apparatus 10 for some or all of the functions. It may be realized by.

The flare image determination unit 41 determines whether or not there is an image containing flare among the two fisheye images, and if so, which is the flare image. The correction exclusion map creation unit 42 creates a correction exclusion map in which pixels that should not be corrected in the fisheye image are specified. The correction amount calculation unit 43 calculates the overlapping area and the correction amount inside the overlapping area. The out-of-region correction value calculation unit 44 calculates the correction value outside the overlapping region. The correction exclusion map application unit 45 applies the correction exclusion map to the fisheye image. The correction unit 46 corrects the brightness of the pixels for which the correction is not excluded based on the correction amount and the correction value (correction is performed so that the brightness of the two fisheye images is about the same). The composite image generation unit 47 synthesizes two fisheye images into one equirectangular image.

<About fisheye images taken with a fisheye lens>
A fisheye lens will be described with reference to FIG. Since the fisheye lens 11 is the same, only one fisheye lens 11 will be described in FIG. The fisheye image captured by the image sensors 22A and 22B having the fisheye lens having an angle of view of more than 180 ° is an image of a subject substantially hemisphere centered on the image pickup position.

Here, as shown in FIG. 4A, assuming that the angle of incidence of light on the fisheye lens 11 is φ, the distance between the center of the image and the image point is the image height h, and the projection function is f, these relationships are established. It can be expressed as the following formula (A).

h = f (φ) ・ ・ ・ Equation (A)

The projection function f differs depending on the properties of the fisheye lens 11. For example, when an equidistant projection type fisheye lens is adopted, as shown in FIG. 4B, the larger the incident angle φ is as shown by the arrow line, the more the image is displayed. It is expressed as a proportional relationship in which the high h increases. In FIG. 4B, the black-filled region outside the circle is a region where light is not incident.

Since the fisheye lens 11 has an angle of view of more than 180 °, the fisheye images captured by the image pickup devices 22A and 22B include overlapping regions.

Next, the spherical image will be described with reference to FIG. The fish-eye image has a format in which approximately half of the spherical surface shown in FIG. 5A is represented by a circle, the longitude in the globe corresponds to the horizontal angle θ, and the latitude corresponds to the vertical angle φ. The horizontal angle θ is in the range of 0 to 360 °, and the vertical angle φ is in the range of 0 to 180 °.

The spherical image is in the format represented by the rectangle shown in FIG. 5B, and is generated by the imaging device combining two hemispherical images in which the horizontal direction is the horizontal angle and the vertical direction is the vertical angle. It is an image. Actually, the image is larger than the hemispherical image due to the overlapping area, but here, it is called a hemispherical image.

The two hemispherical images have the same pixel values as the pixels corresponding to the horizontal and vertical angles of the fisheye image, but have the same horizontal and vertical angles in the rectangular format shown in FIG. 5 (b). It is generated as an image that the corresponding pixel has. This hemispherical image can be generated by the imaging device 10 projecting and transforming the fisheye image, and by combining the two generated hemispherical images, 360 ° omnidirectional directions can be obtained in the horizontal and vertical directions. It is possible to generate a spherical image to represent.

A conversion table used for projective transformation of a fisheye image is illustrated in FIG. As shown in FIG. 6A, the conversion table pixels the coordinate values of the horizontal and vertical angles of the fisheye image, which is the image before the change, and the coordinate values of the hemispherical image, which is the image after the change. It is a table associated with each. The coordinate value of the image before the change is represented by (x, y), and the coordinate value of the image after the change is represented by (θ, φ). As shown in FIG. 6B, in each image, the pixels in the image before the change and the corresponding pixels in the image after the change are determined based on the coordinates (0.0) in the upper left corner, and the pixels of those pixels are determined. The combination of coordinate values is stored as data in the conversion table. The correspondence relationship can be obtained from the projection relationship between the pre-conversion image and the post-conversion image.

The conversion table can be created in advance by the designer or the like based on the lens design data or the like for each of the two fisheye lenses 11 and the image sensors 22A and 22B, and is stored in the SRAM 25 shown in FIG. , The image sensor 10 can read and use it as needed. By using this conversion table, the fisheye image can be projected and transformed to correct the distortion of the fisheye image. By combining the corrected images, a spherical image can be generated.

The flow of the process for generating the spherical image will be described. The processing is started by capturing two fisheye images with the two image sensors 22A and 22B and inputting the two fisheye images. First, the composite image generation unit 47 performs a distortion correction by projecting and transforming each fisheye image using a conversion table as shown in FIG. 6A stored in the SRAM 25. By this distortion correction, two hemispherical images shown in FIG. 6B can be obtained.

Next, the composite image generation unit 47 detects the connection position in order to connect the two obtained hemispherical images in the overlapping region. The above conversion table is corrected based on the detection result of the connection position. Next, the composite image generation unit 47 performs rotation conversion on the corrected conversion table to create a conversion table for image generation. The purpose of the rotation conversion is to match the vertical direction of the image with the zenith direction of the image pickup apparatus 10 in the conversion table for image generation.

Next, the composite image generation unit 47 performs projective transformation on the two fisheye images using a conversion table for image generation, and corrects the distortion of the images. Then, a blending process for synthesizing the two distortion-corrected images is executed. The two images are combined in the overlapping area, but when the data exists only in the overlapping area of one image, the data is used as it is for the combination. A spherical image is generated by the above flow.

<About overlapping areas>
The overlapping region of the two fisheye images 13A and 14a captured by the image pickup devices 22A and 22B will be described with reference to FIGS. 7A and 7B.

FIG. 7A shows fisheye images 13A and 14a captured by the image sensors 22A and 22B. The black areas are areas where light is not incident, the white areas are areas up to an incident angle of 90 °, and diagonal lines. The region shown indicates a region where the incident angle exceeds 90 °.

The area shown by the diagonal line in FIG. 7 (a) can be defined as an overlapping area because it indicates an overlapping image area of the two fisheye images. However, in the fisheye lens 11, the image height h becomes large, and the farther the image point is from the center of the image, the more likely it is that distortion or aberration occurs. In addition, the outer frame or the like around the fisheye lens 11 may be reflected. Images such as areas where distortion and aberration have occurred and outer frames cannot be used to join the images together.

Therefore, as shown in FIG. 7B, the overlapping region 50 can be limited to a ring-shaped region represented by vertical stripes having a predetermined width inside the overlapping region 50. This is because distortion and aberration are likely to occur as the image point is farther from the center of the image, and the outer frame of the fisheye lens may be displaced. Therefore, as shown in FIG. 7B, the overlapping region 50 is limited to the diagonally shaded region having a predetermined width inside the overlapping region 50.

Incidentally, since the two fisheye images 13A and 14a in FIG. 7B expose and image the image pickup elements 22A and 22B at the same time, the overlapping region 50 is basically an image of the same subject. The overlapping region 50 is preferably set on a pixel-by-pixel basis. By setting in pixel units, it is possible to absorb deviations due to errors in optical design.

FIG. 8 shows an overlapping region 50 when two rectangular image data are combined. As described above, the composite image generation unit 47 corrects the distortion of the fisheye image to obtain rectangular image data. FIG. 8 shows a composite image generation unit 47 that distorts and corrects two fisheye images into rectangular image data and superimposes overlapping regions 50.

In the overlapping area 50, similar parts are detected by processing such as template matching so that the same subject of two rectangular image data overlaps. The unit for performing template matching may be any divided image size unit. When the relative position is determined by template matching, the composite image generation unit 47 synthesizes the two images by blending. With the above, the spherical image is obtained.

Although the processing of FIGS. 6 to 8 is actually performed after the brightness correction of the fisheye image, it has been described here to explain the overlapping region 50.

<Whole processing>
FIG. 9 is an example of a flowchart for explaining the overall flow of processing performed by the image pickup apparatus 10. The image pickup apparatus 10 determines a flare image (step S1), creates a correction exclusion map (step S2), calculates a correction amount (step S3), and outside the overlapping region for the two fisheye images processed by the image processing unit 1. Calculation of the correction value (step S4), application of the correction exclusion map (step S5), correction (step S6), and generation of a composite image (step S7) are performed. Each process will be described in detail below.

<Judgment of S1 flare image>
A method of determining a flare image will be described with reference to FIGS. 10 to 12. FIG. 10 is image data used for determining a flare image. As described above, in the fisheye image, the area near the outer frame of the image is the overlapping area 50. The flare image determination unit 41 determines which of the two fisheye images is the flare image based on the pixel value of the overlapping region 50. The input image data format may be RGB or YCbCr.

First, the flare image determination unit 41 processes the input fisheye image in area units as shown in FIG. The size of the area 201 is set to an arbitrary value by a register or the like. Area 201 is one cell divided at equal intervals on the top, bottom, left, and right. The flare image determination unit 41 calculates information necessary for determining a flare image, such as an average value of pixel values and a dispersion value, for each area.

FIG. 11 is an evaluation area 202 in which the information necessary for determining the flare image is calculated. The average value differs depending on where the calculated area is located in the image data. As described above, the flare image determination unit 41 uses the calculated value of the evaluation region 202 near the overlapping region in order to determine which of the two fisheye images is the correction target.

Specifically, in two fisheye images, the flare image determination unit 41 obtains the absolute value of the difference between the average value and the dispersion value of the areas in which the same subject appears, and when the value exceeds the threshold value, flare Judge that there is an image. Further, it can be determined that the fisheye image having the larger average or variance is the image to be corrected (flare image). If one is the image to be corrected (an example of the second image), the other image is determined as the correction reference image (an example of the first image).

Basically, when images are taken by simultaneous exposure, the images of the same subject should be taken with the same brightness in the overlapping area 50, but when flare occurs in one of the images, the average value of the overlapping areas 50 is Since it becomes large, the above-mentioned threshold value is set, and a fisheye image larger than the threshold value is judged to be a flare image.

FIG. 12 is an example of a flowchart for explaining a process of determining whether or not the flare image determination unit 41 is a flare image.

The flare image determination unit 41 calculates, for example, an average value with respect to the brightness value of the evaluation region 202 near the overlapping region 50 (S1-1). The flare image determination unit 41 compares the average value of the brightness values of the two fisheye images (S1-2).

Next, the flare image determination unit 41 determines whether or not the difference between the average values is larger than the threshold value (S1-3).

When the difference between the average values is larger than the threshold value, the flare image determination unit 41 determines that the larger average value is the flare image (S1-4). The one with the smaller average value is judged to be the correction reference image.

When the difference between the average values is equal to or less than the threshold value, the flare image determination unit 41 determines that the flare image is not a flare image (S1-5).

<Creation of S2 correction exclusion map>
13 to 15 are diagrams illustrating a method of creating a correction exclusion map. FIG. 13 is a diagram showing a correction exclusion map in an initial state when it is determined that there is a flare image. When the correction unit 46 corrects the flare image, the entire flare image is corrected, so that the image becomes dark as a whole. Then, when the two fisheye images are combined, the image may be visually unnatural. Therefore, in order to distinguish the parts (light source, etc.) that do not need to be flared corrected, the correction exclusion map creation unit 42 creates a correction exclusion map.

The correction exclusion map can be obtained by the maximum brightness value in the area in the fisheye image outside the overlapping area described above. The condition of the correction exclusion area is
(i) "Brighter than threshold and achromatic" or
(ii) "Brightness higher than the threshold value and the maximum value of the brightness is above the threshold value"
And. The correction exclusion map creation unit 42 makes this determination for each square area 201. The threshold value for whether or not the brightness is high may be set by a register or the like.

It is desirable to judge both (i) and (ii) as the correction exclusion condition. When the entire evaluation area has uniform brightness and color, it can be judged only by (i). However, if a high-brightness subject such as a light source and a low-brightness subject such as a tree exist in one evaluation area, the image of the light source is corrected because it does not conform to (i), and the light source is dark and unnatural. This is because it becomes an image. By including (ii), an image including a light source and a tree can be appropriately extracted as a correction exclusion area.

FIG. 14 schematically shows a correction exclusion map. The correction exclusion map indicates whether or not to exclude it from the correction target by the value of the flag (1,0). The correction exclusion map is divided in the same manner as when the correction target image is divided into a plurality of areas 201. The correction exclusion map creation unit 42 arranges 1 in the area to be corrected and excluded, and 0 in the block that does not. Further, the correction exclusion map stores not only the flags 1 and 0 indicating whether or not to exclude, but also the average value or the variance value of the pixel values for each block.

The correction exclusion map may be stored in the external storage device (SRAM 25) as shown in FIG. 2, or may be stored in the circuit as a register.

FIG. 15 is an example of a flowchart showing a procedure in which the correction exclusion map creation unit 42 creates a correction exclusion map.

The correction exclusion map creation unit 42 creates a correction exclusion map in the initial state divided into areas 201 for the correction target image (S2-1).

The correction exclusion map creation unit 42 determines whether or not the correction is excluded from the correction based on the judgment conditions (i) and (ii) above for each area (S2-2).

In the case of the correction exclusion target, the correction exclusion map creation unit 42 sets the flag 1 in the area of interest (S2-3).

If it is not the correction exclusion target, the correction exclusion map creation unit 42 sets the flag 0 in the area of interest (S2-4).

The correction exclusion map creation unit 42 also sets an evaluation value (average value or variance value) in the area of interest (S2-5).

The correction exclusion map creation unit 42 determines whether or not all the areas have been completed (S2-6), and if not, returns to step S2-2.

<Calculation of S3 correction amount>
FIG. 16 is a diagram illustrating a method of calculating the correction amount. The correction amount is calculated by comparing the overlapping region 50 of the flare image and the reference image. In FIGS. 16A and 16B, the regions 210 and 211 connected by the overlapping region 50 of the fisheye images 13A and 13B are shown by a broken line frame. 16 (c) and 16 (d) show enlarged views of the regions 210 and 211, respectively.

Since the two image sensors are back-to-back in the spherical image sensor, the same subject appears in the areas 210 and 211 at the same position when the fisheye images 13A and 13B are back-to-back. A to L in FIGS. 16C and 16D show corresponding pixels in the two fisheye images 13A and 13B.

When joining the two fisheye images, the value of the evaluation region 202 shown in FIG. 11 shows a high value because flare has occurred on one side and a lower value on the other side. The correction amount calculation unit 43 makes a correction so as to make this the same value. The value used for correction is calculated by any of the following equations (1) and (2).
Correction amount = Evaluation value of reference image / Evaluation value of flare image… (1)
Correction amount = Evaluation value of reference image − Evaluation value of flare image… (2)
Equation (1) uses the ratio as the correction amount, and equation (2) uses the difference as the correction amount. The correction method performed by the correction unit 46 also changes depending on how the correction amount is obtained. The correction amount calculation unit 43 stores the calculated correction amount value in the evaluation area 202 of FIG. 11 of the flare image. The correction amount calculation unit 43 performs this process on all the overlapping areas 50.

<S4 Calculation of correction value outside the overlapping area>
17 to 22 are diagrams for explaining the calculation of the correction value outside the overlap area with the correction map. FIG. 17 shows a method of creating a correction map for the inside of the overlapping region 50. As shown in FIG. 17, the correction map is calculated by performing a weighting calculation based on the distance toward the center of the image based on the value of the correction amount calculated for the overlapping region 50. The closer to the center, the smaller the correction amount. Correction amount calculation unit 43 In this way, the correction map inside the overlapping region is temporarily created. Since it is a temporary creation, the final correction map will be created later.

FIG. 18 is a diagram illustrating a correction map outside the overlapping region 50 as a prior art (Patent Document 2). Conventionally, there is a method of copying the value of the overlapping area 50 as it is on the outside of the overlapping area 50. In FIG. 18, the pixel x uses the same value as the correction amount of the pixel one above, and the pixel y uses the same value as the correction amount of the pixel one left. However, this correction method cannot completely correct the brightness color difference of the joint due to the difference in the amount of light.

With reference to FIGS. 19 and 20, the reason why the conventional method of creating the correction map outside the overlapping region 50 is not preferable will be described. FIG. 19 is a diagram showing an overlapping region 50. As shown in FIGS. 19A and 19B, the overlapping region 50 is taken inside the boundary between the effective image region 51 and the vignetting regions 52A and 52B in the two input fisheye images. The vignetting area is the area where the lens hood and filter appear black in the corners of the screen.

As described above, the two fisheye images 13A and 13B are converted into one equirectangular image by the projective transformation based on the overlapping region 50. At the time of compositing, the composite image generation unit 47 synthesizes the overlapping region 50 at α = 50% in the alpha blend. At the time of synthesis, as shown in FIGS. 19 (c) and 19 (d), the vignetting regions 52A and 52B outside the overlapping region 50 are not discarded. Actually, the value is α <50% or less, which is the region used for compositing with the other fisheye image.

FIG. 20 is a diagram illustrating the synthesis of the overlapping region 50 and the vignetting regions 52A and 52B outside the overlapping region 50. 20 (a) is an equirectangular image of the fisheye image 13A of FIG. 19, and FIG. 20 (b) is an equirectangular image of the fisheye image 13B of FIG. The band-shaped hatching represents the overlapping region 50 and the vignetting regions 52A and 52B outside the overlapping region 50. As described above, the overlapping region 50 of the fisheye images 13A and 13B is synthesized with α = 50%, the vignetting region 52A is synthesized with the fisheye image 13B and α <50%, and the vignetting region 52B is combined with the fisheye image 13A. Synthesized with α <50%. FIG. 20 (c) shows one equirectangular image obtained by synthesizing two images.

The pixel value of the overlapping region 50 is shown below when the value of the correction map is h and the correction target is the fisheye image 13A side. However, the overlapping area 50 of h = pixB / pixA α = 50% is set.
pixo = α x pixA xh + (1−α) x pixB
Since the outside of the overlapping area 50 is also used for compositing in this way, if the correction value outside the overlapping area 50 is simply copied from the correction value of the overlapping area 50, a brightness step may occur at the joint portion of the composite image. There is sex.

Therefore, in the present embodiment, the out-of-area correction value calculation unit 44 also creates a correction map by calculation outside the overlapping area 50. Specifically, by obtaining the correction value inside the overlapping region 50 of the image that is not the correction target, the correction value outside the image to be corrected is obtained.

FIG. 21 shows a method of calculating the correction value outside the overlapping region 50. As shown in FIG. 21, the inside of the overlapping region 50 in one fisheye image 13A corresponds to the outside of the overlapping region 50 in the other fisheye image 13B (see FIG. 20). Using this, the out-of-region correction value calculation unit 44 obtains the correction value outside the overlapping region 50. In FIG. 21, the fisheye image 13B is used as a flare image. The correction value h is a correction value of the pixels outside the overlapping region 50 of the comparative image (flare image).
-Correction value h = Pixel value outside the overlapping area of the comparison image (flare image) / Pixel value inside the overlapping area of the reference image ... (3)
The correction value of the pixels a'to o'outside the overlapping region 50 of the fisheye image 13B is this h.

The correction value h'for the reference image is converted by the following.
・ Correction value h'= 1 / h… (4)
That is, the reciprocal of the correction value h is set as the correction value outside the overlapping region 50 of the fisheye image 13A. The correction value of the pixels a to o outside the overlapping region 50 of the fisheye image 13A is the correction value h'.

Since one of the correction maps is a map for creating the original correction map, the reference interpolation line and the like are used on the original correction map side. In FIG. 21, the correspondence between the pixels a to o of the correction map on one side and the pixels a'to o'of the correction map to the other side is an example, and which region of the correction amount to use is changed by presetting or the like. It is possible.

Pixels a to o inside the overlapping region 50 of the fisheye image 13A include pixels in contact with the overlapping region 50, and pixels a'to o'of the fisheye image 13B in which the same subject as the pixels a to o is captured. Includes pixels in contact with the overlapping region 50. The correction values of the pixels a to o and the pixels outside the pixels a'to o'may be set to be smaller as the distance from the overlapping region is increased, or may be calculated in the same manner as in Eqs. (3) and (4). ..

FIG. 22 is a diagram illustrating another method of obtaining h'. FIG. 22A shows a conversion table having blend values, and FIG. 22B is a diagram illustrating the correspondence between the synthesized equirectangular image and the pixels of the two fisheye images 13A and 13B.

As a method of obtaining h', the blend value of the conversion table as shown in FIG. 22 may be used as a reference. The conversion table is referred to when creating the above-mentioned composite image, and has a blend value α with respect to the original fisheye image. The blend value α is defined as 0 to 1.0 and
Refer to the blend value and
h'= (1-α) x1 / h (h> 1.0)
h'= (1 + α) x1 / h (h <1.0)
The correction value is set to the outside of the overlapping region 50 of the reference image.

By doing so, the blend coefficient on the correction side can be added, and the difference in brightness and color of the joint becomes smaller in the pixel value of the joint position after blending.

The correction map may be stored in the external storage device (SRAM 25) as shown in FIG. 2, or may be stored in the circuit as a register.

<Application of S5 correction exclusion map>
FIG. 23 shows an example of the temporarily created correction map (FIG. 23 (a)), the correction exclusion target map (FIG. 23 (b)), and the correction map (FIG. 23 (c)). The correction exclusion map application unit 45 creates a final correction map using the temporary correction map and the correction exclusion map created as described above for the overlapping area 50, the inside of the overlapping area 50, and the outside of the overlapping area 50. do.

That is, the correction exclusion map application unit 45 replaces the value of the provisionally created correction map with 1.0 if the correction exclusion map is 1 in each area of the temporarily created correction map. If it is 0, the value obtained from the temporarily created correction map is used as it is. The correction exclusion map application unit 45 performs this for the entire area and creates a correction map.

Note that FIG. 23 shows a correction map when the equation (1) is used for calculating the correction amount, and the difference is stored in the correction map when the equation (2) is used. If the correction exclusion map is 1, the correction exclusion map application unit 45 replaces the temporarily created correction map value with 0.0. If the correction exclusion map is 0, the value obtained from the temporarily created correction map is used as it is.

The value of the correction map after applying the correction exclusion map may fluctuate greatly between the divided areas. Therefore, a smoothing filter such as a Gaussian filter may be applied to the calculated correction map.

FIG. 24 is a diagram illustrating a method of converting the resolution of the correction map. The created correction map has a resolution of an arbitrary block size (resolution of area 201). In order to apply this to the entire correction target image, the correction exclusion map application unit 45 scales the correction map and converts it to the resolution of the correction target image. As the scaling method, a scaling method such as nearest neighbor, bilinear, or bicubic is used.

<S6 correction>
The correction unit 46 applies the created correction map to the entire correction target image to obtain a corrected image. If Eq. (1) is used when calculating the correction amount, the correction amount is multiplied by the corrected image. If equation (2) is used, the correction amount is added to the corrected image.

The amount of the calculated value may be limited by setting an upper limit and a lower limit so that the calculated value does not become extremely small or too large.

<Generation of S7 composite image>
The composite image generation unit 47 synthesizes the two fisheye images corrected as described above as described with reference to FIGS. 6 to 8.

FIG. 25 is an example of a flowchart illustrating a flow of processing in which the composite image generation unit 47 generates a composite image.

The composite image generation unit 47 performs distortion correction for converting a fisheye image into an equirectangular image with reference to the table of FIG. 6A (S7-1). Next, the composite image generation unit 47 pattern-matches the overlapping regions 50 to determine the connecting portion (S7-2). Next, the blend processing unit blends the overlapping region 50 using the α value (S7-3).

<Main effect>
Since a spherical camera uses a wide-angle lens or a fisheye lens to image a wide range, it is easy for a light source such as the sun or illumination to enter the imaging range. For this reason, it is known that there is a high possibility that a part of the image becomes white and blurry, and flare that appears as if the light is blurred occurs. Since flare does not occur uniformly in each image, not only a difference in brightness but also a color difference occurs between an image in which flare occurs and an image in which flare does not occur.

It is considered that the prior art is effective in correcting individual differences between a plurality of image pickup devices 22A and 22B. However, when a strong light source is included in any of the plurality of image sensors 22A and 22B, the bright image containing the light source is corrected to be dark, so that the image becomes darker than the actual brightness and color, which gives a sense of discomfort. It may become an image.

Unlike the prior art, this embodiment can make the joints more inconspicuous while maintaining the natural color when a plateau such as the sun or lighting enters the imaging range.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, in the present embodiment, the image in which flare is generated is corrected, but the image in which flare is not generated may be corrected.

Further, in the present embodiment, all the corrections are performed by the imaging device, but all or part of the processing related to the corrections may be performed by the information processing device (server) communicated via the network.

Further, the configuration examples such as FIG. 3 shown in the above examples are divided according to the main functions in order to facilitate understanding of the processing of the image pickup apparatus 10. However, the present invention is not limited by the method and name of division of each processing unit. The image pickup apparatus 10 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

10 Imaging device 41 Flare image judgment unit 42 Correction exclusion map creation unit 43 Correction amount calculation unit 44 Out-of-area correction value calculation unit 45 Correction exclusion map application unit 46 Correction unit 47 Composite image generation unit 50 Overlapping area

Special Table 2013-540390 Japanese Unexamined Patent Publication No. 2015-226144

Claims (8)

An image pickup device that creates one image by synthesizing images captured by a plurality of image pickup elements.
A correction amount calculation unit that calculates a correction amount inside the overlapping area and the overlapping area during synthesis, and a correction amount calculation unit.
Out-of-region correction that calculates the correction value of the pixel outside the overlapping region of the second image, in which the same subject as the pixel is captured, based on the pixel value of the pixel inside the overlapping region of the first image. Value calculation unit and
A correction unit that corrects the first image and the second image using the correction amount and the correction value,
A composite image generation unit that overlaps and synthesizes the overlapping region of the first image corrected by the correction unit and the second image, and a composite image generation unit.
An imaging device characterized by having. The pixels inside the overlapping region of the first image include pixels in contact with the overlapping region, and the pixels of the second image showing the same subject as the pixels are in contact with the overlapping region. The imaging device according to claim 1, further comprising pixels. The out-of-area correction value calculation unit
The imaging device according to claim 2, wherein the correction value is calculated based on the pixel value outside the overlapping region of the second image / the pixel value inside the overlapping region of the first image. .. The out-of-area correction value calculation unit
The imaging apparatus according to claim 3, wherein the reciprocal of the correction value is a correction value outside the overlapping region of the first image. It has a table for converting the first image and the second image captured by a fisheye lens into an equirectangular image, and the blend value of the first image and the second image is displayed on the table for each pixel. If registered,
The out-of-area correction value calculation unit
h'= (1−α) x1 / correction value (correction value> 1.0)
h'= (1 + α) x1 / correction value (correction value <1.0)
The imaging device according to claim 3, wherein the correction value is set to the outside of the overlapping region of the first image. The imaging device according to any one of claims 1 to 5, wherein the second image is an image in which flare is detected. This is a correction method performed by an image pickup device that creates one image by synthesizing images captured by a plurality of image pickup elements.
A step in which the correction amount calculation unit calculates the overlapping area overlapping during synthesis and the correction amount inside the overlapping area, and
Based on the pixel values of the pixels inside the overlapping region of the first image, the out-of-area correction value calculation unit captures the same subject as the pixels, and of the pixels outside the overlapping region of the second image. Steps to calculate the correction value and
A step in which the correction unit corrects the first image and the second image using the correction amount and the correction value.
A step in which the composite image generation unit superimposes and synthesizes the overlapping region of the first image and the second image corrected by the correction unit.
A correction method characterized by having. An image pickup device that synthesizes images captured by a plurality of image pickup elements to create one image.
A correction amount calculation unit that calculates a correction amount inside the overlapping area and the overlapping area during synthesis, and a correction amount calculation unit.
Out-of-region correction that calculates the correction value of the pixel outside the overlapping region of the second image, in which the same subject as the pixel is captured, based on the pixel value of the pixel inside the overlapping region of the first image. Value calculation unit and
A correction unit that corrects the first image and the second image using the correction amount and the correction value,
A program for functioning as a composite image generation unit that overlaps and synthesizes the overlapping region of the first image corrected by the correction unit and the second image.

Images