JP2018174431A - Image processing device, photographing device, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of improving access efficiency to a memory by embedding information related to an amount of correction into any vignetting region of a plurality of images, a photographing device, and an image processing method.SOLUTION: A photographing device comprises two imaging elements, an image processing device, and an external memory. An image processing device comprises an image processing unit, an amount-of-correction creation unit, an image correction unit, a projective transformation unit, an image output unit, a CPU, and an external memory. The amount-of-correction creation unit comprises a correction target determination unit, an amount-of-correction calculation unit, a correction map creation unit, and an embedding unit. The correction target determination unit determines a correction target image from a plurality of images on the basis of a pixel vale of each pixel of the plurality of images. The amount-of-correction calculation unit calculates an amount of correction for correcting the correction target image on the basis of a pixel value of each image. The correction map creation unit creates a correction map (M4) for correcting the correction target image. The embedding unit embeds information related to the amount of correction such as data of the correction map (M4) into any vignetting region of the plurality of images.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an invention for performing image processing on each image acquired from a plurality of image sensors.

  In recent years, there has been provided a special digital camera that obtains two hemispherical image data based on a 360-degree spherical image by one shooting (see Patent Document 1). This special omnidirectional digital camera creates one equirectangular projection image data based on two hemispheric image data, and transmits the equirectangular projection image data to a communication terminal such as a smartphone. The communication terminal that has obtained the equirectangular projection image data creates an omnidirectional image based on the equirectangular projection image data. However, since the image is curved as it is, it is difficult for the user to see. Therefore, by displaying a predetermined area image indicating a predetermined area of the omnidirectional image on the communication terminal, the user is photographed with a general digital camera. Can be viewed with the same feeling as a flat image.

  In addition, when creating an omnidirectional image, a process for joining two hemispherical images is required. Since each hemisphere image is obtained through a fisheye lens having an angle of view of 180 ° or more, a special digital camera has conventionally used a joint of each hemisphere image to create a 360 ° omnidirectional image. Image quality is improved by extracting a region image and performing distortion correction processing and then performing template matching (see Patent Document 2). As a joint process by this technique, corresponding regions are cut out as a reference image and a comparison image, and template matching is performed on the image after distortion correction.

  By the way, since the digital camera for omnidirectional photography has two image sensors facing different directions, one image sensor photographs a very bright landscape such as the sun, and the other image sensor captures a back alley or the like. You may be shooting a very dark landscape. In such a case, the luminance and color difference are very different between hemispherical images. If the two hemispherical images are combined as they are, the luminance difference between the images becomes conspicuous, giving a sense of incongruity. In order to solve this problem, there is already known an image processing device that reduces a luminance difference between a plurality of images by determining a high luminance image from a plurality of images and correcting the high luminance image ( (See Patent Document 3).

  However, an image processing apparatus that determines a high-luminance image from a plurality of images so far and corrects the high-luminance image to reduce the luminance difference between the plurality of videos, a path for transferring image data, A path for determining a high-intensity image and transferring correction data obtained at that time exists independently. For this reason, both write to the same memory via the interconnect circuit, resulting in a problem of poor access efficiency to the memory.

  In order to solve the above-described problem, the invention according to claim 1 is an image processing apparatus that performs image processing on each image acquired from a plurality of image sensors, and the pixel value of each pixel of the plurality of images. A correction target determination unit that determines a correction target image from the plurality of images, and a correction amount calculation unit that calculates a correction amount for correcting the correction target image based on the pixel value of each image. And an embedding unit that embeds information on the correction amount in any vignetting area of the plurality of images.

  According to the present invention, since the information about the correction amount is embedded in any vignetting area of a plurality of images, the memory access efficiency is improved.

(A) is a right side view of the special photographing apparatus, (b) is a rear view of the special photographing apparatus, (c) is a plan view of the special photographing apparatus, and (d) is a bottom view of the special photographing apparatus. It is. It is a usage image figure of a special imaging device. (A) is a hemispheric image (front) photographed by a special photographing device, (b) is a hemispheric image (rear) photographed by a special photographing device, and (c) is an image represented by equirectangular projection. FIG. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of a virtual camera and a predetermined area | region at the time of making a spherical image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. It is a hardware block diagram of the image processing apparatus in the imaging device concerning this embodiment. It is a functional block diagram of the image processing apparatus in the imaging device according to the present embodiment. It is a flowchart which shows the basic process for producing an omnidirectional image. It is a conceptual diagram of the image data imaged with the single image sensor of the digital camera for omnidirectional photography. It is a conceptual diagram of the image data imaged with two image sensors of the digital camera for omnidirectional photography. It is the conceptual diagram which showed the process which converts a circular image into a rectangular image. It is a figure explaining the overlapping area | region of each rectangular image. It is a conceptual diagram of the image data output from an image sensor. It is the conceptual diagram which showed the vignetting area | region (black area | region) used for black level correction | amendment. It is the conceptual diagram which showed the vignetting area | region (black area | region) used for black level correction | amendment. It is the flowchart which showed the judgment process of the flare image. It is the conceptual diagram which showed all the division areas in an image. It is the conceptual diagram which showed the overlap area and the division area near an overlap area. It is the sequence diagram which showed the process which produces a correction | amendment exclusion map and a temporary correction map. It is the conceptual diagram which divided | segmented the flare image into the several area. It is the conceptual diagram which showed the state by which the exclusion flag was stored in each division area of a flare image. (A) is a conceptual diagram showing an overlapping area and a divided area near the overlapping area in a flare image, (b) is an enlarged view showing only a part of the divided area of (a), and (c) is an overlapping in a reference image. The conceptual diagram which showed the division area of the area | region and the overlapping area vicinity, (b) is an enlarged view which showed only the partial division area of (c). It is a conceptual diagram for determining the correction amount of a flare image. It is the flowchart which showed the production | generation of a correction map, and the process which embeds this correction map. It is a conceptual diagram for determining the correction amount of a flare image. It is a conceptual diagram at the time of scaling the division area of the correction map according to the resolution of the correction target image. It is a conceptual diagram which shows the data for each line of the hemispherical image before embedding a correction map, and the vignetting area | region in a hemispherical image. It is a conceptual diagram which shows the data for each line of the hemispherical image after embedding a correction map, and the vignetting area | region in a hemispherical image. It is the conceptual diagram which showed the transfer data per frame. It is a conceptual diagram of the modification of the image data output from an image pick-up element.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< Summary of Embodiment >>
First, the outline of this embodiment will be described with reference to FIGS. 1 to 7.

  First, the external appearance of the photographing apparatus 1 will be described with reference to FIG. The photographing apparatus 1 is a digital camera for obtaining a photographed image that is a source of the omnidirectional image. 1A is a right side view of the special photographing apparatus, FIG. 1B is a rear view of the special photographing apparatus, and FIG. 1C is a plan view of the special photographing apparatus. (D) is a bottom view of the special photographing apparatus.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and (d), a fish-eye lens is placed on the front side (front side) of the special photographing apparatus 1a. A fisheye lens 102b is provided on the rear side (rear side) 102a. Imaging devices (image sensors) 103a and 103b, which will be described later, are provided inside the special imaging device 1a, and a hemispherical image (angle of view 180 °) is obtained by shooting a subject or a landscape through the lenses 102a and 102b, respectively. Above). A shutter button 115a is provided on the surface opposite to the front side of the special imaging device 1a. Further, a power button 115b, a WiFi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the special imaging device 1a. Each time the power button 115b and the WiFi button 115c are pressed, they are switched on and off. The shooting mode switching button 115d switches between a still image shooting mode and a moving image shooting mode each time the button is pressed. The shutter button 115a, the power button 115b, the WiFi button 115c, and the shooting mode switching button 115d are a kind of the operation unit 115, and the operation unit 115 is not limited to these buttons.

  Further, a tripod screw hole 151 for attaching the special imaging device 1a to the camera tripod or the general imaging device 3a is provided at the center of the bottom 150 of the special imaging device 1a. A micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom 150. On the right end side of the bottom 150, an HDMI (High-Definition Multimedia Interface) terminal is provided. HDMI is a registered trademark.

  Next, the use situation of the special imaging device 1a will be described with reference to FIG. In addition, FIG. 2 is a use image figure of a special imaging device. As shown in FIG. 2, the special imaging device 1 a is used, for example, for a user (photographer) to take a subject around the user while holding it in his hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, an outline of processing from the image captured by the special imaging device 1a to the creation of the equirectangular projection image EC and the omnidirectional image CE will be described with reference to FIGS. 3A is a hemispheric image (front side) photographed by the special photographing apparatus 1a, FIG. 3B is a hemispheric image photographed by the special photographing apparatus (rear side), and FIG. It is the figure which showed the image (henceforth "an equirectangular projection image") represented by the cylindrical projection. FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fish-eye lens 102a described later. Also, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) are synthesized by the special imaging device 1a, and as shown in FIG. 3C, the equirectangular projection image EC. Is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems) or the like, the equirectangular projection image is pasted so as to cover the spherical surface as shown in FIG. An omnidirectional image CE as shown in (b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, by displaying a predetermined area (hereinafter referred to as “predetermined area image”) of a part of the omnidirectional image CE as a flat image with little curvature, a display that does not give a sense of discomfort to humans can be achieved. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. In FIG. 6A, the omnidirectional image shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the angle of view α. The zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or away from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 7).

  Then, the predetermined area image Q shown in FIG. 6A is displayed on the predetermined display as an image of the photographing area of the virtual camera IC as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. The predetermined area T may be indicated not by the angle of view α and the distance f but by the imaging area (X, Y, Z) of the virtual camera IC that is the predetermined area T.

  Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. As shown in FIG. 7, the center point CP when the diagonal field angle of the predetermined region T represented by the field angle α of the virtual camera IC is α is the (x, y) parameter of the predetermined region information. It becomes. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following (formula 1) is generally established.

L / f = tan (α / 2) (Formula 1)
Hereinafter, the detailed content of the embodiment of the present invention will be described.

<< Details of Embodiment >>
<Hardware configuration>
Next, a hardware configuration diagram of the image processing apparatus 10 in the photographing apparatus 1 according to the present embodiment will be described with reference to FIG. FIG. 8 is a hardware configuration diagram of the image processing apparatus in the photographing apparatus according to the present embodiment.

  As shown in FIG. 8, the photographing apparatus 1 includes two imaging elements 103 a and 103 b, an image processing device 20, and an external memory 29. Further, the image processing apparatus 10 includes an image processing unit 21, a correction amount creation unit 22, an image correction unit 23, a projective conversion unit 24, an image output unit 25, a CPU 26, and an external memory I / F, each connected to a bus. Has been. Note that the imaging apparatus 1 may be an imaging device that can capture overlapping image areas (overlapping areas) with a plurality of imaging elements and connect the captured images in the overlapping areas. In addition, the process of joining a plurality of images obtained by photographing at the overlapping area can be executed using an image processing IC or software.

  The image sensors 103a and 103b output the digital data of the hemispherical images A and B converted by the A / D converter to the image processing unit 21.

  The CPU 26 controls the entire image processing apparatus 10. The ROM 27 stores a program for starting the imaging apparatus 1, a conversion table, and the like.

  The image processing unit 21 executes predetermined image processing together with the CPU 26 and the like. As the image processing unit 21, an ASIC (Application Specific Integrated Circuit) that is an integrated circuit for a specific application can be used. The image processing unit 21, together with the CPU 26 and the like, performs processing for connecting images together as predetermined image processing to generate an omnidirectional image, and the generated omnidirectional image data is externally transmitted via the external memory I / F 28. Store in the memory 29. The image processing unit 21 stores the images input from the image sensors 103 a and 103 b in the external memory 29 and delivers them to the correction amount creating unit 22.

  The correction amount creation unit 22 creates a correction amount based on the images from the image sensors 103a and 103b. In creating the correction amount, a correction target image determination process is executed, and the evaluation value of the image of the previous frame is referred to in the determination process. The correction amount is created before the image is stored in the external memory 29.

  The image correction unit 23 reads the image stored in the external memory 29 after the correction amount creation processing by the correction amount creation unit 22 is completed, and performs image correction based on the created correction amount. An equirectangular projection image (Equi image) as shown in FIG. 3C is created.

  As shown in FIG. 4, the projective transformation unit 24 transforms the equirectangular projected image into an omnidirectional image. That is, correction processing is performed before input to the projective conversion unit 24.

  The image output unit 25 outputs the image that has undergone the projective transformation by the projective transformation unit 24.

<Functional configuration>
Next, the functional configuration of the image processing apparatus 10 will be described with reference to FIG. FIG. 9 is a functional block diagram of the image processing apparatus 10 in the photographing apparatus 1 according to the present embodiment. Here, a functional configuration of the correction creating unit 22 described in detail in the present embodiment will be described.

  As shown in FIG. 9, the correction amount creation unit 22 can realize the mechanisms of the correction target determination unit 31, the correction amount calculation unit 32, the correction map creation unit 33, and the embedding unit 34.

  The correction target determination unit 31 determines a correction target image from the plurality of images based on the pixel values of the pixels of the plurality of images. The correction amount calculation unit 32 calculates a correction amount for correcting the correction target image based on the pixel value of each image. The correction map creation unit 33 creates a correction map (M4) for correcting the correction target image. The embedding unit 34 embeds information regarding the correction amount such as data of the correction map (M4) in any vignetting area of the plurality of images. Each part 31-34 is demonstrated in detail later.

<< Process of Embodiment >>
Subsequently, the processing of this embodiment will be described with reference to FIGS. 10 to 31.

  First, basic processing for creating an omnidirectional image will be described with reference to FIGS. 10 to 17. FIG. 10 is a flowchart showing a basic process for creating an omnidirectional image. As illustrated in FIG. 10, the image processing unit 21 performs distortion correction on the hemispheric images A and B acquired from the two imaging elements 103 a and 103 b, thereby converting each hemispheric image (circular image) into each planar image. Conversion into (rectangular image) (step S11). This process will be described with reference to FIGS.

  FIG. 11 is a conceptual diagram of image data captured by a single image sensor of a digital camera for omnidirectional photography.

  As shown in FIG. 11, hemispherical image data obtained by the image sensor 103a can be shown as in FIG. Since the lens 102a of the imaging device 1 is a fisheye lens, a circular image with a white outline is formed on the imaging surface of the imaging element 103a. In addition, the black part has shown the location which is not irradiated with light.

  FIG. 12 is a conceptual diagram of image data captured by the two imaging elements of the omnidirectional digital camera. When an omnidirectional image is formed from two images obtained by the two image sensors 103a and 103b, there must be an overlapping area between the two images. The photographing apparatus 1 uses a fisheye lens having an angle of view exceeding 180 degrees. Therefore, like the hemispherical image A (see FIG. 12A) and the hemispherical image (see FIG. 12B), overlapping regions (lane portions in the figure) are generated on each imaging surface. Two pieces of image data are joined using this overlapping region. However, distortion and aberration tend to occur as the image point moves away from the center of the image. In addition, the outer frame of the fisheye lens may move in. Therefore, it is not desirable to use the outer frame (near the boundary between the hatched area and the black area) of the hemispherical image (fisheye image). Therefore, as in the hemispherical image A (see FIG. 12C) and the hemispherical image (see FIG. 12D), the overlapping region 30 is limited to the region indicated by the diagonal lines having a predetermined width inside each.

  FIG. 13 is a conceptual diagram showing processing for converting a circular image, which is a hemispherical image, into a rectangular image. As shown in FIG. 13, the circular image formed on the imaging surface is converted into rectangular image data by distortion correction using a conversion table. Thereby, the process of step S11 is complete | finished.

  Next, the image processing unit 21 searches for a connecting portion between the rectangular images a and b converted from the two hemispherical images A and B (step S12). Then, the image processing unit 21 overlaps the overlapping areas of the rectangular images a and b, and performs a blending process (synthesizing process), thereby obtaining one equirectangular projected image as shown in FIG. Create (step S13).

  FIG. 14 is a diagram illustrating an overlapping area of each rectangular image. When two rectangular images a and b are superimposed in step S13, one equirectangular projection image as shown in FIG. 14 is created. At this time, the image processing unit 21 detects a similar portion between the rectangular image a and the rectangular image b in the overlapping region 30 by processing such as template matching. A unit for performing template matching may be an arbitrary divided image size unit. However, the image processing unit 21 combines the two rectangular images a and b after template matching.

  Subsequently, before explaining the next processing, vignetting areas (black areas) in image data output from the image sensor will be described with reference to FIGS. 15 to 17.

  FIG. 15 is a conceptual diagram of image data output from the image sensor. As described above, the image data output from the image sensor is a conceptual diagram as shown in FIG. 11 because a fisheye lens is used. In FIG. 11, the middle white portion is a region where an image is formed, and the black portion is a black region (vignetting region) where no image is formed. Therefore, in order to reduce useless areas, the image sensors 103a and 103b of the present embodiment output image data in which the left and right black areas are deleted as shown in FIG. The broken line in FIG. 15 shows the deleted black area. However, useless black areas still remain in the image data.

  FIG. 16 is a conceptual diagram showing a vignetting area (black area) used for black level correction. This remaining black area is used by the ISP (Image Signal Processing) unit in the image processing unit 21 for correcting the black level of the image data. In FIG. 16, the area used for black level correction is a square area, but the position is not limited to this. However, useless black areas still remain in the image data.

  FIG. 17 is a conceptual diagram showing a vignetting area (black area) used for black level correction. After the black level correction, that is, after passing through the ISP section, this useless black area (vignetting area) is not used for various image processing at all. However, the image data is once stored in the external memory 29 including this useless black area. That is, useless data is stored in the external memory 29. In the present embodiment, as will be described later, the access efficiency of the external memory 29 is improved by embedding “additional information” in this useless black area (vignetting area) after passing through the ISP. This additional information is, for example, an image (flare image) in which the flare is generated when flare is generated in one hemisphere image by imaging the sun, illumination, etc. of the two hemispheric images A and B. This is information on the amount of correction for matching the luminance of the other hemispherical image.

  Subsequently, in embedding the additional information, first, a process for determining a flare image as a correction target image among the hemispherical images A and B will be described with reference to FIGS. FIG. 18 is a flowchart showing a flare image determination process. The processing of FIG. 18 is all performed by the correction target determination unit 31 of FIG.

  First, the correction target determination unit 31 calculates an average value for the luminance value of the overlapping region 30 (step S21). This will be supplementarily described with reference to FIGS. 19 and 20.

  FIG. 19 is a conceptual diagram showing all the divided areas in the image. As described above, in the hemispherical image, the vicinity of the outer frame is the overlapping region 30. The correction target determination unit 31 measures the pixel value (luminance value) of the overlapping region 30 and calculates an average value (or variance value).

  Next, the correction target determination unit 31 compares the absolute values of the average values between the relative areas of the hemispherical images A and B (step S22). Then, the correction target determining unit 31 determines whether or not the compared difference is larger than the threshold (step S23).

  In step S23, if larger, the correction target determining unit 31 determines that the hemispherical image that exceeds the threshold is a correction target image (here, a flare image) (step S24). In step S23, if not large, the correction target determination unit 31 determines that there is no correction target image (here, a flare image) (step 25).

  The image data format may be RGB or YCbCr. In this case, the result of calculating the average value differs depending on where the calculated area is located in the image data. This will be described with reference to FIG. FIG. 20 is a conceptual diagram showing an overlapping area and divided areas near the overlapping area. As shown in FIG. 20, in the case of the vicinity of the overlapping region, it is used to determine which of the two hemispherical image data is to be corrected. Specifically, the above average and variance values are taken as absolute values of the difference between the relative areas of the two hemispheric images, and an image whose value exceeds the threshold is determined as a correction target image, and correction is performed when there is such an image. It is determined that there is a target image (flare image). In the case of a camera for omnidirectional imaging, if one image is a correction target image, the other image can be determined as a correction reference image. Basically, when imaged by simultaneous exposure, the overlapping region 30 should be taken with the same brightness as the image of the same subject, but when a flare occurs in one hemisphere image, the average value of the overlapping region 30 Therefore, a threshold value is provided as described above, and a flare image is determined when it is larger than the threshold value.

Subsequently, when the correction target determination unit 31 determines that there is a flare image, the correction amount calculation unit 32 creates a correction map. Here, the creation of the correction map will be described in detail.
First, processing for creating a correction exclusion map and a temporary correction map for use in creating a correction map will be described with reference to FIGS. FIG. 21 is a sequence diagram showing processing for creating a correction exclusion map and a temporary correction map. The correction amount calculation unit 32 performs all the processing in FIG.

  When correction is performed on the flare image, the entire flare image is corrected, so that the image becomes dark overall. Then, when the two hemispherical images A and B are superimposed and combined, an unnatural image may be obtained. Therefore, a correction exclusion map M2 is created in order to distinguish a portion (such as a light source) that does not require flare correction.

  The correction exclusion map M2 can be obtained by the maximum value of the luminance value in the area in the hemispheric image other than the overlapping region 30 described above. The condition of the correction exclusion region can be “high luminance and achromatic color” indicated by (condition 1) or “high luminance and maximum luminance value equal to or greater than a threshold value” indicated by (condition 2). The threshold value for high brightness may be set by a register or the like.

  As the correction exclusion condition, it is desirable to make a determination based on both (Condition 1) and (Condition 2). When the entire evaluation area has uniform brightness and color, it can be determined only by (Condition 1). However, if a high-luminance subject such as a light source and a low-luminance subject such as a tree are present in one evaluation area, the light source image is corrected and the light source is dark because the calculation according to (Condition 1) is not satisfied. This is because the image becomes unnatural. By including (Condition 2), an image including a light source and a tree can be appropriately extracted as a correction exclusion region.

  FIG. 22 is a conceptual diagram in which a flare image is divided into a plurality of areas. FIG. 23 is a conceptual diagram showing a state in which an exclusion flag is stored in each divided area of the flare image. The correction exclusion map M2 has the same form as when a correction target image (here, a flare image) is divided into a plurality of evaluation areas. The correction amount calculation unit 32 determines whether there is a correction exclusion target (step S31). Then, when there is a correction exclusion target (YES), the correction amount calculation unit 32 sets “1” for the block to be excluded from correction, as shown in FIG. Is stored (step S32). On the other hand, when there is no correction exclusion target (NO), the correction amount calculation unit 32 sets “1” for the blocks that are not excluded from the correction, as shown in FIG. ". At this time, the correction amount calculation unit 32 not only includes flags “1” and “0” indicating whether or not to exclude, but also an evaluation value for each block (for example, an average value or a variance value of pixel values (luminance values)) ) Is also stored (step S34). And the correction amount calculation part 32 repeats the process of step S31-34 with respect to all the division areas (step S35, NO). When the processing in all the divided areas is finished, the correction amount calculation unit 32 finishes creating the correction exclusion map M2 (step S36).

  Subsequently, the correction amount calculation unit 32 calculates a correction amount (step S37) and creates a temporary correction map M1 (step 38). Here, the calculation of the correction amount and the creation of the temporary correction map M1 will be described in detail with reference to FIGS. In FIG. 24, (a) is a conceptual diagram showing an overlapping area and a divided area near the overlapping area in the flare image, (b) is an enlarged view showing only a part of the divided area of (a), and (c) is an illustration. The conceptual diagram which showed the overlap area in the reference | standard image, and the division area of the overlap area vicinity, (b) is an enlarged view which showed only the partial area of (c).

  The correction amount is calculated by comparing overlapping areas of the flare image and the reference image. A region where the overlapping regions 30 of Ch0 (see FIG. 24 (a)) and Ch1 (see FIG. 24 (c)) are joined is indicated by a broken-line frame, and each enlarged view (FIG. 24 (b), ( d) see). The digital camera for omnidirectional imaging shoots at the same time. Therefore, the same image in the overlapping region 30 is at a position rotated by 180 degrees as shown in FIGS. 3 (a) and 3 (b). At the time of joining, the value of the evaluation area indicated by area A is high because one has flare, and the other is lower than that. In the present embodiment, correction is performed so as to make this the same value. As a value used for correction, either of the following (Expression 2) or (Expression 3) is used.

Correction amount = (Evaluation value of reference image / Evaluation value of flare image) (Equation 2)
Correction amount = evaluation value of reference image−evaluation value of flare image (Equation 3)
The calculated correction value is stored in the area A of the flare image. This process is performed for all the overlapping areas 30.

  FIG. 25 is a conceptual diagram for determining the correction amount of the flare image. A correction amount inside the overlapping region 30 is calculated using the calculated correction amount of the overlapping region 30. Specifically, as shown in FIG. 25, the correction amount is interpolated from the overlap region 30 toward the center in the hemisphere image, and the correction amount of each region inside the overlap region is determined.

  Next, generation of the correction map M4 and processing for embedding the correction map M4 in the vignetting area of the correction target image (hemisphere image) will be described with reference to FIGS. The correction map creation unit 33 performs the processes in steps S41 and S42 in FIG. The embedding unit 34 performs the process of step S43 in FIG.

  As shown in FIG. 26, the correction map creating unit 33 applies the correction exclusion map M2 to the temporary correction map M1 (step S41). This process will be described in detail with reference to FIG.

  FIG. 27 is a conceptual diagram for determining the correction amount of the flare image. The correction map creation unit 33 creates a correction map M3 using the temporary correction map M1 and the correction exclusion map (high brightness map) M2. A specific method is described in, for example, Japanese Patent Application Laid-Open No. 2015-226144, and will be briefly described here. In each divided area of the temporary correction map M1 shown in FIG. 27A, if the exclusion flag of the corresponding divided area of the correction exclusion map M2 in FIG. 27B is “1”, the correction in FIG. Replace the map value with "1.0". On the other hand, if the exclusion flag is “0”, the value obtained from the temporary correction map M1 is used as it is. This process is performed for all the divided areas, and a correction map M3 is created.

  Note that the value of the correction map M3 after application of the correction exclusion map M2 may vary greatly between the divided areas. Therefore, a smoothing filter such as a Gaussian filter may be applied to the calculated correction map M1.

  Next, the correction map creation unit 33 scales the correction map M3 (step S42). This process will be described in detail with reference to FIG.

  FIG. 28 is a conceptual diagram when the division area of the correction map is scaled according to the resolution of the correction target image. The correction map M3 has a resolution of an arbitrary block size. In order to apply this to the entire correction target image, the correction map M3 shown in FIG. 28A is scaled to create a correction map M4 converted to the resolution of the correction target image as shown in FIG. To do. As a scaling method, a scaling method such as nearest neighbor, bilinear or bicubic is used. The corrected correction map M4 is applied to the entire correction target image to obtain a corrected image.

  If (Equation 2) is used when calculating the correction amount, the correction image is multiplied by the correction amount. On the other hand, if (Equation 3) is used, the correction amount is added to the corrected image. As disclosed in Japanese Patent Application Laid-Open No. 2015-226144 regarding the above correction calculation, the amount of calculation value calculated with an upper limit or a lower limit is set so that the calculation value does not become extremely small or too large. You may restrict.

  Finally, the embedding unit 34 embeds the correction map M4 in the vignetting area of the correction target image (hemispheric image) (step S43). This process will be described in detail with reference to FIGS.

  FIG. 29 is a conceptual diagram showing data for each line of the hemispherical image before embedding the correction map and the vignetting area in the hemispherical image. All hemispherical images including the vignetting area are transferred to the external memory 29. However, as described above, the vignetting area in the hemispherical image after passing through the ISP is unnecessary in the subsequent processing, and thus is a useless area. In this case, even if the transfer is the same one line, as shown in FIG. 29, one line (L1) of the image data of the central portion of the hemispherical image and 1 of the image data at the end of the hemispherical image. The ratio of the vignetting area differs from the line (L2) even if the transfer is the same for one line. Further, the vignetting area is not used in the subsequent image correction unit 23, the projection conversion unit 24, or the like. Therefore, it is possible to embed data in this vignetting area.

  FIG. 30 is a conceptual diagram showing the data for each line of the hemispherical image after embedding the correction map and the vignetting area in the hemispherical image. The embedding unit 34 embeds the correction map M4 in the vignetting area like the lower part of the hemispherical image, so that it is not necessary to transfer the correction map M to the external memory 29 separately from the hemispherical image. As a result, as shown in FIG. 31, the memory bandwidth per frame may be two hemispherical images (fisheye images). That is, it is only necessary to transfer two images, a hemispherical image A (for example, not a correction target image) and a hemispherical image B (for example, a correction target image) in which the correction map M4 is embedded. FIG. 31 is a conceptual diagram showing transfer data per frame. FIG. 31 shows an example in which the correction map M4 is embedded on the hemispherical image B side in the case of the correction target image, but it may be embedded on the hemispherical image A side when it is not the correction target image.

<< Effects of Embodiment >>
As described above, according to the present embodiment, by embedding information related to correction such as the output data of the correction map M4 in the vignetting area of the hemispherical image, the access efficiency to the external memory 29 is improved.

[Modification]
FIG. 32 is a conceptual diagram of a modification of image data output from the image sensor. The imaging surfaces of the imaging elements 103a and 103b are shown as white circular images, and this deformed image is shifted to the left side. In such a case, it may be embedded in the vignetting area on the last line (right side) of each divided area. In the present embodiment, since the correction map M4 is embedded in the vignetting area to reduce the memory bandwidth, the position in the vignetting area is not limited.

  The data to be embedded is not limited to the correction map M4 described above. For example, in FIG. 8, when the image correction unit 23 operates when an arbitrary number of lines of image data are written in the external memory 29, the image correction unit 23 is activated at the end of the embedded correction map M4. It is also possible to embed trigger information. When the external memory I / F 28 detects the trigger information, the activation trigger can be output to the image correction unit 23 and operated. Further, since it is not necessary to count the number of lines written, there is an effect of reducing the circuit scale.

DESCRIPTION OF SYMBOLS 1 Imaging device 20 Image processing device 21 Image processing part 22 Correction amount creation part 23 Image processing part 24 Projection conversion part 25 Image output part 26 CPU
27 ROM
28 External memory I / F
29 External memory 31 Correction target determination unit (an example of correction target determination means)
32 Correction amount calculation unit (an example of correction amount calculation means)
33. Correction map creation unit (an example of correction map creation means)
34 Embedding part (an example of embedding means)

JP, 2014-120878, A Japanese Patent Laid-Open No. 2006-027744 JP2013-243610A

Claims (7)

An image processing apparatus that performs image processing on each image acquired from a plurality of image sensors,
Correction target determination means for determining a correction target image from the plurality of images based on pixel values of respective pixels of the plurality of images;
Correction amount calculation means for calculating a correction amount for correcting the correction target image based on the pixel value of each image;
Embedding means for embedding information on the correction amount in any vignetting area of the plurality of images;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein any one of the plurality of images is a vignetting area of the correction target image.   The image processing apparatus according to claim 1, wherein the information related to the correction amount is a correction map for correcting the correction target image.   The image processing apparatus according to claim 1, wherein the position of the vignetting area is set in a register.   5. The image processing apparatus according to claim 1, wherein the embedding unit selects a predetermined image in which information relating to the correction amount is embedded in the vignetting area among the plurality of images. . An image processing apparatus according to any one of claims 1 to 5,
A first image sensor for sending first image data obtained by photographing a subject to the image processing device;
A second image sensor for sending second image data obtained by photographing a subject to the image processing device;
A photographing apparatus having An image processing method for performing image processing on each image acquired from a plurality of image sensors,
A correction target determination step of determining a correction target image from the plurality of images based on pixel values of respective pixels of the plurality of images;
A correction amount calculating step for calculating a correction amount for correcting the correction target image based on the pixel value of each image;
An embedding step of embedding information on the correction amount in any vignetting area of the plurality of images;
The image processing method characterized by performing.