JP6080503B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that combines a plurality of image data to expand the dynamic range of the image data.

  Conventionally, a process for obtaining a composite image with an expanded dynamic range by combining a plurality of images with different exposures has been proposed. The processing mainly uses high-exposure images for low-brightness areas, and mainly uses low-exposure images for high-brightness areas, thereby reducing the effects of overexposure and blackout, It expands the dynamic range.

  In the above-described processing, each of a plurality of captured image data has a relative positional shift due to camera shake or the like, and therefore a process for correcting the positional shift before combining is essential. If a positional shift occurs between the image data at the time of combining, the images are duplicated in the combined image, and the image quality of the combined image is deteriorated. Therefore, alignment of a plurality of captured image data is an important technique.

  With regard to the above-described image data alignment technique, for example, in Patent Document 1, when determining the amount of positional deviation between images, the positional deviation detection is performed by using only the common luminance region of each image. Techniques to do are proposed.

  In Patent Document 2, a low-contrast region that causes erroneous detection of motion vectors is excluded. Further, by looking at the distribution of motion vectors for each small area determined to be appropriate for motion detection in the exclusion process, only the motion vectors determined to accurately represent the motion of the entire image are used for alignment. Technology has been proposed.

JP 2003-319240 A JP 2008-118555 A

  However, in Patent Literature 1 and Patent Literature 2, it is expected that the positional deviation cannot be corrected due to the minute movement of the subject and the difference in the movement due to the difference in the distance of the subject, and the positional deviation remains.

  Also, in image synthesis for the purpose of apparent dynamic range expansion, a high exposure image synthesis ratio is set high in the low luminance region, and a low exposure image synthesis ratio is set high in the high luminance region. Further, in order to prevent a sudden change occurring in the composite image, the composite ratio is set so that the high-exposure image and the low-exposure image are weighted and added in the intermediate luminance region.

  Therefore, when a positional deviation remains in an area where the high-exposure image and the low-exposure image are weighted and added, the positional deviation becomes noticeable at the time of synthesis, and the image quality of the synthesized image deteriorates. is there.

  In view of the above problems, an object of the present invention is to reduce a positional shift that occurs in an area where image data is weighted and added.

  In order to achieve the above object, an image processing apparatus of the present invention comprises the following arrangement.

Image input means for inputting at least two or more images;
A positional deviation amount detecting means for detecting a positional deviation amount between the input images;
A coordinate transformation coefficient for correcting the positional deviation is calculated from the detected positional deviation amount, and coordinate transformation is performed on the input image using the coordinate transformation coefficient, thereby correcting the positional deviation between the input images. Position misalignment correcting means for correcting;
A positional deviation residual area detecting means for detecting an area where the positional deviation remains from an input image in which the positional deviation is corrected;
A composition ratio setting means for setting a composition ratio of the input image for each predetermined region;
Image compositing means for compositing the input image in which the positional deviation is corrected based on the set compositing ratio.

  According to the present invention, when combining a plurality of image data, it is possible to reduce a positional shift that occurs in an area where the image data is weighted and added.

It is a block diagram showing the structure in the imaging device of 1st Embodiment. It is a graph showing the gradation correction characteristic in 1st Embodiment. 6 is a flowchart illustrating a processing flow of a positional deviation detection unit according to the first embodiment. It is a figure which shows position shift detection block division | segmentation in 1st Embodiment. 6 is a flowchart showing a flow of processing of a positional deviation correction unit 107 in the first embodiment. It is a figure which shows the detection method of a misalignment residual area | region in 1st Embodiment. 6 is a flowchart illustrating a processing flow of a composition ratio setting unit 109 according to the first embodiment. It is a figure which shows the synthetic | combination ratio table in FIG. It is a figure which shows the switching area | region made into the subject in 2nd Embodiment. 10 is a flowchart showing a flow of processing of a composition ratio setting unit 109 in the second embodiment. It is a figure which shows the determination method of the synthetic | combination ratio of a position shift | offset | difference revealing area | region in 2nd Embodiment.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration applicable to the image processing apparatus according to the first embodiment of the present invention. The present embodiment is an imaging apparatus that expands the dynamic range of image data.

  The imaging unit 101 includes an imaging lens group and a semiconductor imaging element such as a CMOS or CCD. In the imaging unit 101 (image acquisition unit), a plurality of shootings are performed in order to acquire at least two input images while changing the exposure condition for the same subject, and the captured video signal is A / D. The data is output to the conversion unit 102. The exposure conditions can be controlled by exposure time (shutter speed), aperture value (F value), ISO sensitivity, etc., and the brightness obtained from an image previously acquired from a photometric sensor (not shown) or the imaging unit 101 can be controlled. Each parameter is controlled based on the information. The A / D conversion unit 102 converts the input video signal into digital image data and outputs the digital image data to the image processing unit 103. The image processing unit 103 corrects the positional deviation of the plurality of input image data, and performs a composition process of the image data in which the positional deviation is corrected. The image display unit 111 displays the image data generated by the image processing unit 103 on, for example, a liquid crystal monitor included in the imaging apparatus main body. Further, the image storage unit 112 records the image data generated by the image processing unit 103 on a recording medium.

  Next, the image processing unit 103 will be described in detail. The image processing unit 103 includes a gradation correction unit 104, a memory unit 105, a position shift detection unit 106 (position shift amount detection means), a position shift correction unit 107, a position shift residual area detection unit 108, a composition ratio setting unit 109, an image. The composition unit 110 is configured.

  The gradation correction unit 104 performs a process for adjusting the luminance level for a plurality of input image data. Here, assuming that the number of input image data is two, an image with a relatively large exposure is an over image, and an image with a small exposure with respect to an over image is an under image with respect to input image data having different exposure conditions. To do. The gradation correction unit 104 performs gradation conversion having different gradation conversion characteristics as shown in FIG. 2 on each of the over image and the under image, thereby matching the luminance levels of the over image and the under image. . However, in this embodiment, the luminance level of the under image is matched to that of the over image. The image data processed by the gradation correction unit 104 is stored in the memory unit 105.

  The image data stored in the memory unit 105 is accompanied by a relative positional shift due to camera shake or the like. Therefore, the positional deviation between the image data is corrected by the positional deviation detection unit 106 and the positional deviation correction unit 107. Detailed processing of each part will be described later.

<Position shift detection unit 106>
FIG. 3 is a flowchart showing a flow of processing for detecting a positional shift amount between input images in the positional shift detection unit 106. As shown in FIG. 3, a reference image serving as a reference for obtaining a positional deviation is set in step S302 for the image data input in step S301. Here, the under image is set as the reference image. That is, the positional shift in the present embodiment means a positional shift of the over image with respect to the under image.

  Next, in step S303, as shown in FIG. 4, the image data is divided into a plurality of blocks that are divided into N parts in the horizontal direction and M parts in the vertical direction. Here, for description of future processing, each divided block is referred to as blk [0] to blk [M * N-1]. Subsequently, in step S304, i = 0 is set as the value of the counter i, and in step S305, a motion vector representing a relative positional shift amount between the under image and the over image for the i-th divided block blk [i]. Is detected. As a motion vector detection method, for example, a pattern matching process using a sum of absolute differences as an evaluation value has been proposed. Further, in steps S306 and S307, motion vectors in all the divided blocks are calculated while i is incremented.

<Position shift correction unit 107>
FIG. 5 is a flowchart showing the flow of processing in the positional deviation correction unit 107. As shown in FIG. 5, only the motion vector with high reliability is extracted in step S502 for all the motion vectors in each divided block input in step S501. The reliability described here means a degree representing the movement of the main subject due to camera shake.

  Examples of the extraction method include a method of performing extraction using the ratio of overexposed and black-out pixels in each divided block and the edge integral value as an evaluation value. The purpose of this is to reduce the influence of erroneous detection of motion vectors by pattern matching (Japanese Patent Laid-Open No. 2008-118555). As another extraction method, there is a method of taking a motion vector distribution and extracting only a motion vector having a value close to the motion vector of the main subject. This is because even if the movement is the same for the same camera shake, if the distance between the main subject and the background is significantly different, the amount of movement will be different. The purpose is to eliminate.

  The above-described highly reliable motion vector extraction method is merely an example, and is not particularly limited here.

  Next, in step S503, the positional deviation correction parameter is estimated using the highly reliable motion vector output from step S502 described above as an input. The misregistration correction parameter is a coordinate conversion coefficient for deforming one image in accordance with the other image, and includes, for example, an affine conversion coefficient and a projective conversion coefficient. As a method for calculating an image conversion coefficient, a method has been proposed in which a plurality of motion vectors are input, and the image conversion coefficient is estimated from the input motion vectors by a least square method.

  Next, in step S504, the over image is deformed using the image conversion coefficient estimated in step S503.

  Through the series of processes, an over image is generated that is aligned with the under image (reference image).

<Position displacement residual area detection unit 108>
The misregistration residual area detection unit 108 receives an under image and an over image that have been corrected for misregistration by the misregistration correction unit 107, and detects an area in which the misregistration remains in the two images. Here, the reason why the positional deviation remains is that, as described above, the difference in the amount of movement due to the difference in distance between the subject and the background, or the minute movement of the subject itself remains uncorrected.

  FIG. 6 shows a schematic diagram of a method for detecting a misregistration residual area. As shown in FIG. 6, a difference image is generated by taking the difference between the under image and the over image that have been subjected to positional deviation correction. The difference here is assumed to be the absolute value of the luminance difference. However, if the luminance of the background and the main subject are equal, the detection accuracy decreases, and therefore the difference of the color difference signal may be used together. A region where a difference occurs in the difference image is a misalignment remaining region. Further, since a difference that is not caused by a residual positional deviation is detected in a region where overexposure or blackout occurs, it is necessary to devise a method such as excluding from the detection target region in advance. The method for detecting the misaligned residual region is not limited to the above, and any calculation method can be used as long as it can correlate two images and evaluate the height of the correlation for each region.

<Composition ratio setting unit 109>
FIG. 7 is a flowchart showing the flow of processing in the composition ratio setting unit 109. As shown in FIG. 7, first, in step S701, a reference luminance image is set. The reference luminance image is a luminance image that is referred to in order to set a composition ratio of input image data. Here, the luminance component of the under image is used as the reference luminance image, but an over image may be used as the reference luminance image.

  Next, as shown in step S702, based on the pixel value of the reference luminance image, the composition ratio of the under image and the over image in each pixel is set based on the composition ratio table. The composition ratio table is a table for setting the composition ratio of the under image and the over image with respect to the reference luminance value. FIG. 8 shows a synthesis ratio table in the present embodiment. Here, as shown in FIG. 8, when the reference luminance value is Y, 100% of the over image is used in the region of Y ≦ Y1, and 100% of the under image is used in the region of Y2 ≦ Y. Further, in the region of Y1 <Y <Y2, the composition ratio table is set so that the under image and the over image are weighted and added. Here, Y1 and Y2 are preset brightness values.

  Here, in the area where the residual positional deviation that appears as a problem in this case becomes apparent (divided in multiple divisions), the positional deviation remains in the area that has the luminance for weighted addition of the over image and the under image. Occurs when. Therefore, by detecting a region where the positional deviation becomes apparent and controlling the composition ratio so that an under image is output 100% in the corresponding region, the residual positional deviation can be reduced. Here, the reason for performing control to output 100% of the under image is that the under image is a reference image for alignment in the present embodiment and to avoid overexposure of the 100% output area. .

  Based on the above background, next, as shown in step S703, a weighted addition region and a misaligned residual region are detected. Here, the weighted addition area is at least 2 which is not a composition ratio (100: 0 or 0: 100 [%]) in which only one of the under image and the over image is used in step S702 described above. It is assumed that this is an area indicating weighted addition in which a single image is synthesized. However, the definition of the weighted addition region is not limited to this, and may be determined within the range of the weighted addition coefficient, for example.

  Next, as shown in step S <b> 704, the composition ratio of the detected weighted addition area and misaligned residual area is changed so that an under image is output 100%. Further, for the purpose of making it inconspicuous, the object is achieved if the ratio of the under image is equal to or higher than a predetermined ratio, for example, 90% or higher.

  The above is the processing of the composition ratio setting unit 109, and a composition ratio that reduces the influence of the positional deviation that becomes apparent in the weighted addition region is output.

  In the image composition unit 110, the under image and the over image whose alignment has been completed are composed for each pixel in accordance with a pre-computed composition ratio.

  As a result of the above processing, the image processing unit 103 corrects the positional deviation between the under image and the over image, and synthesizes the under image corrected for the positional deviation and the over image, so that the composite image data having an expanded dynamic range is obtained. Generated.

  As described above, according to the first embodiment, the weighted addition region and the region where the positional deviation remains are determined, and the composition ratio is controlled so as to output 100% of the under image in the region. It is possible to reduce the problem of misalignment in the composite image.

[Second Embodiment]
In the first embodiment, a misalignment manifestation area (weighted addition area and misalignment remaining area) is detected, and the composition ratio is controlled so that 100% of the under image is output in the area. However, as shown in FIG. 9, when 100% of the over image is output around the misalignment revealing area, if 100% of the under image is output according to the first embodiment, the misalignment revealing area and the surrounding area are output. The under image and the over image are switched at the boundary of the region. This may cause misalignment at the boundary. In such a case, it is considered preferable to output 100% of the over image giving priority to the connection of images. Based on the above background. In the second embodiment, in addition to the first embodiment, a case where an over image is output 100% around the position deviation manifestation area is detected. In the above case, an over image is detected in the position deviation detection area. Consider a process of controlling the composition ratio so that 100% is output. As a similar process, when 100% of the under image is output in the vicinity of the misalignment manifestation region, it is possible to control such that the corresponding region outputs 100% of the under image.

  In the block diagram shown in FIG. 1, only the composition ratio setting unit 109 changes the processing contents in the second embodiment as compared with the first embodiment. Hereinafter, the process of the synthesis ratio setting unit 109 will be described in detail.

  FIG. 10 is a flowchart illustrating a processing flow of the composition ratio setting unit 109 according to the second embodiment. As shown in FIG. 10, steps S1001 to S1002 perform the same processing (S701 to S702 in FIG. 7) as in the first embodiment. In the present embodiment, steps S1003 and S1004, which are subsequent processes, are different from those of the first embodiment.

  In steps S1003 and S1004, a positional deviation manifestation area is detected with respect to the composition ratio set in the previous process, and an under image is output 100% for each detected positional deviation manifestation area. Select whether to output 100% over image. FIG. 11 shows an example of a method for determining an output image in the positional deviation manifesting region.

  As shown in FIG. 11, first, an image is divided into small blocks, and it is determined whether or not each block is a positional deviation revealing block. As an example of the determination method, the ratio of the pixels that are the weighted addition area where the positional deviation becomes obvious and the positional deviation residual area is calculated with respect to the number of pixels included in the block. And when the ratio exceeds a predetermined threshold value, the block is determined to be a positional deviation obvious block.

  Next, when the positional deviation revealing blocks are adjacent to each other over a plurality of blocks, the set of the adjacent blocks is set as a positional deviation actualizing area.

Next, paying attention to the blocks around each positional deviation manifestation area determined in the processing, the blocks are divided into any of the following blocks.
-X block: block with no residual displacement-O block: block where positional displacement remains but over image is output 100%-U block: positional displacement remains but under image is 100 % Output block As an example of the determination method of each block, the ratio of pixels in which the positional deviation included in the positional deviation remaining area remains is calculated with respect to the number of pixels included in the block, and the ratio is predetermined. If the value is equal to or less than the threshold value, the block is designated as an X block. Further, for a block that is not determined to be an X block, a ratio of pixels in which an over image is output 100% is calculated with respect to the pixels included in the misaligned residual area in the block, and the ratio is equal to or greater than a predetermined threshold In this case, it is O block, and in other cases, it is U block.

  Next, the ratio of O blocks is obtained for the surrounding blocks determined in the above process. When the block ratio is equal to or greater than a predetermined threshold value, the composition ratio is controlled so that an over image (specific input image) is output 100% in the focused positional deviation manifesting region (FIG. 11, (1)). In cases other than (1), the composition ratio is controlled so as to output 100% of the under image in the positional deviation manifesting region of interest (FIGS. 11, (2) -1, 2). The case other than the above (1) is a case where the periphery is almost an X block or a U block. When the surrounding area is an X block, there is no positional deviation in the first place, so that the switching that is the problem does not occur even if 100% of either the over image or the under image is output. However, since there is a possibility that the over image includes whiteout pixels, in this embodiment, the under image is output 100%.

  As described above, according to the second embodiment, the weighted addition region and the positional deviation revealing region where the positional deviation remains are determined, and the composite ratio in the vicinity region around the positional deviation manifesting region is further determined. To control the composition ratio. Specifically, the composition ratio of the region is set so that a specific input image that is a composition ratio dominant in the surroundings is synthesized at a predetermined ratio or more. As a result, it is possible to reduce the problem that the positional deviation is visible in the composite image, and it is possible to reduce the influence of the positional deviation that occurs at the boundary portion of the positional deviation manifesting region.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  In addition, the present invention is not limited to devices such as digital cameras, but includes built-in or external connection of imaging devices such as mobile phones, personal computers (laptop type, desktop type, tablet type, etc.), game machines, etc. It can be applied to any device. Therefore, the “imaging device” in this specification is intended to include any electronic device having an imaging function.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 A / D conversion part 103 Image processing part 104 Gradation correction | amendment part 105 Memory part 106 Position shift detection part 107 Position shift correction part 108 Position shift residual area detection part 109 Composition ratio setting part 110 Image composition part 111 Image display Part 112 Image recording part

Claims (6)

Image acquisition means for acquiring at least two input images;
A positional deviation amount detecting means for detecting a positional deviation amount between the input images;
A coordinate transformation coefficient for correcting the positional deviation is calculated from the detected positional deviation amount, and coordinate transformation is performed on the input image using the coordinate transformation coefficient, thereby correcting the positional deviation between the input images. Position misalignment correcting means for correcting;
A positional deviation residual area detecting means for detecting an area where the positional deviation remains from an input image in which the positional deviation is corrected;
Setting means for setting the composition ratio of the input image for each predetermined area;
Image synthesis means for synthesizing the input image in which the positional deviation is corrected based on the set synthesis ratio;
The setting means, mixing ratio of the pixels belonging to the positional deviation remaining regions, to indicate the weighted addition at least two images are synthesized, the front Symbol position location offset remaining regions, the synthesis of one of the images The ratio is changed so as to be equal to or greater than a predetermined ratio , and when the ratio of pixels in which the composite ratio of the specific input image is equal to or greater than the predetermined ratio in the vicinity of the misalignment residual area is dominant, An image processing apparatus , wherein a pixel belonging to a misregistration residual region is changed so that a composition ratio of the specific input image is a predetermined ratio or more . The image processing apparatus according to claim 1, wherein the setting unit sets a composition ratio of the input image for each predetermined region based on brightness of each region. The image processing apparatus according to claim 1, wherein the plurality of input images are images captured under different exposure conditions. The setting means refers to a luminance value of a reference luminance image, uses an image with a large exposure from the plurality of input images for the region with the low luminance value, and applies to the region with the large luminance value. 4. The method according to claim 3, wherein the image is controlled to use an image having a low exposure, and the luminance image is generated from an image having the lowest exposure among the plurality of input images. The image processing apparatus described.   5. The image processing apparatus according to claim 1, wherein the misregistration residual area detection unit detects the misregistration residual area by taking a correlation between the input images. 6. An image acquisition step of acquiring at least two input images;
A positional shift amount detecting step for detecting a positional shift amount between the input images;
A coordinate transformation coefficient for correcting the positional deviation is calculated from the detected positional deviation amount, and coordinate transformation is performed on the input image using the coordinate transformation coefficient, thereby correcting the positional deviation between the input images. A position misalignment correction step to be corrected;
A position misalignment region detection step for detecting a region in which the position misalignment remains from an input image in which the position misalignment is corrected;
A setting step of setting a composition ratio of the input image for each predetermined region;
An image synthesis step of synthesizing the input image in which the positional deviation is corrected based on the set synthesis ratio;
In the setting step, mixing ratio of the pixels belonging to the positional deviation remaining regions, to indicate the weighted addition at least two images are synthesized, the front Symbol position location offset remaining regions, the synthesis of one of the images The ratio is changed so as to be equal to or greater than a predetermined ratio , and when the ratio of pixels in which the composite ratio of the specific input image is equal to or greater than the predetermined ratio in the vicinity of the misalignment residual area is dominant, An image processing method , comprising: changing a composition ratio of the specific input image to a predetermined ratio or more for pixels belonging to a misregistration remaining area .

Images