JP2011182084A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve positioning precision in superimposing and synthesizing a plurality of images. <P>SOLUTION: A criterion region is set in a criterion image, and a search region being the region to search, in a reference image, a region corresponding to the criterion region is set. A reliability in deriving a motion vector from the criterion region and the corresponding search region is derived. In this case, the reliability is derived by performing dissolution into respective axial components in two axial directions. Regions corresponding to the respective criterion regions are searched in the search region, so as to derive the motion vectors MV1-MV4. The motion vector MV3 is dissolved into the motion vectors of the axial components in the two axial directions thereby to derive the vectors MV3x', MV3y' of the respective axial components. When the reliability lower than a prescribed threshold exists in the reliability of the respective axial components, the reliability of the axial component is determined to be low. Then, the motion vector (MV3y') of the axial components corresponding to the reliability of the axial component determined to be low is corrected (MV4y'). <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing technique used when a plurality of images generated by an image input device such as an electronic photographing apparatus are combined.

  As a technique for reducing image blur, expanding the dynamic range of an image, or obtaining an image with a resolution that exceeds the number of pixels in the image sensor and the resolution of the taking lens, multiple images are aligned and superimposed. A technique for synthesizing (hereinafter, synthesizing and synthesizing a plurality of images is referred to as “combining a plurality of images”) is known.

  In order to reduce the image blur, the exposure operation is performed a plurality of times in a relatively short time in a time shorter than the shutter speed at which the appropriate exposure amount is obtained. By photographing in this way, it is possible to obtain an image with little image blur even if each image is an image with insufficient exposure. By synthesizing a plurality of images thus obtained, an image with less image blur can be obtained with an appropriate exposure amount. This technique is hereinafter referred to as an electronic blur correction technique.

  In the dynamic range expansion, an exposure operation is performed a plurality of times, and at that time, an exposure operation is performed with an underexposure exposure amount and an overexposure exposure amount with respect to a standard exposure amount. In an image obtained with an underexposure exposure amount, whiteout in a highlight portion is suppressed, and in an image obtained with an overexposure exposure amount, blackout in a shadow portion is suppressed. By synthesizing these plural images, it is possible to obtain an image with a wide dynamic range in which blackout and whiteout are suppressed. Hereinafter, this technique is referred to as a dynamic range expansion technique.

  To obtain an image with a resolution that exceeds the number of pixels of the image sensor and the resolution of the taking lens, multiple images obtained from multiple shots are combined, thereby complementing the missing information in each other's images. Each other. Thereby, it is possible to obtain an image having a higher resolution than that of an image obtained by one shooting. This technique is hereinafter referred to as super-resolution technique.

  As described above, a technique for combining a plurality of images is a basic technique in the image processing field, and various methods have been proposed.

  Patent Document 1 discloses a technique for electronic blur correction. In the technique disclosed in Patent Document 1, a representative value of an image is obtained for an image movement vector and a magnification variation amount, and the positional deviation of the image is corrected. However, in actual shooting, camera shake or subject blur occurs during a series of a plurality of exposure operations. For this reason, the position of the subject (subject position) changes in each of the plurality of images. The change in the subject position caused by the camera shake of the photographer becomes a uniform overall movement in the image. Therefore, it is possible to easily synthesize a plurality of images by obtaining a movement vector at a representative position of the image and aligning it. On the other hand, in subject blurring, only a moving part in a subject is displaced between a plurality of images. Such locally generated positional deviation is difficult to correct with a method for correcting the overall movement between a plurality of images as disclosed in Japanese Patent Application Laid-Open No. 2004-133620.

  In order to cope with this problem, Patent Documents 2 and 3 disclose a method of simultaneously correcting the local movement caused by the movement of the subject as well as the overall positional deviation of the image caused by the camera shake of the photographer. Proposed. The methods disclosed in Patent Document 2 and Patent Document 3 divide an image into small regions, obtain a movement vector in each region, and then smoothly deform the image. By doing so, it becomes possible to simultaneously correct blurring (blurring of camera shake) existing in the entire image and blurring (subject blurring) existing locally in the image.

JP 2006-319578 A JP 2006-099467 A Japanese Patent No. 3935500

  In the above-described method, the determination process of the reliability of the movement vector obtained in each region greatly affects the performance. For example, FIG. 1 illustrates an area where it is difficult to correctly determine a movement vector. FIG. 1 is a diagram illustrating an example of an image having a region where it is difficult to correctly obtain a movement vector when aligning a plurality of images.

  In FIG. 1, a low contrast region 1, a repetitive pattern region 2, an opening problem region 3 and the like are shown. The opening problem area can be defined as an area that has contrast but lacks clues for alignment. In such an area, a position different from the original corresponding position may be detected as a matching position in the template matching process. As a result, there are cases where the required movement vector is not always accurate. The combined image may become unnatural due to the influence of such an inaccurate movement vector.

  Japanese Patent Application Laid-Open No. 2004-228667 discloses a technique for deforming a grid after dividing an image into a grid, obtaining a movement vector at each grid point, and further weighting according to the magnitude of the luminance gradient of each grid point. . This suppresses the adverse effect of the movement vector obtained in a region with a small luminance gradient, such as the low contrast region 1. However, in the technique of Patent Document 2, it is not always sufficient to consider the movement vector obtained in the repeated pattern area 2 and the opening problem area 3.

  In Patent Document 3, an image is divided into small regions, movement vectors are obtained in each region, and the reliability of the movement vectors according to the size of the edge or the corner angle and the correlation between the surrounding movement vectors are calculated. A technique for smoothing a movement vector while weighting according to the degree of correlation to be expressed is disclosed.

  The technique disclosed in Patent Document 3 does not use the movement vector obtained in the low contrast area 1, the repeated pattern area 2, and the opening problem area 3. Therefore, there is a possibility that the influence of errors occurring in the movement vector required in these areas can be eliminated. However, there is a possibility that the information contained in these excluded movement vectors cannot be used, and there are fewer clues for alignment. As a result, the alignment process may not be stable.

  The present invention has been made in view of the above problems, and evaluates the reliability of a movement vector in more detail, and extracts and uses information effective for alignment from a movement vector with low reliability. It is an object to provide a technique capable of obtaining a more stable image alignment result.

According to the first aspect of the present invention, a movement vector measurement region setting unit that sets a reference region in a reference image and sets a search region that is a region for searching for a region corresponding to the reference region in a reference image. When,
When a movement vector is derived from the reference area and the corresponding search area set by the movement vector measurement area setting unit, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving unit derived by analyzing at least one of the regions;
A motion vector deriving unit for searching in the search region for a region corresponding to the reference region set by the movement vector measurement region setting unit, and deriving a motion vector;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived by the movement vector deriving unit is decomposed into axial components along two axial directions. The movement vector correction processing unit for deriving the movement vector of each axis component and correcting the movement vector of the axis component with lower reliability among the movement vectors of each axis component solves the above-described problem. .

A second aspect of the present invention is an image processing program for causing a computer to execute a process of superimposing by reducing a positional deviation between a plurality of images. This image processing program
A movement vector measurement region setting step for setting a reference region in a reference image and setting a search region which is a region for searching for a region corresponding to the reference region in a reference image;
When a movement vector is derived from the reference area and the corresponding search area set in the movement vector measurement area setting step, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving step derived by analyzing at least one of the regions;
A motion vector deriving step of searching in the search region for a region corresponding to the reference region set in the motion vector measurement region setting step, and deriving a motion vector;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived in the movement vector deriving step is decomposed into two axial components. A movement vector correction step of deriving a movement vector of each axis component and correcting a movement vector of the axis component having lower reliability among the movement vectors of each axis component.

A third aspect of the present invention is an image processing program for causing a computer to execute a process of superimposing by reducing a positional deviation between a plurality of images. This image processing program
A movement vector measurement region setting step for setting a reference region in a reference image and setting a search region which is a region for searching for a region corresponding to the reference region in a reference image;
A motion vector deriving step of searching in the search region for a region corresponding to the reference region set in the motion vector measurement region setting step, and deriving a motion vector;
When a movement vector is derived from the reference area and the corresponding search area set in the movement vector measurement area setting step, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving step derived by analyzing at least one of the regions;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived in the movement vector deriving step is decomposed into two axial components. A movement vector correction step of deriving a movement vector of each axis component and correcting a movement vector of the axis component having lower reliability among the movement vectors of each axis component.

  According to the present invention, when it is determined that the reliability is high only in one axial direction, it is possible to effectively use the movement vector of the component in the axial direction, and the positional deviation between a plurality of images can be reduced. It can be reduced.

It is a figure explaining the example of the image element which can anticipate exact position alignment by template matching, and the image element to which exact position alignment is difficult. It is a schematic block diagram explaining the example in which the image processing part of 1st Embodiment is integrated in an imaging device. It is a schematic block diagram explaining the example of an internal structure of the position alignment process part which the image process part of 1st Embodiment has. It is a figure which shows typically the image used as the object of the alignment process, (a) is a reference | standard image, (b) is a figure which shows a reference image. It is a figure explaining the relationship between the pattern of the image element in the template area | region of a reference | standard image, and the reliability of the movement vector derived | led-out using this pattern as a template, (a) is reliability of both axial components of a biaxial direction. (B) is an example in which only the axial component in one axial direction is highly reliable, and (c) and (d) are examples in which both the axial components in two axial directions are low in reliability. It is a schematic flowchart explaining the procedure of a reliability derivation process. It is a schematic flowchart explaining the procedure of opening problem area | region determination processing. It is a figure explaining the relationship between the edge gradient direction of an image pattern, and the direction where the reliability of the derived movement vector becomes high. It is a figure which illustrates notionally the movement vector correction process performed in a movement vector correction process part, (a) shows the movement vector before correction | amendment processing, (b) shows the movement vector after correction | amendment processing. It is a figure explaining a mode that correction processing is made to a movement vector of an axis component of a low reliability direction by movement vector correction processing. It is a figure explaining movement vector interpolation processing notionally, (a) shows a standard picture and (b) shows a reference picture. It is a schematic flowchart explaining the process sequence which synthesize | combines a some image. FIG. 10 is a diagram for explaining the relationship between the pattern of an image element in a template region of a reference image and the reliability of a movement vector derived using this pattern as a template in the second embodiment. An example in which both the axial components along the axis and the Y axis are highly reliable, (b) an example in which only the axial component along the X axis is highly reliable, and (c) in the axial direction along the Y axis. FIGS. 4A and 4B are diagrams showing an example in which only the component has high reliability, and FIGS. 4D and 4E show examples in which the components in both axial directions along the X axis and the Y axis are low in reliability. It is a schematic flowchart explaining the procedure of the reliability derivation process in 2nd Embodiment. In a 2nd embodiment, it is a figure explaining the movement vector correction processing performed in a movement vector correction processing part notionally, (a) shows a movement vector before correction processing, and (b) is after correction processing. The movement vector of is shown. It is a schematic block diagram explaining the example in which the image processing part of 3rd Embodiment is integrated in an imaging device, and is a figure explaining the example in which a movement vector interpolation process and an image composition process are performed by an external processing device. It is a schematic block diagram explaining the example of an internal structure of the position alignment process part which the image process part of 3rd Embodiment has. It is a schematic block diagram explaining the example in which the image processing part of 4th Embodiment is integrated in an imaging device. It is a schematic block diagram explaining the example of an internal structure of the position alignment process part which the image process part of 4th Embodiment has. It is a schematic flowchart explaining the procedure of the reliability derivation processing of 4th Embodiment. It is a schematic flowchart explaining the process sequence which synthesize | combines the some image in 4th Embodiment. It is a schematic block diagram explaining the example of another internal structure of the position alignment process part which the image process part of 4th Embodiment has. It is a schematic flowchart explaining another example of the synthetic | combination processing procedure of the some image in 4th Embodiment.

− First embodiment −
FIG. 2 is a diagram schematically showing a configuration of the photographing apparatus 100 having the image processing unit according to the first embodiment of the present invention. In FIG. 2, the arrows attached to the lines connecting the components indicate the direction of data flow. The photographing apparatus 100 is configured to be able to perform a series of multiple exposure operations in response to a single photographing operation (an operation of pressing a release switch) by a user. Then, the image capturing apparatus 100 can perform alignment and composition processing on images for a plurality of frames generated by a plurality of exposure operations, and can generate an image with an expanded dynamic range, an image with reduced blurring, and the like. Configured. Here, “alignment” refers to a process of identifying pixels or areas having a correspondence relationship between a plurality of images having the same subject, and in this embodiment, for a certain area of the reference image. In other words, the process specifies a pixel or region having a corresponding relationship in a reference image.

  The imaging apparatus 100 includes an optical system 104, an imaging unit 105, an analog front end (hereinafter referred to as AFE) 106, a signal processing circuit 107, a storage unit 108, a CPU 109, and an image processing unit 300. The elements are electrically connected via the system bus 110.

  The storage unit 108 includes a RAM 108A and a ROM 108B. The ROM 108B is configured by a flash memory or the like. The ROM 108B stores shooting parameters, image processing parameters (these parameters will be described later), and an image processing program. The ROM 108B also stores image data obtained by photographing. Note that the storage unit 108 may include a storage medium such as a memory card or a hard disk drive that is detachably attached to the photographing apparatus 100. In that case, the image data can be stored in the ROM 108B or a storage medium.

  The RAM 108A is used as a work area. For example, the image processing program is read from the ROM 108B and copied to the RAM 108A, and the image processing program is sequentially read, interpreted and executed by the CPU 109. The image processing unit 300 can be configured by a dedicated hardware logic or an application specific integrated circuit (ASIC), but in the following, the CPU 109 interprets and executes the above-described image processing program so that the image processing unit 300 It will be described as being configured.

  Of the shooting parameters stored in the ROM 108B, those related to the focal length, shutter speed, aperture (F value), and the like are output to the optical system 104. The optical system 104 includes a variable focal length lens that forms a subject image on the light receiving surface of the imaging unit 105, and a shutter unit that can adjust a shutter speed and an aperture. The optical system 104 is configured to be able to adjust the focal length, shutter speed, aperture, etc. based on the parameters related to the focal length, shutter speed, aperture, etc. read from the ROM 108B.

  The imaging unit 105 is configured by an image sensor such as a CCD or a CMOS. The imaging unit 105 may have a so-called multi-plate configuration having a color separation optical system such as a dichroic prism and a plurality of image sensors, or on-chip color such as red, green, and blue on the light receiving surface. You may have what is called a single plate type structure which has an image sensor in which a filter is formed. In the following description, it is assumed that the imaging unit 105 has a single-plate configuration and the image sensor has a Bayer array on-chip color filter. The imaging unit 105 photoelectrically converts the subject image formed by the optical system 104 to generate an analog image signal.

  Among the shooting parameters stored in the ROM 108 </ b> B, those related to ISO sensitivity (amplifier gain) and the like are output to the AFE 106. The AFE 106 performs processing such as correlated double sampling, amplification, and A / D conversion on the analog image signal output from the imaging unit 105 to generate a digital image signal, and temporarily stores it in the RAM 108A. At this time, the AFE 106 amplifies the analog image signal output from the imaging unit 105 with a gain setting based on an amplifier gain parameter output from the ROM 108B.

  As described above, the signal processing circuit 107 performs noise removal, demosaicing, tone correction processing, and the like on the digital image signal output from the AFE 106 and stored in the RAM 108A to generate image data. The image data is stored in the storage unit 108 (RAM 108A or ROM 108B). The series of processes described above is performed every time the exposure operation is performed once. When a plurality of exposure operations are continuously performed, the above processing is performed for the number of times of exposure.

  The image processing unit 300 includes an alignment processing unit 309 and an image composition processing unit 310. Among the parameters stored in the ROM 108B, the image processing parameter is read by the alignment processing unit 309. The alignment processing unit 309 performs movement vector derivation processing on image data for a plurality of frames generated by performing a series of multiple exposure operations based on the image processing parameters as described in detail below. I do. The alignment processing unit 309 outputs information on the derived movement vector to the storage unit 108.

  The image composition processing unit 310 performs image composition processing based on the image data for a plurality of frames stored in the storage unit 108 and information on the movement vector, and generates a composite image. The generated composite image can be output to a display device, a printer, or the like. When the storage unit 108 includes a storage medium such as a memory card or a hard disk, the generated composite image can be stored in the storage medium.

  FIG. 3 shows an internal configuration of the alignment processing unit 309. FIG. 4 schematically shows two images to be combined by the image processing unit 300. 4A shows a standard image, and FIG. 4B shows a reference image, that is, an image to be superimposed on the standard image. The reference image is an image serving as a reference for alignment. A reference coordinate system is set for the reference image. The reference image is an image to be subjected to deformation processing so that an image element in the standard image and an image element in the reference image overlap each other.

  As shown in FIG. 4, a plurality of templates are provided on the reference image. The template is an image element that serves as a reference in the process of finding an image element that matches an image element in the reference image in the reference image. An alignment reference block is also provided on the reference image with the position where the template exists as a node.

  FIG. 4A shows an example in which one alignment reference block is provided corresponding to one rectangular area surrounded by four templates adjacent to each other. However, the alignment reference block may have a triangular shape surrounded by three templates, or may have an arbitrary polygonal shape. In addition, each alignment reference block may have the same shape or area, or may have a different shape or area. For example, the alignment reference block may be set smaller in the center of the screen where there is a high probability that there is an object of interest to the observer, and the alignment reference block may be set larger in the periphery of the screen. Similarly, it is also possible to set the alignment reference block smaller in a region with a higher spatial frequency in the image and to set a larger alignment reference block in a region with a lower spatial frequency.

  Further, in an application where two images are overlapped to obtain one large panoramic image, a template or an alignment reference block may be provided only in a portion where the images are overlapped.

  In the reference image shown in FIG. 4B, a search area is provided corresponding to each template provided on the standard image. The search area is an area that defines a search range when searching for an image element that matches each template provided on the reference image in the reference image.

  Referring to FIG. 3, alignment processing unit 309 includes movement vector measurement region setting unit 311, alignment reference region setting unit 312, movement vector derivation unit 313, reliability derivation unit 314, and movement vector correction processing. Unit 315 and a movement vector interpolation processing unit 316.

  The movement vector measurement region setting unit 311 is a template region or search region for performing a movement vector derivation process based on image processing parameters such as the image size read from the storage unit 108, the number of alignment templates, and the search range. Set the movement vector measurement area. That is, the movement vector measurement region setting unit 311 sets a plurality of templates on the standard image and sets a search region on the reference image.

  The alignment reference area setting unit 312 divides the reference image into a plurality of small areas based on image processing parameters such as the image size and the number of area divisions read from the storage unit 108 to obtain alignment reference blocks. . In the example shown in FIG. 4A, a rectangular region surrounded by four templates adjacent to each other on the reference image is one alignment reference block. In FIG. 4A, six alignment reference blocks are shown.

  The movement vector deriving unit 313 performs template matching processing between the reference image and the reference image in each of the movement vector setting regions set by the movement vector measurement region setting unit 311. Then, a movement vector corresponding to each movement vector measurement region is derived. The template matching process is a process of calculating a matching index while scanning a template in a search area and detecting a position where the matching index is the highest. Here, the positional deviation between the position with the highest matching index and the corresponding position of the template is calculated, and is output as a movement vector that defines the direction of positional deviation and the amount of positional deviation. As the coincidence index, a known technique such as SAD (Sum of Absolute Intensity Difference), SSD (Sum of Squared Intensity Difference), or NCC (Normalized Cross-Correlation) may be used.

  The reliability deriving unit 314 derives the reliability of each movement vector derived by the movement vector deriving unit 313. Here, an example of a pattern included in an image element in a template will be described with reference to FIG. When template matching is performed using the low contrast image illustrated in FIG. 5C or the repetitive pattern image illustrated in FIG. 5D as a template, there is a high possibility of deriving a movement vector different from the original one. Become.

  That is, when the image element in the template is such a low-contrast image or a repetitive pattern image, the reliability of the derived movement vector is low in any direction in the image. Here, the reliability of the derived movement vector can be defined as the probability or possibility that the derived movement vector matches the original direction and size. Alternatively, it may be defined as a probability or possibility that an accurate movement vector is derived.

  When template matching is performed using an image as shown in FIG. 5A, that is, an image having a clear pattern boundary in two directions, ideally, two directions orthogonal to each other, the corresponding position is unique. Can find out. That is, the reliability of the movement vector derived from the image element as shown in FIG. 5A is high in any two-axis direction.

  When template matching is performed using an image element as shown in FIG. 5B, that is, an image having a linear pattern boundary along a certain direction as a template, the amount of misalignment is correctly corrected in the direction orthogonal to the pattern boundary. Can be derived. However, there is no clue to alignment in the direction along the pattern boundary, and the amount of misalignment cannot be obtained correctly. Therefore, the reliability of the motion vector derived from the image element as shown in FIG. 5B is high only in one axial direction orthogonal to the pattern boundary and low in the direction parallel to the pattern boundary. As described above, an area in which a movement vector can be accurately obtained only in one direction is generally referred to as an “opening problem area”.

  The reliability deriving unit 314 can derive reliability by analyzing at least one of the image elements in the template of the standard image and the image elements in the search area of the reference image. In the following, for ease of understanding, the reliability deriving unit 314 is described as deriving reliability by analyzing image elements in the template.

  A processing flow of the reliability deriving unit 314 is shown in FIG. The processing shown in FIG. 6 is executed by the CPU 109 when an image processing program for executing the image processing unit 300 is executed by the CPU 109 and the reliability derivation processing is called in the image processing program.

  In S600, the CPU 109 analyzes the image of the currently selected template area and determines whether or not the area is a low contrast area. For example, the absolute value of the differential intensity is obtained for each pixel with respect to the image in the region. The sum of the absolute values of the differential intensities is compared with a threshold. If the sum of the absolute values of the differential intensities is higher than the threshold, the contrast is high, and if the sum of the absolute values of the differential intensities is less than or equal to the threshold, the contrast is low. It is possible to determine.

  If the determination in S600 is affirmative, the process proceeds to S602, whereas if the determination is negative, the process proceeds to S604. In step S <b> 604, the CPU 109 determines whether the image of the currently selected template area has a repeated pattern. Specifically, the autocorrelation is calculated for the image of the currently selected template region. Then, when the maximum position of the correlation value occurs periodically or when the maximum position varies, it is determined as a repetitive region.

  If the determination in S604 is affirmed, the process proceeds to S602, whereas if the determination is negative, the process proceeds to S606. In S602, which is a branch destination when the determinations in S600 and S604 are affirmed, the CPU 109 determines that the reliability of the motion vector derived from the currently selected template is low in both the two-axis directions, and is related to this determination. Attributes to be assigned to the template currently being processed.

  In S606, which is a branch destination when the determination in S604 is negative, the CPU 109 determines whether the image of the currently selected template area is an opening problem area. The opening problem area determination process performed in S606 will be described with reference to FIG. FIG. 7 is a flowchart for explaining the contents of the opening problem determination process performed in S606.

  In S700, the CPU 109 performs differentiation processing in the X direction, that is, the row direction of the two-dimensionally arranged pixels that form the image. In step S <b> 702, the CPU 109 performs differentiation processing in the Y direction, that is, in the column direction of the two-dimensionally arranged pixels. The differential processing here is processing of a first-order differential system, and may be, for example, a Sobel filter or a Prewit filter.

  In S704, the CPU 109 obtains the edge gradient direction θij based on the X-direction differential value dxij and the Y-direction differential value dyij obtained in S700 and S702. Here, i and j represent the position of the pixel in the row direction and column direction. The edge gradient direction θij can be obtained by the following equation.

θij = tan ⁻¹ (dyij / dxij) (1)

  In S706, the CPU 109 determines whether or not the processing of S700 to S704 has been performed for all the pixels in the image of the currently selected template region, and the process returns to S700 while this determination is negative. On the other hand, if the determination in S706 is affirmative, the process proceeds to S708.

  In step S <b> 708, the CPU 109 performs statistical processing on θij obtained as described above, and obtains dispersion of θij (variation of distribution of θij). Then, it is determined whether or not the variance of θij exceeds a predetermined threshold value.

  If the determination in S708 is affirmative, the process proceeds to S710, the determination result that “reliability is high in the two-axis direction” is returned, the process in FIG. 7 is terminated, and the process returns. On the other hand, if the determination result in S708 is negative, the process proceeds to S712, and returns the determination result of “open problem area” and information on the direction θ in which high reliability is obtained in the derived motion vector. After the process of FIG.

  Here, the direction θ in which high reliability is obtained will be described with reference to FIG. High alignment accuracy can be expected in the image pattern as exemplified in FIG. 8 in the direction orthogonal to the direction in which the boundary of the pattern extends. By using the angle θ representing the direction, the angle θ in the direction in which the above-described high reliability can be obtained can be defined. This angle θ can be obtained as follows, for example. That is, θij is obtained based on the above formula (1) for all the pixels in the region from which reliability is derived. It is possible to statistically process these θij and set the mode value to θ. Alternatively, the average value or median value of these θij can be set to θ.

  When the process of S710 is performed, the determination in S606 of FIG. 6 is denied and the process branches to S610. When the process in S712 is performed, the determination in S606 in FIG. 6 is affirmed, and the process branches to S608.

  In step S <b> 608, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is high only in one axis direction, and that the direction having high reliability is θ, and currently selects an attribute related to this determination. It is given to the template that has been made.

  In S610, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is high in both the biaxial directions, and assigns an attribute related to this determination to the currently selected template.

  In step S612, the CPU 109 determines whether or not analysis for deriving reliability has been completed for all templates in the reference image. While this determination is negative, the processing from S600 to S610 is repeated. If the determination in S612 is affirmed, the CPU 109 returns.

  The movement vector correction processing unit 315 performs a process of correcting the movement vector as necessary based on the movement vector derived by the movement vector deriving unit 313 and the reliability derived by the reliability deriving unit 314. Here, “correction of a movement vector” means that the movement vector is replaced with another movement vector, a movement vector derived using another movement vector, or the movement vector and another movement. This includes replacement with a movement vector derived using a vector, replacement with a movement vector derived using only the movement vector, and the like.

  The content of processing performed by the movement vector correction processing unit 315 will be described with reference to FIG. This process is a process of correcting the movement vector based on the movement vector derived by the movement vector deriving unit 313 and the reliability derived by the reliability deriving unit 314.

  FIG. 9A and FIG. 9B respectively show four adjacent templates (template 1, template 2, template 3, template 4). In each template, an image pattern and movement vectors MV1, MV2, MV3, and MV4 obtained corresponding to the template are shown. FIG. 9A shows a state before the correction process, and FIG. 9B shows a state after the correction process.

  Below, the example which correct | amends as needed with respect to the movement vector of only the template 1, the template 2, the template 3, and the template 4 is demonstrated. However, in practice, correction is added to the movement vectors of all templates in the reference image as necessary.

  The template 1 and the template 4 are shown as an example having an image element in which the reliability derived by the reliability deriving unit 314 is high in both biaxial directions. The template 2 shows an example having image elements whose reliability derived by the reliability deriving unit 314 is low in both biaxial directions. The template 3 shows an example having an opening problem area, that is, an image element in which the reliability derived by the reliability deriving unit 314 is high only in one axis direction.

  In Template 1 and Template 4, which have high reliability in both the two-axis directions, there is no change before and after the movement vector correction process, and they remain MV1 and MV4, respectively. In template 2 and template 3, correction is applied as described below.

  The movement vector obtained by the movement vector deriving unit 313 using the template 2 having low reliability in both the two-axis directions is MV2 as shown in FIG. The reliability of the movement vector MV2 is low in both biaxial directions. In this case, a movement vector of a template having high reliability in both axes among the templates existing around the template 2 is set as a corrected movement vector in the template 2.

  For example, as shown in FIG. 9B, the movement vector MV1 of the template 1 that is located next to the template 2 and has high reliability in both two-axis directions is set as a corrected movement vector corresponding to the template 2. By correcting the movement vector in this way, it can be expected to obtain a more preferable alignment result.

  Next, correction of the movement vector in the template 3 whose reliability is high only in one axis direction and whose high reliability direction is θ will be described. As described above, a highly reliable movement amount can be obtained in the high reliability direction, but a highly reliable movement amount cannot be obtained in the direction orthogonal to the high reliability direction. Therefore, the movement vector MV3 before the correction processing is changed from the movement vector MV3x ′ of the axial component along the high reliability direction and the movement of the axial component along the direction orthogonal to the high reliability direction as shown in FIG. It decomposes into the vector MV3y ′. In the present specification, a direction orthogonal to the high reliability direction is referred to as a low reliability direction.

  The axial component movement vector MV3x ′ along the high reliability direction and the axial component movement vector MV3y ′ along the low reliability direction can be obtained as follows. In the xy coordinate system defined by the row direction (x-axis direction) and the column direction (y-axis direction) of the array of pixels constituting the image element of the template, the axis components along the x-axis direction of the movement vector MV3 are represented by MV3x, y The axial component along the axial direction is MV3y.

  The axis component movement vector MV3x along the x-axis direction and the axis component movement vector MV3y along the y-axis direction are rotated by the angle θ, and the high-reliability direction (x′-axis direction) and low reliability are obtained. The movement vectors MV3x ′ and MV3y ′ of the axis components along the two-axis direction in the x′y ′ coordinate system defined by the direction (y′-axis direction) are obtained by coordinate rotation processing. The following equation (2) can be used for this coordinate rotation process. That is, x ′ and y ′ obtained as x = MV3x and y = MV3y in Equation (2) become MV3x ′ and MV3y ′.

  Of the biaxial component movement vectors obtained by the equation (2), high reliability cannot be expected for the axial component vector MV3y ′ along the low reliability direction. In this case, a template having sufficient reliability with respect to the low reliability direction of the template 3 is searched from templates existing in the vicinity of the template 3. In the example of FIG. 9, the template 4 corresponds to such a template.

  Note that it is possible to determine whether a neighboring template has sufficient reliability with respect to the low reliability direction of the template 3 as follows. That is, the high reliability direction of the template (template 4 in this example) located in the vicinity of the template to be corrected (template 3) is, for example, 45 degrees or more with respect to the low reliability direction of the template to be corrected, preferably 60. Whether or not the angle difference is equal to or greater than the degree can be used as a criterion. Alternatively, a template having high reliability in both two-axis directions may be searched from templates existing in the vicinity of the template to be corrected.

  Next, as shown in FIG. 9A, the above-described equation (2) is also used for the movement vector MV4, and the vector of the axis component of the movement vector MV4 along the low reliability direction of the template 3 is obtained. The MV4y ′ in FIG. 9A is thus obtained.

  Subsequently, as shown in FIG. 9B, the vector MV3y ′ of the axial component along the low reliability direction of the movement vector MV3 of the template 3 is replaced with MV4y ′ obtained as described above. Hereinafter, a description will be given with reference to FIG. 10 in which the movement vector of the template 3 shown in FIG. As a result of correcting the movement vector as described above, the vector of the axial component along the high reliability direction corresponding to the movement vector MV3 remains MV3x ′, and the vector in the axial direction along the low reliability direction is changed from MV3y ′. It is corrected to MV4y ′. A combination of these two axial component vectors MV3x 'and MV4y' is a corrected movement vector MV3new.

  A coordinate conversion process is performed on the above-described two axial component vectors MV3x ′ and MV4y ′ using Equation (3), and the corrected movement vector MV3new is converted into axial component vectors MV3xnew and MV3new along the xy coordinate axes. Ask. That is, x and y obtained as x ′ = MV3x ′ and y ′ = MV4y ′ in the expression (3) become the vector MV3xnew and MV3new of axial components along the xy coordinate axis of MV3new.

  The example in which the axis component MV3y ′ along the low reliability direction of the movement vector 3 is replaced with the axis component MV4y ′ of the movement vector MV4 of the template 4 located in the vicinity of the template 3 has been described above. However, the present invention is not limited to this, and in the above example, MV3y ′ and MV4y ′ may be arithmetically averaged or geometrically averaged to obtain the corrected axis component vector. At this time, not only a simple average but also a weighted average or the like may be performed to obtain a corrected axis component vector.

  Further, when there are a plurality of templates having relatively high reliability of the axial component along the low reliability direction of the template 3 in the vicinity of the template 3, the movement vector of the template 3 is based on the movement vector corresponding to each of the plurality of templates. Corrections may be made to.

  Referring to FIG. 3 again, the movement vector interpolation processing unit 316 derives a movement vector for each pixel based on the movement vector (corrected as necessary) obtained for each search area in the reference image. To do. That is, based on the movement vectors derived for each template located at the four corners of one alignment reference block, the movement vectors corresponding to all the pixels existing in the alignment reference block are obtained. To derive.

  At this time, an interpolation process is performed according to the distance between the position of the pixel for which the movement vector is to be obtained and the center position of the four templates, and a movement vector corresponding to each pixel is obtained. The above processing is performed for all alignment reference blocks. In this way, a movement vector is obtained corresponding to each pixel constituting the reference image.

  Details of the above processing in the movement vector interpolation processing unit 316 will be described with reference to FIG. FIG. 11A shows a standard image, and FIG. 11B shows a reference image. The movement vector after the correction processing is performed as necessary in the movement vector correction processing unit 315 is shown superimposed on the corresponding template in FIG.

  For the movement vector shown in FIG. 11A, the absolute value is increased (the length of the directed line segment is increased) for easy viewing. In the following description, it is assumed that the pixel position of the pixel located at the upper left corner of the image is (0, 0), and the coordinate value increases in the right direction and the downward direction. For example, the template image element located in the upper left corner on the reference image in FIG. 11A is composed of 100 pixels vertically and horizontally, and is located from the pixel position (101, 101) to the pixel position (200, 200). To do.

  As a result of the template matching process, it is assumed that the image element matching the template is located from the pixel position (151, 151) to the pixel position (250, 250) in the reference image. In this case, the derived movement vector is (−50, −50). The movement vector corresponding to this is directed to the upper left direction as shown on the template in the upper left corner of FIG.

  As is clear from the above, the movement vector shifts the image element in the template corresponding area before deformation by what distance (number of pixels), that is, if the deformation is deformed, it overlaps the image element in the template on the reference image. Is shown. The movement vector is derived corresponding to each template. In the reference image of FIG. 11B, the template corresponding region in the reference image before the deformation is indicated by a white rectangle. Then, in accordance with the movement vector shown in FIG. 11A, how to move each template corresponding region is indicated by a hatched rectangle.

  By the way, in the interpolation processing described below, it is considered that the above-described movement vector is given to one pixel located at the center (center of gravity or centroid) of a region defining each template. The movement vector for pixels other than one pixel located at the center (center of gravity or centroid) of the region defining each template can be obtained by known interpolation processing or extrapolation processing. In the present specification, interpolation processing and extrapolation processing are generically referred to simply as interpolation processing.

  The reference image is deformed so as to shift the pixel position in the direction and distance defined by the movement vector for each pixel, which is derived by performing interpolation processing corresponding to each pixel in the reference image, and the image after deformation By superimposing with the reference image, it is possible to obtain a sharp image in which deviation is suppressed.

  In FIG. 11, a rectangular alignment reference block is formed by cutting one mesh with four neighboring templates as nodes, but the shape of the alignment reference block is arbitrary as described above. It can be a polygon. In addition, each alignment reference block may have the same size and the same shape, or may have different sizes and shapes.

  The image composition processing unit 310 illustrated in FIG. 2 deforms the reference image based on the movement vector for each pixel derived by the movement vector interpolation processing unit 316. As a result, the alignment reference block surrounded by the four template regions before deformation is deformed as shown in FIG. The image composition processing unit 310 performs image composition by superimposing the reference image modified in this way on the standard image. Referring to FIG. 11B, the deformed reference image may seem to be deviated from the base image, but this is not the case. As a result of deforming the reference image shown in FIG. 11B, the deformed reference image becomes the same as the standard image shown in FIG. As a result, the images can be superimposed with almost no deviation.

  An image processing program for performing the above-described processing for combining a plurality of images will be described with reference to FIG. FIG. 12 is a schematic flowchart for explaining an image processing procedure for combining a plurality of images, which is executed by the CPU 109. The program for executing this image processing procedure is stored in the ROM 108B as described above. Alternatively, this program may be stored in a computer-readable storage medium such as a memory card, magnetic disk, optical disk, or magneto-optical disk. It is also possible to download the program through a communication line such as a network and store it in the ROM 108B.

  When the program is stored in a computer-readable storage medium such as a magnetic disk, an optical disk, or a magneto-optical disk, the photographing apparatus 100 has a drive device for reading information from these storage media. Alternatively, it may be configured to be connectable to an external drive device. When the program is stored in a memory card or the like, the photographing apparatus 100 may be provided with an interface or the like for reading information from the memory card or the like.

  In S1200, the CPU 109 performs a movement vector measurement region setting process. This process corresponds to the process performed by the movement vector measurement region setting unit 311 described with reference to FIG. In step S1202, the CPU 109 performs alignment reference area setting processing. This processing corresponds to the processing in the alignment reference area setting unit 312 described with reference to FIG.

  In step S1204, the CPU 109 performs a movement vector derivation process. This process corresponds to the process in the movement vector deriving unit 313 described with reference to FIG. In step S1206, the CPU 109 performs reliability derivation processing. This process corresponds to the process in the reliability deriving unit 314 described with reference to FIG.

  In step S1208, the CPU 109 performs a movement vector correction process. This processing corresponds to the processing in the movement vector correction processing unit 315 described with reference to FIG. In step S1210, the CPU 109 performs movement vector interpolation processing. This process corresponds to the process in the movement vector interpolation processing unit 316 described with reference to FIG.

  In step S <b> 1212, the CPU 109 performs image composition processing. This processing corresponds to the processing in the image composition processing unit 310. After completing the process of S1212, the CPU 109 completes the process of transforming the reference image and superimposing it on the standard image. At this time, when three or more images are to be superimposed, the number of images to be superimposed (two for three images, three for four images,...) In FIG. The process is repeated.

  During the processing of the flowchart shown in FIG. 12, the order of the processing may be reversed for the movement vector derivation processing in S1204 and the reliability derivation processing in S1206. That is, the reliability derivation process may be performed in S1204 and the movement vector derivation process may be performed in S1206.

  Further, the processing of S1212 can be omitted. That is, it is possible to omit the image composition processing unit 310 in the imaging apparatus 100 of FIG.

  In this case, the image data is stored in the storage unit 108 or a memory card together with information on the movement vector for each pixel. Here, a combination of image data and movement vector information for each pixel is referred to as a data set for convenience of explanation. The image composition processing unit 310 can be included in a personal computer, a storage viewer, or the like. The data set is transferred to a personal computer, a storage viewer, or the like via a wired or wireless communication means, or via a memory card. Then, the image composition processing unit 310 in the personal computer, the storage viewer, or the like can perform image composition processing based on the data set to generate a composite image.

  In the above-described processing for combining a plurality of images, a movement vector is obtained for each alignment reference block obtained by dividing the image into a plurality of small regions. A favorable result can be obtained even if has moved. For example, even when the background car is moving between multiple images or the arm of a person who is the subject is moving, the image can be locally deformed, so the accuracy of image composition can be improved. It becomes possible.

  As described above, according to the first embodiment of the present invention, it is highly reliable for a motion vector that is “highly reliable only in one axis direction” that has not been sufficiently utilized in the past. It is possible to effectively use the vector of the axial component along the direction. At this time, with respect to the vector of the axis component along the low reliability direction, it is possible to stabilize the image alignment processing by using the information of the movement vector obtained from the neighboring template.

− Second Embodiment −
In 1st Embodiment, the example using the reliability of the two components decomposed | disassembled along arbitrary two axial directions was demonstrated. That is, the reliability is “high reliability in both biaxial directions” shown in FIG. 5A, “high reliability only in one axial direction” shown in FIG. 5B, and FIG. 5C. A description has been given of processing performed by classifying into three patterns “low reliability in both two-axis directions” shown in FIG. When the pattern boundary extends obliquely with respect to the direction of the arrangement of the pixels constituting the image as shown in FIG. 5B, the reliability is high in the direction orthogonal to the pattern boundary. An example of classification has been described.

  On the other hand, in the second embodiment, an example in which the reliability along the column direction and the reliability along the row direction of the arrangement of the pixels constituting the image will be described. Hereinafter, the row direction of the arrangement of pixels constituting the image is referred to as an X-axis direction, and the column direction is referred to as a Y-axis direction. Note that the imaging apparatus having the image processing unit according to the second embodiment has the same configuration as that described with reference to FIGS. 2 and 3 in the first embodiment, and thus illustration and description thereof are omitted. To do.

  The difference of the second embodiment from the first embodiment is only the processing contents in the reliability deriving unit 314 and the movement vector correction processing unit 315. Hereinafter, these differences will be mainly described.

  In the second embodiment, the reliability deriving unit 314 (FIG. 3) performs processing by classifying into four patterns as described below with reference to (a) to (e) of FIG. An image element as shown in FIG. 13A has a clear pattern boundary in both the X-axis direction and the Y-axis direction. When template matching is performed using such image elements, it is possible to derive a highly reliable movement vector in both the X-axis direction and the Y-axis direction. That is, it is determined that the reliability of the movement vector derived from the image element as shown in FIG. 13A is high in both the X-axis direction and the Y-axis direction.

  The image element illustrated in FIG. 13B has a clear pattern boundary in the Y-axis direction. When template matching is performed using such image elements, it is possible to derive a highly reliable movement vector in the X-axis direction. However, there is no clue for alignment in the Y-axis direction, and the amount of positional deviation cannot be obtained correctly. That is, it is determined that the reliability of the movement vector derived from the image element as shown in FIG. 13B is high only in the X-axis direction.

  The image element illustrated in FIG. 13C has a clear pattern boundary in the X-axis direction. When template matching is performed using such image elements, it is possible to derive a highly reliable movement vector in the Y-axis direction. However, there is no clue for alignment in the X-axis direction, and the amount of positional deviation cannot be obtained correctly. That is, it is determined that the reliability of the movement vector derived from the image element as shown in FIG. 13C is high only in the Y-axis direction.

  When template matching is performed using the low contrast image illustrated in FIG. 13D or the repetitive pattern image illustrated in FIG. 13E as a template, a movement vector different from the original one may be derived. Therefore, when the image element in the template is such a low-contrast image or a repetitive pattern image, it is determined that the reliability of the derived movement vector is low in both the X-axis direction and the Y-axis direction. Is done.

  In addition to those shown in FIG. 13, the pattern boundaries of the image elements in the template may extend obliquely with respect to the X axis direction or the Y axis direction. For these, any of the above-described reliability is given to each template by the process of the reliability deriving unit 314 described below with reference to FIG. That is, if the extending direction of the pattern boundary is angularly close to the X-axis direction, it is determined that the reliability is high with respect to the Y-axis direction. Similarly, if the extending direction of the pattern boundary is angularly close to the Y-axis direction, it is determined that the reliability is high with respect to the X-axis direction.

  Also in the second embodiment, the image processing unit 300 can be configured by dedicated hardware logic or an application specific integrated circuit (ASIC). However, in the following, the CPU 109 executes the above-described image processing program. Description will be made assuming that the image processing unit 300 is configured by interpretation and execution.

  The processing flow of the reliability deriving unit 314 is shown in FIG. The processing shown in FIG. 14 is executed by the CPU 109 when an image processing program for executing the image processing unit 300 is executed by the CPU 109 and the reliability derivation processing is called in the image processing program. When the process of FIG. 14 is executed once, reliability is derived corresponding to all templates in one reference image.

  The CPU 109 performs differentiation processing in the X axis direction in S1400, and performs differentiation processing in the Y axis direction in S1402. In the above-described X-axis direction differentiation process and Y-axis direction differentiation process, the absolute value of the differential value is derived. In S1404, the CPU 109 determines whether or not the differential processing of all the images in the region has been completed, and repeatedly performs the processing from S1400 to S1404 while this determination is denied.

  If the determination in S1404 is affirmed, the CPU 109 determines in S1406 whether or not the sum of the differential values in the X-axis direction exceeds a predetermined threshold value. If the determination in S1406 is affirmed, the process proceeds to S1408, whereas if the determination is negative, the process proceeds to S1414.

  In step S1408, the CPU 109 determines whether or not the sum of the differential values in the Y-axis direction exceeds a predetermined threshold value. If the determination in S1408 is affirmed, the process proceeds to S1410. If the determination is negative, the process proceeds to S1412. In S1414, which is a branch destination when the determination in S1406 is negative, the CPU 109 determines whether or not the sum of the differential values in the Y-axis direction exceeds a predetermined threshold value. If the determination in S1414 is affirmed, the process proceeds to S1416, whereas if the determination is negative, the process proceeds to S1418.

  Here, the threshold when the determination process is performed in S1406, S1408, and S1414 will be described. The threshold value is set so that when the sum of the differential values in the X-axis direction or the Y-axis direction exceeds these threshold values, it can be determined that the image element in the analysis target region has sufficient contrast, that is, has high reliability. Predetermined. That is, in S1406, it is determined whether or not the reliability in the X-axis direction is high, and in S1408 and S1414, it is determined whether or not the reliability in the Y-axis direction is high.

  The method for determining whether or not the pattern is a repetitive pattern has been described in the first embodiment and will not be described. However, the sum of the differential values in the X-axis direction and the Y-axis direction exceeds a predetermined threshold. It is desirable to determine a repetitive pattern when determining whether or not it exists. In that case, the determination of S1406, S1408, and S1414 may be denied when a repeated pattern is detected in at least one of the X-axis direction and the Y-axis direction.

  Depending on the combination of determination results in S1406, S1408, and S1414, the process branches to one of S1410, S1412, S1416, and S1418 described below.

  The process of S1412 is a process performed when it is determined in S1406 that the reliability in the axial direction along the X axis is high, and in S1408, the reliability in the axial direction along the Y axis is determined to be low. In step S1412, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is an opening problem region having high reliability only in the axial direction along the X axis. Give to the template.

  The process of S1410 is a process performed when it is determined that the reliability in the axial direction along the X axis is high in S1406, and the reliability in the axial direction along the Y axis is determined in S1408. In step S1410, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is high in the two axial directions, the axial direction along the X axis and the axial direction along the Y axis. Assign related attributes to the template.

  The process of S1416 is a process performed when it is determined in S1406 that the reliability in the axial direction along the X axis is low, and in S1414, the reliability in the axial direction along the Y axis is determined to be high. In step S1416, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is an opening problem region having high reliability only in the axial direction along the Y axis, and sets the attribute related to this determination as the attribute. Give to the template.

  The process of S1418 is a process performed when it is determined in S1406 that the reliability in the axial direction along the X axis is low, and in S1414, the reliability in the axial direction along the Y axis is determined to be low. In step S1418, the CPU 109 determines that the reliability of the movement vector derived from the currently selected template is low in both the axial direction along the X axis and the axial direction along the Y axis. Give to that template.

  When any one of the processes of S1412, S1410, S1416, and S1418 described above is completed, the process proceeds to S1420. In step S1420, the CPU 109 determines whether or not the assignment of reliability to all the templates has been completed, and repeats the above-described processing from S1400 to S1418 while the determination is negative. If the determination in S1420 is affirmed, the CPU 109 returns.

  The processing content performed by the movement vector correction processing unit 315 will be described with reference to FIG. The processing shown in FIG. 15 is similar to that described with reference to FIG. 9 in the first embodiment. However, the process shown in FIG. 15 is simplified as compared with the process in the first embodiment, as will be apparent from the following description.

  The CPU 109 corrects the movement vector based on the movement vector derived by the movement vector deriving unit 313 and the reliability derived by the reliability deriving unit 314. FIG. 15A and FIG. 15B show templates of four adjacent regions (template 1, template 2, template 3, template 4), respectively.

  In each template, an image pattern and movement vectors MV1, MV2, MV3, and MV4 obtained corresponding to the template are shown. FIG. 15A shows a state before the correction process, and FIG. 15B shows a state after the correction process. Below, the example which correct | amends only the movement vector of the template 1, the template 2, the template 3, and the template 4 as needed is demonstrated. However, in practice, correction is applied to the movement vectors of all templates in the reference image as necessary.

  Examples of the template 1 and the template 4 include an image element having a pattern in which the reliability derived by the reliability deriving unit 314 is high in the two axial directions along the X axis and the Y axis. Has been. The template 2 shows an example having image elements having a pattern in which the reliability derived by the reliability deriving unit 314 is low in both biaxial directions. The template 3 has an opening problem area, more specifically, an example having an image element having a pattern in which the reliability derived by the reliability deriving unit 314 is high only in the axial direction along the X axis.

  In Template 1 and Template 4, which have high reliability in both the axial direction along the X axis and the axial direction along the Y axis, there is no change before and after the movement vector correction processing, and they remain MV1 and MV4, respectively. In template 2 and template 3, correction is applied as described below.

  The reliability of the movement vector MV2 obtained by the movement vector deriving unit 313 using the template 2 having low reliability in both the biaxial directions is low in both the biaxial directions. In this case, a template movement vector having high reliability in the axial direction along the X axis and the axial direction along the Y axis among the templates existing around the template 2 is set as a corrected movement vector. For example, as shown in FIG. 15B, the movement vector MV1 of the template 1 that is located next to the template 2 and has high reliability in both the axial direction along the X axis and the axial direction along the Y axis corresponds to the template 2. It is set as the corrected movement vector. By correcting the movement vector in this way, it can be expected to obtain a more preferable alignment result.

  Next, correction of the movement vector in the template 3 having high reliability only in the axial direction along the X axis will be described. The CPU 109 decomposes the movement vector MV3 before the correction processing into an axis component movement vector MV3x along the X axis and an axis component movement vector MV3y along the Y axis as shown in FIG.

  Since the template 3 has high reliability only in the axial direction along the X axis, no correction is performed on the movement vector MV3x of the axial component along the X axis. On the other hand, since the reliability of the axial component along the Y axis is low, sufficient reliability with respect to the low reliability direction of the template 3 (in this example, the axial direction along the Y axis) from among the templates existing near the template 3 Search for templates that have sex. In the example of FIG. 15, the template 4 corresponds to such a template. 15B, the axis component vector MV3y along the Y axis of the movement vector MV3 of the template 3 is replaced with the axis component vector MV4y along the Y axis direction of the template 4. As a result, the movement vector of template 3 is corrected from MV3 to MV3new.

  The example in which the axis component MV3y along the Y axis of the movement vector 3 is replaced with the axis component MV4y along the Y axis of the movement vector MV4 corresponding to the template 4 located in the vicinity of the template 3 has been described above. However, the present invention is not limited to this, and the corrected vector component may be obtained by arithmetically averaging or geometrically averaging MV3y and MV4y in the above example. At this time, not only a simple average but also a weighted average or the like may be performed to obtain a corrected vector component.

  In addition, when there are a plurality of templates having relatively high reliability of the axial component along the Y axis in the vicinity of the template 3, the movement vector of the template 3 is corrected based on the movement vector obtained from each of the plurality of templates. May be.

  Although not shown in FIGS. 15 (a) and 15 (b), a movement vector corresponding to a template in which only the reliability of the axial component along the Y axis is high is also applied by the same method as described above. Corrections can be made.

  Note that an image processing program for performing a series of processes for combining a plurality of images is the same as that described in the first embodiment with reference to FIG.

  As described above, in the second embodiment, as described in the first embodiment, the movement vector is converted into an axial component (in a highly reliable direction) along the edge gradient direction of the image element in the template. The axial component) is not decomposed into the axial component along the direction perpendicular to the edge gradient direction (the axial component along the low reliability direction). Instead, the movement vector is decomposed into an X-axis direction component and a Y-axis direction component, and correction processing is performed according to the level of reliability in the X-axis direction and the Y-axis direction. By doing so, it is possible to omit the arc tangent calculation and the coordinate conversion processing performed in the first embodiment, thereby reducing the calculation cost.

  Also in the second embodiment, the process of S1212 can be omitted in the flowchart shown in FIG. That is, it is possible to omit the image composition processing unit 310 in the imaging apparatus 100 of FIG. As in the first embodiment, a personal computer, a storage viewer, or the like can include the image composition processing unit 310.

  Note that the pixel arrangement of the imaging unit 105 may be a staggered arrangement or an arrangement in which non-rectangular polygons are tiled. In that case, the outline shape of the template may be a rectangle, and the X-axis direction and the Y-axis direction may be directions along the horizontal and vertical sides of the rectangular shape. Or it is good also as a direction along the standard image which has a rectangular outline shape, the horizontal side of a reference image, and a vertical side. In short, the X-axis direction and the Y-axis direction may be determined so that the arc tangent calculation during processing, the matrix calculation for coordinate conversion, and the like described in the first embodiment can be omitted.

− Third embodiment −
In the first and second embodiments, the image processing unit 300 includes the alignment processing unit 309 and the image composition processing unit 310, and the image processing unit 300 is configured to be able to generate a composite image. In the first and second embodiments, the alignment processing unit 309 includes a movement vector interpolation processing unit 316 and is configured to output a movement vector for each pixel to the storage unit 108. .

  On the other hand, in the third embodiment, as shown in FIG. 16, the image processing unit 300 </ b> A does not have an image composition processing unit 310 inside. Further, the alignment processing unit 309A does not have the movement vector interpolation processing unit 316 as shown in FIG. The image processing unit 300 </ b> A outputs the corrected movement vector processed by the movement vector correction processing unit 315 (FIG. 17) instead of the movement vector for each pixel to the storage unit 108.

  Note that among the constituent elements of the photographing apparatus 100A shown in FIG. 16, the same constituent elements as those of the photographing apparatus 100 shown in FIG. Similarly, among the components included in the alignment processing unit 309A illustrated in FIG. 17, the same components as those included in the alignment processing unit 309 illustrated in FIG. .

  The external processing device 350 can be a personal computer, a storage viewer, a digital photo frame, or the like. The external processing device 350 includes a movement vector interpolation processing unit 316 and an image composition processing unit 310. Image data is transferred from the imaging apparatus 100 </ b> A to the external processing apparatus 350. Wired or wireless communication means can be used for transferring image data. Alternatively, image data may be stored in a memory card attached to the photographing apparatus 100A, and the image data transferred by attaching the memory card to the external processing device 350. This image data includes the corrected reference vector and the information of the alignment reference block.

  FIG. 17 shows an internal configuration of the alignment processing unit 309A. The difference from the alignment processing unit 309 in the first and second embodiments shown in FIG. 3 is that the movement vector interpolation processing unit 316 in FIG. Then, it is a point to be omitted.

  The imaging apparatus 100A and the external processing apparatus 350 have the configuration described above. The imaging apparatus 100A performs correction of the movement vector, and the subsequent processing is performed by the external processing apparatus 350. The composite image generated by the image composition processing unit 310 is output to an image output device such as a display device or a printer.

  Also in the third embodiment, the image processing unit 300A can be configured by a dedicated hardware logic or an application specific integrated circuit (ASIC), but below, the CPU 109 interprets the above image processing program, In the following description, it is assumed that the image processing unit 300A is configured by execution.

  In the image processing unit 300A, processing corresponding to the processing from S1200 to S1208 is performed among the processing in the flowchart shown in FIG. Then, processing corresponding to the processing of S1210 and S1212 is performed by the external processing device 350.

  As described above, since the movement vector interpolation process is not performed in the image processing unit 300A, the processing load on the image processing unit 300A can be reduced. In addition, the storage capacity of the RAM 108A can be saved.

  As in the first and second embodiments, the movement vector derivation process in S1204 shown in FIG. 12 and the reliability derivation process in S1206 are performed in the reverse order shown in FIG. May be.

-Fourth embodiment-
In the first to third embodiments, the example in which the reliability is derived corresponding to each template in the standard image every time the alignment process for superimposing the reference image on the standard image is described.

  For example, when performing the process of aligning and overlaying three images, the following processes are performed as an example in the first to third embodiments. That is, when the first image is a standard image and the second image is a reference image, reliability is derived based on the first image. Then, based on the first and second images, movement vector correction and interpolation processing are performed, and alignment and composition processing are performed. Next, reliability is newly derived based on the synthesized image. Then, based on the synthesized image and the third image, correction of the movement vector and interpolation processing are performed, and alignment and synthesis processing are performed.

  On the other hand, in the fourth embodiment, the case where the process of aligning and superimposing three images as described above will be described as an example. First, any one of the three images is selected. Reliability is derived based on a single image.

  This image may be any of the first to third images. When image processing is performed immediately after the start of imaging, it is desirable to derive the reliability from the first image in order to reduce the processing time.

  Alternatively, when image processing is performed after a series of shootings is completed, the second image, that is, a plurality of images obtained by a series of shootings, is obtained in the vicinity of the timing between the shooting start timing and the shooting completion timing. An image is desirable. The reason is that the second image is taken at substantially equal intervals in time with respect to the first and third images. Therefore, the difference between the first image and the second image, and the difference between the second image and the third image are larger than the difference between the first image and the third image. This is because it can be expected that there is little, that is, a high degree of matching of images.

  Then, the reliability derived based on one specific image as described above is commonly used for the movement vector correction process performed several times thereafter, thereby reducing the processing load related to the reliability derivation. It becomes possible to do. This effect becomes more prominent as the number of superimposed images increases to 4, 5, and so on.

  FIG. 18 is a block diagram illustrating a configuration of an imaging device 100B having an image processing unit 300B according to the fourth embodiment. FIG. 19 is a block diagram illustrating a configuration of the alignment processing unit 309B. Among the constituent elements of the photographing apparatus 100A shown in FIG. 18, the same constituent elements as those of the photographing apparatus 100 shown in FIG. Similarly, among the components of the alignment processing unit 309B shown in FIG. 19, the same components as those of the alignment processing unit 309 shown in FIG. .

  Referring to FIG. 18, the image processing unit 300B includes an alignment processing unit 309B and an image composition processing unit 310. The image processing unit 300B can be configured by a dedicated hardware logic or an application specific integrated circuit (ASIC). In the following, the image processing unit 300B is configured by the CPU 109 interpreting and executing the image processing program. It will be described as a thing.

  Referring to FIG. 19, the reliability deriving unit 314A in the alignment processing unit 309B performs the same processing as the reliability deriving unit 314 in the first or second embodiment, and a plurality of units provided in the image. Reliability is derived for each of the templates. At this time, the reliability deriving unit 314A is different from the other embodiments in that the reliability deriving unit 314A derives the reliability corresponding to one predetermined image as described above. Then, when a series of processing for alignment is performed a plurality of times thereafter, the reliability derived by the reliability deriving unit 314A is used in common.

  From the alignment processing unit 309B, a movement vector for each pixel is output corresponding to each of a plurality of reference images. The image composition processing unit 310 performs image composition processing based on the image data and the movement vector for each pixel, and generates a composite image.

  An image processing program for performing a series of processes for image composition is shown in FIGS. 20 and 21 are flowcharts schematically showing the procedure of the reliability deriving process and the process of synthesizing a plurality of images executed by the CPU 109.

  The reliability derivation processing procedure will be described with reference to the flowchart of FIG. The process in FIG. 20 is a process performed on a predetermined image as described above. In S1200, the CPU 109 performs a movement vector measurement region setting process. This process is the same as the process in S1200 of FIG. In S2000, the CPU 109 performs reliability derivation processing. The reliability derived by this processing may be decomposed into two axial components along the high reliability direction and the low reliability direction described in the first embodiment. It may be decomposed into two axial components along the X-axis direction and the Y-axis direction described in the second embodiment. After completing the above processing, the CPU 109 completes the reliability derivation processing.

  The image composition process will be described with reference to the flowchart of FIG. In the flowchart shown in FIG. 21, the same processes as those shown in FIG. 12 are denoted by the same reference numerals as those assigned to the respective processing steps in FIG. 12, and detailed description thereof is omitted. The difference from the processing in FIG. 12 will be mainly described.

  In the process of FIG. 21, there is no process of S1206 in the process of FIG. Instead, for example, when the image composition processing is performed three times, the processing in FIG. 21 is performed three times. At this time, the reliability obtained by the processing in FIG. 20 performs the processing in FIG. It is applied uniformly each time.

  As described above, in the fourth embodiment, reliability is derived and retained in correspondence with a template in a predetermined image, so that alignment and image overlay processing are performed. It is possible to reduce the processing load when the process is performed a plurality of times. That is, the processing time can be shortened. As a result, it is possible to reduce the waiting time from the end of one shooting until the next shooting is possible. In addition, it is possible to increase the number of frames (frame speed) that can be shot per unit time during continuous shooting. In addition, when the processing for combining a plurality of images is performed by an external processing device such as a personal computer, the waiting time until the image is displayed on the monitor display device after the image processing is started is shortened. It becomes possible.

  By the way, when the image composition processing is performed by the image processing unit 300B of the fourth embodiment, for example, when the subject is a human and the arm or face moves greatly during a series of photographing operations, or the subject is There is a possibility that a desired processing result may not be obtained when a flower shakes due to the influence of wind during a series of shooting operations.

  For example, it is assumed that a certain template in an image used for reliability derivation includes an image of a part of an arm, and the reliability that the reliability is high in the biaxial direction is given. However, it is assumed that the region corresponding to the template in the reference image does not include the arm image and only the plain background image is shown.

  In such a case, the reliability of the motion vector obtained by template matching does not necessarily match the reliability. In order to cope with such a situation, it is desirable that the alignment processing unit is configured as shown in FIG. Among the components included in the alignment processing unit 309C illustrated in FIG. 22, the same components as those included in the alignment processing unit 309 illustrated in FIG.

  The alignment processing unit 309C shown in FIG. 22 is different from the alignment processing unit 309B shown in FIG. 19 in the points described below. That is, the alignment processing unit 309C further includes a validity verification unit 317 in which the movement vector deriving unit 313 of the alignment processing unit 309B is replaced with a movement vector deriving unit 313A.

  The movement vector deriving unit 313A performs a process of deriving a matching degree of template matching in addition to deriving a movement vector by performing a template matching process. That is, after finding an image element that matches the image element of the template in the reference image in the reference image, the degree of coincidence between the image element of the template and the image element determined to match the image element of the template is derived. A known technique can be used to derive the degree of coincidence.

  The validity verification unit 317 verifies the validity of the movement vector derived by the movement vector deriving unit 313A based on the degree of coincidence information derived by the movement vector deriving unit 313A. That is, when the coincidence degree is lower than a predetermined threshold (the coincidence degree is relatively low), it is determined that the reliability of the obtained movement vector is low. The reliability given to the template that is currently determined is “high reliability in both axial components in two axes”, “high reliability only in one axial component”, etc. Even if it is a thing, the said reliability is changed into "the reliability of both the axial components of 2 axial directions is low." That is, the validity verification unit 317 performs a process of correcting the reliability as necessary based on the degree of coincidence of the template matching process result derived by the movement vector deriving unit 313A.

  The movement vector correction processing unit 315A performs a process of correcting the movement vector as necessary based on the movement vector before correction and the reliability after correction.

  FIG. 23 is a flowchart schematically showing a processing procedure of image composition processing executed by the CPU 109. In the flowchart shown in FIG. 23, the same processes as those shown in FIG. 12 are denoted by the same reference numerals as those assigned to the respective processing steps in FIG. The difference from the processing in FIG. 12 will be mainly described.

  In the image composition processing of FIG. 23, there is no processing (reliability derivation processing) of S1206 in the image composition processing of FIG. Instead, for example, when the image composition processing is performed three times, the processing in FIG. 21 is performed three times. At this time, the reliability obtained by the processing in FIG. 20 performs the processing in FIG. It is applied uniformly each time. This point is the same as that shown in FIG.

  Further, in the image composition process of FIG. 23, the process of S1204 (movement vector derivation process) in the image composition process of FIG. 12 or FIG. 21 is replaced with the process of S1203 (movement vector derivation / matching degree derivation process). Processing (validity verification / reliability correction processing) is added.

  In step S1203, the CPU 109 performs movement vector derivation and coincidence degree derivation processing. This process corresponds to the process in the movement vector deriving unit 313A described with reference to FIG. In S1205, the CPU 109 performs validity verification and reliability derivation processing. This processing corresponds to the processing in the validity verification unit 317 described with reference to FIG.

  When the subject is a still life, it is effective to perform the image composition processing described with reference to FIGS. On the other hand, when the subject has a motion, a more preferable result can be obtained by performing the image composition processing described with reference to FIGS.

  As described above, in the fourth embodiment, the imaging apparatus 100B is described as including the image processing unit 300B. However, the imaging apparatus 100B includes an image processing unit 300B corresponding to an external processing device that is different from the imaging apparatus 100B. It may be. Alternatively, the external processing device may have some of the functions of the image processing unit 300B.

  The image processing technique according to the present invention can be applied to a digital still camera, a digital movie camera, and the like. Furthermore, the present invention can be applied to a camera incorporated in a mobile phone, a personal computer, a portable information terminal device, or the like.

100, 100A, 100B Imaging device 104 Optical system 105 Imaging unit 106 Analog front end (AFE)
107 Signal Processing Circuit 108 Storage Unit 108A RAM
108B ROM
109 CPU
110 System bus 300, 300A, 300B Image processing unit 309, 309A, 309B, 309C Registration processing unit 310 Image composition processing unit 311 Movement vector measurement region setting unit 312 Registration reference region setting unit 313, 313A Movement vector deriving unit 314, 314A Reliability deriving unit 315, 315A Movement vector correction processing unit 316 Movement vector interpolation processing unit 317 Validity verification unit 350 External processing device MV1, MV2, MV3, MV4 Movement vector

Claims (7)

A movement vector measurement region setting unit that sets a reference region in a reference image and sets a search region that is a region for searching for a region corresponding to the reference region in a reference image;
When a movement vector is derived from the reference area and the corresponding search area set by the movement vector measurement area setting unit, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving unit derived by analyzing at least one of the regions;
A motion vector deriving unit for searching in the search region for a region corresponding to the reference region set by the movement vector measurement region setting unit, and deriving a motion vector;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived by the movement vector deriving unit is decomposed into axial components along two axial directions. And a movement vector correction processing unit for deriving a movement vector of each axis component and correcting a movement vector of the axis component with lower reliability among the movement vectors of each axis component. .   The movement vector correction unit includes an axial component along the edge gradient direction of at least one of the reference region and the corresponding search region set by the movement vector measurement region setting unit as the axial component in the two-axis direction. The movement vector and the movement vector of the axial component along the direction orthogonal to the edge gradient direction are configured to correct the movement vector of the axial component with lower reliability. Image processing apparatus.   The movement vector correction unit is more reliable as an axis component in the two-axis direction, of a movement vector of an axis component along the column direction and a movement vector of an axis component along the row direction of the arrangement of pixels constituting the image. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to correct a movement vector having a low axial component. An image processing program for causing a computer to execute a process of superimposing by reducing positional deviation between a plurality of images,
A movement vector measurement region setting step for setting a reference region in a reference image and setting a search region which is a region for searching for a region corresponding to the reference region in a reference image;
When a movement vector is derived from the reference area and the corresponding search area set in the movement vector measurement area setting step, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving step derived by analyzing at least one of the regions;
A motion vector deriving step of searching in the search region for a region corresponding to the reference region set in the motion vector measurement region setting step, and deriving a motion vector;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived in the movement vector deriving step is decomposed into two axial components. An image processing program comprising: a movement vector correcting step of deriving a movement vector of each axis component and correcting a movement vector of an axis component having lower reliability among the movement vectors of each axis component. An image processing program for causing a computer to execute a process of superimposing by reducing positional deviation between a plurality of images,
A movement vector measurement region setting step for setting a reference region in a reference image and setting a search region which is a region for searching for a region corresponding to the reference region in a reference image;
A motion vector deriving step of searching in the search region for a region corresponding to the reference region set in the motion vector measurement region setting step, and deriving a motion vector;
When a movement vector is derived from the reference area and the corresponding search area set in the movement vector measurement area setting step, the reliability of the derived movement vector is determined from the reference area and the corresponding search area. A reliability deriving step derived by analyzing at least one of the regions;
Based on the analysis result obtained by analyzing at least one of the reference area and the corresponding search area, the movement vector derived in the movement vector deriving step is decomposed into two axial components. An image processing program comprising: a movement vector correcting step of deriving a movement vector of each axis component and correcting a movement vector of an axis component having lower reliability among the movement vectors of each axis component.   In the movement vector correction step, as each axis component decomposed in the two-axis direction, in the edge gradient direction of at least one of the reference area and the corresponding search area set in the movement vector measurement area setting step 6. The movement vector of the axial component with lower reliability is corrected among the movement vector of the axial component along the axis and the movement vector of the axis component along the direction orthogonal to the edge gradient direction. The image processing program described in 1.   In the movement vector correction step, as each axis component decomposed in the two-axis direction, out of the movement vector of the axis component along the column direction and the movement vector of the axis component along the row direction of the arrangement of the pixels constituting the image The image processing program according to claim 4, wherein the movement vector of the axial component with lower reliability is corrected.