JP2007272459A - Image processor and processing method, and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor, capable of further appropriately performing superimposing composition of a plurality of images. <P>SOLUTION: The image processor for superimposing and composing a plurality of related images 10a-10m into one image comprises a position detection part 1 for determining a relative displacement each between images; a reliability detection part 2 for determining a reliability of the displacement determined by the position detection part from the displacement; an image selection part 3 for selecting a plurality of images to be used for superimposing composition based on the reliability determined by the reliability detection part; and an image composition part 4 for positioning and composing the plurality of images selected by the image selection part based on the displacement determined by the position detection part so that each subject is matched. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, and an imaging apparatus. More specifically, the present invention relates to an image processing apparatus and method that superimposes and combines a plurality of images on one image, and an imaging apparatus.

  In recent years, digitalization of cameras has progressed rapidly, and attempts have been made to expand the shooting field of cameras, such as dynamic range expansion using digital signal processing and camera shake correction.

  Patent Document 1 discloses a method for performing so-called dynamic range expansion. Patent Document 1 discloses that a plurality of images with different exposure amounts are acquired, and those signals are combined to avoid so-called overexposure and underexposure and to substantially expand the dynamic range. .

  Patent Document 2 discloses a shake removal apparatus using a plurality of images. According to Japanese Patent Application Laid-Open No. 2004-228688, after each still image constituting a moving image is temporarily stored in a memory, the correlation between the still image of the next frame and the still image on the memory is obtained. Thereby, the movement of the image is detected, and the recording is performed with the position adjusted. As a result, a high-quality moving image can be obtained by correcting image blur due to so-called camera shake.

  Patent Documents 3 and 4 disclose methods for improving the estimation accuracy of a movement vector between a plurality of images. According to the invention disclosed in Patent Document 3, when a movement vector is calculated in a plurality of blocks, the vector estimation accuracy is improved by utilizing a vector of an adjacent block. According to the invention disclosed in Patent Document 4, the reliability of the detected vector is estimated based on the frequency of the vector. By applying the methods disclosed in Patent Documents 3 and 4, it is possible to appropriately perform image processing with increased accuracy of estimation of a movement vector between a plurality of images.

Japanese Patent Laid-Open No. 61-219270 Japanese Patent Laid-Open No. 01-109970 Japanese Patent Application Laid-Open No. 07-170496 Japanese Unexamined Patent Publication No. 07-007721

  However, when the inventions of Patent Documents 1 and 2 are used for a digital camera or the like, there has been a problem that when the SN ratio of the image is not good, the estimation accuracy of the movement vector is lowered and the image cannot be processed appropriately.

  In addition, when the invention of Patent Document 3 is used for a digital camera or the like, if the SN ratio is not good in many blocks, for example, when shooting a night scene, the estimation accuracy is sufficiently improved even if adjacent block information is used. It is difficult to do.

  In addition, when the invention of Patent Document 4 is used for a digital camera or the like, if a large amount of camera shake occurs instantaneously, the correct condition is removed based on the occurrence frequency information, and the estimation accuracy is sufficiently improved. There was a problem that I could not.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus and method, and an imaging apparatus capable of performing overlay synthesis of a plurality of images more appropriately.

  In order to achieve the above object, an image processing apparatus of the present invention for superimposing and synthesizing a plurality of images related to each other on a single image includes position detection means for obtaining a relative displacement between the images, and the position detection A reliability detecting means for obtaining the reliability of the displacement determined by the means, and a selecting means for selecting a plurality of images used for overlay synthesis based on the reliability obtained by the reliability detecting means; And combining means for aligning and synthesizing the plurality of images selected by the selection means so that the subject matches based on the displacement obtained by the position detection means.

  The image processing method of the present invention for superimposing and synthesizing a plurality of images related to each other into a single image includes a position detection step for obtaining a relative displacement between the images, and a displacement obtained in the position detection step. Reliability is selected from the reliability detection step for obtaining reliability from the displacement, a selection step for selecting a plurality of images to be used for overlay synthesis based on the reliability obtained in the reliability detection step, and the selection step. A combining step of aligning and synthesizing the plurality of images based on the displacement obtained in the position detection step so that the subject matches.

  According to another configuration, the image processing apparatus of the present invention for superimposing and synthesizing a plurality of images related to each other on a single image includes position detection means for obtaining a relative displacement between the images, and the position The displacement between the reliability detection means obtained from the displacement obtained by the detection means and the low reliability image obtained by the reliability detection means is determined by the displacement between the images other than the images. And replacing means for replacing, and combining means for aligning and synthesizing the plurality of images so that the subject matches based on the displacement obtained by the position detecting means and the displacement replaced by the replacing means.

  The image processing method of the present invention for superimposing and synthesizing a plurality of images related to each other into a single image includes a position detection step for obtaining a relative displacement between the images, and a displacement obtained in the position detection step. A reliability detection step for obtaining reliability from the displacement, a replacement step for replacing displacement between images with low reliability obtained in the reliability detection step by displacement between images other than between the images, and the position Based on the displacement obtained by the detection step and the displacement replaced by the replacement step, the composition step of aligning and synthesizing the plurality of images so that the subject matches.

  Furthermore, according to another configuration, the image processing apparatus of the present invention that superimposes and combines a plurality of images related to each other into a single image includes a dividing unit that divides the plurality of images into a plurality of blocks, and the division For each block divided by the means, a first position detecting means for obtaining a relative displacement between images, and a first position detecting means for obtaining the reliability of the displacement obtained by the first position detecting means from the displacement. Relative relationships between the images obtained by the first position detecting means of the blocks other than the reliability detecting means and the blocks having low reliability obtained by the first reliability detecting means among the plurality of blocks. A second position detecting means for obtaining a movement amount or mapping matrix between the entire images based on the displacement, and a movement amount or mapping matrix obtained by the second position detecting means. And selecting a plurality of images to be used for overlay synthesis based on the reliability obtained by the second reliability detecting means for obtaining the movement amount or the reliability of the mapping matrix, and the reliability obtained by the second reliability detecting means. And a combining means for aligning and synthesizing the plurality of images selected by the selecting means based on the amount of movement or the mapping matrix obtained by the second position detecting means so that the subject matches. And have.

  An image processing method for superimposing and combining a plurality of images related to each other on a single image includes a dividing step of dividing the plurality of images into a plurality of blocks, and an image for each block divided in the dividing step. A first position detecting step for obtaining a relative displacement between the first position detecting step, a first reliability detecting step for obtaining the reliability of the displacement obtained in the first position detecting step from the displacement, Of the blocks other than the low-reliability block obtained in the first reliability detection step, based on the relative displacement between the images obtained in the first position detection step, between the whole images A second position detecting step for obtaining a moving amount or mapping matrix; and a reliability of the moving amount or mapping matrix based on the moving amount or mapping matrix obtained in the second position detecting step. Selected in the selection step, a selection step for selecting a plurality of images to be used for overlay synthesis based on the reliability obtained in the second reliability detection step A combining step of aligning and synthesizing the plurality of images based on the movement amount or mapping matrix obtained in the second position detection step so that the subject matches.

  An imaging apparatus according to the present invention includes any one of the image processing apparatuses described above.

  It is possible to provide an image processing apparatus and method, and an imaging apparatus capable of performing overlay synthesis of a plurality of images more appropriately.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
A first embodiment of the present invention will be described with reference to FIGS.

Configuration of Image Processing Device FIG. 1 is a block diagram showing an electrical configuration of the imaging device according to the first embodiment of the present invention. In FIG. 1, 31 is an optical system comprising a lens and a diaphragm, 32 is a mechanical shutter, 33 is a solid-state image sensor such as a CCD or CMOS sensor, and 34 is a CDS circuit that performs analog signal processing. Reference numeral 35 denotes an A / D converter that converts an analog signal into a digital signal. Reference numeral 36 denotes a timing signal generation circuit that generates a signal for operating the image sensor 33, the CDS circuit 34, and the A / D converter 35. 37 is a drive circuit for the optical system 31, shutter 32 and image sensor 33, 38 is a signal processing circuit for performing signal processing necessary for captured image data, and 39 is an image memory for storing the signal processed image data.

  Reference numeral 40 denotes a recording medium that can be removed from the imaging apparatus, 41 denotes a recording circuit that records signal processed image data on the recording medium 40, 42 denotes a display apparatus that displays the signal processed image data, and 43 denotes an image on the display apparatus 42. Is a display circuit. A system control unit 44 controls the entire imaging apparatus. 45 is a nonvolatile memory for storing a program describing a control method executed by the system control unit 44, control data such as parameters and tables used when executing the program, and correction data such as a scratch address (ROM). A volatile memory (RAM) 46 is used to transfer and store the program, control data, and correction data stored in the nonvolatile memory 45 and used when the system control unit 44 controls the imaging apparatus.

Shooting Operation Hereinafter, a shooting operation when the mechanical shutter 32 is used with the imaging apparatus configured as described above will be described. Prior to the photographing operation, necessary programs, control data, and correction data are transferred from the nonvolatile memory 45 to the volatile memory 46 and stored at the start of the operation of the system control unit 44 such as when the imaging apparatus is turned on. Shall be kept. These programs and data are used when the system control unit 44 controls the imaging apparatus. Furthermore, as necessary, additional programs and data are transferred from the nonvolatile memory 45 to the volatile memory 46, or the system control unit 44 directly reads and uses the data in the nonvolatile memory 45. .

  First, the optical system 31 drives a diaphragm and a lens in accordance with a control signal from the system control unit 44 to form a subject image set to an appropriate brightness on the image sensor 33. Next, the mechanical shutter 32 is driven by a control signal from the system control unit 44 so as to shield the image sensor 33 in accordance with the operation of the image sensor 33 so as to have a necessary exposure time. At this time, when the image sensor 33 has an electronic shutter function, it may be used together with the mechanical shutter 32 to secure a necessary exposure time.

  The image sensor 33 is driven by a drive pulse based on the operation pulse generated by the timing signal generation circuit 36 controlled by the system control unit 44, and converts the subject image into an electrical signal by photoelectric conversion as an analog image signal. Output. The analog image signal output from the image pickup device 33 is subjected to an operation pulse generated by a timing signal generation circuit 36 controlled by the system control unit 44, and the clock synchronization noise is removed by the CDS circuit 34. The A / D converter In 35, it is converted into a digital image signal.

  Next, the signal processing circuit 38 controlled by the system control unit 44 performs image processing such as color conversion, white balance correction, and gamma correction (not shown), resolution conversion processing, image compression processing, and the like on the digital image signal. Do. Further, a process of generating a single image by superposing and synthesizing a plurality of images is performed. This synthesis process will be described later in detail. The image memory 39 is used for temporarily storing a digital image signal during signal processing or for storing image data which is a digital image signal subjected to signal processing.

  The image data signal-processed by the signal processing circuit 38 and the image data stored in the image memory 39 are converted into data suitable for recording on the recording medium 40 (for example, file system data having a hierarchical structure) in the recording circuit 41. And recorded on the recording medium 40. Further, after the resolution conversion processing is performed by the signal processing circuit 38, the display circuit 43 converts the signal into a signal suitable for the display device 42 (for example, an NTSC analog signal) and displays it on the display device 42.

  Here, the signal processing circuit 38 may output the digital image signal as it is as image data to the image memory 39 or the recording circuit 41 without performing signal processing by the control signal from the system control unit 44. Further, when requested by the system control unit 44, the signal processing circuit 38 outputs information on digital image signals and image data generated in the signal processing process to the system control unit 44. The information output here is, for example, information such as the spatial frequency of the image, the average value of the designated area, the data amount of the compressed image, or information extracted from them. Further, the recording circuit 41 outputs information such as the type of the recording medium 40 and the free capacity to the system control unit 44 when requested by the system control unit 44.

Reproduction Operation A reproduction operation when image data is recorded on the recording medium 40 will be described. In response to a control signal from the system control unit 44, the recording circuit 41 reads image data from the recording medium 40. Similarly, in response to a control signal from the system control unit 44, the signal processing circuit 38 performs image expansion processing and stores the image data in the image memory 39 when the image data is a compressed image. The image data stored in the image memory 39 is subjected to resolution conversion processing by the signal processing circuit 38, converted to a signal suitable for the display device 42 by the display circuit 43, and displayed on the display device 42.

Image Synthesis Processing FIG. 2 is a block diagram showing the functional configuration of the signal processing device 38 when a plurality of images are synthesized. Note that the synthesis processing performed here is overlay synthesis, and can be used for known dynamic range expansion, noise reduction, camera shake correction, and the like. Further, it is not always necessary to perform this synthesis process, and it may be performed according to the necessity or according to the setting of the operator.

  As shown in FIG. 2, a plurality of images 10 a, 10 b,..., 10 m are given to the position detection unit 1 and the image selection unit 3. The position detection unit 1 detects a positional deviation between a plurality of images by, for example, a method disclosed in Japanese Patent Laid-Open No. 2000-341582. The reliability detection unit 2 determines the reliability of position detection based on the output of the position detection unit 1 and obtains an image suitable for image synthesis. In the image selection unit 3, based on the reliability information obtained by the reliability detection unit 2, an image used for composition is selected from the plurality of images 10a, 10b,. It passes to the synthesis unit 4. The image composition unit 4 performs alignment and image exposure correction on a plurality of images selected by the image selection unit 3 based on the information of the position detection unit 1 and composes it into one image.

  Next, with reference to FIG. 3, the composition of multiple images and the movement vector and reliability detection unit 2 will be described.

  In FIG. 3, (a) and (b) are schematic diagrams showing two images to be combined, and (c) is an overlay of the images shown in (a) and (b) without alignment. FIG. 4D is a schematic diagram illustrating a case where the images illustrated in FIGS. 2A and 2B are aligned and superimposed.

  Reference numerals 21a and 21b denote subjects, respectively. Reference numerals 20a and 20b in FIG. 3C denote movement vectors between the images shown in FIGS.

  If a plurality of images taken in a state where there is a factor that changes the angle of view, such as camera shake, are superimposed as they are as shown in FIG. 3C, the subjects 21a and 21b are displaced. The positional deviation between the two images can be obtained by taking the correlation between the images by the method disclosed in Japanese Patent Laid-Open No. 2000-341582. In the present invention, a vector indicating a shift between images is referred to as a “movement vector”. It is possible to perform alignment by mapping the image using the obtained movement vector, and it is possible to obtain a combined image after performing alignment as shown in FIG.

  In FIG. 3, for the sake of simplicity of explanation, the composition of two images and the movement vector have been described. However, the same processing is sequentially performed for more images, so that three or more images can be composed. Can do.

  Next, the principle of the reliability estimation of the movement vector between a plurality of images performed by the reliability detection unit 2 will be described with reference to FIG.

In the following description, a movement vector from the p-th sheet to the q-th sheet is denoted as v _pq . FIG. 4 shows vectors from the first sheet to the nth sheet. FIG. 4A shows a case where there is no failure or error in detection of the movement vector from the first image to the n-th image, and FIG. 4B shows n−1 among the first to n-th images. The cases where there is a failure or error in the detection of the movement vector from the first sheet to the nth sheet are shown. For an ideal subject, there is no error or failure in detecting the movement vector, but if the subject in the original image has movement or is affected by the noise of the image sensor, the movement vector is detected by that effect. Errors and failures may occur. As can be seen from FIG. 4, there is redundancy in determining the movement vector between the images. That is, the vector from the first sheet to the nth sheet can be obtained as v _1n , but can also be obtained as v ₁₂ + v ₂₃ +... + V _{(n−1) n} . That means
v _1n = v ₁₂ + v ₂₃ + ... + v _{(n-1) n} (1)

Holds. In the state where there is no detection failure or error as shown in FIG.
_{v 12 + v 23 + ... +} v (n1) n -v 1n = v 12 + v 23 + ... + v (n1) n + v n1 = 0 ... (2)
Holds. However, since there is generally a detection error, as shown in FIG. 4B, v _1n ≠ v ₁₂ + v ₂₃ +... + V _{(n−1) n} (3)
And as a result,
_{v 12 + v 23 + ... +} v (n1) n -v 1n = v 12 + v 23 + ... + v (n1) n + v n1 ≠ 0 ... (4)
It becomes. On the other hand considering the motion vector from the first sheet to n-1 th, 4 in the example of _{(b) v 12 + v 23} + ... + v (n-2) (n-1) -v 1 (n- _{1) = v 12 + v 23} + ... + v (n-2) (n-1) + v (n-1) 1 = 0 ... (5)
Holds. Here, the following expression (6) is used as an evaluation value of the reliability of the identification of the movement vector from the first sheet to the nth sheet.

(Reliability evaluation _{value) = | v 12 + v 23} + ... + v (n1) n + v n1 | ... (6)
As described with reference to FIG. 4, when there is a detection error or a large error in the identification of the movement vector, the reliability evaluation value takes a large value, and becomes zero when there is no error at all. That is, it is possible to estimate the vector identification accuracy by the reliability evaluation value.
Further, a method of determining a movement vector and an image used for image synthesis using the reliability evaluation value will be described with reference to FIGS.

  In FIG. 5, for the sake of simplicity, a case where four images are combined will be described as an example. However, the present invention can be similarly applied to a case where five or more images are combined. FIG. 5A shows the case where there is no failure or error in detection of the movement vector from the first to the fourth sheet, and FIG. 5B shows the third to fourth of the first to fourth sheets. The cases where there are failures or errors in the detection of the movement vector to the eye are shown. Here, it is assumed that the vectors other than the third to fourth vectors can be accurately identified.

  As described with reference to FIG. 4, in the state of FIG.

(Reliability evaluation value) = | v ₁₂ + v ₂₃ + v ₃₄ + v ₄₁ | ≠ 0 (7)
It is.
The reliability detection unit 2 performs synthesis when the reliability evaluation value calculated as described above is larger than a predetermined value (reliability is low) as in the example illustrated in FIG. The evaluation value is calculated again while skipping any one of the four images to be processed. The allowable reliability evaluation value may be determined in consideration of an allowable circle of confusion and the like. Of the movement vectors shown in FIG. 5B, the case where the first image is skipped is shown in FIG. 5C, the case where the second image is skipped is shown in FIG. FIG. 5E shows the case where the image is skipped, and FIG. 5F shows the case where the fourth image is skipped.

From FIG. 5, clearly
(Reliability evaluation value when the first sheet is removed) = | v ₂₃ + v ₃₄ + v ₄₂ | ≠ 0
(Reliability evaluation value when the second sheet is removed) = | v ₁₃ + v ₃₄ + v ₄₁ | ≠ 0
(Reliability evaluation value when the third sheet is removed) = | v ₁₂ + v ₂₄ + v ₄₁ | = 0

(Reliability evaluation value when the fourth sheet is removed) = | v ₁₂ + v ₂₃ + v ₃₁ | = 0 (8)
It becomes. In general, since the identification includes an error, the reliability evaluation value is not strictly zero. However, by synthesizing an image based on a vector that becomes small, the alignment accuracy of the image can be improved. it can.
The position detection unit 1 shown in FIG. 2 detects all the necessary movement vectors in order to calculate the reliability evaluation value excluding each image as shown in the equation (8) by the reliability detection unit 2. Output to the sex detector 2. Since the number of necessary movement vectors is determined by a combination of two arbitrary points, n (n−1) / 2 vectors are obtained for n images. For example, when there are four images as described above, they are v ₁₂ , v ₁₃ , v ₂₃ , v ₂₄ , v ₃₄ , v ₄₁ .

As described above, the reliability evaluation unit 2 obtains the reliability evaluation value as described above, and obtains a highly reliable vector. Next, the image selection unit 3 shown in FIG. 2 determines an image to be combined and a movement vector used for the combination based on the value from the reliability detection unit 2. In the example shown in Expression (8), the reliability evaluation value when the third sheet is removed and the reliability evaluation value when the fourth sheet is removed are both 0. Synthesis is performed using the removed three images (that is, the first, second, fourth, or first, second, third). As can be seen from FIG. 5B, since the detection of the movement vector from the third image to the fourth image has failed here, any of the third image and the fourth image can be excluded during image synthesis. movement vector v ₃₄ can be prevented from being used. In practice, since the reliability evaluation value rarely becomes 0, a combination of images having a smaller reliability evaluation value (higher reliability) may be selected.

  Once the image and the movement vector used for the composition are determined, the image composition unit 4 then composes one image from a plurality of images by performing alignment and image exposure correction. As a result, a high-quality composite image can be obtained. This high-quality image is recorded on the recording medium 40 using the recording circuit 41 shown in FIG.

  As described above, according to the first embodiment, a plurality of images can be superimposed and combined more appropriately.

<Modification>
In the first embodiment, based on the reliability evaluation value shown in Expression (8), the image and the movement vector used for the synthesis are selected from among the plurality of images. However, the movement vector is replaced as described below. Anyway. In the example of Expression (8), when v ₃₄ is included, the reliability evaluation value becomes large (the reliability becomes low), so it can be evaluated that v ₃₄ is a movement vector that has failed to be detected.

Accordingly, the image selection unit 3 selects a movement vector that has failed in detection or a movement vector that has succeeded in detection, and outputs a movement vector in which v ₃₄ is replaced with another movement vector and replaced with the image composition unit 4. Also good. Specifically, it is replaced with v ₃₁ + v ₁₄ or v ₃₂ + v ₂₄ . The image synthesis unit 4 can also synthesize all the first to fourth images using the movement vector obtained from the position detection unit 1 and the movement vector replaced by the image selection unit 3. Although the case where four images are combined has been described here, it is needless to say that the same processing may be performed when combining images other than four images.

  As described above, it is possible to appropriately synthesize a larger number of images as compared with the first embodiment.

<Second Embodiment>
Next, a second embodiment of the present invention will be described with reference to FIGS. 1, 2, and 6 to 8. FIG.

  In the second embodiment, the operations of the position detector 1 and the reliability detector 2 are different from those of the first embodiment. Since other configurations and operations are the same as those in the first embodiment, the position detection unit 1 and the reliability detection unit 2 will be described below, and the description of the other components will be omitted.

  FIG. 6 is a block diagram illustrating a functional configuration of the position detection unit 1 in the second embodiment. As shown in FIG. 6, the position detection unit 1 includes a division unit 5, a movement vector detection unit 6, a block reliability detection unit 7, and an affine map generation unit 8.

  Next, the operation of the position detection unit 1 having the above configuration will be described.

  First, the dividing unit 5 divides each of the plurality of images 10a to 10m into n regions (blocks) set in advance. Instead of dividing by the dividing unit 5, some feature points in the image may be extracted using a feature point extracting unit. There is no essential difference between the two methods when it comes to using movement vectors at multiple points in the image. Therefore, although the case where the dividing unit 5 is used will be described in the second embodiment, the second embodiment can be applied to a method using feature points.

  Next, the movement vector detection unit 6 detects misalignment between a plurality of images for each block using, for example, the method disclosed in Japanese Patent Application Laid-Open No. 2000-341582. The block reliability detection unit 7 determines the reliability of position detection based on the output of the movement vector detection unit 6 and obtains a block suitable for identifying an affine map. The affine mapping generation unit 8 generates an affine mapping matrix representing mapping between a plurality of images from the block obtained from the block reliability detection unit 7 and position information on the image of the block.

  The above operation will be specifically described with reference to FIG.

  In FIG. 7, (a) and (b) are schematic diagrams showing two images to be combined, and (c) is a direct overlay of the images shown in (a) and (b) without alignment. FIG. 4D is a schematic diagram illustrating a case where the images illustrated in FIGS. 2A and 2B are aligned and superimposed. FIG. 7C shows a plurality of blocks.

  In FIG. 7, reference numerals 21a and 21b denote subjects, and reference numerals 20a, 20b, 20c, 20d, and 20e in (c) denote movement vectors between images shown in (a) and (b).

  If a plurality of images taken in a state where there is a factor that changes the angle of view, such as camera shake, are superimposed as they are as shown in FIG. 7C, the subjects 21a and 21b are displaced. Therefore, the positional deviation between the two images in each region can be obtained by taking the correlation between the images in each divided region. Also in the second embodiment, a vector indicating a shift between images is referred to as a movement vector.

  In FIG. 7C, it is assumed that a block in which the movement vector 20e exists does not have a so-called subject. In a block having no subject, correlation between images may not be obtained well. In such a case, an inappropriate movement vector may be obtained as shown in 20e. If the affine mapping matrix of the entire image is identified including the case where the identification is inappropriate like the movement vector 20e, it is difficult to accurately perform the alignment.

The operation of the affine map generation unit 8 will be described with reference to FIG. In FIG. 8, O indicates a reference point on the image. In the second embodiment, in the p-th image, a vector from the calculation reference point O provided on the image to the i-th region is represented by v _pi, and in the i-th region, the block position detection unit 4 Let u _{ipq be} the movement vector between the p-th image and the q-th image obtained by the above.

FIG. 7 illustrates the movement vector and the vector to the block obtained in the first to fifth blocks between the first and second sheets. In FIG. 8, it is assumed that the fifth block is not properly identified with the motion vector, and the other blocks are properly identified. Here, if the matrix representing the affine mapping from the p-th to the q-th is expressed as A _pq , it is apparent from the above definition that v _qi = v _pi + u _ipq = A _pq v _pi (9)

In the above equation (9), u _ipq is obtained by the movement vector detection unit 6, and v _pi is a known value determined by the block division method. Moreover, since the above equation holds regardless of the block i,

_{(V q1, v q2, ...} , v qm) = (v p1 + u 1pq, v p2 + u 2pq, ..., v pm + u mpq)
= A _pq (v _p1 , v _p2 , ..., v _pm ) (10)
In the above equation (10), when (v _q1 , v _q2 ,..., V _qm ) = M _q and (v _p1 , v _p2 ,..., V _pm ) = M _p , ) Is M _q = A _pq M _p (11)

Can be written. Solving this equation (11), A _pq is
A _pq = M _q M _p ^T (M _p M _p ^T ) ⁻¹ (12)
Can be obtained as By appropriately processing so that (M _p M _p ^T ) ⁻¹ in Expression (12) does not drop in rank, A _pq can be obtained. Since A _pq is obtained so that the residual sum of squares is minimized, if there is an improperly identified block such as u ₅₁₂ in FIG. 7, the identification accuracy of the affine matrix is lowered. Therefore, the first embodiment is applied to the movement vector obtained between n images in each block to improve the affine matrix identification accuracy. More specifically, the movement vectors u _i12 , u _i23 ,..., U _{i (n−1) n} obtained in the i-th block.
In contrast, the method of the first embodiment is applied. According to the method of the first embodiment, the block reliability detection unit 7 selects a block suitable for identifying an affine matrix. For example, although the fifth block in FIG. 7 is inappropriately identified, the background portion and the like are likely to be inappropriately identified as shown in FIG. Such blocks can be easily excluded because the movement vector is obtained randomly between images, and the evaluation value shown in the first embodiment does not become small. By identifying the affine matrix after removing the movement vector of such a block, an accurately superimposed image as shown in FIG. 7D can be obtained.
As shown in FIG. 2, the affine matrix obtained by the position detection unit 1 is passed to the reliability detection unit 2. In the second embodiment, the reliability detection unit 2 performs an evaluation using an affine matrix determinant. That is, when the affine matrix is identified without error as in the case of the movement vector in the first embodiment, A _1n = A ₁₂ A ₂₃ ... A _{(n−1) n} (13)

Holds. When this equation (13) is transformed, E = A ₁₂ A ₂₃ ... A _{(n-1) n} A _n1 (14)
It becomes. However, E on the left side of Equation (14) represents a unit matrix. In other words, when the mapping from the nth sheet to the first sheet is further performed on the one obtained by sequentially superimposing the mappings from the first sheet to the nth sheet, the unit matrix is restored. Considering the determinant on both sides of the above equation

(Reliability evaluation value) = det (E) = det (A ₁₂ A ₂₃ ... A _{(n−1) n} A _n1 ) = 0 (15)
Therefore, the value obtained by Expression (15) is defined as the reliability evaluation value. And
(Reliability evaluation value excluding the kth sheet) = det (A ₁₂ A ₂₃ ... A _{(k−2) (k−1)} A _{(k−1) (k + 1)} ... A _{(n−1) n} A _n1 ) (16)
Are sequentially calculated, and an appropriate affine map can be selected by reducing this value in the same manner as the reliability evaluation value of the first embodiment.
In the reliability detection unit 2 shown in FIG. 2, in the second embodiment, a reliability evaluation value is obtained by Expression (16), and a highly reliable affine map is obtained. Next, the image selection unit 3 shown in FIG. 2 determines an image to be combined and an affine map used for the combination based on the values from the reliability detection unit 2.

  Then, finally, the image composition unit 4 performs alignment, image exposure correction, and the like to compose one image from a plurality of images. As a result, a high-quality composite image can be obtained. This high-quality image is recorded on the recording medium 40 using the recording circuit 41 shown in FIG.

  As described above, according to the second embodiment, a plurality of images can be superimposed and combined more appropriately.

<Other embodiments>
In the first and second embodiments, the case where the signal processing circuit of the present invention is mounted on the imaging apparatus has been described, but the present invention is not limited to this. For example, the present invention can be applied when an image input from an imaging device is processed by an external signal processing device such as a computer.

  In that case, the object of the present invention can be achieved, for example, as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment of the present invention. It is a block diagram which shows the function structure of the image processing apparatus in embodiment of this invention. It is a figure for demonstrating the superimposition synthesis | combination of a several image. It is a figure explaining the definition of the vector in embodiment of this invention. It is explanatory drawing in the case of extracting a highly reliable vector in the 1st Embodiment of this invention. It is a block diagram which shows the function structure of the position detection part in the 2nd Embodiment of this invention. It is a figure for demonstrating the superimposition synthesis | combination of the several image in the 2nd Embodiment of this invention. It is a figure explaining the definition of the vector in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Position detection part 2 Reliability detection part 3 Image selection part 4 Image synthetic | combination part 5 Dividing part or feature point extraction part 6 Movement vector detection part 7 Block reliability detection part 8 Affine map generation part 10a-10m image 11 Output image 20 Movement Vector 21 Subject 31 Optical system 32 Mechanical shutter 33 Image sensor 34 CDS circuit 35 A / D converter 36 Timing signal generation circuit 37 Drive circuit 38 Signal processing circuit 39 Image memory 40 Recording medium 41 Recording circuit 42 Display device 43 Display circuit 44 System Control unit 45 Non-volatile memory (ROM)
46 Volatile memory (RAM)

Claims (26)

An image processing apparatus that superimposes and combines a plurality of mutually related images on a single image,
Position detecting means for obtaining a relative displacement between the images;
Reliability detection means for determining the reliability of displacement obtained by the position detection means from the displacement;
Selection means for selecting a plurality of images to be used for overlay synthesis based on the reliability obtained by the reliability detection means;
An image processing apparatus comprising: combining means for aligning and synthesizing a plurality of images selected by the selection means so that a subject matches based on the displacement obtained by the position detection means. An image processing apparatus that superimposes and combines a plurality of mutually related images on a single image,
Position detecting means for obtaining a relative displacement between the images;
Reliability detection means for determining the reliability of displacement obtained by the position detection means from the displacement;
Replacement means for replacing displacement between images with low reliability obtained by the reliability detection means by displacement between images other than between the images;
An image processing method comprising: combining means for aligning and synthesizing the plurality of images so that a subject matches based on the displacement obtained by the position detection means and the displacement replaced by the replacement means. apparatus.   3. The image according to claim 1, wherein the position detection unit obtains the displacement as a movement vector, and the reliability detection unit obtains reliability using synthesis of movement vectors between the plurality of images. Processing equipment. In the case where n images are synthesized, the reliability detecting means expresses a movement vector from the p-th sheet to the q-th sheet as v _pq .
_{| V 12 + ... + v (} n1) n + v n1 |
If the reliability is lower than the preset reliability, further | v ₁₂ + ... + v while sequentially removing one of the plurality of images (k = 1 to n) _{(k-1) (k + 1)} + ... + v _{(n-1) n} + v _n1 |
The image processing apparatus according to claim 3, wherein the image processing device is obtained as reliability.   The image processing apparatus according to claim 4, wherein the predetermined reliability is a function of an allowable circle of confusion. An image processing apparatus that superimposes and combines a plurality of mutually related images on a single image,
Dividing means for dividing the plurality of images into a plurality of blocks, respectively.
First position detecting means for obtaining a relative displacement between images for each block divided by the dividing means;
First reliability detection means for determining the reliability of the displacement determined by the first position detection means from the displacement;
Based on the relative displacement between the images obtained by the first position detection means of the blocks other than the low reliability blocks obtained by the first reliability detection means among the plurality of blocks. Second position detecting means for obtaining a movement amount or mapping matrix between the entire images;
Second reliability detection means for determining the reliability of the movement amount or mapping matrix based on the movement amount or mapping matrix obtained by the second position detection means;
Selection means for selecting a plurality of images used for overlay synthesis based on the reliability obtained by the second reliability detection means;
Combining a plurality of images selected by the selection unit based on a movement amount or a mapping matrix obtained by the second position detection unit so as to match the subject so as to match with each other. A featured image processing apparatus.   The first position detection unit obtains the displacement as a movement vector, and the first reliability detection unit obtains the reliability using synthesis of movement vectors between the plurality of images. 6. The image processing apparatus according to 6. The first reliability detection unit, when combining n images, represents a movement vector from the p-th to the q-th as v _pq ,
_{| V 12 + ... + v (} n1) n + v n1 |
If the reliability is lower than the preset reliability, further | v ₁₂ + ... + v while sequentially removing one of the plurality of images (k = 1 to n) _{(k-1) (k + 1)} + ... + v _{(n-1) n} + v _n1 |
The image processing apparatus according to claim 7, wherein the image processing device is obtained as reliability.   The image processing apparatus according to claim 8, wherein the preset reliability is a function of an allowable circle of confusion. The second reliability detection means, in the case of synthesizing the n images, when a matrix representing the affine transformation to the q-th from p th is expressed as A _{pq,
det (A 12 + ... + A} (n1) n + A n1)
If the reliability is lower than a preset reliability, further removing one of the plurality of images (k = 1 to n) in order.
_{det (A 12 ... A (k} -1) (k + 1) + ... + A (n1) n + A n1 |
The image processing apparatus according to claim 6, wherein the image processing apparatus is obtained as reliability.   The image processing apparatus according to claim 10, wherein the predetermined reliability is a function of an allowable circle of confusion.   An imaging apparatus comprising the image processing apparatus according to claim 1. An image processing method for superimposing and synthesizing a plurality of images related to each other on a single image,
A position detecting step for obtaining a relative displacement between the images;
A reliability detection step for determining the reliability of the displacement obtained in the position detection step from the displacement;
A selection step for selecting a plurality of images used for overlay synthesis based on the reliability obtained in the reliability detection step;
An image processing method comprising: combining a plurality of images selected in the selection step so as to align and synthesize a subject based on the displacement obtained in the position detection step. An image processing method for superimposing and synthesizing a plurality of images related to each other on a single image,
A position detecting step for obtaining a relative displacement between the images;
A reliability detection step for determining the reliability of the displacement obtained in the position detection step from the displacement;
Replacing the displacement between images with low reliability determined in the reliability detection step by displacement between images other than between the images; and
An image processing method comprising: a combining step of aligning and combining the plurality of images so that a subject matches based on the displacement obtained by the position detection step and the displacement replaced by the replacement step. Method.   The image according to claim 13 or 14, wherein the displacement is obtained as a movement vector in the position detection step, and the reliability is obtained using synthesis of movement vectors between the plurality of images in the reliability detection step. Processing method. In the reliability detection step, when n images are synthesized, when a movement vector from the p-th sheet to the q-th sheet is expressed as v _pq ,
_{| V 12 + ... + v (} n1) n + v n1 |
If the reliability is lower than the preset reliability, further | v ₁₂ + ... + v while sequentially removing one of the plurality of images (k = 1 to n) _{(k-1) (k + 1)} + ... + v _{(n-1) n} + v _n1 |
The image processing method according to claim 15, wherein the image processing method is obtained as reliability.   The image processing method according to claim 16, wherein the preset reliability is a function of an allowable circle of confusion. An image processing method for superimposing and synthesizing a plurality of images related to each other on a single image,
A dividing step of dividing each of the plurality of images into a plurality of blocks;
A first position detecting step for obtaining a relative displacement between images for each block divided in the dividing step;
A first reliability detection step for determining the reliability of the displacement determined in the first position detection step from the displacement;
Based on the relative displacement between the images obtained in the first position detection step of blocks other than the low reliability block obtained in the first reliability detection step among the plurality of blocks. A second position detecting step for obtaining a movement amount or mapping matrix between the entire images;
A second reliability detecting step for determining the reliability of the moving amount or mapping matrix based on the moving amount or mapping matrix obtained in the second position detecting step;
A selection step of selecting a plurality of images to be used for overlay synthesis based on the reliability obtained in the second reliability detection step;
Combining a plurality of images selected in the selection step so as to align and synthesize the subject based on the movement amount or mapping matrix obtained in the second position detection step. A featured image processing method.   The first position detection step obtains the displacement as a movement vector, and the first reliability detection step obtains the reliability using synthesis of movement vectors between the plurality of images. The image processing method according to claim 18. In the first reliability detection step, when n images are synthesized, when a movement vector from the p-th sheet to the q-th sheet is expressed as v _pq ,
_{| V 12 + ... + v (} n1) n + v n1 |
If the reliability is lower than the preset reliability, further | v ₁₂ + ... + v while sequentially removing one of the plurality of images (k = 1 to n) _{(k-1) (k + 1)} + ... + v _{(n-1) n} + v _n1 |
The image processing method according to claim 19, wherein the image processing method is obtained as reliability.   21. The image processing method according to claim 20, wherein the preset reliability is a function of an allowable circle of confusion. In the second reliability detection step, when n images are synthesized, when a matrix representing an affine mapping from the p-th image to the q-th image is expressed as A _pq ,
_{det (A 12 + ... + A} (n1) n + A n1)
If the reliability is lower than a preset reliability, further removing one of the plurality of images (k = 1 to n) in order.
_{det (A 12 ... A (k} -1) (k + 1) + ... + A (n1) n + A n1 |
The image processing method according to claim 18, wherein the image processing method is obtained as reliability.   The image processing method according to claim 22, wherein the preset reliability is a function of an allowable circle of confusion.   A program that can be executed by an information processing apparatus, and that causes the information processing apparatus that has executed the program to function as the image processing apparatus according to any one of claims 1 to 11.   24. A program executable by an information processing apparatus, comprising program code for realizing each step of the image processing method according to claim 13.   26. A storage medium readable by an information processing apparatus, wherein the program according to claim 24 or 25 is stored.

Images