JP4779491B2 - Multiple image composition method and imaging apparatus - Google Patents

Description

  The present invention relates to a multiple image synthesis method and an imaging apparatus that synthesize images that are continuously captured in time series.

  Focusing on the fact that the random noise component included in the signal is attenuated by applying the smoothing filter, the respective still images constituting the moving image are integrated in the time axis direction, the SN ratio is improved, and the imaging sensitivity is substantially increased. Conventionally, an image composition method for improving the image quality has been provided. This method is also used because the output image is used as one input image for image composition processing, the newly acquired image is used as the other input image, and the processing method of averaging both input images is efficient. . This method is mathematically well known as an infinite impulse response function. A video signal processing device that realizes a high-sensitivity method using this processing method and a method that obtains a telephoto effect by two-dimensionally expanding still images constituting the operation by switching and using a common storage means is provided. (For example, Patent Document 1).

  There is also a synthesis method that achieves the expansion of the apparent dynamic range by continuously acquiring multiple images with different exposure amounts and combining images suitable for reproducing dark areas and images suitable for reproducing bright areas. There is provided an imaging screen composition device that performs image composition by combining the method for realizing the expansion of the apparent dynamic range and the method for correcting the movement between images due to the displacement of the imaging device (for example, Patent Document 2). ).

Further, a video signal that is implemented by performing position correction by calculating a motion vector for each block by dividing the image into a plurality of blocks by the same principle as the method disclosed in Patent Document 1. A processing apparatus is also provided (for example, Patent Document 3).
Japanese Patent No. 2781936 (paragraphs 0084, 0085) Japanese Patent No. 3110797 (paragraphs 0019 to 0022) JP 2000-13643 A (paragraphs 0043 and 0044)

  By the way, in the above infinite impulse response function, past information remains while being attenuated. Therefore, when image synthesis using this infinite impulse response function is performed as in the video signal processing apparatus disclosed in Patent Document 1, the image is displayed. An afterimage such as a tail is generated in a subject moving with respect to the frame. In addition to such an example, when a new image is generated by synthesizing using a plurality of images obtained in time series, the infinite impulse response function described above is caused by the movement of the subject. Then, since past information remains attenuated, if the image synthesis using this infinite impulse response function is performed as in the video signal processing device disclosed in Patent Document 1, the subject moving with respect to the image frame There is an afterimage that has a tail. In addition to such an example, when a new image is generated by combining a plurality of images obtained in time series, there is a problem that some trouble occurs due to the movement of the subject.

  In the imaging screen composition device disclosed in Patent Document 2, motion correction is performed using a single motion vector that represents the entire screen. Therefore, the imaging screen composition device may be inappropriate for an imaging object that moves differently from the background. There was a problem of high nature.

  In addition, as in the video signal processing device disclosed in Patent Document 3, if the points for obtaining the motion vector are increased in detail, the motion vector of each pixel on the image can be obtained. There was a problem of a decrease in accuracy.

  The present invention has been made in view of the above points, and it is an object of the present invention to provide a multiple image composition processing method and a multiple image composition apparatus that can obtain a highly accurate motion vector with a small amount of calculation.

In order to achieve the above object, in the invention of the multiple image composition method according to claim 1, motion correction is performed using motion vectors between a plurality of images including the same subject imaged in time series. A multi-image synthesis method for synthesizing an image with a second image, wherein a rectangular area including a subject is identified on the image by a difference between the plurality of images or face detection processing, and the position of the area If you configure multiple motion vector measurement point to a specific location within the region defined by the size, and performing image motion compensation by combining the motion vector measurement of these motion vector measurement point.

  According to the multiple image composition method of the present invention, the motion vector is measured at a plurality of measurement points arranged in accordance with the area of the target subject on the image, and the motion vector measurement value is combined to perform motion correction. Thus, a highly accurate motion vector can be obtained with a small amount of calculation.

According to a second aspect of the present invention, there is provided a multi-image composition method according to the first aspect, wherein a rectangular subject region is obtained by binarizing a difference image between the first image and the second image used for composition. And a motion vector measurement point is set so as to be dense in the upper half of the area .

According to the invention of the multiple image composition method of the second aspect, it is possible to measure a motion vector corresponding to a human motion accompanied by partial movement or deformation of a limb or the like .

According to a third aspect of the present invention, the face area extracted by the face detection process is set as a subject area, and the face feature points substantially coincide with the area. The motion vector measurement point is set so as to do this.

  According to the multiple image composition method of the present invention, the motion vector can be measured with respect to the movement of the human face.

According to a fourth aspect of the present invention, there is provided a multi-image composition method according to any one of the first to third aspects, wherein the representative value of the motion vector of the entire region of the target subject is obtained by combining the motion vector measurement values at the motion vector measurement points. In other words, the image motion correction is performed using the representative value .

According to the invention of the multiple image composition method of claim 4 , reliability is enhanced by handling a human motion by one motion vector measurement .

According to a fifth aspect of the present invention, there is provided a method for synthesizing a plurality of images according to any one of the first to fourth aspects . The image motion correction is performed using an image whose block matching error with the image is equal to or less than a predetermined threshold value .

According to the invention of claim 5, it is possible to synthesize images having high similarity.

In the invention of the multiple image composition device according to claim 6, the motion correction is performed using the motion vector between the imaging unit and a plurality of images including the same subject imaged in time series by the imaging unit. An image composition processing unit for combining the image and the second image, and the image composition processing unit includes a rectangular region including a subject on the image by a difference between a plurality of images or a face detection process. identified, to configure multiple motion vector measurement point to a specific location within the region defined by the position and size of the area, an image of the motion compensation by combining the motion vector measurement of these motion vector measurement point It has the means to perform, It is characterized by the above-mentioned.

According to the invention of the multiple image composition device of claim 6 , the motion vector is measured at a plurality of measurement points arranged in accordance with the area of the target subject on the image, and the motion vector measurement value is combined to perform motion correction. Thus, it is possible to provide a multi-image synthesis device that can obtain a highly accurate motion vector with a small amount of calculation.

According to a seventh aspect of the present invention, there is provided a multi-image composition apparatus according to the sixth aspect , wherein at least the imaging means is fixed to a building.

According to the invention of claim 7 , since it is not necessary to consider the displacement of the imaging means, the amount of calculation can be reduced.

  The multiple image composition method and the multiple image composition device of the present invention measure a motion vector at a plurality of measurement points arranged in accordance with a region of an object subject on an image, and perform motion correction by combining the motion vector measurement values. Thus, there is an effect that a highly accurate motion vector can be obtained with a small amount of calculation.

Embodiments of the present invention will be described below.
(Embodiment 1)
As shown in FIG. 1 (b), the imaging apparatus of the present embodiment is attached to the ceiling of a building, and the imaging unit 1 that continuously images a predetermined imaging region including the subject Mn in time series, and the imaging unit 1 An image composition processing unit 2 that captures the captured moving image and performs image composition is connected to an image display unit 3 that displays an output image from the image composition processing unit 2.

  As shown in FIG. 1A, the image composition processing unit 2 obtains the image storage means 4 for saving the output image, the saved image (first image) saved in the image storage means 4, and the imaging means 1. Image synthesizing means 5 for synthesizing with the latest acquired image (second image) among a plurality of still images constituting the moving image, attitude control means 6 for controlling the attitude of the imaging means 1, and imaging means Imaging condition control means 7 that controls one imaging condition, and image motion conversion means that converts the attitude information of the imaging means 1 from the attitude control means 6 and the condition information from the imaging condition control means 7 into a motion vector in the image. 8, a motion vector measurement point that sets a motion vector measurement point by referring to the target subject area on the image, and calculates a motion vector between the acquired image and the saved image, and a target subject area Moving body detection means 10 and face detection hand 11 to be composed of.

  Here, each means constituting the image composition processing unit 2 may be configured by a function realized by executing a program in a microcomputer, or may be configured independently by a hardware configuration. . The image composition processing unit 2 is separate from the imaging unit 1 as shown in FIG. 1B, but may be integrated with the imaging unit 1 to constitute an imaging apparatus. Furthermore, the image display means 3 may be integrated into the image pickup apparatus.

  Next, the operation of each unit of the image composition processing unit 2 will be described.

  First, the moving object detection unit 10 creates a difference image between, for example, an acquired image and a saved image to be synthesized, extracts a change area, binarizes the change area, and combines the change areas of the proximity distances into one. A circumscribed rectangle is generated as an integrated region, and for each integrated region, density pattern matching is performed between the acquired image and the saved image, and an integrated region with a low similarity is extracted as a moving object candidate as shown in FIG. This is given to the image motion calculation means 9 as subject area information.

  On the other hand, the face detection unit 11 compares the color histogram generated from the image data with a template histogram of the face part created in advance, for example, using a known technique such as a method for determining the face part area. This face area information is given to the image motion calculating means 9.

  The image motion calculation means 9 sets a plurality of motion vector measurement points for the moving object (target subject area) detected by the moving object detection means 10. Here, when the subject Mn is an article, even if the subject Mn moves within the frame of the captured image, there is no deformation, but when the subject Mn is a person, the limb or the like may be partially moved or deformed. In order to measure the overall movement of a person while avoiding these effects, it is necessary to measure the movement of the face and the torso, so the image motion calculation means 9 is a region set as a moving body (FIG. 2 (a ), The motion vector measurement points (indicated by □ in FIG. 2A) are set so as to be dense in the upper half of the circumscribed rectangular area indicated by the broken line), and the motion vector at each motion vector measurement point is set by the block matching method. As shown by “→” in FIG.

  Further, a representative value is selected from the obtained plurality of information motion vectors. In the selection of the representative value, for example, a motion vector equal to a majority or more in the obtained motion vector is set as the representative value motion vector M1.

  In this embodiment, since a plurality of motion vectors are handled as will be described later, the representative value is not limited to a unique value, and a plurality of motion vector measurement values may be output to the image synthesis means 5.

  The image motion calculation means 9 moves the motion vector measurement points as shown so as to correspond to the shadowed regions such as the eyes, ears, and nose with respect to this face region (inside the rectangle indicated by the broken line in FIG. 2C). (Indicated by □ in FIG. 2C) is set, and the motion vector at each motion vector measurement point is measured by the block matching method as indicated by “→” in FIG. In this case, the mouth periphery is deformed when a word is generated, so the motion vector measurement points around the mouth need to be set closely, but the mouth periphery is not important when determining the displacement of the entire face. It is not necessary to set motion vector measurement points in the periphery. At each motion vector measurement point, an accurate position can be obtained by recognizing facial parts such as eyes, ears, and nose by a method of extracting a facial region. Depending on the required set position accuracy, the relative position of each face part with respect to the face outline may be obtained in advance from the face shape statistical information, and geometrically applied to the rectangle that will be the face outline.

  FIG. 2D shows a motion vector distribution M2 (x, y) <hereinafter referred to as motion vector M2> obtained discretely at each motion vector measurement point set in the target subject region (face region). Assuming that the face is a rigid body without deformation except for the deformed portion of the mouth, the movement of the face between the acquired image and the saved image can be approximated by a primary transformation. However, as shown in the drawing, in order to cope with a state in which the movement is not uniform around the mouth, processing such as data mapping is required.

As shown in FIG. 3, the image pickup means 1 is controlled by the attitude control means 6 with a drive means (not shown) such as a motor, thereby controlling the attitude in the horizontal direction (pan) H and the vertical direction (tilt) V. However, the image motion conversion means 8 takes in the posture information of the imaging means 1 from the posture control means 6 and measures the relative shake angle θ at the time of acquisition of each image used for image synthesis. Of course, when the posture control of the imaging camera is not performed, zero displacement is adopted as the posture information. The image motion conversion unit 8 converts the shake angle θ obtained as described above into a motion vector using condition information including the focal length l of the optical system of the imaging unit 1 obtained from the imaging condition control unit 7. 4A and 4B show conversion examples. In this figure, in order to make the event easy to understand, the coordinates from the center in the image are represented only by x, the above-mentioned deflection angle θ, and the above-mentioned focus. The amount of movement Δx on the image plane α due to the difference in shooting time between the acquired image and the stored image using the distance l is defined as Δx = [(l ² + x ² ) / l] tan θ, which is necessary Accordingly, distortion information from the posture control means 6 is also added to set a motion vector distribution M3 (x, y) <referred to as a motion vector M3>.

  The image synthesizing unit 5 obtains a weighted average of m: n pixel values of the same coordinates on the acquired image and the saved image, outputs an image based on the averaged pixel value as a synthesized image, and outputs the synthesized image The image is stored in the image storage unit 2 and used as a new saved image when processed with a new image acquired from the photographing unit 1. This series of processing can be shown as an infinite length impulse response IIR filter shown in FIG. 5, and this filter acts as a smoothing filter to reduce noise. By the way, the addition ratio of the individual images acquired in time series in the latest composite image is as shown in FIG. 6, and the moving information is blurred because the past information remains so as to have a tail. In addition, the bar graph of the left end in FIG. 6 shows a bar of an image acquired in a form that goes back in the past in the right direction from the right side of the acquired image, and the bar length of this image is the latest composite image, That is, the addition ratio in the stored image is shown.

  Therefore, the image synthesizing unit 5 performs motion correction using the motion vectors M1 and M2 from the image motion calculating unit 9 and the motion vector M3 from the image motion converting unit 8 as motion vector candidates.

  FIG. 7 is a diagram showing the principle of motion correction. In the saved image shown in FIG. 7A and the acquired image shown in FIG. 7B, the subject Mn captured over time has a motion. Here, as shown in FIG. 8D, a processing unit consisting of one or more pixels (□ shown in FIG. 8D) is set on the acquired image, the subject Mn is a person, and the torso of the moving person In the processing unit corresponding to, since the similarity with the acquired image is high in the saved image in which the motion is corrected by the motion vector M1 obtained by moving object detection from the acquired image (FIG. 8D), the motion vector M1 is applied ( FIG. 8 (c)). Further, in the processing unit corresponding to the face of a moving person, the motion vector M2 is applied because the saved image whose motion is corrected by the motion vector M2 obtained by the face detection has high similarity with the acquired image (FIG. 8B). )). Furthermore, in the processing unit corresponding to the background, the saved image that has been motion-corrected by the motion vector M3 obtained by the displacement of the imaging unit 1 has high similarity to the acquired image, and therefore motion correction by the motion vector M3 is applied.

  Next, the image synthesizing unit 5 performs the process of evaluating the similarity between the corrected image after the motion correction and the acquired image as described above before synthesizing the image.

  In this case, the absolute value of the difference between the pixel values between the corrected image and the acquired image is integrated within the processing unit or in the vicinity of the processing unit to obtain a block matching error for evaluation, or in the spatial difference image Similarly, block matching errors are obtained and evaluated. In this case, in the block matching error, the similarity decreases as the value increases.

In this way, through the process of evaluating the similarity, the addition ratio of the pixel values of the saved image and the acquired image at the time of composition is determined based on the evaluation result, and image composition is performed. Here, the ratio of the added value is determined by, for example, comparing the similarity evaluation value for the saved image subjected to motion correction with the motion vector M1 and the similarity evaluation value for the saved image subjected to motion correction with the motion vector M2. If the similarity on the side of the motion vector M1 is high, the acquired image and the saved image corrected with the motion vector M1 are weighted and averaged at a ratio of 1: 3 (see (I) in FIG. 9). Also the stored images when the high similarity of motion vectors M2 side and the motion compensation in the acquired image and motion vector M2 1: taking a weighted average at a rate of 3 ((II) see in Figure 9). Further, if any evaluation value does not exceed a predetermined threshold value set in advance, the pixel value of the saved image is not used and the pixel value of the acquired image is applied as it is (see (III) in FIG. 9). .
(Embodiment 2)
FIG. 10A is a diagram illustrating the relationship between the evaluation result and the image addition ratio in the present embodiment. In the first embodiment, the ratio of the pixel values of the stored image is selected from zero and 3/4 with the threshold as the boundary. In the present embodiment, however, the stored image is selected according to the similarity evaluation value. The ratio of the pixel value is changed stepwise. 10A is divided into a stored image (i) without motion correction, a stored image (iii) with motion correction, and a new acquired image (ii) shown in FIG. 10 (b). Based.

  The present embodiment is the same as the first embodiment except that the image synthesizing unit in FIG. As a result, there is a problem that the boundary between processing units is noticeable due to the difference in processing between adjacent processing units, or that there is a sense of incongruity in isolated processing units when an erroneous selection is made due to sudden noise. Can be reduced.

  Although the addition ratio is set stepwise in FIG. 10A, a continuous value may be set depending on the pixel value calculation method.

Alternatively, the brightness level correction may be performed with reference to the pixel value in the brightness level measurement region set for each of the stored image and the acquired image, and the motion vector candidates may be evaluated after the brightness level correction.
(Embodiment 3)
As shown in FIG. 11B, the photographing apparatus according to the present embodiment is configured by integrating the imaging unit 1 and the image processing unit 2 into a portable housing 12, and the housing 12 is used by the user. U has a structure that is directed in any direction in his hand, no attitude control means for changing the attitude of the imaging means 1, instead of the attitude control means, as shown in FIG. 11 (a), Except that the displacement measuring means 13 of the imaging means 1 is provided, it is the same as the first embodiment shown in FIG. The displacement measuring means 13 measures the angle of the optical axis of the optical system at the time of acquisition of an image used for synthesis by an angular velocity sensor (not shown). If necessary, an acceleration sensor is provided to measure the relative position. To do. In an imaging device that is fixed to a building or a tripod and does not change its orientation, the displacement measuring means 13 is unnecessary, and zero displacement is adopted as information that replaces the output of the displacement measuring means 13.

  The method of using this displacement information is the same as the displacement information of the imaging means 1 output from the attitude control means 6 in the first embodiment.

  Since other configurations are the same as those of the first embodiment, the same components are denoted by the same reference numerals and description thereof is omitted.

(A) is a circuit block diagram of Embodiment 1, (b) is a usage example figure of Embodiment 1. FIG. (A) is a setting example diagram of the motion vector measurement point of the target subject area of the first embodiment, (b) is a measurement result diagram of each motion vector measurement point of the target subject area of the first embodiment, and (c) is the first embodiment. FIG. 6D is a diagram illustrating an example of setting motion vector measurement points in the face area, and FIG. 6D is a measurement result diagram of each motion vector measurement point in the face area according to the first embodiment. It is explanatory drawing of the attitude | position control of the imaging means used for Embodiment 1. FIG. It is explanatory drawing of the example of motion vector conversion of the imaging means used for Embodiment 1. FIG. 6 is an equivalent circuit diagram of an infinite impulse response IIR filter corresponding to the processing operation of the image synthesizing unit of Embodiment 1. FIG. FIG. 6 is an explanatory diagram of an addition ratio between a stored image of a composite image and an acquired image according to Embodiment 1. FIG. 3 is a diagram for explaining the principle of motion correction in the first embodiment. FIG. 3 is a diagram illustrating the relationship between motion correction and each motion vector in the first embodiment. FIG. 10 is an explanatory diagram of similarity of correction by a motion vector according to the second embodiment and a ratio of a stored image and an acquired image in a composite image. FIG. 10 is an explanatory diagram of similarity of correction by a motion vector according to the second embodiment and a ratio of a stored image and an acquired image in a composite image. (A) is a usage example diagram of the third embodiment, (b) is a circuit configuration diagram of the third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging camera 2 Image composition process part 3 Image display means 4 Image storage means 5 Image composition means 6 Attitude control means 7 Imaging condition control means 8 Image motion conversion means 9 Image motion calculation means 10 Moving object detection means 11 Face detection means Mn Subject

Claims (7)

A multi-image synthesis method for synthesizing a first image and a second image that have been motion-corrected using a motion vector between a plurality of images including the same subject captured in time series,
Identify a rectangular region including the object on the image by the difference or the face detection processing between a plurality of images, the multiple motion vector measurement point to a specific location within the region defined by the position and size of the region A multi-image synthesis method comprising: setting and performing motion correction of an image by combining motion vector measurement values at these motion vector measurement points. A rectangular subject area is identified from the binarized difference image between the first image and the second image used for synthesis, and the motion vector measurement points are dense in the upper half of the area. The multi-image composition method according to claim 1, wherein: 3. The face vector extracted by the face detection process is set as a subject area, and a motion vector measurement point is set so as to substantially match a feature point of the face with respect to the area. Multiple image synthesis method. 4. A representative value of a motion vector of the entire area of the target subject is obtained by a combination of motion vector measurement values at a motion vector measurement point, and image motion correction is performed using the representative value. A method for synthesizing multiple images. Among the motion vector measurement values at the motion vector measurement point, the motion correction of the image is performed using a block matching error between the first image after the motion correction and the second image that is equal to or less than a predetermined threshold. The multi-image composition method according to claim 1, wherein the method is performed. The first image and the second image that have been subjected to motion correction are combined using a motion vector between the imaging unit and a plurality of images including the same subject that is captured in time series by the imaging unit. An image composition processing unit, and the image composition processing unit identifies a rectangular region including a subject on the image by a difference between a plurality of images or face detection processing, and is determined by the position and size of the region imaging to configure multiple motion vector measurement point to a specific location within the region, characterized in that it comprises means for performing image motion compensation by combining the motion vector measurement of these motion vector measurement point apparatus. The imaging apparatus according to claim 6, further comprising a structure for fixing at least the imaging unit to a building.

Images