JP4654817B2 - Multiple image composition method and multiple image composition device - Google Patents

Description

  The present invention relates to a multi-image synthesis method and a multi-image synthesis apparatus that synthesize a plurality of images that are captured in time series (continuously in the time axis direction) including the same subject.

  Focusing on the fact that the random noise component included in the signal is attenuated by applying the smoothing filter, the respective still images constituting the moving image are integrated in the time axis direction, the SN ratio is improved, and the imaging sensitivity is substantially increased. Conventionally, an image composition method for improving the image quality has been provided. This method is also used because the output image is used as one input image for image composition processing, the newly acquired image is used as the other input image, and the processing method of averaging both input images is efficient. . This method is mathematically well known as an infinite impulse response function. And there is provided a video signal processing apparatus that realizes a high sensitivity technique by this processing method and a technique for obtaining a telephoto effect by two-dimensionally enlarging a still image constituting an operation by switching a common storage means. (For example, Patent Document 1).

  There is also a synthesis method that achieves the expansion of the apparent dynamic range by continuously acquiring multiple images with different exposure amounts and combining images suitable for reproducing dark areas and images suitable for reproducing bright areas. Conventionally, there has been provided an imaging screen synthesizing apparatus that performs image synthesis by using both a method for realizing the expansion of the apparent dynamic range and a method for correcting a motion between images due to displacement of the imaging device (for example, patents). Reference 2).

Further, a video that is implemented by performing noise reduction processing based on the same principle as the video signal processing device disclosed in Patent Document 1 by dividing an image into a plurality of blocks, calculating a motion vector for each block, and performing position correction. A signal processing apparatus is also provided (for example, Patent Document 3).
Japanese Patent No. 2781936 (paragraphs 0084, 0085) Japanese Patent No. 3110797 (paragraphs 0019 to 0022) JP 2000-13643 A (paragraphs 0043 and 0044)

  By the way, in the above infinite impulse response function, past information remains while being attenuated. Therefore, when image synthesis using this infinite impulse response function is performed as in the video signal processing apparatus disclosed in Patent Document 1, the image is displayed. An afterimage such as a tail is generated in a subject moving with respect to the frame. In addition to such an example, when a new image is generated by combining a plurality of images obtained in time series, there is a problem that some trouble occurs due to the movement of the subject.

  In the imaging screen composition device disclosed in Patent Document 2, motion correction is performed using a single motion vector that represents the entire screen. Therefore, when there are a plurality of subjects moving in different directions, rotation, enlargement / reduction, and the like. There has been a problem that the arrangement cannot be corrected when there is a deformation such as.

  In addition, as in the video signal processing device disclosed in Patent Document 3, if the points for obtaining the motion vector are increased in detail, the motion vector of each pixel on the image can be obtained. However, if the motion vector of the part to be processed is obtained, the effect of motion correction can be improved. However, the shadowed part or the part where the wrong motion vector is calculated breaks down due to the movement of the subject. There was a problem that it would end up.

  The present invention has been made in view of the above-described problems, and the purpose of the present invention is to reduce image defects such as afterimages that cannot be accommodated by motion correction even if there is a complicated subject movement. An object of the present invention is to provide a multi-image composition method and a multi-image composition apparatus that can prevent the image from failing when the image is generated.

In order to achieve the above-mentioned object, in the invention of the multiple image composition method according to claim 1, a plurality of weighted average processing of pixel values is performed between a plurality of images including the same subject imaged in time series. An image composition method, wherein a plurality of motion vector candidates for a subject are calculated between a first image used for composition and another second image used for composition, Correspondence of the processing unit of the second image in the first image with reference to the value of the pixel included in the processing unit currently being processed in the second image for each processing unit composed of one or more pixels The absolute value of the difference from the value of the pixel included in the plurality of processing units whose position has been corrected with a plurality of motion vector candidates from the position is integrated for the region including the processing unit, and the integrated value is obtained as an evaluation value , If any of the evaluated value also exceeds a predetermined threshold value In the processing unit adopts the value of each pixel included in the processing unit of the second image as a composite processing result, in the processing unit when at least one of the evaluation value is equal to or less than the threshold value a second image The weighted average processing is performed using the value of each pixel included in the processing unit and the value of each pixel included in the processing unit on the first image after position correction with the smallest evaluation value. .

  According to the multiple image composition method of the first aspect of the present invention, it is possible to reduce problems such as afterimages even if there is a complicated movement of the subject that cannot be dealt with by correction with a single motion vector. It is possible to prevent the image from failing when the image changes.

In the invention of the multiple image composition method of claim 2, in the invention of claim 1, when the weighted average process is performed, the addition ratio of the second image increases as the evaluation value in the processing unit increases. The weighted average processing for setting the addition ratio for each processing unit so as to increase is performed.

According to the multiple image composition method of the second aspect of the present invention, it is possible to reduce problems such as afterimages even if there is a complicated movement of the subject that cannot be dealt with by correction with a single motion vector, and motion compensation cannot be dealt with. It is possible to prevent image failure when image changes occur, especially when processing unit boundaries are conspicuous due to differences in processing between adjacent processing units, or when erroneous selections are made due to sudden noise. It is possible to reduce inconvenience that an uncomfortable feeling occurs in an isolated processing unit.

According to a third aspect of the present invention, there is provided the invention of claim 1 or 2, wherein one of the motion vector candidates is determined from posture information of the imaging apparatus itself .

According to the invention of the multiple image composition method of the third aspect, it is possible to take into account the displacement of the posture of the imaging means.

According to a fourth aspect of the present invention, there is provided the method for synthesizing a plurality of images according to any one of the first to third aspects, wherein a pixel value in a luminance level measurement region set for each of the first image and the second image is referred to Then, after performing luminance level correction, a difference between pixel values in the processing unit is obtained between the first image and the second image.

According to the fourth aspect of the present invention, motion vector candidates can be evaluated without being affected by the luminance variation because the luminance variation is corrected before the evaluation.

The invention of claim 5 is a multiple image composition method for performing weighted average processing of pixel values between a plurality of images including the same subject imaged in time series. Means for calculating a plurality of motion vector candidates for a subject between a first image used and another second image used for synthesis, and a process comprising a part of the image and one or more pixels For each unit, a plurality of motion vectors from the corresponding position of the processing unit of the second image in the first image with reference to the value of the pixel included in the processing unit currently processed in the second image. the absolute value of the difference between the values of pixels included in the plurality of processing units located corrected by candidate integrates the area including the processing unit, means for determining the integrated value as an evaluation value, none of the evaluation value When a predetermined threshold is exceeded, the processing unit Adopts the value of each pixel included in the processing unit of the second image as a composite processing result, the processing of the second image in the processing unit when at least one of the evaluation value is equal to or less than the threshold value Image synthesizing means for performing weighted average processing based on the value of each pixel included in the unit and the value of each pixel included in the processing unit on the first image after position correction having the smallest evaluation value; It is characterized by that.

According to the invention of claim 5, a plurality of images that can reduce defects such as afterimages even when there is a complicated movement of the subject, and can prevent image failure when an image change that cannot be compensated for motion correction occurs. A synthesizer can be provided.

According to a sixth aspect of the present invention, in the fifth aspect of the invention, when the weighted average processing is performed, the image combining means increases the evaluation value in the processing unit as the second evaluation value . A weighted average process is performed in which the addition ratio is set for each processing unit so that the addition ratio of the image of the image becomes larger.

According to the invention of claim 6 , even if there is a complicated movement of the subject that cannot be dealt with by correction with a single motion vector, it is possible to reduce problems such as afterimages, and image changes that cannot deal with motion correction occur. This is an isolated processing unit especially when the boundary between processing units is conspicuous due to the difference in processing between adjacent processing units, or when an erroneous selection is made partially due to sudden noise. It is possible to provide a multi-image composition device that can reduce the problem of uncomfortable feeling.

  The multi-image composition method and multi-image composition apparatus of the present invention can reduce defects such as afterimages even when there is a complicated movement of a subject that cannot be dealt with by correction using a single motion vector, and motion compensation cannot be handled. There is an effect that it is possible to prevent the failure of the image when the image changes.

Embodiments of the present invention will be described below.
(Embodiment 1)
As shown in FIG. 1 (b), the imaging apparatus of the present embodiment is attached to the ceiling of a building, and the imaging unit 1 that continuously images a predetermined imaging region including the subject Mn in time series, and the imaging unit 1 An image composition processing unit 2 that captures the captured moving image and performs image composition is connected to an image display unit 3 that displays an output image from the image composition processing unit 2.

  As shown in FIG. 1A, the image composition processing unit 2 obtains the image storage means 4 for saving the output image, the saved image (first image) saved in the image storage means 4, and the imaging means 1. Image synthesizing means 5 for synthesizing with the latest acquired image (second image) among a plurality of still images constituting the moving image, attitude control means 6 for controlling the attitude of the imaging means 1, and imaging means Imaging condition control means 7 that controls one imaging condition, and image motion conversion means that converts the attitude information of the imaging means 1 from the attitude control means 6 and the condition information from the imaging condition control means 7 into a motion vector in the image. 8, a motion vector measurement point that sets a motion vector measurement point by referring to the target subject area on the image, and calculates a motion vector between the acquired image and the saved image, and a target subject area Moving body detection means 10 and face detection hand 11 to be composed of.

  Here, each means constituting the image composition processing unit 2 may be configured by a function realized by executing a program in a microcomputer, or may be configured independently by a hardware configuration. . The image composition processing unit 2 is separate from the imaging unit 1 as shown in FIG. 1B, but may be integrated with the imaging unit 1 to constitute an imaging apparatus. Furthermore, the image display means 3 may be integrated into the image pickup apparatus.

  Next, the operation of each unit of the image composition processing unit 2 will be described.

  First, the moving object detection unit 10 creates a difference image between, for example, an acquired image and a saved image to be synthesized, extracts a change area, binarizes the change area, and combines the change areas of the proximity distances into one. A circumscribed rectangle is generated as an integrated region, and for each integrated region, density pattern matching is performed between the acquired image and the saved image, and an integrated region with a low similarity is extracted as a moving object candidate as shown in FIG. This is given to the image motion calculation means 9 as subject area information.

  On the other hand, the face detection unit 11 compares the color histogram generated from the image data with a template histogram of the face part created in advance, for example, using a known technique such as a method for determining the face part area. This face area information is given to the image motion calculating means 9.

  The image motion calculation means 9 sets a plurality of motion vector measurement points for the moving object (target subject area) detected by the moving object detection means 10. Here, when the subject Mn is an article, even if the subject Mn moves within the frame of the captured image, there is no deformation, but when the subject Mn is a person, the limb or the like may be partially moved or deformed. In order to measure the overall movement of a person while avoiding these effects, it is necessary to measure the movement of the face and the torso, so the image motion calculation means 9 is a region set as a moving body (FIG. 2 (a ) Set the motion vector measurement points (indicated by □ in FIG. 2A) so as to be dense in the upper half of the circumscribed rectangle indicated by the broken line in Fig. 2), and block matching the motion vectors at each motion vector measurement point This is obtained by the method as shown by “→” in FIG.

  Further, a representative value is obtained from these obtained motion vectors. The representative value is obtained by, for example, setting a motion vector equal to more than a majority of the obtained motion vectors as the representative value motion vector M1. In this embodiment, since a plurality of motion vectors are handled as will be described later, the representative value is not limited to a unique value, and a plurality of motion vector measurement values may be output to the image synthesis means 5.

  The image motion calculation means 9 moves the motion vector measurement points as shown so as to correspond to the shadowed regions such as the eyes, ears, and nose with respect to this face region (inside the rectangle indicated by the broken line in FIG. 2C). (Indicated by □ in FIG. 2C) is set, and the motion vector at each motion vector measurement point is measured by the block matching method as indicated by “→” in FIG. In this case, the mouth periphery is deformed when a word is generated, so the motion vector measurement points around the mouth need to be set closely, but the mouth periphery is not important when determining the displacement of the entire face. It is not necessary to set motion vector measurement points in the periphery. At each motion vector measurement point, an accurate position can be obtained by recognizing facial parts such as eyes, ears, and nose by a method of extracting a facial region. Depending on the required set position accuracy, the relative position of each face part with respect to the face outline may be obtained in advance from the face shape statistical information, and geometrically applied to the rectangle that will be the face outline.

  FIG. 2D shows a motion vector distribution M2 (x, y) <hereinafter referred to as motion vector M2> obtained at each motion vector measurement point set in the target subject region (face region). Assuming that the face is a rigid body without deformation except for the part, the movement of the face between the acquired image and the saved image can be approximated by a primary transformation. However, as shown in the drawing, in order to cope with a state in which the movement is not uniform around the mouth, processing such as data mapping is required.

As shown in FIG. 3, the image pickup means 1 is controlled by the attitude control means 6 with a drive means (not shown) such as a motor, thereby controlling the attitude in the horizontal direction (pan) H and the vertical direction (tilt) V. However, the image motion conversion means 8 takes in the posture information of the imaging means 1 from the posture control means 6 and measures the relative shake angle θ at the time of acquisition of each image used for image synthesis. Of course, when the posture control of the imaging camera is not performed, zero displacement is adopted as the posture information. The image motion conversion unit 8 converts the shake angle θ obtained as described above into a motion vector using condition information including the focal length l of the optical system of the imaging unit 1 obtained from the imaging condition control unit 7. 4A and 4B show conversion examples. In this figure, in order to make the event easy to understand, the coordinates from the center in the image are represented only by x, the above-mentioned deflection angle θ, and the above-mentioned focus. The amount of movement Δx on the image plane α due to the difference in shooting time between the acquired image and the stored image using the distance l is defined as Δx = [(l ² + x ² ) / l] tan θ, which is necessary Accordingly, distortion information from the posture control means 6 is also added to set a motion vector distribution M3 (x, y) <referred to as a motion vector M3>.

  The image synthesizing unit 5 obtains a weighted average of m: n pixel values of the same coordinates on the acquired image and the saved image, outputs an image based on the averaged pixel value as a synthesized image, and outputs the synthesized image The image is stored in the image storage unit 2 and used as a new saved image when processed with a new image acquired from the photographing unit 1. This series of processing can be shown as an infinite length impulse response IIR filter shown in FIG. 5, and this filter acts as a smoothing filter to reduce noise. By the way, the addition ratio of the individual images acquired in time series in the latest composite image is as shown in FIG. 6, and the moving information is blurred because the past information remains so as to have a tail. In addition, the bar graph of the left end in FIG. 6 shows a bar of an image acquired in a form that goes back in the past in the right direction from the right side of the acquired image, and the bar length of this image is the latest composite image, That is, the addition ratio in the stored image is shown.

  Therefore, the image synthesizing unit 5 performs motion correction using the motion vectors M1 and M2 from the image motion calculating unit 9 and the motion vector M3 from the image motion converting unit 8 as motion vector candidates.

  FIG. 7 is a diagram showing the principle of motion correction. In the saved image shown in FIG. 7A and the acquired image shown in FIG. 7B, the subject Mn captured over time has a motion. Here, as shown in FIG. 8D, a processing unit consisting of one or more pixels (□ shown in FIG. 8D) is set on the acquired image, the subject Mn is a person, and the torso of the moving person In the processing unit corresponding to, since the similarity with the acquired image is high in the saved image in which the motion is corrected by the motion vector M1 obtained by moving object detection from the acquired image (FIG. 8D), the motion vector M1 is applied ( FIG. 8 (c)). Further, in the processing unit corresponding to the face of a moving person, the motion vector M2 is applied because the saved image whose motion is corrected by the motion vector M2 obtained by the face detection has high similarity with the acquired image (FIG. 8B). )). Furthermore, in the processing unit corresponding to the background, the saved image that has been motion-corrected by the motion vector M3 obtained by the displacement of the imaging unit 1 has high similarity to the acquired image, and therefore motion correction by the motion vector M3 is applied.

  Next, the image synthesizing unit 5 performs the process of evaluating the similarity between the corrected image after the motion correction and the acquired image as described above before synthesizing the image.

  In this case, the absolute value of the difference between the pixel values between the corrected image and the acquired image is integrated within the processing unit or in the vicinity of the processing unit to obtain a block matching error for evaluation, or in the spatial difference image Similarly, block matching errors are obtained and evaluated. In this case, in the block matching error, the similarity decreases as the value increases.

In this way, through the process of evaluating the similarity, the addition ratio of the pixel values of the saved image and the acquired image at the time of composition is determined based on the evaluation result, and image composition is performed. Here, the ratio of the added value is determined by, for example, comparing the similarity evaluation value for the saved image subjected to motion correction with the motion vector M1 and the similarity evaluation value for the saved image subjected to motion correction with the motion vector M2. If the similarity on the side of the motion vector M1 is high, the acquired image and the saved image corrected with the motion vector M1 are weighted and averaged at a ratio of 1: 3 (see (I) in FIG. 9). Also the stored images when the high similarity of motion vectors M2 side and the motion compensation in the acquired image and motion vector M2 1: taking a weighted average at a rate of 3 ((II) see in Figure 9). Further, if any evaluation value does not exceed a predetermined threshold value set in advance, the pixel value of the saved image is not used and the pixel value of the acquired image is applied as it is (see (III) in FIG. 9). .
(Embodiment 2)
FIG. 10A is a diagram illustrating the relationship between the evaluation result and the image addition ratio in the present embodiment. In the first embodiment, the ratio of the pixel values of the stored image is selected from zero and 3/4 with the threshold as the boundary. In the present embodiment, however, the stored image is selected according to the similarity evaluation value. The ratio of the pixel value is changed stepwise. 10A is divided into a stored image (i) without motion correction, a stored image (iii) with motion correction, and a new acquired image (ii) shown in FIG. 10 (b). Based.

  The present embodiment is the same as the first embodiment except that the image synthesizing unit in FIG. As a result, there is a problem that the boundary between processing units is noticeable due to the difference in processing between adjacent processing units, or that there is a sense of incongruity in isolated processing units when an erroneous selection is made due to sudden noise. Can be reduced.

  Although the addition ratio is set stepwise in FIG. 10A, a continuous value may be set depending on the pixel value calculation method.

Alternatively, the brightness level correction may be performed with reference to the pixel value in the brightness level measurement region set for each of the stored image and the acquired image, and the motion vector candidates may be evaluated after the brightness level correction.
(Embodiment 3)
As shown in FIG. 11B, the photographing apparatus according to the present embodiment is configured by integrating the imaging unit 1 and the image processing unit 2 into a portable housing 12, and the housing 12 is used by the user. U has a structure that is directed in any direction in his hand, no attitude control means for changing the attitude of the imaging means 1, instead of the attitude control means, as shown in FIG. 11 (a), Except that the displacement measuring means 13 of the imaging means 1 is provided, it is the same as the first embodiment shown in FIG. The displacement measuring means 13 measures the angle of the optical axis of the optical system at the time of acquisition of an image used for synthesis by an angular velocity sensor (not shown). If necessary, an acceleration sensor is provided to measure the relative position. . In an imaging device that is fixed to a building or a tripod and does not change its orientation, the displacement measuring means 13 is unnecessary, and zero displacement is adopted as information that replaces the output of the displacement measuring means 13.

  The method of using this displacement information is the same as the displacement information of the imaging means 1 output from the attitude control means 6 in the first embodiment.

  Since other configurations are the same as those of the first embodiment, the same components are denoted by the same reference numerals and description thereof is omitted.

(A) is a circuit block diagram of Embodiment 1, (b) is a usage example figure of Embodiment 1. FIG. (A) is a setting example diagram of the motion vector measurement point of the target subject area of the first embodiment, (b) is a measurement result diagram of each motion vector measurement point of the target subject area of the first embodiment, and (c) is the first embodiment. FIG. 6D is a diagram illustrating an example of setting motion vector measurement points in the face area, and FIG. 6D is a measurement result diagram of each motion vector measurement point in the face area of the first embodiment. It is explanatory drawing of the attitude | position control of the imaging means used for Embodiment 1. FIG. It is explanatory drawing of the example of motion vector conversion of the imaging means used for Embodiment 1. FIG. 6 is an equivalent circuit diagram of an infinite impulse response IIR filter corresponding to the processing operation of the image synthesizing unit of Embodiment 1. FIG. FIG. 6 is an explanatory diagram of an addition ratio between a stored image of a composite image and an acquired image in the first embodiment. FIG. 3 is a diagram for explaining the principle of motion correction in the first embodiment. FIG. 3 is a diagram illustrating the relationship between motion correction and each motion vector in the first embodiment. FIG. 10 is an explanatory diagram of similarity of correction by a motion vector according to Embodiment 2 and a ratio of a stored image and an acquired image in a composite image. FIG. 10 is an explanatory diagram of similarity of correction by a motion vector according to the second embodiment and a ratio of a stored image and an acquired image in a composite image. (A) is a usage example diagram of the third embodiment, (b) is a circuit configuration diagram of the third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging camera 2 Image composition process part 3 Image display means 4 Image storage means 5 Image composition means 6 Attitude control means 7 Imaging condition control means 8 Image motion conversion means 9 Image motion calculation means 10 Moving object detection means 11 Face detection means Mn Subject

Claims (6)

A multiple image synthesis method for performing weighted averaging of pixel values between a plurality of images including the same subject imaged in time series,
Calculating a plurality of motion vector candidates between the first image used for composition and another second image used for composition;
For each processing unit that is a part of the image and that includes one or more pixels, the second value in the first image is determined based on the value of the pixel included in the processing unit that is the current processing target in the second image. The absolute values of the differences from the values of the pixels included in the plurality of processing units whose positions have been corrected by the plurality of motion vector candidates from the corresponding position of the processing unit of the image for the region including the processing unit, and the integration Find the value as the evaluation value ,
When any evaluation value exceeds a predetermined threshold value, the processing unit adopts the value of each pixel included in the processing unit of the second image as a synthesis processing result, and at least one of the evaluation values is equal to or lower than the threshold value. In this case, in the processing unit, the value of each pixel included in the processing unit of the second image and each pixel included in the processing unit on the first image after position correction having the smallest evaluation value A multi-image synthesis method, wherein weighted averaging is performed according to a value. When performing the weighted average processing, weighted average processing is performed in which the addition ratio is set for each processing unit so that the addition ratio of the second image increases as the evaluation value in the processing unit increases. The multi-image synthesis method according to claim 1. 3. The multiple image synthesis method according to claim 1, wherein one of the motion vector candidates is determined from posture information of the imaging apparatus itself. After performing brightness level correction with reference to the pixel value in the brightness level measurement region set in each of the first image and the second image, the processing unit in the first image and the second image 4. The method for synthesizing a plurality of images according to any one of claims 1 to 3, wherein a difference between pixel values is obtained . A multiple image synthesis method for performing weighted averaging of pixel values between a plurality of images including the same subject imaged in time series,
Means for calculating a plurality of motion vector candidates for a subject between a first image used for composition and another second image used for composition;
For each processing unit that is a part of the image and that includes one or more pixels, the second value in the first image is determined based on the value of the pixel included in the processing unit that is the current processing target in the second image. The absolute values of the differences from the values of the pixels included in the plurality of processing units whose positions have been corrected by the plurality of motion vector candidates from the corresponding position of the processing unit of the image for the region including the processing unit, and the integration Means for obtaining a value as an evaluation value;
When any evaluation value exceeds a predetermined threshold value, the processing unit adopts the value of each pixel included in the processing unit of the second image as a synthesis processing result, and at least one of the evaluation values is equal to or lower than the threshold value. In this case, in the processing unit, the value of each pixel included in the processing unit of the second image and each pixel included in the processing unit on the first image after position correction having the smallest evaluation value A multi-image synthesizing apparatus comprising: an image synthesizing unit that performs weighted average processing according to values . When performing the weighted average process, the image composition unit sets an addition ratio for each processing unit so that the addition ratio of the second image increases as the evaluation value in the processing unit increases. 6. The multi-image composition apparatus according to claim 5, wherein weighted average processing is performed .

Images