JP2007036742A - Method and apparatus for synthesizing multiple images - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and apparatus for synthesizing multiple images which can avoid drawbacks such as blur or an after-image generated by a change in a subject when synthesizing multiple images consecutively picked up in time series. <P>SOLUTION: When information about the change in the subject from a motion detecting means 10 or a voice information acquiring means 11 changes when an image synthesizing means 5 is in a first operation state with image synthesizing, a processing method determining means 12 executes determination for shifting the means 5 to a second operation state without image synthesizing. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a multi-image synthesis method and a multi-image synthesis apparatus that synthesize a plurality of images that are captured in time series (continuously in the time axis direction) including the same subject.

Focusing on the fact that the random noise component included in the signal is attenuated by applying the smoothing filter, the respective still images constituting the moving image are integrated in the time axis direction, the SN ratio is improved, and the imaging sensitivity is substantially increased. Conventionally, an image composition method for improving the image quality has been provided. This method is also used because the output image is used as one input image for image composition processing, the newly acquired image is used as the other input image, and the processing method of averaging both input images is efficient. . This method is mathematically well known as an infinite impulse response function. There has been provided a video signal processing apparatus using a high sensitivity technique by this processing method (for example, Patent Document 1).

  In video equipment that handles moving images, such as VCRs and TV monitors, there is a method that achieves noise reduction by averaging signals (pixel values) of pixels in the same position in old and new frames. This is known as “three-dimensional digital noise reduction” combined with a two-dimensional smoothing filter in a still image, and a video signal processing apparatus using this three-dimensional digital noise reduction is provided (for example, Patent Document 2).

Further, a composite processing method that continuously acquires a plurality of images having different exposure amounts and combines an image suitable for reproducing a dark portion and an image suitable for reproducing a bright portion, thereby realizing an expansion of an apparent dynamic range. Also, there has been provided an imaging screen composition device that performs image composition using a method that realizes the expansion of the apparent dynamic range (for example, Patent Document 3).
Japanese Patent No. 2781936 (paragraphs 0084, 0085) JP 2000-13643 A (paragraphs 0043 and 0044) Japanese Patent No. 3110797 (paragraphs 0019 to 0022)

  By the way, in the above infinite impulse response function, past information remains while being attenuated. Therefore, when image synthesis using this infinite impulse response function is performed as in the video signal processing apparatus disclosed in Patent Document 1, the image is displayed. An afterimage such as a tail is generated in a subject moving with respect to the frame. In addition to such an example, when a new image is generated by combining a plurality of images obtained in time series, there is a problem that some trouble occurs due to the movement of the subject.

  Further, in the imaging screen composition device disclosed in Patent Document 3, since motion correction is performed using a single motion vector that represents the entire screen, there may be a plurality of subjects moving in different directions, rotation, and enlargement. There is a problem that the arrangement cannot be corrected when there is a deformation such as reduction. In addition, when calculating the motion vector by comparing two images, if you increase the number of motion vector points, you can determine the motion vector of each pixel on the image, but the processing time increases and the local accuracy There is a problem of decline.

  The present invention has been made in view of the above-described points, and the object of the present invention is, for example, blurring or afterimages caused by a change in a subject when a plurality of images continuously captured in time series are combined. It is an object of the present invention to provide a multiple image composition processing method and a multiple image composition apparatus that can avoid the above-mentioned problems.

  In order to achieve the above object, in the invention of the multiple image composition processing method of claim 1, an image that includes the same subject and that calculates a pixel value between a plurality of images that are continuously captured in time series. In the multiple image composition method for obtaining a composite image by composition processing, when there is a change in information regarding the change of the subject in the first operation state involving the image composition processing, the second without the image composition processing. It is characterized by shifting to the operating state.

  According to the multiple image composition processing method of the first aspect of the present invention, the process of generating a composite image using a plurality of images that are continuously captured in time series is performed during image synthesis. If there is a change in information relating to a change in the subject that causes a problem, an operation state in which the image composition processing is not performed is performed, so that problems such as blurring and afterimages can be avoided.

  According to a second aspect of the invention, there is provided the second image processing method according to the first aspect, wherein the information regarding the change in the subject is an imaging condition of the subject, and the second operation is performed when the change in the imaging condition is equal to or greater than a predetermined value. It is characterized by shifting to a state.

  According to the second aspect of the present invention, it is possible to avoid a case where a malfunction occurs due to a change in imaging conditions.

  According to a third aspect of the invention, there is provided the multiple image composition processing method according to the first aspect, wherein the information on the subject change is an audio signal acquired simultaneously with a moving image in which the subject is imaged. When the value is equal to or greater than a predetermined value, the second operation state is entered.

  According to the multiple image composition processing method of the third aspect of the present invention, it is possible to avoid the occurrence of problems when the mouth of a subject made up of people changes.

  According to a fourth aspect of the invention, there is provided the multiple image composition processing method according to the first aspect of the invention, wherein the information related to the change of the subject is a detection signal from a moving body detecting unit that detects the movement of an object within the imaging range including the subject. When there is a detection signal indicating that a moving object is detected from the moving object detection means, the second operation state is entered.

  According to the invention of the multiple image composition processing method of the fourth aspect, it is possible to avoid the occurrence of problems when the subject moves.

  According to a fifth aspect of the present invention, there is provided the multiple image composition processing method according to the first aspect of the present invention, wherein the information related to the change in the subject is the luminance of a moving image obtained by imaging the subject, It shifts to the 2nd operation state, It is characterized by the above-mentioned.

  According to the multiple image composition processing method of the present invention, inconvenience when the brightness around the subject changes can be avoided.

  According to a sixth aspect of the present invention, there is provided the multiple image composition processing method according to the first aspect, wherein the information related to the change in the subject is a motion amount or a change amount of a moving image obtained by imaging the subject. If is greater than or equal to a predetermined value, the second operation state is entered.

  According to the sixth aspect of the present invention, it is possible to avoid the occurrence of troubles due to miscalculation or the like when the similarity is low, the shift amount is large, or the moving image is synthesized.

  8. The multi-image composition apparatus according to claim 7, wherein the image pickup means for picking up a predetermined range including the subject, the image storage means for storing the output image, the newly picked up image and the image storage means. An image composition unit that calculates a pixel value of the output image and combines both images, a change information acquisition unit that acquires information about changes in the subject, and a change in the information acquired by the change information acquisition unit Accordingly, the image processing apparatus includes processing method determination means for determining whether the operation state is a first operation state involving image composition processing by the image composition means or a second operation state not involving image composition processing.

  According to the invention of the multiple image composition device according to claim 7, there is a problem in the image composition during the continuous execution of the process of generating the composite image using the plurality of images continuously captured in time series. If there is a change in the information related to the change in the subject that causes an image, an operation state in which the image composition processing is not performed is performed, so that problems such as blurring and afterimages can be avoided.

  The multi-image composition method and multi-image composition device of the present invention have a problem in image composition while continuously performing a process of generating a composite image using a plurality of images that are continuously captured in time series. If there is a change in the information related to the change in the subject that causes an image, an operation state in which the image composition processing is not performed is performed, and thus there is an effect that problems such as blurring and afterimages can be avoided.

Embodiments of the present invention will be described below.
(Embodiment 1)
As shown in FIG. 1 (b), the imaging apparatus of the present embodiment is attached to the ceiling of a building, and the imaging unit 1 that continuously images a predetermined imaging region including the subject Mn in time series, and the imaging unit 1 An image composition processing unit 2 that captures the captured moving image and performs image composition is connected to an image display unit 3 that displays an output image from the image composition processing unit 2.

  As shown in FIG. 1A, the image composition processing unit 2 includes an image storage unit 4 that stores an output image, a plurality of images that are stored in the image storage unit 4 and a moving image obtained by the imaging unit 1. Image synthesizing means 5 for synthesizing the latest acquired image with the latest acquired image, attitude control means 6 for controlling the attitude of the imaging means 1, and imaging condition control means 7 for controlling the imaging conditions of the imaging means 1. The image motion conversion means 8 for converting the motion information in the image from the posture information of the image pickup means 1 from the posture control means 6 and the condition information from the image pickup condition control means 7, and the target subject area on the image. The motion vector measuring point is set, the motion vector calculating means 9 for calculating the motion vector between the acquired image and the saved image, the moving object detecting means 10 for capturing the change of the subject as the motion of the subject, and the change of the subject by voice. To capture Based on the output of the change information acquisition means comprising the audio information acquisition means 11, the image motion conversion means 8, the image motion calculation means 9, the moving object detection means 10, and the audio information acquisition means 11. It is comprised with the processing method judgment means 12 which judges whether it performs.

  Here, each means constituting the image composition processing unit 2 may be configured by a function realized by executing a program in a microcomputer, or may be configured independently by a hardware configuration. . The image composition processing unit 2 is separate from the imaging unit 1 as shown in FIG. 1B, but may be integrated with the imaging unit 1 to constitute an imaging apparatus. Furthermore, the image display means 3 may be integrated into the image pickup apparatus.

  Next, the operation of each unit of the image composition processing unit 2 will be described.

  First, the image motion calculation unit 9 sets a motion vector measurement point for the target subject region extracted from the image, and obtains a newly acquired image acquired from the imaging unit 1 and a stored image stored in the image storage unit 5. The motion vector is calculated. FIG. 3 shows an example of a motion vector obtained by the image motion calculating means 9, FIG. 3 (a) shows a saved image, FIG. 4 (b) shows an acquired image, and the target subject extracted with the saved image. A plurality of motion vector measurement points are set on the Mn image, and a plurality of motion vectors are measured by the block matching method at the motion vector measurement points (indicated by “→” in FIG. 2C). The measurement result is output to the processing method determination unit 12 and the image composition unit 5. In addition, when the difference of each motion vector between an acquisition image and a preservation | save image is small, the motion which has arisen in the image is close | similar uniformly, and when the difference is large, it means that a motion changes with places. In the latter case, there are portions that appear behind or appear behind moving objects between images with different acquisition times.

  As shown in FIG. 3, the image pickup means 1 is controlled by the attitude control means 6 with a drive means (not shown) such as a motor, thereby controlling the attitude in the horizontal direction (pan) H and the vertical direction (tilt) V. However, the image motion conversion means 8 takes in the posture information of the imaging means 1 from the posture control means 6 and measures the relative shake angle θ at the time of acquisition of each image used for image synthesis. Of course, when the posture control of the imaging camera is not performed, zero displacement is adopted as the posture information.

The image motion conversion unit 8 converts the shake angle θ obtained as described above into a motion vector using condition information including the focal length l of the optical system of the imaging unit 1 obtained from the imaging condition control unit 7. 4A and 4B show conversion examples. In this figure, in order to make the event easy to understand, the coordinates from the center in the image are represented only by x, the above-mentioned deflection angle θ, and the above-mentioned focus. The amount of movement Δx on the image plane α due to the difference in shooting time between the acquired image and the stored image using the distance l is defined as Δx = [(l ² + x ² ) / l] tan θ, which is necessary In response to this, distortion information from the posture control means 6 is also added to measure the motion vector distribution. The measurement result is output to the processing method determination unit 12 and the image composition unit 5.

  The image synthesizing unit 5 performs motion correction on the stored image using the motion vector from the image motion calculating unit 9 and the motion vector distribution from the image motion converting unit 8 as motion vector candidates, and then stores the acquired image and the stored image. M: n weighted average of pixel values of the same coordinates on the image with the image is obtained, an image based on the averaged pixel value is output as a composite image, and the output composite image is stored in the image storage means 2; When a new image acquired from the photographing means 1 is processed, it is used as a new saved image. This series of processing can be shown as an infinite length impulse response IIR filter shown in FIG. 5, and this filter acts as a smoothing filter to reduce noise. By the way, the addition ratio of the individual images acquired in time series in the latest composite image is as shown in FIG. 6, and the moving information is blurred because the past information remains so as to have a tail. In addition, the bar graph of the left end in FIG. 6 shows a bar of an image acquired in a form that goes back in the past in the right direction from the right side of the acquired image, and the bar length of this image is the latest composite image, That is, the addition ratio in the stored image is shown.

  As shown in FIG. 7A, the moving body detection unit 10 includes the imaging unit 1 regardless of the image acquired by the imaging unit 1 using heat rays (infrared rays) generated by a subject Mn made of a person or reflected ultrasonic waves. Detects a moving object (subject Mn) within a shooting range where the is facing, and outputs a detection signal to the processing method determination means 12 as a signal indicating a change in the subject Mn.

  The audio information acquisition unit 11 including a microphone or the like acquires audio information in a direction in which the imaging unit 1 is directed before and after the time when an image is acquired from the imaging unit 1 as shown in FIG. Yes, audio information is output to the processing method determination means 12 as a signal for detecting a change in the subject Mn.

  The processing method determination unit 12 adds a pixel value of the acquired image and the saved image by the image synthesis unit 5 to perform image synthesis and outputs a synthesized image, and the pixel of the acquired image without performing image synthesis This is a processing method determination for instructing the image composition means 5 to select one of the second operation states for outputting an image using the value as it is, and has the following functions.

  That is, when a moving object detection signal is input from the moving object detection means 10, it is determined that image combining is not performed in order to prevent image combining problems caused by movement of the subject Mn, and the determination result is used as the image combining. When the voice information is included in the voice information from the voice information acquisition means 11 and the function to be outputted to the means 5, the person who is the subject Mn to be photographed utters a voice by deforming the mouth. A function of judging that there is a possibility that the image composition is not performed in order to prevent the occurrence of a malfunction of the image composition due to the deformation of the subject Mn, and outputting the judgment result to the image composition means 5; A function is provided for determining that image composition is not performed even when any of the following applies to a plurality of motion vectors obtained by the motion calculation means 7.

  Here, it is determined that image synthesis is not performed for a plurality of motion vectors in any of the following cases. That is, when the similarity of the measured motion vectors is lower than the predetermined condition, the occurrence of a malfunction in the shadow part (arm or head) Mn ′ as shown in FIG. When the motion vector representing a plurality of motion vectors exceeds a predetermined size and the amount of deviation is too large to reduce the effective portion of the composition, the motion vector representing the plurality of motion vectors and the image motion conversion means 8 And the difference between the motion vectors obtained in (1) exceeds a predetermined size, and it is estimated that there is a high possibility that the image motion calculation means 8 has erroneously calculated the motion vector. Note that the above-described method of obtaining a representative motion vector is, for example, a motion vector equal to a majority or more in the obtained motion vector as a representative value motion vector.

  Thus, according to the present embodiment, the processing method determining means 12 determines that image composition is not performed when a problem occurs in image composition due to a change in the subject Mn, and the image composition processing in the image composition means 5 is performed. Since the image using the pixel value of the acquired image as it is is output without performing the above-described processing, problems such as blurring and afterimage can be avoided.

  Further, the moving body detection means 10 is not limited to a means using heat rays or ultrasonic waves, and any means capable of detecting a moving body may be used. For example, an image sensor may be used.

  In addition, imaging conditions such as exposure, angle, and color balance variation may be used as conditions for determining state transition. Further, a change in brightness such as illumination, that is, a change in luminance may be used as a condition.

(Embodiment 2)
As shown in FIG. 8A, the photographing apparatus according to the present embodiment is configured by integrating the imaging unit 1 and the image processing unit 2 into a portable housing 14, and the housing 14 is used by the user. U is configured to be held in a hand and directed in an arbitrary direction, and does not have an attitude control means for changing the attitude of the imaging means 1, but instead of the attitude control means as shown in FIG. Except that the displacement measuring means 13 of the imaging means 1 is provided, it is the same as the first embodiment shown in FIG. The displacement measuring means 13 measures the angle of the optical axis of the optical system at the time of acquisition of an image used for synthesis by an angular velocity sensor (not shown). If necessary, an acceleration sensor is provided to measure the relative position. . In an imaging device that is fixed to a building or a tripod and does not change its orientation, the displacement measuring means 13 is unnecessary, and zero displacement is adopted as information that replaces the output of the displacement measuring means 13.

  The method of using this displacement information is the same as the displacement information of the imaging means 1 output from the attitude control means 6 in the first embodiment.

  Since other configurations are the same as those of the first embodiment, the same components are denoted by the same reference numerals and description thereof is omitted.

(A) is a circuit block diagram of Embodiment 1, (b) is a usage example figure of Embodiment 1. FIG. FIG. 6 is an explanatory diagram for measuring a motion vector of a target subject area according to the first embodiment. It is explanatory drawing of the attitude | position control of the imaging means used for Embodiment 1. FIG. It is explanatory drawing of the example of motion vector conversion of the imaging means used for Embodiment 1. FIG. 6 is an equivalent circuit diagram of an infinite impulse response IIR filter corresponding to the processing operation of the image synthesizing unit of Embodiment 1. FIG. FIG. 6 is an explanatory diagram of an addition ratio between a stored image of a composite image and an acquired image in Embodiment 1. (A) is operation | movement explanatory drawing of the moving body detection means in Embodiment 1, (b) is operation | movement explanatory drawing of an audio | voice information acquisition means. (A) is a usage example diagram of the second embodiment, (b) is a circuit configuration diagram of the third embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging camera 2 Image composition process part 3 Image display means 4 Image storage means 5 Image composition means 6 Attitude control means 7 Imaging condition control means 8 Image motion conversion means 9 Image motion calculation means 10 Moving object detection means 11 Audio | voice information acquisition means 12 Process Method determination means Mn Subject

Claims (7)

In a multiple image composition method for obtaining a composite image by image composition processing that calculates pixel values between a plurality of images that include the same subject and are continuously captured in time series, the first method that involves the image composition processing When there is a change in information related to the change in the subject while in the operating state, the multi-image combining method is shifted to the second operating state without the image combining process. 2. The multi-image composition method according to claim 1, wherein the information related to the change in the subject is an imaging condition of the subject, and the transition to the second operation state is made when the change in the imaging condition is greater than or equal to a predetermined value. The information regarding the change in the subject is an audio signal acquired simultaneously with a moving image obtained by imaging the subject, and the change to the second operation state is made when the change in the audio signal is equal to or greater than a predetermined value. A method for synthesizing multiple images. The information related to the change in the subject is a detection signal from a moving object detection unit that detects the movement of an object within the imaging range including the subject, and there is a detection signal indicating that the moving object is detected from the moving object detection unit. And a transition to the second operating state. 2. The method of synthesizing multiple images according to claim 1. 2. The multiple images according to claim 1, wherein the information related to the change in the subject is a luminance of a moving image obtained by imaging the subject, and the change to the second operation state is performed if the luminance change is equal to or greater than a predetermined value. Synthesis method. The information regarding the change of the subject is a motion amount or a change amount of a moving image obtained by imaging the subject, and if the motion amount or the change amount is equal to or greater than a predetermined value, the process shifts to the second operation state. 2. The method for synthesizing multiple images according to 1. An image pickup means for picking up a predetermined range including a subject, an image storage means for storing an output image, and calculating both pixel values of a newly picked up image and an output image stored in the image storage means An image composition means for performing the composition processing, a change information obtaining means for obtaining information relating to a change in the subject, and an image composition processing by the image composition means according to the degree of change in the information obtained by the change information obtaining means. A multi-image composition apparatus, comprising: a processing method judging means for judging whether the first operation state or the second operation state without image composition processing.

Images