JP2007259106A - Method of detecting moving object in picked-up image and apparatus thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain the position of a moving object and a motion vector in a picked-up image at high speed and with less operation quantity. <P>SOLUTION: Image of a first screen and a second screen are each divided into a plurality of same regions, a correlation value between a reference image segmented from the first screen and each reference image segmented from the second screen while shifting a position is obtained for each divided region, motion vectors k1-k16 are obtained for each of the divided regions from the difference between the position of a reference image indicating the maximum value of the correlation value and the position of a reference image in the divided region, the motion vectors for each of the divided regions are differentiated into first type motion vectors k1, k2, ... which come into the same direction and the same length within a required range, and second type motion vectors k9, k15 which do not come into the required range, and it is determined that an image of the moving object exists in the position of the divided region where the second type motion vectors k9, k15 are calculated. A mean value of the first type motion vectors is obtained, and a difference vector between the second type motion vectors k9, k15 and the mean value is regarded as the motion vector of the image of the moving object. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a moving object detection method and apparatus for detecting a moving object in a captured image.

  For example, when a moving picture is shot with a digital still camera or a video camera having a moving picture shooting function and a moving image is compressed in the MPEG (Moving Picture Expert Group) format, a moving area in the screen is defined as a stationary area. It is necessary to distinguish.

  FIG. 6 is a diagram for explaining the principle of motion detection. If the image in the block 2 indicated by the predetermined address in the image 1 of the Nth frame shown in FIG. 6A is a reference image, the same predetermined in the image 3 of the (N + 1) th frame shown in FIG. 6B. If the comparison image cut out by the block 4 indicated by the address is the same as the reference image, the image in the block 2 is not moved.

  However, if the image captured in the block 2 is moving, this image does not exist in the block 4 in the N + 1 frame image. Therefore, the blocks 4a, 4b, 4c,... Are shifted in the X direction (horizontal direction) and the Y direction (vertical direction) by one pixel unit or several pixel units in the N + 1 frame image. 4c,... Are compared with the reference image (the image in block 2), and the block position where the comparison image having the highest correlation with the reference image is cut out is obtained.

  In the example shown in FIG. 6B, if the comparison image having the highest correlation with the reference image of block 2 is cut out at block 4c, the movement amount and the movement direction of the movement of the reference image in block 2 are as follows. It can be represented by the vector K.

  Incidentally, as a conventional technique related to motion detection, there is the following Patent Document 1.

JP 2005-228273 A

  The calculation for obtaining the correlation between the reference image and the comparison image is performed by comparing one pixel and one pixel of the reference image and the comparison image, and calculating the minimum value of the sum (total value) of the total pixels in the block of the difference (mismatch value) between the pixel data. Since it is performed by calculating, the calculation load is high and the time required for the calculation also becomes long.

  In addition, since it is not known where in the image 1 the moving image exists, the block 2 for extracting the reference image also needs to be moved in units of one pixel or several pixels, and a comparison for comparing each reference image Since the cut-out block of the image is also moved in units of one pixel or several pixels, unless a high-performance and high-cost arithmetic processing unit is used, it is not in time to sequentially compress the images that are input one after another.

  An object of the present invention is to provide a method and apparatus for detecting a moving object in a captured image, which can detect the position of the moving object and its motion vector in the captured image at high speed with a small amount of computation.

  The method and apparatus for detecting a moving object in a captured image according to the present invention divides each image of the first screen and the second screen into the same plurality of regions, and extracts a reference image from the first screen for each divided region. And the respective correlation images cut out while shifting the position from the second screen, and for each divided area, the position of the comparative image showing the highest correlation value and the position in the divided area A movement vector is calculated from the difference from the position of the reference image, and the movement vector for each divided region is the first type of movement vector that falls in the same direction and length within the required range and the first movement vector that does not fall within the required range. A distinction is made between two types of movement vectors, and it is determined that a moving body image exists at the position of the divided area where the second type of movement vector is calculated.

  The method and apparatus for detecting a moving object in a captured image according to the present invention obtains an average value of the first type of movement vector and uses a difference vector between the second type of movement vector and the average value as a motion of the moving object image. It is characterized by being a vector.

  The method and apparatus for detecting a moving object in a captured image according to the present invention evaluates a reference image cut out from each of the divided regions, and the movement vector of the divided region where a reference image whose evaluation value does not reach a predetermined threshold exists. Is stopped, and the movement vector of the divided region where the reference image equal to or greater than the predetermined threshold exists is calculated.

  In the method and apparatus for detecting a moving object in a captured image according to the present invention, the evaluation value is calculated by any one or a combination of a dynamic range evaluation value, a sharpness evaluation value, and a spatial frequency evaluation value of a reference image. It is characterized by.

  In the method and apparatus for detecting a moving object in a captured image according to the present invention, the dynamic evaluation value is low when the dynamic range of the reference image is small.

  The method and apparatus for detecting a moving object in a captured image according to the present invention is characterized in that the sharpness evaluation value is low when the sharpness of the reference image is small.

  In the method for detecting a moving object in a captured image of the present invention and within that, when the spatial frequency of the reference image is concentrated on a single frequency, concentrated on the high frequency side, or concentrated on the low frequency side The spatial frequency evaluation value is low.

  According to the present invention, it is possible to calculate the position of a moving object in a captured image and its motion vector at high speed with a small amount of calculation.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a configuration diagram of a main part of a digital still camera with a moving image shooting function equipped with a motion detection device according to the first embodiment of the present invention. Image data output from a solid-state imaging device (not shown) is stored in the frame memory 10, subjected to various image processing, compressed by the image compression circuit 11, and stored in the recording medium 12.

  When the image compression circuit 11 compresses the image data, the motion vector (motion vector) calculated by the motion detection device is used, and the moving image compression data is created and stored in the recording medium 12.

  The motion detection apparatus includes a pre-processing unit 21 that captures and thins out image data output from a solid-state imaging device, and two memories 22 and 23 that store reduced image data generated by the thinning-out processing by the pre-processing unit 21. And a switch 24 for switching and connecting one of the two memories 22 and 23 to the preprocessing unit 21.

  This movement vector calculation device is further provided between the reference image cutting circuit 26, the comparative image cutting circuit 27, the reference image cutting circuit 26 and the comparative image cutting circuit 27, and the memories 22, 23. , 23 is connected to the reference image cut-out circuit 26, and the connection switching circuit 28 that connects the other of the memories 22, 23 to the comparison image cut-out circuit 27 is compared with the output image data of both the cut-out circuits 26, 27. And a comparison calculation processing unit 29 that calculates the position and motion vector (movement vector) of the moving object in the captured image by performing correlation calculation processing and the like, and outputs the result to the image compression circuit 11.

  FIG. 2A is a diagram showing an example in which one screen is divided into a total of 16 divided regions 5 of 4 rows and 4 columns. The motion detection apparatus according to the present embodiment has the same division of the next frame in which the N-th frame reference image cut out in the fixed block 2 in the divided region 5 is cut out in the same size block for every 16 divided regions 5. Compared with the comparative image in the region 5, as shown in FIG. 2B, movement vectors k1 to k16 for each divided region 5 are calculated.

  Nth frame image data, N + 1 frame image data, N + 2 frame image data, and so on are successively input to the motion detection apparatus. The preprocessing unit 21 thins out the image data of each frame one after another to generate and output reduced image data, and the switch 24 is controlled to be switched by a frame switching signal.

  As a result, when the reduced image data of the Nth frame is stored in the memory 22 and the reduced image data of the (N + 1) th frame is stored in the memory 23, the memory 23 stores the image data stored in the memory 22 (current reference image data). A movement vector of the stored image data (current comparison image data) is calculated.

  Next, when the reduced image data of the (N + 2) th frame is overwritten in the memory 22, the stored image data (the next comparison) of the image stored in the memory 23 with respect to the stored image data (the reduced image data of the (N + 1) th frame: the next reference image data). The movement vector of (image data) is calculated, and the same processing is repeated thereafter.

  As described above, when calculating the movement vector, it is preferable to use the reduced image data in order to reduce the processing load of the comparison calculation processing unit 29 and increase the processing speed. It is also possible to do this.

  The connection switching circuit 28 selects one of the memories 22 and 23 storing the reduced image data to be used as the reference image by the frame switching signal and connects it to the reference image cutting circuit 26, and the other of the memories 22 and 23 is connected. The comparison image cutting circuit 27 is connected.

  First, the reference image cutout circuit 26 selects the divided region 5 in the first row and first column shown in FIG. 2A in the reduced image data of N frames, cuts out the reference image by the block 2, and performs a comparison operation. The data is output to the processing unit 29.

  Similarly, the comparison image cutting circuit 27 selects the divided area 5 in the first row and the first column from the reduced image data of N + 1 frames as a search range, and compares the same size as the block 2 from the search range. A comparison image is cut out by the image cutout block and output to the comparison calculation processing unit 29.

  The comparison calculation processing unit 29 compares both images captured from the cut-out circuits 26 and 27 on a pixel-by-pixel basis, for example, a difference for each pixel, for example, a difference in luminance, a difference in color difference, or RGB color in the case of a primary color filter. In the case of a difference for each color or a complementary color filter, a difference for each complementary color is obtained, an integrated value of total pixels in each block of each difference is calculated, and stored in an internal memory (not shown).

  After the integrated value is calculated, the comparison calculation processing unit 29 outputs a movement command to the comparison image cutting circuit 27. As a result, the comparison image cutout block moves to the next position within the search range (because the reduced image data is used, the movement is in units of one pixel on the reduced image), and the next comparison image is cut out. The comparison calculation processing unit 29 performs a comparison calculation between the comparison image and the reference image, and similarly obtains the integrated value of the difference between the pixels.

  In the same manner, the comparison images in the search range (division region 5 in the first row and first column) are cut out one after another, the integrated value for each comparative image is obtained, and the minimum value (correlation value) of each integrated value is obtained. Maximum value = minimum value of mismatch amount)

  After the processing in the divided region of the first row and the first column is completed, the comparison calculation processing unit 29 next outputs a search range switching instruction to the reference image cutting circuit 26 and the comparative image cutting circuit 27. Thereby, the divided region 5 in the first row and the second column shown in FIG. 2A becomes the search range, and the above processing is repeated.

  Similarly, the minimum values Mi (i = 1 to 16) in the 16 search ranges i (i = 1 to 16) divided into 4 rows and 4 columns shown in FIG.

  Then, in each divided region 5, the difference (moving direction, moving distance) between the reference image cut-out block position and the block position from which the comparative image from which the minimum integrated value has been obtained is cut out is obtained as shown in FIG. As shown in b), movement vectors k1, k2,..., k16 for each divided region 5 are obtained.

  In the illustrated example, 14 movement vectors out of 16 movement vectors have the same direction and the same length within a predetermined allowable range, and the directions of the two movement vectors k9 and k15 are different.

  Camera shake occurs when the photographer is shooting a video with the camera in hand. The camera shake is shifted by substantially the same direction and the same distance on the same screen (since there is a slight rotation component in the camera shake component, it is rare that the upper right corner and the lower left corner of the screen exactly match each other). That is, it can be determined that the above 14 movement vectors are movement vectors due to camera shake.

  On the other hand, it can be determined that the movement vectors k9 and k15 are not caused by camera shake, and the moving object image is included in this divided region. For this reason, the comparison calculation processing unit 29 has a moving body image in the divided region of the third row and first column where the movement vector k9 is calculated, and the divided region of the fourth row and third column where the movement vector k15 is calculated. It is determined that there is also a moving body image.

  Then, an average value L of the 14 movement vectors is obtained and used as a movement vector L due to camera shake, and a difference between the movement vector k9 and the average value L is used as a motion vector of the divided region in the third row and first column. Notify the compression circuit 11. Further, the difference between the movement vector k15 and the average value L is notified to the image compression circuit 11 as a motion vector of the divided region in the fourth row and third column.

  In addition, after the reference image cut out for each divided region is cut out by the block 2 at the fixed position and the divided region where the moving body image exists is specified, the position of the block 2 from which the reference image is cut out in this divided region A configuration in which a block is moved in units of several pixels, and the size of a block from which a reference image and a comparison image are cut out is adaptively enlarged or reduced to detect a moving object position and a motion vector with high accuracy, as in the known motion detection technology. It is also possible. Even in this case, since the search range is limited, the position and motion vector of the moving object image can be calculated at high speed with a small amount of calculation.

(Second Embodiment)
FIG. 3 is a flowchart showing a motion detection processing procedure according to another embodiment performed by the comparison calculation processing unit shown in FIG. In the first embodiment described above, the calculation processing for calculating the integrated value is executed in all the 16 divided regions shown in FIG. 2A, and 16 movement vectors are calculated.

  However, depending on the type of reference image in each divided region, there are images that are inappropriate for calculating motion vectors, and useful information can be obtained even if comparison calculation processing is executed based on such images. Therefore, it becomes useless arithmetic processing. Therefore, in the present embodiment, the divided areas on which the comparison calculation process is performed are not limited to all the 16 divided areas, but are limited as follows.

  First, the variable i is set to 1 (step S1), and the reference image i is captured (step S2). For example, in FIG. 2A, the search range of the first row and first column is i = 1, the search range of the first row and second column is i = 2,..., And the search range of the fourth row and fourth column is i = 2. 16 is set.

  In the next step S3, the reference image i is evaluated. FIG. 4 is a flowchart showing a detailed procedure of the evaluation process for the reference image i.

  In the evaluation process of the reference image i, first, in step S31, the dynamic range Di of the reference image i is evaluated. The dynamic range Di is obtained, for example, as the difference between the highest luminance value and the lowest luminance value in the reference image i.

  An image with a large dynamic range is an image with a large difference in brightness. For this reason, it is easy to detect a comparative image having a high correlation (a small amount of mismatch) with respect to a reference image having a large dynamic range.

  However, an image with a small dynamic range is, for example, a noppelli image with little change in brightness, such as a wall, and an image that does not know whether the correlation is high or low even if the reference image and the comparison image are compared. I can say that. That is, it can be said that an image with a small dynamic range has low reliability for calculating a movement vector.

  In the next step S32, the sharpness Si of the reference image i is evaluated. The sharpness Si can be obtained as a difference (differentiation) between adjacent pixels, and the evaluation value Si increases as the difference increases.

  An image having a large sharpness is, for example, an image with a clear edge in the image. Therefore, it is easy to detect the position of a comparative image having a high correlation (small amount of mismatch) with respect to the reference image.

  However, an image with low sharpness is an image with unclear outlines, so it can be said that it is not clear whether the correlation is high or low. That is, it can be said that an image with a small sharpness has low reliability for calculating a movement vector.

  In the next step S33, the spatial frequency Fi of the reference image i is evaluated. The spatial frequency can be obtained by, for example, FFT (Fast Fourier Transform), DCT (Discrete Cosine Transform), Orthogonal Transform, Karhunen-Loeve Transform, Hadamard Transform, etc.

  If a strong repetitive pattern exists in the reference image i, a peak (large value) appears at one frequency in the spatial frequency distribution. In this case, if the shift amount of the block from which the comparison image is cut out coincides with the cycle of the repeated pattern, the minimum value of the integrated value is periodically generated (this includes a large number of candidates that can be movement vectors). The reliability of the movement vector calculation is reduced.

  Further, even when the spatial frequency is concentrated on the high frequency side, it indicates that a repetitive pattern exists in the reference image i, and the minimum value of the mismatch amount is the period with respect to the shift amount of the block from which the comparison image is cut out. It means to occur automatically. That is, an image in which the spatial frequency is concentrated on the high frequency side can be said to be an image with low reliability for calculating the movement vector.

  In addition, when the spatial frequency is concentrated on the low frequency side, the amount of change in correlation (the amount of mismatch) with respect to the shift amount of the block from which the comparison image is cut out becomes small, and the accuracy for obtaining the minimum value of the mismatch is high. It will decline. That is, it can be said that it is an image with low reliability for calculating the movement vector.

  In the next step S34, a comprehensive evaluation value Ti is calculated based on the dynamic range Di, sharpness Si, and spatial frequency Fi. The comprehensive evaluation value Ti may be obtained by a linear combination formula of Di, Si, and Fi or by a product value.

  When the comprehensive evaluation value Ti of the reference image i is obtained, the process proceeds to step S4 in FIG. 3 to determine whether i = 16. If i = 16 is not satisfied, i = i + 1 is set in step S5, and the process returns to step S2. Thereby, the comprehensive evaluation value Ti for the reference image i in the 16 search ranges i (i = 1 to 16) shown in FIG. 2A is calculated.

  After the respective comprehensive evaluation values Ti for the reference image i in the 16 search ranges i are calculated, the process proceeds from step S4 to step S6, and the reference image i having the total evaluation value Ti equal to or greater than the predetermined threshold value α is determined.

  In the next step S7, the comparison image extraction circuit 27 is notified of the search range (divided region) i indicated by the label i determined in step S6.

  As a result, for example, as shown in FIG. 5, only the movement vectors k1, k3, k4, k5, k6, k7, k8, k9, k10, k14, and k16 of the divided areas where the total evaluation value Ti is equal to or greater than the predetermined threshold value α are obtained. The movement vector calculation process of i = 2, 11, 12, 13, and 15 is skipped. Therefore, in this embodiment, the calculation load is reduced, and a highly accurate motion detection process can be performed at high speed.

  After obtaining each movement vector shown in FIG. 5, the comparison calculation processing unit 29 calculates an average value L of movement vectors other than the movement vector k9 different from the others, and uses this average value L as a movement vector due to camera shake between two screens. And Then, the vector of the difference between the movement vector k9 and the average value L is set as the motion vector of the moving object in the captured image.

  In the above-described embodiment, an example in which one screen is divided into 16 has been described, but the number of divisions is not limited to this.

  The motion detection method and apparatus according to the present invention can obtain a motion vector with high accuracy and high speed and with a small amount of calculation, and thus are useful when applied to a digital still camera or a digital video camera having a moving image shooting function. .

It is a principal part block diagram of the digital still camera with a video recording function carrying the motion vector detection apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the example of a screen division used with the motion vector detection method which concerns on 1st Embodiment of this invention. It is a flowchart which shows the motion vector detection process procedure which the comparison calculation process part shown in FIG. 1 performs. 4 is a flowchart showing details of a reference image evaluation processing procedure shown in FIG. 3. It is a figure which illustrates the movement vector for every division area calculated | required by the motion vector detection process which concerns on 2nd Embodiment of this invention. It is a figure explaining the detection principle of a motion vector.

Explanation of symbols

10 frame memory 11 image compression circuit 22, 23 memory for storing reduced image 26 reference image extraction circuit 27 comparison image extraction circuit 28 connection switching circuit 29 comparison operation processing unit

Claims (14)

  Each image of the first screen and the second screen is divided into the same plurality of regions, and for each divided region, a reference image cut out from the first screen and each comparative image cut out while shifting the position from the second screen, For each divided region, a movement vector is calculated from the difference between the position of the comparative image showing the highest correlation value and the position of the reference image in the divided region for each divided region. The movement vector for each region is distinguished into a first type of movement vector that falls in the same direction and the same length within the required range and a second type of movement vector that does not fall within the required range. A method of detecting a moving object in a captured image, characterized in that it is determined that a moving object image is present at the position of the divided region for which the value is calculated.   2. The captured image according to claim 1, wherein an average value of the first type of movement vector is obtained, and a difference vector between the second type of movement vector and the average value is used as a motion vector of the moving object image. Detection method for moving objects inside.   A division in which a reference image cut out from each of the divided regions is evaluated, calculation of the movement vector in the divided region in which there is a reference image whose evaluation value does not reach a predetermined threshold value is stopped, and a reference image having the predetermined threshold value or more exists A method for detecting a moving object in a captured image, wherein the movement vector of a region is calculated.   The said evaluation value is calculated by any one or a combination of a dynamic range evaluation value, a sharpness evaluation value, and a spatial frequency evaluation value of a reference image. Detection method.   The method according to claim 3, wherein the dynamic evaluation value is low when a dynamic range of a reference image is small.   The method for detecting a moving object in a captured image according to claim 3, wherein the sharpness evaluation value is low when the sharpness of the reference image is small.   The spatial frequency evaluation value is low when the spatial frequency of the reference image is concentrated on a single frequency, when concentrated on the high frequency side, or when concentrated on the low frequency side. Item 4. A method for detecting a moving object in a captured image according to Item 3.   Each of the images on the first screen and the second screen is divided into the same plurality of regions, and a reference image cut out from the first screen for each divided region and each comparative image cut out while shifting the position from the second screen. And a second means for calculating a movement vector from the difference between the position of the comparison image showing the highest correlation value for each divided area and the position of the reference image in the divided area. And a second type of movement vector that is classified into a first type of movement vector that falls within the same direction and length within the required range and a second type of movement vector that does not fall within the required range. And a third means for determining that the moving image is present at the position of the divided region where the movement vector of the seed is calculated.   9. The third means obtains an average value of a first type of movement vector and calculates a difference vector between the second type of movement vector and the average value as a motion vector of the moving object image. The detection apparatus of the moving body in the captured image of description.   A division in which a reference image cut out from each of the divided regions is evaluated, calculation of the movement vector in the divided region in which there is a reference image whose evaluation value does not reach a predetermined threshold value is stopped, and a reference image having the predetermined threshold value or more exists An apparatus for detecting a moving object in a captured image, wherein the movement vector of an area is calculated.   11. The moving object in the captured image according to claim 10, wherein the evaluation value is calculated by one or a combination of a dynamic range evaluation value, a sharpness evaluation value, and a spatial frequency evaluation value of a reference image. Detection device.   The apparatus according to claim 10, wherein the dynamic evaluation value is low when a dynamic range of a reference image is small.   The apparatus for detecting a moving object in a captured image according to claim 10, wherein the sharpness evaluation value is low when the sharpness of the reference image is small.   The spatial frequency evaluation value is low when the spatial frequency of the reference image is concentrated on a single frequency, when concentrated on the high frequency side, or when concentrated on the low frequency side. Item 11. A detection device for a moving object in a captured image according to Item 10.

Images