JP5361703B2 - Motion detection apparatus and motion detection method - Google Patents

Description

  The present invention relates to a motion detection device, and more particularly to a motion detection device and a motion detection method for detecting the motion of a moving object by image matching.

  Various techniques for tracking (tracking) a moving object in the screen and recognizing where the target moving object is located in the screen have been proposed. For example, in Patent Document 1, it is determined that there is a correlation when the difference between the reference image that is the reference and the evaluation image is smaller than the reference value, and further, the correlation is found when the absolute difference between the reference image and the continuous evaluation image is greater than the reference value An intra-image moving body tracking device that determines that there is no image and updates the image data of the reference image is disclosed. As a result, even if the target subject is hidden behind the object, a similar object is not set as the tracking target, and a reduction in identification accuracy due to erroneous determination can be suppressed.

JP-A-9-65193

  According to the intra-image moving object tracking device disclosed in Patent Literature 1 described above, the identification accuracy of a tracking target with a large background change is improved. However, for a tracking target whose shape changes during tracking, the variation in correlation value becomes large, so that the correlation value cannot be obtained properly, and tracking accuracy is reduced, and a tracking error may occur. is there.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a motion detection device and a motion detection method in which tracking accuracy does not decrease even for a tracking target having a large background change.

  In order to achieve the above object, the motion detection apparatus according to the first aspect of the present invention uses the imaging unit, the memory, and the image data output in time series by the imaging unit as the latest image data as an evaluation image, which is output before this. Correlation calculation for calculating a correlation value and a motion amount based on an image data update unit that performs an update operation to record the image data as a reference image in the memory, and the evaluation image and the reference image stored in the memory And a correlation value calculated by the correlation calculation unit in a time series in the memory, a reference value calculation unit for calculating a reference value based on these time series correlation values, and the correlation calculation unit calculated by the correlation calculation unit The correlation value is compared with the reference value calculated by the reference value calculation unit, and if the correlation value is less than the reference value, the update operation by the image data update unit is permitted, and if the correlation value is greater than or equal to the reference value, If And a control unit for prohibiting an update operation by serial image data updating unit.

In the motion detection apparatus according to a second aspect of the present invention, in the first aspect, the reference value calculation unit has a degree of variation from an average value of time-series correlation values stored in the memory for storing the correlation values. The reference value is calculated according to the above.
In the motion detection apparatus according to the third invention, in the second invention, the reference value calculation unit obtains a difference value between a maximum value and a minimum value of the correlation value as a degree of variation of the correlation value, and the difference value is calculated. The reference value is calculated by determining a coefficient corresponding to the above and multiplying the correlation value by this coefficient.

  The motion detection method according to the fourth invention is an update operation in which the latest image data is used as an evaluation image from the image data output in time series by the imaging unit, and the image data output before this is recorded in the memory as a reference image. The correlation value and the amount of motion are calculated based on the evaluation image and the reference image stored in the memory, the correlation value is stored in the memory in time series, and the time-series correlation values are calculated. The reference value is calculated based on the correlation value, and the correlation value is compared with the reference value. If the correlation value is less than the reference value, the update operation is permitted. If the correlation value is equal to or greater than the reference value, the update operation is prohibited. To

  According to the present invention, it is possible to provide a motion detection device and a motion detection method in which tracking accuracy does not decrease even for a tracking target having a large background change.

1 is a block diagram mainly showing an electrical configuration of an electronic camera according to an embodiment of the present invention. 6 is a flowchart illustrating control of camera operation in the electronic camera according to the embodiment of the present invention. 6 is a flowchart showing control of reference value calculation in the electronic camera according to the embodiment of the present invention. It is a figure explaining calculation of a correlation value in the electronic camera concerning one embodiment of the present invention. It is a figure which shows the relationship between the change of a correlation value, and a reference value in the electronic camera in one Embodiment of this invention.

  A preferred embodiment will be described below using an electronic camera to which the present invention is applied according to the drawings. An electronic camera according to a preferred embodiment of the present invention is a digital camera, and includes an imaging unit. The imaging unit converts a subject image into image data, and the subject image is converted based on the converted image data. Live view is displayed on the display unit on the back of the main unit. The photographer determines the composition and the photo opportunity by observing the live view display. During the release operation, image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

  FIG. 1 is a block diagram mainly showing an electrical configuration of a digital camera 100 according to the present embodiment. On the optical axis of the photographing lens 101, an aperture 103a, a shutter 105a, and an image sensor 107 are disposed. The image sensor 107 is connected to an image sensor interface circuit (hereinafter referred to as an image sensor IF circuit) 109, and the image sensor IF circuit 109 is connected to the system controller 110.

  The photographing lens 101 is an optical system for condensing a subject light beam on the image sensor 107 and forming a subject image. The photographing lens 101 is moved in the optical axis direction by a focus adjustment mechanism 102 that operates in accordance with an instruction from the system controller 110, and the focus state changes. The aperture of the aperture 103 a is adjusted by an aperture driving mechanism 103 that operates in accordance with an instruction from the system controller 110, and the amount of incident light of the subject incident on the image sensor 107 via the imaging lens 101 is adjusted. The shutter 105 a opens and closes the subject light flux, and controls the projection time of the subject light flux on the image sensor 107. The shutter 105 a has an opening / closing time (shutter speed) controlled by a shutter driving mechanism 105 that operates in accordance with an instruction from the system controller 110.

  The image sensor 107 includes a Bayer array color filter disposed on the front surface and a photoelectric conversion element such as a photodiode array corresponding to the color filter. Each color filter and each corresponding photoelectric conversion element constitute each pixel. The image sensor 107 receives the light collected by the photographing lens 101 by each pixel, converts it into a photocurrent, accumulates this photocurrent with a capacitor, and outputs it to the image sensor IF circuit 109 as an analog voltage signal (image signal). To do. The image sensor IF circuit 109 performs photocurrent accumulation control and read control of the image sensor 107 in accordance with an instruction from the system controller 110, AD converts the read image signal, and outputs digital image data.

  The system controller 110 includes an ASIC, and includes a CPU 111, an AF control circuit 113, an AE control circuit 115, an image processing circuit 117, a face recognition circuit 119, a compression / expansion circuit 121, a correlation calculation circuit 123, and a memory control circuit 125. The CPU 111 operates in accordance with a program code 135a stored in a flash ROM 135, which will be described later, and performs overall control of the electronic camera.

  The AF control circuit 113 in the system controller 110 extracts a high frequency component based on the digital image data output from the image sensor IF circuit 109, and passes through the focus adjustment mechanism 102 so that the high frequency component becomes a peak. Then, automatic focusing of the taking lens 101 is performed. Note that the subject for which automatic focusing is performed by the AF control circuit 113 is a human face portion recognized by a face recognition circuit 119 described later, or a portion designated by the photographer using the touch panel 145. When these subjects move, they are tracked by the correlation calculation circuit 123, and even if the subject moves, automatic focus adjustment is performed on the designated portion.

  The AE control circuit 115 detects subject brightness based on the digital image data output from the image sensor IF circuit 109, and calculates exposure control values for the aperture 103a, the shutter 105a, and the like so as to obtain appropriate exposure. Based on the exposure control value, the diaphragm 103 a is controlled through the diaphragm driving mechanism 103, and the shutter 105 a is controlled through the shutter driving mechanism 105.

  The image processing circuit 117 performs digital amplification (digital gain adjustment processing), white balance, color correction, gamma (γ) correction, contrast correction, and live view display image generation of digital image data output from the image sensor IF circuit 109. Various image processing such as moving image generation is performed.

  The face recognition circuit 119 uses the digital image data output from the image sensor IF circuit 109 or the digital image data image-processed by the image processing circuit 117 to determine whether a human face portion exists in the image. If it exists, its position and size are detected.

  The compression / decompression circuit 121 is a circuit for compressing still image or moving image image data temporarily stored in an SDRAM 151, which will be described later, by a compression method such as JPEG or TIFF, and for decompression for display or the like. Note that image compression is not limited to JPEG or TIFF, and other compression methods can be applied.

  The correlation calculation circuit 123 is a circuit that performs a known correlation calculation from the evaluation image data and the reference image data temporarily stored in the SDRAM 151, and calculates a motion amount (motion vector) and a correlation value. The memory control circuit 125 controls writing and reading of data to and from the memory in the electronic camera, that is, the recording medium 131, the flash ROM 135, and the SDRAM 151.

  A recording medium 131, a flash ROM 135, a display element driving circuit 141, a touch panel driving circuit 147, a camera operation switch 149, and an SDRAM 151 are connected to the system controller 110. The display element driving circuit 141 is connected to the display element 143, receives image data from the system controller 110, displays live view on the display element 143, displays a captured image, displays an operation mode setting of the electronic camera, and the like. Display control. The display element 143 includes a display element such as an LCD display element, an organic EL display element, or a plasma display element.

  The touch panel 145 is disposed on the front surface of the display element 143 or is configured integrally with the display element 143, and detects a position where the display surface of the display element 143 is touched with a finger, a stylus pen, or the like. The touch panel 145 is connected to the touch panel drive circuit 147, is driven by this, and the detected contact position is output to the system controller 110. The camera operation switch includes switches necessary for camera operation such as a power switch, a release switch, a mode setting switch, and the operation state of these switches is output to the system controller 110.

  The SDRAM 151 is a memory for temporarily storing data necessary for the operation of the electronic camera. The work area 151a is used for temporarily storing image data, for example, when executing image processing. The evaluation image area 151b is used for storing an evaluation image when the subject tracking operation is performed. The reference image area 151c is used for storing a reference image. In the correlation history area 151d, past data calculated by the correlation calculation circuit 123 is stored. The number of past correlation value data stored is the latest predetermined number, and the old correlation value data is overwritten and deleted sequentially.

  As described above, the flash ROM 135 stores the program code 135a executed by the system controller 110, the control parameters 135b necessary for controlling the electronic camera, and the like. The recording medium 131 includes a nonvolatile memory, a small HDD, and the like, and is detachable from the electronic camera. In the recording medium 131, an image file 131a such as a captured still image file (JPEG) or a moving image file (MPEG) is recorded.

  Next, the operation of this electronic camera will be described with reference to the flowcharts shown in FIGS. This flowchart is executed by the CPU 111 in accordance with the program code 135a stored in the flash ROM 135.

  When the power switch of the electronic camera is turned on, the flow shown in FIG. 2 starts. When this flow starts, first, a live view operation is started (S100). In this step, the image sensor IF circuit 109 is controlled to acquire image data at a predetermined frame rate and output it to the display element drive circuit 141. By this operation, the subject image is displayed on the display element 143 in real time, and the user can observe the subject.

  Subsequently, the outputs of the touch panel drive circuit 147 and the face recognition circuit 119 are acquired (S102), and it is determined whether or not a subject is detected (S104). In these two steps, the subject to be tracked is detected. That is, if the photographer observes the live view display and touches the subject with a finger or the like, or if there is a face portion in the live view display image, that portion becomes the subject to be tracked. When a face part is detected and touched with a finger, the touched part is given priority as a tracking target. If the result of determination in step S104 is that no subject has been detected, processing returns to step S100.

  On the other hand, if the result of determination in step S104 is that a subject has been detected, next, a reference image area is set (S106). Here, when there is a touch position detected by the touch panel drive circuit 147, the image data of the portion is set as the reference image region, and when the face is recognized by the face recognition circuit 119, the image data of the portion is set as the reference image region. To do. Now, when the image data of the Nth frame is acquired as shown in FIG. 4A, when the character portion A is detected as a subject, the portion including the character A is set as the reference image region 300. Is done. The image data of the reference image area 300 is temporarily stored in the reference image area 151c. In the present embodiment, the area of the reference image area 300 is a fixed area of 6 × 6 pixels as shown in the figure, but the area width may be appropriately selected as a design value. It may be variable according to the size of the subject.

  Once the reference image area is set, display of a target mark indicating the position of the tracking subject is started (S108). In this step, the target mark is superimposed and displayed on the live view image on the display element 143 based on the position and area (reference image area 300) information of the tracking subject (face) set in step S106. The target mark moves on the screen in accordance with the movement of the subject, and notifies the photographer of the operation state of subject tracking.

  Once the display of the target mark is started, it is next determined whether or not an evaluation image has been acquired (S110). Since the image sensor IF circuit 109 outputs image data at a predetermined frame rate, the process waits at this step until the next image data is output. The evaluation image is acquired in synchronization with this frame rate, and the image data of the acquired evaluation image is temporarily stored in the evaluation image area 151b in the SDRAM 151. As shown in FIG. 4A, when the image data of the reference image area 300 is acquired at the Nth frame, the image data of the evaluation image area 301 is acquired at the N + 1th frame as shown in FIG. 4B. . Since the correlation calculation is performed using the image data of the evaluation image region 301 and the image data of the reference image region 300, the evaluation image region 301 is wider than the reference image region 300. The evaluation image may be acquired in synchronization with the frame rate of 1/2 or 1/3 according to the processing capability of the correlation calculation circuit 123. The area of the evaluation image area 301 is a fixed area of 17 × 13 pixels as shown in the figure, but the area width may be appropriately selected as a design value, and the size of the subject. In addition, it may be variable.

  If an evaluation image is acquired as a result of the determination in step S110, a motion amount (motion vector) and a correlation value are calculated based on the evaluation image and the reference image (S112). Here, using the image data of the reference image temporarily stored in the reference image area 151c and the image data of the evaluation image temporarily stored in the evaluation image area 151b, the correlation calculation circuit 123 uses the motion amount (motion vector). ) And a correlation value are calculated. In this calculation, for example, in FIG. 4B, the reference image region 300 is overlaid on the upper left corner of the evaluation image region 301, and in this state, a cumulative value of difference values is obtained for each pixel. Thereafter, the reference image 300 is moved one pixel at a time on the evaluation image region 301 to obtain the cumulative value of the difference values, and the smallest cumulative value is used as the correlation value. The amount of movement at that time is the amount of motion (including the direction). Then, the motion vector). Therefore, the smaller the correlation value, the higher the degree of matching with the reference image. When the correlation value is 0, the matching with the reference image is complete.

  Once the correlation value and the amount of motion are obtained, a reference value is calculated (S114). In steps S116 and S117, which will be described later, when the correlation value is smaller than the reference value, that is, when the correlation degree is high, the reference image is updated. In step S114, the reference value to be compared is used. Is calculated. In calculating the reference value, a reference value (determination value) is calculated based on a plurality of correlation values temporarily stored in the correlation value history area 151 d in the SDRAM 115. The reference value calculation flow will be described later with reference to FIG.

  Once the reference value has been calculated, it is next determined whether or not the correlation value is greater than the reference value (S116). Here, the correlation value calculated in step S112 is compared with the reference value calculated in step S114. If the correlation value is smaller than the reference value as a result of the comparison, next, the reference image is updated and the history data of the correlation value is updated (S117). That is, in the evaluation image area 301, an area corresponding to the reference image area 300 and having the smallest correlation value is set as a new reference image area 300, and image data there is stored in the reference image area 151c. Further, the correlation value calculated at this time is temporarily stored in the correlation value history area 151d. When the correlation value temporarily stored in the correlation value history area 151d exceeds a predetermined number (for example, several to about 10), the oldest data is erased and written. If the predetermined number of correlation values to be temporarily stored is several tens, the accuracy is improved. However, since the reference value cannot be determined until the predetermined number is reached, the number is set to several to about 10 in the present embodiment.

  If the reference image is updated and the correlation value history data is updated, or if the correlation value is greater than the reference value as a result of the determination in step S116, then the subject position information is updated and the target mark display position is changed. (S118). In this step, the subject position information is changed based on the amount of motion calculated in step S112. Further, the target mark display position is changed based on the subject position information.

  Next, it is determined whether or not a shooting start operation has been performed (S120). Here, the state of the release switch in the camera operation switch 149 is detected and determined. If the result of this determination is that a shooting start operation has not been performed, processing returns to step S110, and subject tracking described above is executed.

  On the other hand, if the result of determination in step S120 is that a shooting start operation has been performed, a shooting preparation operation is next carried out (S122). Here, based on the tracked subject position information, the subject at that position is focused, and exposure control is performed so that an appropriate exposure is obtained. That is, the AF control circuit 113 adjusts the photographing lens 101 so that the contrast of the image data at the position based on the subject position information is maximized. Further, the AE control circuit 115 obtains the subject brightness based on the position image data based on the subject position information, and sets the conditions of the aperture 103a and the shutter 105a so as to achieve appropriate exposure with respect to the subject brightness. Note that the automatic focus adjustment operation and the exposure condition setting operation may be performed in parallel with the subject tracking operation in steps S110 to S118 described above, and in this case, the time lag until the start of exposure is shortened. Can do.

  When the shooting preparation operation is finished, still image data is acquired and an image file is generated (S124). Here, subject light flux is received on the image sensor 107 under the exposure conditions (aperture value and shutter speed) obtained in step S122, photocurrent accumulation is performed, and the image signal at this time is read out by the image sensor IF circuit 109. The read image signal is AD converted into digital image data, subjected to image processing by the image processing circuit 117, compressed by the compression / expansion circuit 121 to generate an image file, and this image file is recorded on the recording medium 131. To record. When the image file is recorded, a series of operations are terminated, and if the power switch is on, the process starts again from step S100.

  Next, the reference value calculation flow in step S114 will be described with reference to the flowchart shown in FIG. When the flow for calculating the reference value is entered, first, the average value of the correlation values is calculated from the correlation value history data (S200). Here, a predetermined number of correlation values temporarily stored in the correlation value history area 151d of the SDRAM 151 are read, and an average value of these correlation values is calculated.

  Once the average value of the correlation values is calculated, the maximum value and the minimum value of the correlation values are then obtained from the temporarily stored correlation value history data (S202). Here, the maximum value and the minimum value are obtained from the predetermined number of correlation values read in step S200. Subsequently, a correction coefficient corresponding to the difference value between the maximum value and the minimum value is determined (S204). Here, a difference value between the maximum value and the minimum value obtained in step S202 is obtained, and a correction coefficient corresponding to the difference value is determined based on a correction calculation table created in advance. For example, when the difference value is smaller than a preset threshold value, the correction is 1, and when the difference value is larger than the preset threshold value, the correction coefficient is 2.

  Next, a reference value is calculated from the average value and the correction coefficient (S206). Here, the calculation is performed by multiplying the average value calculated in step S200 by the correction coefficient determined in step S204. When the reference value is calculated, the process returns to the original flow.

  As described above, in one embodiment of the present invention, a correlation value is obtained based on the reference image and the evaluation image, the correlation value is compared with the reference value, and the correlation value is smaller than the reference value, that is, the correlation value. When the degree is high, the reference image is updated. The reference value is determined based on the correlation value stored in the correlation value history area 151d in time series and stored in time series. For this reason, even when the background changes and the correlation value fluctuates, the tracking accuracy does not decrease.

  For example, as shown in FIG. 5, when the correlation value fluctuates greatly from time t1 to time t2, the reference value is increased so that the reference image is not updated frequently. Further, when the fluctuation of the correlation value is small, it is easy to permit the update of the reference image, and it responds to the change in the image of the subject to be tracked. For this reason, a correct reference image can be maintained, and tracking accuracy does not deteriorate.

  In one embodiment of the present invention, the correction coefficient is determined using the difference value between the maximum value and the minimum value of the correlation value as a value indicating variation. However, the value indicating the variation is not limited to the difference value, and for example, a dispersion value may be obtained and used.

  In the embodiment of the present invention, the digital camera is used as the electronic camera to which the present invention is applied. However, the electronic camera may be a digital single-lens reflex camera or a compact digital camera. Such a camera for moving images may be used, and an electronic camera built in a mobile phone, a personal digital assistant (PDA), a game machine, or the like may be used.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 101 ... Shooting lens, 102 ... Focus adjustment mechanism, 103 ... Diaphragm drive mechanism, 103a ... Diaphragm, 105 ... Shutter, 105a ... Shutter drive mechanism, 107 ... Imaging element, DESCRIPTION OF SYMBOLS 109 ... Imaging device IF circuit, 110 ... System controller, 111 ... CPU, 113 ... AF control circuit, 115 ... AE control circuit, 117 ... Image processing circuit, 119 ... Face recognition circuit, 121... Compression / decompression circuit, 123... Correlation operation circuit, 125... Memory control circuit, 131... Recording medium, 131 a. 135a ... Program code, 135b ... Control parameter, 141 ... Display element drive circuit, 143 ... Display element, 145 ... Touch panel, 1 7. Touch panel drive circuit, 149 ... Camera operation switch, 151 ... SDRAM, 151a ... Work area, 151b ... Evaluation image area, 151c ... Reference image area, 151d ... Correlation History area, 300 ... reference image area, 301 ... evaluation image area

Claims (4)

An imaging unit;
Memory,
An image data update unit that performs an update operation that records the latest image data from the image data output in time series by the imaging unit as an evaluation image and records the image data output before this as a reference image in the memory;
A correlation calculation unit that calculates a correlation value and a motion amount based on the evaluation image and the reference image stored in the memory;
A correlation value calculated by the correlation calculation unit is stored in the memory in time series, and a reference value calculation unit that calculates a reference value based on the correlation values of these time series; and
The correlation value calculated by the correlation calculation unit is compared with the reference value calculated by the reference value calculation unit, and if the correlation value is less than the reference value, an update operation by the image data update unit is permitted, A control unit that prohibits the update operation by the image data update unit if the correlation value is equal to or greater than the reference value;
A motion detection apparatus comprising:   2. The reference value calculation unit according to claim 1, wherein the reference value calculation unit calculates the reference value according to a degree of variation from an average value of time-series correlation values stored in the memory for storing the correlation value. The motion detection device described.   The reference value calculation unit obtains a difference value between the maximum value and the minimum value of the correlation value as a degree of variation of the correlation value, determines a coefficient according to the difference value, and multiplies the correlation value by the coefficient. The motion detection apparatus according to claim 2, wherein the reference value is calculated by: The latest image data from the image data output in time series by the imaging unit is used as an evaluation image, and an update operation is performed in which image data output before this is recorded in a memory as a reference image.
Based on the evaluation image and the reference image stored in the memory, a correlation value and a motion amount are calculated,
The correlation value is stored in the memory in time series, a reference value is calculated based on the time series correlation values,
The correlation value is compared with the reference value, the update operation is permitted if the correlation value is less than the reference value, and the update operation is prohibited if the correlation value is greater than or equal to the reference value.
A motion detection method characterized by the above.