JP2009027437A - Image processor, image processing method and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor, image processing method and imaging device which can obtain a more appropriate motion vector when a mobile moves in a z direction of an imaging area while taking a moving image. <P>SOLUTION: When the motion vector of a mobile T is detected based on image data obtained by taking an image, a frame obtained by cutting out a predetermined region where at least part of the mobile T is taken in image data is divided into a plurality of areas b0 to b3, and a correction vector search range R is set for each of the plurality of areas b0 to b3. Further, a moving component of the mobile T in the correction vector search range R is detected as correction vectors V0 to V3 for each of the plurality of areas b0 to b3, the motion vector is combined with the correction vectors V0 to V3, and the composited vector is newly set as the motion vector. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  In particular, the present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus including the image processing apparatus that perform pattern matching processing used for a moving body tracking function that tracks a moving subject during moving image capturing.

  Currently, some imaging apparatuses such as digital still cameras and digital video cameras have a moving image shooting function. Some imaging apparatuses having a photographing function have a moving body tracking function for performing photographing while tracking a subject moving within an imaging area. For example, in order to prevent a so-called camera shake that causes the position of the subject in the imaging area to move due to the shaking of the user's hand holding the imaging device body during imaging, moving object tracking Processing is in progress.

  In the moving body tracking process, a movement vector is calculated by performing a pattern matching process on a subject in a predetermined cutout position (hereinafter referred to as a frame) in an imaging area, and an imaging recorded in a memory based on the movement vector. Controls the position of the subject in the next frame of the image.

Conventionally, as an image processing method for detecting a moving object, for example, there are methods shown in the following patent documents.
Japanese Patent Application Laid-Open No. 2004-228561 performs image processing on an image obtained in a zooming or turning state to detect a moving object. Based on this estimation, the camera movement (framework) is estimated. Then, a process for modifying the past image is performed and compared with the current image.
Japanese Patent Application Laid-Open No. 2004-228561 is a method of widening the search range when the zoom magnification of the camera is large, and narrowing the search range when the zoom magnification is small.

JP 2002-344960 A Japanese Patent Application No. 2006-059943

  By the way, when detecting the movement vector of the moving object, not only the movement of the position of the moving object in the two-dimensional direction (xy direction) parallel to the imaging area but also the imaging area is defined by the x direction and the y direction. It is necessary to consider a moving component in the z direction in the state of being regarded as a two-dimensional plane. FIG. 11 is a diagram illustrating a state of movement of a moving object in the imaging area. In FIG. 11, in the imaging area, the moving object T captured in the previous frame of the tracking image area moves to the upper right direction in the drawing and the near side in the depth direction with respect to the imaging area when the current frame is set. Indicates the state. At this time, the position of each object T in the previous frame and the current frame is compared by a conventional pattern matching process to detect a movement vector.

  However, when the moving object T is also moving in the z direction as shown in FIG. 11, the moving vector includes an error. The reason is that in the pattern matching process, there is a tendency that the correlation is strong in the characteristic part of the target moving object, and in FIG. On the other hand, when the correlation with the outline portion of the mouth that is displayed substantially enlarged or reduced is increased, an erroneous detection component of the movement vector may be detected, and an erroneous movement vector may be detected. FIG. 12 is a diagram illustrating the position of the moving object with respect to the frame. FIG. 12A is a diagram showing the correct frame position and the position of the moving object T. FIG. FIG. 12B is a diagram illustrating the position of the moving object with respect to the current frame. In the conventional pattern matching process, an error equivalent to the difference between the frame position in FIG. 12A and the frame position in FIG. 12B is included in the movement vector. Thus, conventionally, when moving images are captured, an appropriate movement vector cannot be obtained due to the movement of the moving object in the z direction, so that an improvement occurs in that an error occurs in the frame position during moving object tracking processing. There was room.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing apparatus and an image that can obtain a more appropriate movement vector when a moving object moves in the z direction of an imaging area during moving image capturing. A processing method and an imaging apparatus are provided.

The above object of the present invention is achieved by the following configurations.
(1) An image processing apparatus that detects a movement vector of a moving object based on image data obtained by imaging,
A search range setting that divides a frame obtained by cutting out a predetermined area in which at least a part of the moving object is imaged in the image data into a plurality of areas and sets a correction vector search range for each of the divided areas Means,
Correction vector detection means for detecting, as a correction vector, a moving component of the moving object in the correction vector search range for each of the plurality of areas;
An image processing apparatus comprising: a vector synthesizing unit that synthesizes the movement vector and the correction vector, and newly sets the synthesized vector as the movement vector.
(2) luminance information acquisition means for acquiring luminance information of the image data;
Pattern matching processing means for executing pattern matching processing for detecting the movement vector;
The image processing apparatus according to (1), further comprising: a determination unit that determines whether or not the correction vector is detected based on the luminance information.
(3) The pattern matching processing means compares the luminance ranges of a plurality of frames targeted for the pattern matching processing, and executes the pattern matching processing when the luminance range is within a preset threshold value. The image processing apparatus according to (2), characterized in that it is characterized in that
(4) An imaging apparatus including an image processing apparatus that detects a movement vector of a moving object based on image data obtained by imaging with an imaging element,
A search range in which the image processing device divides a frame obtained by cutting out a predetermined region in which at least a part of the moving object is imaged in the image data, and sets a plurality of correction vector search ranges for each of the divided areas. Setting means;
Correction vector detecting means for detecting a moving component of the moving object as a correction vector for each of the plurality of correction vector search ranges;
An image pickup apparatus comprising: vector combining means for combining the movement vector and the correction vector, and newly setting the combined vector as the movement vector.
(5) The luminance information acquisition means for acquiring luminance information of the image data by the image processing device;
Pattern matching processing means for executing pattern matching processing for detecting the movement vector;
The imaging apparatus according to (4), further comprising: a determination unit that determines whether to detect the correction vector based on the luminance information.
(6) The pattern matching processing means compares the luminance ranges of a plurality of frames targeted for pattern matching processing, and executes the pattern matching processing when the luminance range is within a preset threshold value. The imaging apparatus according to (5) above, which is characterized.
(7) An image processing method for detecting a movement vector of a moving object based on image data obtained by imaging,
Dividing a frame obtained by cutting out a predetermined area in which at least a part of the moving object is imaged in the image data, and setting a plurality of correction vector search ranges for each divided area;
Detecting a moving component of the moving object as a correction vector for each of the plurality of correction vector search ranges;
An image processing method comprising: synthesizing the movement vector and the correction vector, and newly setting the synthesized vector as the movement vector.
(8) obtaining luminance information of the image data;
Performing a pattern matching process for detecting the movement vector;
The image processing method according to (7), further comprising a step of determining whether or not to detect the correction vector based on the luminance information.
(9) The method includes a step of performing pattern matching processing when the luminance ranges of a plurality of frames to be subjected to pattern matching processing are compared with each other and the luminance range is within a preset threshold value. The image processing method according to (8).

  According to the present invention, a frame of image data is divided into a plurality of areas, a moving component of a moving object is detected as a correction vector for each divided area, and the correction vector detected for each area is preliminarily obtained by pattern matching. It combines with the detected movement vector. When the correction vector of the moving object is detected for each area, and the error component is included in the movement vector due to a high correlation with a part of the feature of the moving object, the movement vector and the correction vector A new movement vector obtained by synthesizing and can be set, and an error component can be removed. For this reason, when the moving object moves in the z direction with respect to the imaging area, a more appropriate movement vector can be obtained as compared with the movement vector detected by the conventional pattern matching processing.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an imaging apparatus that can obtain a more appropriate movement vector when a moving object moves in the z direction of the imaging area during moving image imaging.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a schematic configuration of a digital camera as an example of an imaging apparatus for explaining an embodiment of the present invention. A digital camera 100 shown in FIG. 1 includes an imaging unit 1 including a photographing lens 1a and an imaging element 1b, an analog signal processing unit 2, an A / D conversion unit 3, a driving unit 4, a lens driving unit 5, and a digital camera. Signal processing unit 6, compression / decompression processing unit 7, display unit 8, system control unit (CPU) 9, internal memory 10, media interface 11, recording medium 12, operation unit 13, pattern matching Device 15. The digital signal processing unit 6, compression / decompression processing unit 7, display unit 8, system control unit 9, internal memory 10, media interface 11, and pattern matching device 15 are connected to a system bus 14.

  The imaging unit 1 images a subject with an optical system 1a including a zoom lens and a photographing lens, and an imaging element 1b such as a CCD type or CMOS type solid-state imaging element, and outputs an analog imaging signal. The analog signal processing unit 2 performs predetermined analog signal processing on the imaging signal obtained by the imaging unit 1. The A / D conversion unit 3 converts the analog signal processed by the analog signal processing unit 2 into a digital signal.

  When the digital camera 100 is set to a photographing mode (a mode in which a subject can be photographed and photographed image data can be recorded), the driving unit 4 is driven by a solid-state imaging device 1b, The analog signal processing unit 2 and the A / D conversion unit 3 are driven. The digital camera 100 can set a moving image shooting mode and a still image shooting mode.

  When an instruction to change the optical zoom magnification is given from the operation unit 13, the lens driving unit 5 moves the zoom lens included in the optical system 1a in accordance with the instruction and changes the photographing magnification set in the image sensor 1b. The lens driving unit 5 is controlled by the system control unit 9.

  The digital signal processing unit 6 performs digital signal processing corresponding to the operation mode set by the operation unit 13 on the digital signal from the A / D conversion unit 3 to generate photographed image data. The processing performed by the digital signal processing unit 6 includes black level correction processing (OB processing), linear matrix correction processing, white balance adjustment processing, gamma correction processing, and synchronization processing. The digital signal processing unit 6 is configured by a DSP, for example. When the moving image shooting mode is set, the digital signal processing unit 6 sequentially outputs, for example, maximum resolution image data at a predetermined frame rate, and stores these in the internal memory 10. When an instruction to change the electronic zoom magnification is given from the operation unit 13, the digital signal processing unit 9 performs image processing according to the instruction and changes the photographing magnification set in the image sensor 1 b.

  The compression / decompression processing unit 7 performs compression processing on the captured image data generated by the digital signal processing unit 6 and performs expansion processing on the compressed image data obtained from the recording medium 12.

  The display unit 8 includes an LCD display device, for example, and displays an image based on photographed image data that has been photographed and subjected to digital signal processing. An image is also displayed based on the image data obtained by decompressing the compressed image data recorded on the recording medium 12. It is also possible to display a through image in the shooting mode, various states of the digital camera, information on operations, and the like.

  The system control unit 9 is mainly configured by a processor that operates according to a predetermined program, and performs overall control of the entire digital camera 100 including shooting operations. When there is an instruction to change the photographing magnification from the operation unit 13, the system control unit 9 notifies the lens driving unit 5 and the digital signal processing unit 6 of this instruction, and performs control to change the photographing magnification.

  The internal memory 10 is, for example, a DRAM and is used as a work memory for the pattern matching device 15, the digital signal processing unit 6, and the system control unit 9, and temporarily stores captured image data recorded on the recording medium 12. It is also used as a buffer memory for displaying image data for display on the display unit 8. The media interface 11 inputs / outputs data to / from a recording medium 12 such as a memory card.

  The operation unit 13 performs various operations when using the digital camera, and includes a release button (not shown) for instructing photographing.

  FIG. 2 is a block diagram showing a schematic configuration of the pattern matching device 15 shown in FIG. The pattern matching device 15 is based on the luminance information acquisition unit 21 that acquires luminance information of captured image data, the pattern matching processing unit 22 that executes pattern matching processing, and the luminance information acquired by the luminance information acquisition unit 21. And a determination unit 23 for determining whether or not to detect a correction vector described in the processing.

  The pattern matching device 15 further calculates a correction vector and combines it with the movement vector in order to correct the movement vector detected by the pattern matching processing unit 22, and further includes a search range setting unit 31 and a correction vector. A detection unit 32, a vector synthesis unit 33, and a frame position determination unit 34 are included. The process of detecting the correction vector and combining it with the movement vector will be described later.

  The luminance information acquired by the image device luminance information acquisition unit 21 and the movement vector detected by the pattern matching processing unit 22 are stored in the memory 35 functioning as a storage unit, and are read out as necessary. As the memory 35, the frame memory 10 of the imaging apparatus 100 shown in FIG. 1 can be used.

Next, a procedure for detecting a correction vector will be described.
FIG. 3 is a diagram illustrating a frame before the moving object moves. FIG. 4 is a diagram illustrating the frame after the moving object has moved.
As illustrated in FIG. 3, the frame F <b> 0 indicates a portion obtained by cutting out a predetermined area where the moving object T is captured in the image data. Note that the frame F0 in FIG. 3 is equivalent to the previous frame at the time of moving image capturing, and the frame F1 in FIG. 4 indicates the current frame.

  Before detecting the correction vector, the frames F0 and F1 are divided into a plurality of areas (here, four as an example) b0 to b3. In the present embodiment, the frame F0 is divided into four equal areas by dividing the frame F0 into two at equal intervals in the vertical and horizontal directions. In FIG. 4, the divided areas of the frame F1 correspond to the divided areas of the frame F0 of FIG.

  After detecting the movement vector of the moving object T in the frames F0 and F1 by the pattern matching processing unit 22, each of the areas divided in the frame F0 in the search range setting unit 31 of the pattern matching device 15 includes the frame F1. A correction vector search range R is newly set around.

5 to 8 are diagrams for explaining the procedure for detecting the correction vector in the search range for the divided areas.
As shown in FIG. 5A, the search range setting unit 31 sets a correction vector search range R that includes the area b0 and has a wide area with respect to the area b0 located at the upper left. Next, the correction vector detection unit 32 performs pattern matching processing on a part of the moving object T imaged in the area b0 of the frame F0 and the moving object T at the time point of the frame F1 shown in FIG. To detect the correction vector V0.

  Subsequently, as shown in FIG. 6A, the search range setting unit 31 sets a correction vector search range R that includes the area b1 and has a wide area with respect to the area b1 located on the upper right. Next, the correction vector detection unit 32 performs pattern matching processing on a part of the moving object T imaged in the area b1 of the frame F0 and the moving object T at the time point of the frame F1 shown in FIG. A correction vector V1 is detected. Similarly, as shown in FIG. 7A, a correction vector search range R including the area b2 and having a wide area is set for the area b2 located at the lower left. Next, the correction vector V2 is detected by performing pattern matching processing on a part of the moving object T imaged in the area b1 of the frame F0 and the moving object T at the time of the frame F1 shown in FIG. 7B. . Similarly, as shown in FIG. 8A, a correction vector search range R including the area b3 and having a wide area is set for the area b3 located at the lower right. Next, the correction vector V3 is detected by performing pattern matching processing on a part of the moving object T imaged in the area b3 of the frame F0 and the moving object T at the time of the frame F1 shown in FIG. 8B. .

  Next, the vector combining unit 33 combines the correction vectors V0 to V3 detected for the areas b0 to b3 and the movement vector detected in advance by the pattern matching process between the frames F0 and F1. The vector calculated by the synthesis is set as a new movement vector.

  Then, based on the new movement vector, the frame position determination unit 34 determines the correct frame position. The frame position is stored in the memory 35 as position information.

  According to the present invention, the frame F0 of the image data is divided into a plurality of areas b0 to b3, the moving components of the moving object T are detected as the correction vectors V0 to V3 for each of the divided areas b0 to b3, and the areas b0 to b3 are detected. The correction vectors V0 to V3 detected for each b3 are combined with the movement vector detected in advance by pattern matching. When the correction vector V0 to V3 of the moving object is detected for each of the areas b0 to b3, and an error component is included in the moving vector due to a high correlation with a part of the characteristic part of the moving object T In addition, an error component can be removed by setting a new movement vector obtained by combining the movement vector and the correction vectors V0 to V3. For this reason, when the moving object T moves in the z direction with respect to the imaging area, a more appropriate movement vector can be obtained as compared with the movement vector detected by the conventional pattern matching processing.

  In the above embodiment, the configuration of the imaging device including the pattern matching device 15 has been described. However, the pattern matching device 15 does not have to be provided in the imaging device, and is a single image processing device according to the present invention. You can also. In this case, after the image data picked up by the external image pickup device is input to the image processing device, the image processing device detects the movement vector and the correction vectors V0 to V3.

Next, the procedure of the image processing method according to the present invention will be described using the imaging apparatus of the present embodiment.
9 and 10 are flowcharts showing the procedure of the image processing method according to the present invention.

  First, the image data picked up by the image pickup device 1 b is subjected to analog signal processing, A / D conversion, and digital signal processing, and then input to the pattern matching device 15.

  The luminance information acquisition unit 21 of the pattern matching device 15 calculates a luminance range and a minimum value Min of the sum of absolute values of luminance differences as luminance information from the input image data.

  Here, the luminance range means a difference between the highest luminance value and the lowest luminance value in a range to be subjected to pattern matching processing in image data. When the brightness range is large, it indicates that the change in the image in the search range of the pattern matching process is large, and when the brightness range is small, it indicates that the change in the image in the search range of the pattern matching process is small.

  The calculation of the luminance range may be performed simultaneously with the pattern matching process described later in order to shorten the processing time.

  The minimum value Min of the sum of absolute values of the luminance differences is calculated by shifting the luminance difference in the search range between frames by a predetermined pixel (for example, one pixel), and all the calculated luminance differences are calculated. Calculate as the sum. Then, the value having the smallest sum is set as the minimum value Min. The minimum value Min of the sum of absolute values of luminance differences is used as the reliability of the movement vector. The smaller the minimum value Min of the sum of absolute values of luminance differences, the higher the correlation and the higher the reliability.

  After calculating the luminance range and the minimum sum Min of the absolute values of the luminance differences, the determination unit 23 determines whether the luminance range is greater than a preset threshold A. Here, when it is determined that the luminance range is smaller than the threshold value A, the luminance change of the input image data is small, and the accuracy of the detected motion vector is reduced. Therefore, the correction vector is not detected, It is determined that processing is impossible, and processing for inputting the next image is continued.

  When it is determined that the luminance range is larger than the threshold value A, the pattern matching processing unit 22 performs a pattern matching process between frames and detects a movement vector.

  After detecting the movement vector, it is determined whether or not the minimum value Min of the sum of absolute values of the previously calculated luminance difference is smaller than a preset threshold value B. When the minimum value Min is smaller than the threshold value B, the moving vector detected by the pattern matching process is adopted, and the tracking process of the moving object T is executed without executing the steps of detecting the correction vectors V0 to V3. .

  On the other hand, when the minimum value Min of the sum of absolute values of luminance differences is equal to or greater than the threshold value B, a flow for detecting a correction vector is executed. Specifically, the search range setting unit 31 divides the frame F0 into a plurality of areas b0 to b3, and sets a correction vector search range R for each of the divided areas b0 to b3. Note that the comparison between the minimum value Min of the sum of the absolute values of the luminance differences and the threshold value B may be performed simultaneously with or before the pattern matching process.

  For each of the plurality of areas b0 to b3, the correction vector detection unit 32 detects the moving components of the moving object T in the correction vector search range R as correction vectors V0 to V3.

  After detecting the correction vectors V0 to V3, the vector combining unit 33 combines the movement vectors previously detected by the pattern matching process and the correction vectors V0 to V3. The frame position determination unit 34 uses the synthesized vector as a new movement vector to determine the frame position by moving object tracking.

  When the correction vector V0 to V3 of the moving object is detected for each of the areas b0 to b3, and an error component is included in the moving vector due to a high correlation with a part of the feature of the moving object T In addition, a new movement vector obtained by combining the movement vector and the correction vectors V0 to V3 can be set to remove the error component. For this reason, when the moving object T moves in the z direction with respect to the imaging area, a more appropriate movement vector can be obtained as compared with the movement vector detected by the conventional pattern matching processing.

It is a figure which shows schematic structure of a structure of an imaging device. It is a figure which shows schematic structure of a pattern matching apparatus. It is a figure which shows the flame | frame before a moving object moves. It is a figure which shows the frame after a moving object moved. It is a figure explaining the procedure which detects a correction vector in the search range with respect to the divided area. It is a figure explaining the procedure which detects a correction vector in the search range with respect to the divided area. It is a figure explaining the procedure which detects a correction vector in the search range with respect to the divided area. It is a figure explaining the procedure which detects a correction vector in the search range with respect to the divided area. It is a flowchart which shows the procedure of an image processing method. It is a flowchart which shows the procedure of an image processing method. It is a figure explaining the state of the movement of the moving object in an imaging area. It is a figure explaining the position of the moving object with respect to a flame | frame.

Explanation of symbols

15 Pattern matching device (image processing device)
100 Digital camera (imaging device)
b0 to b3 Divided areas F0 and F1 Frame T Moving object V0 to V3 Correction vector

Claims (9)

An image processing apparatus that detects a movement vector of a moving object based on image data obtained by imaging,
A search range setting that divides a frame obtained by cutting out a predetermined area in which at least a part of the moving object is imaged in the image data into a plurality of areas and sets a correction vector search range for each of the divided areas Means,
Correction vector detection means for detecting, as a correction vector, a moving component of the moving object in the correction vector search range for each of the plurality of areas;
An image processing apparatus comprising: a vector synthesizing unit that synthesizes the movement vector and the correction vector, and newly sets the synthesized vector as the movement vector. Luminance information acquisition means for acquiring luminance information of the image data;
Pattern matching processing means for executing pattern matching processing for detecting the movement vector;
The image processing apparatus according to claim 1, further comprising: a determination unit that determines whether to detect the correction vector based on the luminance information.   The pattern matching processing unit compares the luminance ranges of a plurality of frames targeted for pattern matching processing, and executes the pattern matching processing when the luminance range is larger than a preset threshold value. Item 3. The image processing apparatus according to Item 2. An imaging apparatus including an image processing apparatus that detects a movement vector of a moving object based on image data obtained by imaging with an imaging element,
A search range in which the image processing device divides a frame obtained by cutting out a predetermined region in which at least a part of the moving object is imaged in the image data, and sets a plurality of correction vector search ranges for each of the divided areas. Setting means;
Correction vector detecting means for detecting a moving component of the moving object as a correction vector for each of the plurality of correction vector search ranges;
An image pickup apparatus comprising: vector combining means for combining the movement vector and the correction vector, and newly setting the combined vector as the movement vector. Luminance information acquisition means for the image processing apparatus to acquire luminance information of the image data;
Pattern matching processing means for executing pattern matching processing for detecting the movement vector;
The imaging apparatus according to claim 4, further comprising: a determination unit that determines whether to detect the correction vector based on the luminance information.   The pattern matching processing unit compares the luminance ranges of a plurality of frames targeted for pattern matching processing, and executes the pattern matching processing when the luminance range is larger than a preset threshold value. Item 6. The imaging device according to Item 5. An image processing method for detecting a movement vector of a moving object based on image data obtained by imaging,
Dividing a frame obtained by cutting out a predetermined area in which at least a part of the moving object is imaged in the image data, and setting a plurality of correction vector search ranges for each divided area;
Detecting a moving component of the moving object as a correction vector for each of the plurality of correction vector search ranges;
An image processing method comprising: synthesizing the movement vector and the correction vector, and newly setting the synthesized vector as the movement vector. Obtaining luminance information of the image data;
Performing a pattern matching process for detecting the movement vector;
The image processing method according to claim 7, further comprising a step of determining whether or not to detect the correction vector based on the luminance information. 9. The method according to claim 8, further comprising the step of performing pattern matching processing when the luminance ranges of a plurality of frames targeted for pattern matching processing are compared and the luminance range is greater than a preset threshold value. Image processing method.

Images