JP4985569B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method capable of detecting a moving body in a screen (image).

  An imaging apparatus such as a video camera has a function of recognizing a motion vector of a moving object. As a method of detecting such a motion vector, for example, a representative point matching method is used (for example, Patent Document 1).

  Such a representative point matching method is an image signal level in a field immediately before an arbitrary pixel defined as a representative point pixel, and an image signal level in a current field of all pixels in an area defined as a search range. The sum of absolute difference values is calculated to calculate a correlation value. Further, a highly correlated sampling point pixel having the smallest correlation value is obtained, and a difference in position between the sampling point pixel and the representative point pixel is determined as a motion vector. It is a method to do.

  When such a representative point matching method is used, for example, a motion vector for each block obtained by dividing a screen (image) into a plurality of blocks can be derived. However, since the actual motion of the object and the motion due to camera shake are superimposed on the motion vector, it is not possible to extract only the motion vector of the object. Under such circumstances, there is a case where the moving object which is a moving object is determined to be a camera shake, or the camera shake is determined to be a moving object, and the camera shake correction is not normally performed.

Therefore, the inventor of the present application has created a technique capable of detecting a motion vector of the entire screen with high accuracy regardless of the presence or absence of a moving object in the screen (for example, Patent Document 2).
Japanese Patent No. 3300500 JP 2007-235769 A

  By the technique of detecting the motion vector of the entire screen described above by the inventor of the present application, it is possible to extract only the motion vector of the entire screen, that is, the camera shake operation, and it is possible to perform highly accurate camera shake correction. By applying such a technique, it is possible to remove a camera shake element from an image being captured, and to distinguish between a stationary subject and a moving subject in the screen. Therefore, in the future, in order to give further added value, an appropriate extraction technique of the moving body for accurately performing the movement derivation and tracking of the moving body is desired.

  In addition, when imaging control such as focus adjustment and exposure adjustment is performed on a predetermined position on the screen, if the moving object crosses the predetermined position, the imaging control works on a subject (moving body) at a different distance. Therefore, there arises a problem that desired image quality cannot be maintained.

  In view of such a problem, the present invention uses the derived motion vector of the entire screen, and further appropriately and quickly extracts the motion vector of the mobile object and the mobile object, thereby performing various image processing using the mobile object. An object of the present invention is to provide an imaging apparatus and an imaging method capable of executing the above.

  In order to solve the above-described problem, a typical configuration of an imaging apparatus according to the present invention includes an imaging unit that captures an image of a subject and generates image data, and generates a motion vector for the entire screen based on the captured image data. A total vector generation unit, a vector subtraction unit that divides the screen into a plurality of blocks, subtracts the motion vector of the entire screen from the motion vector of each block, and generates an absolute motion vector for each block, and a predetermined value from the absolute motion vector A moving body exists in each block by a low-pass filter that extracts a low-frequency component of the cutoff frequency and a moving body entering and leaving the block based on the extracted low-frequency component of the absolute motion vector. And a moving body determination unit that determines whether or not.

  In the present invention, a pure motion vector (absolute motion vector) of a moving object is obtained by subtracting the motion vector of the entire screen from the motion vector of each block and further extracting only the low frequency component for each block to remove the camera shake component. Can be acquired. In addition, the existence of a moving object can be determined based on the low-frequency component of such an absolute motion vector, and various image processing such as derivation and tracking of the moving object by a monitoring camera or the like can be performed on the moving object. It becomes possible.

  The imaging apparatus integrates the low frequency components of the absolute motion vector extracted for each block, and the result is substantially equal to the block size, so that the moving object approach prediction unit predicts the approach of the moving object to the adjacent block. The mobile body determination unit may determine the presence of the mobile body also by predicting the approach of the mobile body.

  The above-described low-pass filter for eliminating a shake component may cause a recognition delay of a moving object in a block that the moving object enters. Such a moving body approach prediction unit can predict the moving body's approach to the block prior to the recognition of the moving body by the filter output, and can perform various image processing for the moving body at an appropriate timing. Become.

  The imaging apparatus further includes a moving object exit prediction unit that predicts the exit of the moving object by a timeout according to arrival of the estimated stay time of the moving object in the block. May be determined.

  When a moving body cannot be recognized due to an obstacle or the like in an arbitrary block, only an entrance prediction of the moving body may be performed by an adjacent block. However, when there is no actual entry determination, no exit prediction is made for the entry prediction. In the present invention, since the exit prediction is executed using a timeout based on the estimated stay time, the moving object determination unit can determine the presence of the moving object at an appropriate timing when the moving object passes.

  When it is determined that a moving body exists, the imaging apparatus may further include a target exclusion unit that excludes a block in which the moving body exists from an imaging control target including one or both of focus adjustment and exposure adjustment.

  With such a configuration, it is possible to eliminate the imaging control for the block in which the moving body exists, and to avoid the phenomenon that the erroneous imaging control is performed when the moving body crosses in front of the subject that is the imaging control target, High-quality image data desired by the photographer can be maintained.

The whole vector generation unit is the same as a block vector deriving unit that divides a screen into a plurality of blocks and derives a motion vector of each block, and a grouping unit that groups a plurality of blocks having similar motion vectors of each block Motion vector reliability that determines whether or not a group includes a moving object based on the motion vector history of a plurality of blocks belonging to the group and the motion vector history of the entire screen and excludes the group including the moving object A sex determination unit and an entire vector deriving unit that derives a motion vector of the entire screen using a motion vector of blocks that form a group that does not include a moving object may be included.

  With this configuration, it is possible to appropriately and quickly extract a motion vector of the entire screen, which is a camera shake component, and perform highly accurate camera shake correction. Further, the extraction accuracy of the moving body in the subsequent stage is improved, and various image processing such as derivation and tracking of the moving body and switching of the imaging control target can be performed.

  A typical configuration of the imaging method according to the present invention is to generate an image data by imaging a subject, generate a motion vector of the entire screen based on the captured image data, divide the screen into a plurality of blocks, The motion vector of the entire screen is subtracted from the motion vector of each block to generate an absolute motion vector for each block, and the low frequency component of the predetermined cutoff frequency is extracted from the absolute motion vector. It is characterized in that it is determined whether or not there is a moving body in each block by determining whether the moving body enters and leaves each block based on the band component.

  The above-described components based on the technical idea of the imaging apparatus and the description thereof can be applied to the imaging method.

  According to the present invention described above, various image processing using a moving object is performed by using the derived motion vector of the entire screen and further appropriately and quickly extracting the motion vector of the moving object and the moving object. it can.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

  Imaging devices such as video cameras have an autofocus (AF) function that automatically focuses the main subject and an autoexposure (AE) function that automatically adjusts exposure according to the shutter speed and aperture size. Thus, it is possible to clearly image the subject by correcting the difference in the imaging environment by such imaging control (focus adjustment, exposure adjustment, etc.). Further, since focus adjustment and exposure adjustment are performed only on a subject located at an arbitrary distance, a subject such as a person desired by the photographer is usually selected as the target.

  However, when a moving body crosses in front of a subject to be subjected to imaging control, there arises a problem that imaging control is applied to the moving body and normal imaging control is not performed. In this embodiment, it is possible to execute various image processing using a moving object by using the derived motion vector of the entire screen and further extracting the moving object and the moving vector of the moving object appropriately and quickly. Further, by removing the moving body from the target of the imaging control, normal imaging control can be performed regardless of the presence or absence of the moving body. Hereinafter, a typical imaging apparatus and imaging method of the present embodiment will be described.

(Imaging device 100)
FIG. 1 is an external view illustrating an example of an imaging apparatus 100 according to the present embodiment. The imaging apparatus 100 is often portable and includes a main body 102, an imaging lens 104, operation keys 106, and a viewfinder 108.

  The main body 102 records the image data captured through the imaging lens 104 so that it can be viewed again, and the recording timing and angle of view are adjusted in accordance with user input to the operation keys 106. In addition, the focus and exposure of a desired subject in the screen in the image data are automatically adjusted to a desired value. The viewfinder 108 is composed of a liquid crystal display, an organic EL (Electro Luminescence), etc., and can visually recognize image data that is actually recorded. The photographer can capture the subject at a desired position and size while referring to the image displayed on the viewfinder 108.

  FIG. 2 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the present embodiment. The imaging apparatus 100 includes an imaging unit 120, a central control unit 122, a signal output unit 124, and a recording unit 126.

  The imaging unit 120 captures a subject through the imaging lens 104 and generates image data. Specifically, the imaging unit 120 converts light incident through the focus lens 140 used for focus adjustment, the aperture 142 used for exposure adjustment, and the imaging lens 104 into an electrical signal, and performs signal processing such as color separation and γ correction. An image pickup circuit (image pickup element) 144, a focus lens 140, and a drive circuit 146 that drives the diaphragm 142 are configured. Image data captured by the image capturing unit 120 is transferred to the central control unit 122.

  The central control unit 122 manages and controls the entire imaging device 100 by a semiconductor integrated circuit including a central processing unit (CPU) and a signal processing device (DSP: Digital Signal Processor), and is based on image data captured by the imaging unit 120. Then, the image processing control such as the motion vector of the entire screen, extraction of the moving body, focus adjustment and exposure adjustment is executed. The detailed configuration of the central control unit 122 will be described later.

  The signal output unit 124 processes the image data subjected to camera shake correction into a viewable image signal and outputs the image signal to the viewfinder 108. The photographer can specify the imaging target while visually recognizing the video of the viewfinder 108. Here, the output destination of the image signal is the viewfinder 108, but it can be connected to various image display devices such as a separate monitor.

  The recording unit 126 generates a recording signal (data stream) encoded through the encoding process, and records the image data on an arbitrary recording medium 148. As an arbitrary recording medium 148, a medium that does not require a power source such as a DVD or a BD, or a medium that requires a power source such as a RAM, an EEPROM, a nonvolatile RAM, a flash memory, or an HDD can be applied. Also, a separate recording medium that can be connected from the outside can be used.

  The central control unit 122 includes a block vector deriving unit 150, a correlation reliability determining unit 152, a grouping unit 154, a motion vector reliability determining unit 156, a distance reliability determining unit 158, and an overall vector deriving unit. 160, a vector subtraction unit 162, a low-pass filter 164, a mobile object entry determination unit 166, a mobile object exit determination unit 168, a mobile object approach prediction unit 170, a mobile object exit prediction unit 172, and a mobile object It also functions as a determination unit 174, a camera shake correction unit 176, and a target exclusion unit 178. In particular, the block vector deriving unit 150, the correlation reliability determining unit 152, the grouping unit 154, the motion vector reliability determining unit 156, the distance reliability determining unit 158, and the overall vector deriving unit 160 are based on the captured image data. It functions as an overall vector generation unit 180 that generates a motion vector of the entire screen.

  The block vector deriving unit 150 divides the image (screen) of the image data captured by the imaging unit 120 into a plurality of blocks, for example, four blocks in the horizontal direction and four blocks in the vertical direction, and blocks through the representative point matching method. Each motion vector is detected. In this representative point matching method, each block is further divided into eight pixels in the horizontal direction and four pixels in the vertical direction to obtain a sampling point pixel highly correlated with the representative point pixel in the previous field, and the representative point pixel The motion vector is obtained from the difference in position between In addition, correlation value data such as the minimum value, average value, and gradient of the correlation value is derived for each block together with the motion vector, and transmitted to the correlation reliability determination unit 152 and the vector subtraction unit 162.

  The correlation reliability determination unit 152 is effective in determining the reliability of each block using the motion vector from the block vector deriving unit 150 and the correlation value data, that is, whether each block generates a motion vector for the entire screen. Determine whether or not. Therefore, the block determined as invalid by the correlation reliability determination unit 152 is not used for the subsequent processing.

  The grouping unit 154 compares the motion vectors of the blocks with respect to the blocks determined to be valid by the correlation reliability determination unit 152, determines whether the motion vectors are similar, and the motion vectors are similar. Group blocks. In this way, all blocks determined to be valid can be divided into an arbitrary number of groups.

  Specifically, the sum of the absolute difference value in the horizontal direction and the absolute difference value in the vertical direction of the motion vectors of the two blocks to be compared (hereinafter referred to as “block correlation value”) is obtained, and the calculated block correlation value is smaller than the correlation threshold value. In some cases, it is determined that the two blocks to be compared belong to the same group (similar motion vectors). More specifically, when the motion vector of an arbitrary block determined to be valid is (X1, Y1) and the motion vector of another block also determined to be valid is (X2, Y2), the block correlation value is | It is calculated from X1-X2 | + | Y1-Y2 |. When the calculated block correlation value is smaller than the correlation threshold, it is determined that the two blocks belong to the same group.

  The motion vector reliability determination unit 156 uses the motion vector history data for the blocks grouped by the grouping unit 154 to determine the reliability of the group, that is, whether the group is caused by the moving object. And invalidate the block of the group caused by such a moving body.

…（数式１）

…（数式２）
ここでは、同一履歴時刻をｎとし、現時刻を０、１フィールド前の時刻を１とし、ｎフィールド前の時刻であればｎとなる。また、ｋはグループ内におけるブロックの識別番号であり、ＮＢＬはグループに含まれる総ブロック数である。従って、ｋは１からＮＢＬまでの値をとる。 More specifically, first, group motion vectors (GXAV [n], GYAV [n] are obtained by averaging the motion vectors (BXk [n], BYk [n]) of blocks belonging to the same group at the same history time. ]) Is calculated using Equation 1 and Equation 2. Here, GXAV [n] is the X component (horizontal component) of the group motion vector, and GYAV [n] is the Y component (vertical component).

... (Formula 1)

... (Formula 2)
Here, the same history time is n, the current time is 0, the time one field before is 1, and the time is n fields before n. K is an identification number of a block in the group, and NBL is the total number of blocks included in the group. Therefore, k takes a value from 1 to NBL.

…（数式３）

…（数式４）
ただし、ＦＸ[n]、ＦＹ[n]は、画面全体の動きベクトルを表し、ＮＲＦは、所定フィールド数である。ここで積算の対象に現時点（ｎ＝０）が含まれていないのは、画面全体の動きベクトルがそもそも最終的な計算値であって、最初のフィールドの動きベクトル（ＦＸ[0]，ＦＹ[0]）は算出されていないからである。 Next, SXP, which is the X component of the first determination parameter, and SYP, which is the Y component, are calculated using Equation 3 and Equation 4, respectively.

... (Formula 3)

... (Formula 4)
However, FX [n] and FY [n] represent the motion vectors of the entire screen, and NRF is the predetermined number of fields. Here, the current target (n = 0) is not included in the target of integration because the motion vector of the entire screen is the final calculated value in the first place, and the motion vector (FX [0], FY [ 0]) is not calculated.

  The first determination parameters SXP and SYP represent the degree of similarity between the group motion vector and the motion vector of the entire screen, and indicate that the similarity between the motion vectors decreases as the values of the first determination parameters SXP and SYP increase. ing. Therefore, when the value of the first determination parameter is large, there is a high possibility that the detected group motion vector is not caused by panning operation or camera shake.

…（数式５）

…（数式６）
ここで、第２判定パラメータは、直近（ＮＲＦ＋１）のフィールドにおけるグループ動きベクトルのＸ成分およびＹ成分の積算値の絶対値であり、グループ動きベクトルが手振れによるものであれば小さな値に収束する。従って、第２判定パラメータＧＸＩまたはＧＹＩの値が大きいときは、そのグループは動く物体を含むか、カメラのパンニング中であるかのいずれかの状態であることが考えられる。 Next, the second determination parameters GXI and GYI are calculated using Equation 5 and Equation 6.

... (Formula 5)

... (Formula 6)
Here, the second determination parameter is the absolute value of the integrated value of the X and Y components of the group motion vector in the most recent (NRF + 1) field, and converges to a small value if the group motion vector is due to camera shake. Therefore, when the value of the second determination parameter GXI or GYI is large, the group may be in a state where either the moving object is included or the camera is panning.

  The motion vector reliability determination unit 156 compares the first determination parameters SXP and SYP and the second determination parameters GXI and GYI calculated as described above with the corresponding predetermined threshold values SXTH, SYTH, GXTH, and GYTH. If the conditions of SXP> SXTH and GXI> GXTH and SYP> SYTH and GYI> GYTH are satisfied, it is determined that the group includes a moving body, and the group is invalidated in the calculation of the motion vector of the entire screen. . This is because, when such a condition is satisfied, it can be determined that the motion vector of the group is not due to camera panning and is not due to camera shake.

  In this way, by excluding the motion vector due to the moving body existing in the screen, the motion vector of the entire screen due to the camera shake can be detected with high accuracy. However, in the determination using the motion vector history by the motion vector reliability determination unit 156, after the mobile object enters an arbitrary block, the mobile object is recognized only after spending several fields of time. Therefore, immediately after entering an arbitrary block, the moving body cannot be detected, and it is determined that the block does not include the moving body, that is, an effective block. Therefore, the imaging apparatus 100 according to the present embodiment also includes a distance reliability determination unit 158 that recognizes and invalidates the moving body immediately after entering.

  The distance reliability determination unit 158 performs reliability determination using a group distance that is a parameter indicating the degree of spread of the group on the screen. Specifically, the sum of the number of blocks in the horizontal direction (horizontal distance) and the number of blocks in the vertical direction (vertical distance) is calculated, and the blocks in the group with the smaller sum are used for the calculation of motion vectors for the entire screen. Do not be. This is because when the moving body enters the screen, it affects only a part of the screen, and the group does not occupy a very wide range for the entire screen. Therefore, the block immediately after entering the moving body also has a small group distance.

  On the other hand, camera shake is a movement that affects the entire screen, and groups representing camera shake occupy a wide area of the screen and are scattered on the screen, so the group distance representing camera shake has a large value (for example, the maximum value). Become. When there are a plurality of groups with the same distance (sum) having the same value, the average value of the motion vectors of all the blocks included in the plurality of effective groups is calculated and used as the motion vector of the entire screen. By determining the group distance in this way, it is possible to exclude a block including a moving body immediately after entering the screen and accurately detect a motion vector due to camera shake.

  The overall vector deriving unit 160 derives the motion vector of the entire screen using the motion vectors of the blocks constituting one or more groups determined to be valid by the motion vector reliability determining unit 156 and the distance reliability determining unit 158. .

  At this time, if there is no effective group in the screen, the motion vector of the entire screen is multiplied by an arbitrary constant α (0 <α <1) by the motion vector of the entire screen one field before. Value. Thus, even when a moving object exists in the screen and the moving object stays on the screen for a certain period of time, or even immediately after the moving object enters the screen, the motion vector of the entire screen is highly accurate. Can be detected.

  The vector subtracting unit 162 subtracts the motion vector of the entire screen derived by the entire vector deriving unit 160 from the motion vector of each block derived by the block vector deriving unit 150 to generate an absolute motion vector for each block. The motion vector of each block derived by the block vector deriving unit 150 indicates a relative value including a shake component. In order to derive an absolute motion vector due to the moving object from the motion vector, the motion vector of the entire screen, which is a shake component, is subtracted as described above.

  However, the block determined to be invalid by the correlation reliability determination unit 152, the motion vector reliability determination unit 156, and the distance reliability determination unit 158 described above is highly likely to present an incorrect vector. A value obtained by multiplying the motion vector before the field (previous value) by an arbitrary constant β (0 <β <1) is set as the current value.

  The low-pass filter (LPF) 164 extracts a low-frequency component having a predetermined cutoff frequency (for example, 1 Hz) for each block in order to further remove a shake component from the absolute motion vector derived by the vector subtraction unit 162. With such a low-pass filter 164, even if the motion vector value of the entire screen is not appropriate and the vector subtraction unit 162 cannot completely eliminate the shake component, it is possible to extract an accurate absolute motion vector of each block. it can. In addition, it is possible to remove, for example, CCD noise of 15 Hz or higher.

…（数式７）
ただし、ａ_０は０．０４９９１０４、ａ_１は０．０４９９１０４、ｂ_１は−０．９００１７９２である。 Usually, camera shake is accompanied by vibration of 1 to 15 Hz. Therefore, a low-pass filter having a cutoff frequency of 1 Hz and a sampling frequency of 60 Hz (NTSC system, field unit) is realized by an IIR (Infinite Impulse Response) filter. The transfer function at this time is expressed by Equation 7.

... (Formula 7)
However, _{a 0} is .0499104, _{a 1} is .0499104, _{b 1} is -0.9001792.

When such a low-pass filter is realized on a program, it is executed according to the following procedure.
out = a ₁ x filIn1 + a ₀ x in-b ₁ x filOut1;
filIn1 = in;
filOut1 = out;
Here, in is a filter input, out is a filter output, and filIn1 and filOut1 are registers for storing previous values. Also, paying attention to the relationship between the coefficients a ₁ = a ₀ and b ₁ = 1−2 × a ₀ ,
out = a ₀ x (in + filIn1 + filOut1 <<1)-filOut1;
Thus, the number of multiplications can be reduced to one. However, “<< 1” indicates arithmetic processing for shifting 1 bit to the left.

  In this way, by removing the hand shake band (1 to 15 Hz) from the absolute motion vector, only a pure moving component of the moving body can be extracted.

  The mobile object entry determination unit 166 determines the entry of the mobile object for each block based on the filter output of the low-pass filter 164.

  However, the filter output is delayed due to the phase characteristics of the low-pass filter 164 described above, and when the output of the low-pass filter 164 reaches an arbitrary threshold that can be recognized as a mobile object, the mobile object has already moved. The mobile object recognition by the mobile object entry determination unit 166 is delayed.

  FIG. 3 is an explanatory diagram for explaining the output delay of the low-pass filter 164. When the signal having the stay time and the input value as shown in FIG. 3A is generated by the vector subtraction unit 162 due to the moving body entering, the filter output of the low-pass filter 164 is a solid line in FIG. A delay with a time constant occurs as shown in Fig. 5, and a delay time d is spent until the threshold value δ at which it can be determined that the mobile body has entered is reached.

  Therefore, the moving body entry determination unit 166 determines the entry of the moving body using the same direction counter in addition to the output of the low-pass filter 164. The same direction counter counts the number of times that the absolute motion vector of the block continuously indicates the same direction, and is reset when indicating a different direction.

  When the moving speed of the moving body is relatively slow (when it is equal to or smaller than a predetermined threshold), the moving body approach determination unit 166 satisfies the condition of the same direction counter> γ and the filter output value> ε (where 0 <γ , 0 <ε), and as shown in FIG. 3B, the moving body determination flag indicating that the moving body exists in the block is turned on.

  Further, when the moving speed of the moving body is relatively fast (when it is equal to or greater than a predetermined threshold) 202, only under the condition of the filter output value> δ (where ε <δ), the moving body as shown in FIG. Turn on the confirmation flag. The reason why the same direction count is not used when the moving speed is high is that the moving object may pass through the target block before the same direction counter exceeds a predetermined threshold value.

  When it is determined that the mobile object has passed through each block, the mobile object exit determination unit 168 turns off the mobile object determination flag set by the mobile object entry determination unit 166. Therefore, when the moving object determination flag is on, it is recognized that a moving object exists in the block.

  The mobile object exit determination unit 168 determines the exit of the mobile object based on the output of the low-pass filter 164 as in the mobile object entrance determination unit 166. In order to compensate for the delay of the low-pass filter 164, the mobile object exit determination unit 168 The transition gradient (slope) is also used.

  FIG. 4 is an explanatory diagram for explaining the gradient of the transition of the filter output. As shown in FIG. 4, the gradient is different between the case 210 in which the moving body leaves each block at the same speed as when entering, and the case 212 in which the speed gradually decreases and stops in the block. In particular, when the moving body passes through the block at a high speed, the gradient becomes tight (the inclination angle becomes large), and the exit of the moving body can be determined when the gradient becomes steep. Specifically, when the absolute value of the gradient obtained by subtracting the filter output before a predetermined time from the current filter output is equal to or greater than a predetermined value, the moving object determination flag is turned off. Also, it can be determined that the moving object has left when the filter output is close to zero.

  Furthermore, in order to avoid such an erroneous determination of leaving due to the gradient, the moving object leaving determination unit 168 constantly updates the maximum value of the filter output while the moving object determination flag is on, and the filter output is the maximum value. The slope is calculated after By doing so, it is not determined that the time when the vehicle enters the slope having a positive value is erroneously exited.

  The mobile object approach prediction unit 170 predicts the entry of the mobile object for each block independently of the mobile object entry determination unit 166. Although the moving object exit determination unit 168 also uses the same direction counter to recognize the moving object somewhat faster, it is possible to reduce the number of fields used to determine whether the motion vector is a shake or a moving object. Can not. Therefore, the moving object approach prediction unit 170 performs the moving object prediction. The moving object approach prediction unit 170 can predict which block the moving object will enter next from the motion vector and the direction of the motion vector. Then, the mobile object prediction flag of the block where the mobile object is predicted to enter is turned on.

  FIG. 5 is an explanatory diagram for explaining the operation of the mobile object approach prediction unit 170. As shown in FIG. 5A, when a truck (lorry vehicle) 222 as a moving body is about to enter the predetermined block 220, the moving body determination flag of the block 220 is immediately turned on by the moving body entering determination unit 166. Don't be. However, since the mobile object approach prediction unit 170 can predict the approach of the track 222 to the block 220, the mobile object prediction flag of the block 220 is turned on. After several fields, the moving body approach determination unit 166 also recognizes the presence of the track 222, and both the moving body prediction flag and the moving body determination flag are turned on as shown in FIG. 5B.

  The moving body approach prediction unit 170 accumulates absolute motion vectors for each block while the moving body determination flag controlled by the moving body entry determination unit 166 and the moving body exit determination unit 168 is on.

  FIG. 6 is an explanatory diagram for explaining integration of absolute motion vectors. FIG. 6 shows a situation in which the track 222 as a moving object moves through the block 220 in the order of FIGS. 6A, 6 </ b> B, 6 </ b> C, and 6 </ b> D and enters the adjacent block 224. . The moving object approach predicting unit 170 performs absolute motion vectors A and B indicated by arrows in FIGS. 6A, 6B, 6C, and 6D while the moving object determination flag of the block 220 is on. , C and D are integrated, and the integrated value becomes substantially equal to the block size (including the block interval when there is an interval between blocks) as shown in FIG. It can be determined that the vehicle has entered or is about to enter the adjacent block 224, and the mobile object prediction flag of the block 224 is turned on.

  Ideally, the integrated value of the absolute motion vector for each block matches the block size, but the time during which the moving object determination flag is on due to the output delay of the low-pass filter 164 is the actual track 222. The block size is not reached because it is different from the stay time. Therefore, it is desirable to set the target value of the integrated value of the absolute motion vector that can be determined that the track 222 enters the block 224 to a value smaller than the block size. This small value may be a value obtained by subtracting a predetermined value from the block size or a value obtained by multiplying the block size by a predetermined value ζ (0 <ζ <1). In addition, since a moving body having a high speed may not be correctly recognized only by accumulating absolute motion vectors, the same direction counter can be used in combination with the moving body determination flag on.

  As described above, the moving object approach prediction unit 170 determines the possibility that the same moving object enters the block 224 adjacent to the moving direction side of the block 220 in which the moving object determination flag is turned on. In order to make a more accurate determination, the similarity between the motion vectors and the minimum correlation value can be confirmed.

  FIG. 7 is an explanatory diagram for explaining confirmation of similarity of motion vectors. In FIG. 7, as in FIG. 6, a track 222 as a moving body passes through a block 224 adjacent to a predetermined block 220. If the same track 222 exists in the block 220 in which the presence of the moving object is detected and the block 224 adjacent to the block 220, the motion vectors indicated by the arrows in the two blocks should show close values. Therefore, if the difference between the motion vectors of the predetermined block 220 and the adjacent block 224 is within a predetermined range, it can be determined that the motion vectors are similar, and the mobile object prediction flag is kept on.

  FIG. 8 is an explanatory diagram for explaining confirmation by the minimum correlation value. In the block vector deriving unit 150 described above, the representative point matching method is used to calculate the motion vector. The representative point matching method is a means for obtaining the position of an image that matches the current image and the previous image. Under such a representative point matching method, when a moving object such as a track 222 enters a block, the current image 226 and the previous image 228 do not match, and the minimum correlation value becomes larger than the previous value. The approach of the moving object can be recognized by such a displacement of the minimum correlation value, and the moving object prediction flag can be kept on.

  The mobile object exit prediction unit 172 predicts the exit of the mobile object for each block independently of the mobile object exit determination unit 168.

  FIG. 9 is an explanatory diagram for explaining the operation of the moving object exit prediction unit 172. FIG. 9A corresponds to FIG. 5B, and FIG. 9B shows a state after a predetermined time of FIG. 5B. When the mobile object exit determination unit 168 recognizes the exit of the track 222 of the target block 220 in a state where both the mobile object prediction flag and the mobile object confirmation flag are on as shown in FIG. As shown in FIG. 9B, in response to the moving object leaving determination unit 168 turning off the moving object determination flag, the moving object leaving prediction unit 172 also turns off the moving object prediction flag in the same block 220.

  On the other hand, if the mobile object is hidden behind the object or the mobile object stops immediately before, the mobile object exit prediction unit 172 falls into a state where the mobile object prediction flag cannot be turned off forever. End up.

  FIG. 10 is an explanatory diagram for explaining the operation of the mobile object prediction flag when the mobile object is hidden behind the object. First, in block 220, the approach of the track 222 to the adjacent block 224 is predicted, and the mobile object prediction flag in the block 224 is turned on as shown in FIG. However, the block 224 has a shielding object 232 having the boundary 230 as an end between the imaging device 100 and the track 222, and the moving object entry determination unit 166 of the block 224 cannot recognize the entry of the track 222. . Accordingly, the moving object determination flag in the block 224 is not turned on, and naturally, the opportunity for the moving object determination flag to be turned off from on does not come.

  In order to avoid such a state, as shown in FIG. 10B, when the moving body determination flag is turned off as shown in FIG. 10B, the track 222 enters the block 220 in which the track 222 having the moving body prediction flag turned on exists. Estimate the stay time of the track 222 in the wax block 224, and at the time out of the estimated time, that is, when the track 222 leaves the block 224 as shown in FIG. Confirm that the mobile object prediction flag is turned off. Such timeout is done using a counter that decrements by one for each field.

  The mobile body determination unit 174 is based on the mobile body determination flag and the mobile body prediction flag controlled by the mobile body entrance determination unit 166, the mobile body exit determination unit 168, the mobile body entrance prediction unit 170, and the mobile body exit prediction unit 172. , It is determined whether or not there is a moving object in each block.

  The camera shake correction unit 176 corrects screen shake due to camera shake by displacing the captured image data in the reverse direction by the motion vector of the entire screen derived by the entire vector deriving unit 160. At this time, it is possible to perform the camera shake correction more reliably and with high accuracy by invalidating the block determined by the moving object determination unit 174 as having a moving object.

  The object excluding unit 178 matches the block in which the subject that is subject to imaging control including either or both of the focus adjustment and the exposure adjustment matches the block in which the moving object determining unit 174 determines that the moving object exists. In this case, the drive circuit 146 is instructed to exclude the block in which the moving body is present from the target of the imaging control. As means for excluding from the target, it is conceivable to move or enlarge the target area of the imaging control or to switch to a predetermined area.

  With such a configuration, it is possible to eliminate the imaging control for the block in which the moving body exists, and to avoid the phenomenon that the erroneous imaging control is performed when the moving body crosses in front of the subject that is the imaging control target, High-quality image data desired by the photographer can be maintained.

(Imaging method)
An imaging method capable of detecting a moving body in the screen is also provided.

  FIG. 11 is a flowchart showing specific processing of the imaging method. First, when a photographer starts imaging using the imaging device 100, the imaging unit 120 of the imaging device 100 captures an object and generates image data (S300).

  Then, the block vector deriving unit 150 divides the captured image into a plurality of blocks, derives a motion vector of each block (S302), and the grouping unit 154 groups a plurality of blocks having similar motion vectors. (S304).

  Subsequently, the motion vector reliability determination unit 156 determines whether the group includes a moving body based on the motion vector history of a plurality of blocks belonging to the same group and the motion vector history of the entire screen. Then, the distance reliability determination unit 158 determines the reliability using the group distance (S308), and the overall vector deriving unit 160 uses the motion vectors of the blocks constituting the group not including the moving object. A motion vector of the entire screen is derived (S310). Such a motion vector of the entire screen can be used for camera shake correction.

  Next, the vector subtracting unit 162 subtracts the motion vector of the entire screen from the motion vector of each block to generate an absolute motion vector for each block (S312), and the low-pass filter 164 determines a predetermined value from the absolute motion vector. A low frequency component of the cutoff frequency is extracted (S314). The moving body determination unit 174 determines whether or not there is a moving body in each block based on the determination of the moving body entering and leaving each block based on the low-frequency component of the extracted absolute motion vector (S316). .

  Finally, when it is determined that a moving body is present in an arbitrary block, the target excluding unit 178 excludes the block in which the moving body is present from the target of imaging control including one or both of focus adjustment and exposure adjustment. (S318).

  Even in such an imaging method, it is possible to execute various image processing using a moving object, and normal imaging control is performed regardless of the presence or absence of the moving object by excluding the moving object from the target of imaging control. be able to.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  Note that each step in the imaging method of the present specification does not necessarily have to be processed in time series in the order described in the flowchart, and may include parallel or subroutine processing.

  The present invention can be used for an imaging apparatus and an imaging method capable of detecting a moving body in a screen (image).

It is the external view which showed an example of the imaging device in this embodiment. It is a block diagram which shows the structure of the imaging device in this embodiment. It is explanatory drawing for demonstrating the output delay of a low-pass filter. It is explanatory drawing for demonstrating the gradient of transition of a filter output. It is explanatory drawing for demonstrating operation | movement of a mobile body approach prediction part. It is explanatory drawing for demonstrating the integration of an absolute motion vector. It is explanatory drawing for demonstrating confirmation of the similarity of a motion vector. It is explanatory drawing for demonstrating confirmation by the minimum correlation value. It is explanatory drawing for demonstrating operation | movement of a mobile body exit prediction part. It is explanatory drawing for demonstrating operation | movement of the mobile body prediction flag when a mobile body is hidden behind a thing. It is the flowchart which showed the specific process of the imaging method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Imaging device 120 ... Imaging part 150 ... Block vector derivation | leading-out part 152 ... Correlation reliability determination part 154 ... Grouping part 156 ... Motion vector reliability determination part 158 ... Distance reliability determination part 160 ... Whole vector derivation part 162 ... Vector Subtractor 164 ... Low-pass filter 166 ... Mobile object entry determination unit 168 ... Mobile object exit determination part 170 ... Mobile object entry prediction part 172 ... Mobile object exit prediction part 174 ... Mobile object determination part 176 ... Camera shake correction part 178 ... Target Exclusion unit 180 ... Whole vector generation unit

Claims (6)

An imaging unit that images a subject and generates image data;
An overall vector generation unit that generates a motion vector of the entire screen based on the captured image data;
Dividing the screen into a plurality of blocks, subtracting the motion vector of the entire screen from the motion vector of each block to generate an absolute motion vector for each block; and
A low-pass filter that extracts a low-frequency component of a predetermined cutoff frequency from the absolute motion vector;
A moving body determination unit that determines whether or not a moving body exists in each block based on a low-frequency component of the extracted absolute motion vector based on the moving body entering and exiting each block; and
An imaging apparatus comprising: A moving body approach prediction unit is further provided that accumulates the low frequency components of the extracted absolute motion vector for each block, and predicts the approach of the moving body to an adjacent block by the result being substantially equal to the block size. ,
The imaging apparatus according to claim 1, wherein the moving body determination unit determines the presence of the moving body also by predicting an approach of the moving body. The mobile unit further includes a mobile unit exit prediction unit that predicts the mobile unit's exit based on a time-out according to reaching the estimated stay time of the mobile unit in the block,
The imaging apparatus according to claim 1, wherein the moving body determination unit determines the presence of a moving body also by predicting the exit of the moving body.   And a target excluding unit that excludes a block in which the moving body is present from an imaging control target including one or both of focus adjustment and exposure adjustment when it is determined that the moving body exists. Item 4. The imaging device according to any one of Items 1 to 3. The overall vector generation unit
A block vector deriving unit that divides the screen into a plurality of blocks and derives a motion vector of each block;
A grouping unit that groups a plurality of blocks having similar motion vectors of each block ;
Motion that determines whether or not the group includes a moving object based on the motion vector history of a plurality of blocks belonging to the same group and the motion vector history of the entire screen, and excludes the group that includes the moving object A vector reliability determination unit;
An overall vector derivation unit for deriving a motion vector of the entire screen using motion vectors of blocks constituting the group not including the moving object;
5. The imaging apparatus according to claim 1, comprising: Capture the subject, generate image data,
Generating a motion vector for the entire screen based on the imaged image data;
Dividing the screen into a plurality of blocks, subtracting the motion vector of the entire screen from the motion vector of each block to generate an absolute motion vector for each block,
Extracting a low frequency component of a predetermined cutoff frequency from the absolute motion vector;
An imaging method comprising: determining whether or not a moving body exists in each block by determining whether the moving body enters and leaves each block based on the low-frequency component of the extracted absolute motion vector.

Images