JP2010211398A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of making an image process to be performed in high speed by accelerating calculation of motion detection when the motion detection of an image block which includes a plurality of frames is performed. <P>SOLUTION: A feature point map 15 is searched into a raster direction, then a table block corresponding to an identification number of a motion information table 17 is found out by using the identification number of a feature point when the feature point exists at the place. Since a next tracking block ID (namely, the number indicating the table block to be processed next) is memorized in the table block, data (feature point coordinate and next tracking coordinate) of the table block which is indicated by the next tracking block ID is utilized, then a next tracking coordinate and a motion vector can be calculated. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that processes an image based on a moving state of image feature points.

  2. Description of the Related Art Conventionally, as an image processing device that captures and processes an image around a vehicle, a device that uses a motion vector of a moving image is known. As a method of detecting this motion vector, in order to reduce false detection of a motion vector, a feature point is determined from an image once stored in an image memory, and a template image surrounding the feature point is located anywhere on the image after one frame. There has been proposed a method for calculating a motion vector indicating whether or not it has moved (for example, see Patent Documents 1, 2, and 3).

These prior arts extract feature points on each image block (unit of image for motion detection), and calculate motion vectors using the feature points.
In recent years, the present inventor has proposed a technique for speeding up image processing by speeding up calculation of motion vectors (see Patent Document 4).

  This technique realizes high-speed processing by optimizing data transfer of image blocks in the raster direction based on a feature point map generated at the time of image capture. Specifically, the data transfer method is switched according to the distance between the feature points on the feature point map.

JP-A-8-242453 JP-A-9-55943 JP-A-9-55944 JP 2008-192060 A

  Since the technique of the cited document 4 describes the motion between two frames, considering the case of detecting the motion of an image block including a feature point across a plurality of frames using this technique, a motion information table (feature point coordinates and motion) is used. It is conceivable to store the starting point of the next tracking in a table in which the vector calculation result is stored.

  However, since the start point of the next tracking is not stored in the order of the raster direction in the motion information table, it is necessary to search for the coordinates of the next tracking so that the data can be transferred in the raster direction. It can be considered as a bottleneck in the process.

  The present invention has been made in order to solve the above-described problems, and an image processing apparatus capable of speeding up image processing by accelerating calculation of motion detection when detecting motion of an image block across a plurality of frames. The purpose is to provide.

  (1) An image processing apparatus according to claim 1, which has been made to solve the above problem, extracts a feature point that is a pixel that is a feature on a past image and has a predetermined size including the feature point An image processing apparatus that calculates movement information (next tracking coordinates, motion vector) indicating where a template image has moved over a plurality of frames on the current image, and is captured by the image processing apparatus A feature point map indicating where the feature points on the image are present in units of image blocks, which are image units for motion detection, and data used when calculating the movement information (feature points) Or a motion information table in which table blocks (elements of motion information table) for storing next tracking coordinates and next tracking block ID) are set in the order of identification numbers of the feature point map, In the information table, for each table block, the feature point position data (feature point coordinates X) extracted in units of pixels and the position of the next tracking point that is the position data after the feature point is moved are stored. Data (next tracking coordinate Y) and a next tracking block identification number indicating a table block storing data to be processed next are stored, and the identification number of the image block stored in the feature point map is used as an index. The table block to be processed is determined in the motion information table, and the table block to be processed next is determined using the next tracking block identification number stored in the table block as an index.

  In the present invention, the identification number of the image block stored in the feature point map is used as an index to determine a table block to be processed in the motion information table, and the next tracking block identification number stored in the table block is used as an index. The table block to be processed next can be determined.

  That is, in the present invention, the order in which motion detection should be performed can be sequentially determined using the identification number of the image block stored in the feature point map or the next tracking block identification number stored in the motion information table as an index. Therefore, even when the data of the position of the next tracking point is not stored in the order of the raster direction, the calculation process can be speeded up when calculating the movement information between a plurality of frames.

Here, the feature point of the image block is a pixel that is a feature on the image (that is, pixel data that makes it possible to distinguish the corresponding image block from other image blocks).
(2) In the invention of claim 2, when a table block to be processed is determined in the motion information table, the position data of the next tracking point stored in the table block and the movement after the determination of the next tracking point The movement information (for example, the next tracking coordinate (T) after the movement of the next tracking coordinate (T-1)) is calculated using the previous and next image data.

The present invention shows the subsequent processing when a table block to be processed is determined in the motion information table. Thereby, movement information can be calculated promptly.
(3) In the invention of claim 3, the first frame of the image calculates movement information indicating where the template image including the feature point on the past image has moved on the current image, and the calculation result Is stored in the first motion information table storing the feature point coordinates of the past image. In the second and subsequent frames of the image, the identification number of the image block stored in the feature point map is executed. The next tracking point position data (next tracking coordinate) and the next tracking block identification number (next tracking block identification number ID) are extracted from the motion information table, and movement information is calculated based on the next tracking coordinate. Then, a second process of storing the calculation result in the second motion information table storing the feature point coordinates of the past image is performed.

The present invention shows the processing procedure in chronological order.
That is, by sequentially writing data using a plurality of motion information tables, it is possible to easily calculate movement information using the data sequentially.

(4) The invention of claim 4 is characterized in that the second process is repeated until the next tracking block identification number reaches a value indicating 0 (for example, NULL).
The present invention indicates the end of processing. As a result, only necessary calculations can be performed.

  The feature point map includes a feature point map in the current image (latest image) and a feature point map in the past image (for example, an image one frame before the latest image), but more feature point maps. May be used.

  In the feature point map, when moving information of the first frame and the second frame is calculated, presence / absence of feature points in image block units (data indicating in which part an image block exists in the image) Can be stored (see FIG. 2). In addition, when calculating movement information for the second and subsequent frames (for example, movement of the second and third frames), the identification number for each image block before the movement of the feature point is set to the position after the movement of the feature point. It can be stored (see FIGS. 5-8). An image block is composed of a plurality of pixels.

The inventions of claims 1 to 4 can be applied to the inventions of claims 1 to 20 described in Japanese Patent Application Laid-Open No. 2008-192060 proposed by the present inventor.
Specifically, for example, an image storage unit that stores image data captured by the image processing apparatus, and necessary image data for extracting feature points and calculating motion vectors are created. A pre-processing unit, a feature point map indicating where the feature points on the image data are in units of image blocks, which are image units for motion detection, and feature point position data extracted in units of pixels; When feature points are extracted in units of image blocks from a motion information table storing motion vector data and image data obtained from the preprocessing unit, if feature points can be extracted, the feature points exist in the image block. A feature point map is set in the feature point map, and the feature point extraction unit that stores the position of the feature point in the motion information table, the image data stored in the image storage unit, and the feature point map An image processing apparatus comprising: a motion detection unit that calculates a motion vector based on the memorized feature point data of the image block unit and the pixel unit feature point position data stored in the motion information table Can be applied to.

  The motion detection unit includes a motion detection image storage unit for storing past image data and current image data necessary for motion detection, and image data stored in the image storage unit for motion detection. Using the image data transfer unit to transfer to the image storage unit and the motion detection image storage unit, calculate a motion vector indicating where the template image including the feature points on the past image has moved on the current image And a motion detection calculation unit that stores the calculation result in a past motion information table that stores feature point coordinates when a past image is input, and a past flag set when a past image is input. In order to process all image blocks in which feature points exist in the raster direction with reference to the feature point map, the image block whose motion vector has been calculated at this time and the features on the image block line Is calculated from the distance value to the next image block in which the feature point exists in the next image block in which the feature point exists. It is possible to adopt a configuration including a motion detection control unit that determines whether it can be used for calculation and instructs the image data transfer unit to transfer only the missing image data.

1 is a block diagram illustrating an image processing apparatus according to Embodiment 1. FIG. It is explanatory drawing (processing result of STEP0) which shows a feature point map. It is explanatory drawing which shows a motion information table. It is explanatory drawing which shows the whole flow of the tracking between 3 frames. It is explanatory drawing which shows the procedure of the process in STEP1. It is explanatory drawing which shows the procedure of the process in STEP2. It is explanatory drawing which shows the procedure of the process in STEP3. It is explanatory drawing which shows the procedure of the process in STEP4. 3 is a flowchart for describing processing related to tracking between one and two frames in the first embodiment. 6 is a flowchart for describing processing and the like related to tracking in the second and subsequent frames in the first embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

a) First, the configuration of the image processing apparatus of this embodiment will be described.
As shown in FIG. 1, the image processing apparatus 1 is an apparatus that detects a motion vector of a moving image shot by a camera 3.

  The image processing apparatus 1 includes an image input I / F unit 5 that captures an image, a smoothing processing unit 7 that smoothes input pixels, a Y differentiation processing unit 9 that performs X differentiation on the smoothed input pixels, Similarly, a Y differentiation processing unit 11 that Y-differentiates the smoothed input pixel, and a feature point that extracts a feature point that is a pixel on the image from the X differentiated input pixel and the Y differentiated input pixel The extraction unit 13 stores a feature point map 15 indicating where the feature points are in image block units, which are image units for motion detection, the coordinate positions of the feature points in pixel units, the next tracking coordinates described in detail later, and the like. A motion information table 17, an image storage unit 19 that stores image data, a motion detection unit 21 that calculates next tracking coordinates and motion vectors, and the like.

  Among these, the motion detection unit 21 stores the image data transfer unit 21a that transfers the image data stored in the image storage unit 19 necessary for calculating the next tracking coordinates and the motion vector, and the transferred image data. Using the motion detection image storage unit 21b and the motion detection image storage unit 21b, the next tracking coordinates and motion vectors (where the template image including the feature point on the image one frame before has moved on the current image) A motion detection calculation unit 21c that stores the calculation result in the motion information table 17, and an image data transfer unit 21a for calculating the next tracking coordinates and motion vectors of feature points on the image. It is comprised by the motion detection control part 21d which controls the motion detection calculating part 21c.

  In the image processing apparatus 1, the image storage unit 19, the motion detection image storage unit 21b, the feature point map 15, and the motion information table 17 are memory portions for storing various data. The conversion processing unit 7, the X differentiation processing unit 9, the Y differentiation processing unit 11, the feature point extraction unit 13, and the motion detection calculation unit 21c are calculation processing portions that perform calculation processing using various data. The smoothing processing unit 7, the Y differentiation processing unit 9, and the Y differentiation processing unit 11 constitute a preprocessing unit 22.

  Here, the feature point of each image block indicates a pixel (for example, the intersection of the edge of the X differential image and the Y differential image) that is a feature on the image. As will be described later, the feature point map is The map shows the presence or absence of feature points in image block units when an image is divided into a plurality of image blocks.

  Note that image data, calculation results, and the like captured by the camera 3 are stored in a memory such as the feature point map 15, the motion information table 17, and the image storage unit 19, and process the image data by the units 7 to 13 and 21. Various operations such as the occasion are executed by a computer.

b) Next, the operation of each block in FIG. 1 will be described.
The camera 3 outputs monochrome image data, and here, the image size is set to a VGA size of 640 × 480 pixels. However, the image size is not limited to this. Further, there is no problem with a color image, and in that case, a function for converting a luminance image needs to be added to the image input I / F unit 5.

  The image input I / F unit 5 extracts the luminance value, x-coordinate, and y-coordinate of each pixel from the output signal from the camera 3 and outputs them to the smoothing processing unit 7. Further, the processed image size may be reduced by thinning out the image data pixel by pixel.

  The smoothing processing unit 7 holds a buffer for four lines inside, and when pixel data of coordinate values (x, y) is input, coordinates are obtained by a convolution operation based on a 5 × 5 smoothing filter coefficient. The smoothed pixel data of the value (x−2, y−2) is output to the X differentiation processing unit 9, the Y differentiation processing unit 11, and the image storage unit 19.

  Similar to the smoothing processing unit 7, the X differentiation processing unit 9 holds a buffer for 4 lines therein, and when pixel data of coordinate values (x, y) is input, 5 × 5 X The X differential pixel data of the coordinate value (x−2, y−2) is output to the feature point extraction unit 3 and the image recording unit 19 by convolution calculation based on the differential filter coefficient.

  Similar to the smoothing processing unit 7, the Y differentiation processing unit 11 holds a buffer for four lines therein, and when pixel data of coordinate values (x, y) is input, 5 × 5 Y The Y differential pixel data of the coordinate value (x−2, y−2) is output to the feature point extraction unit 13 and the image recording unit 19 by the convolution calculation based on the differential filter coefficient.

Needless to say, the filter sizes of the smoothing, X differentiation, and Y differentiation need not be limited to this.
The feature point extraction unit 13 includes two line buffers for an X differential image and a Y differential image for 8 lines when the size of an image block that is an image unit for motion detection is 8 × 8 pixels. . While the image data is being written into the line buffer of one X differential image and the Y differential image, the other X differential image and Y differential image line buffer is used in the image block of 8 × 8 pixels. Extract the most corner-like pixels.

  Specifically, an index indicating the likelihood of a corner of each pixel is calculated by a well-known technique called Harris corner detection, and a pixel having a value higher than an arbitrary threshold and having the highest value is characterized in the image block. Let it be a point.

  As shown in FIG. 2B, the feature point map 15 is a table showing in which image block the feature point exists. For example, the storage unit can set 1 as a flag when a feature point can be detected and set 0 as a flag when the feature point cannot be detected. The feature point map 15 has a two-bank configuration for the past image of the T-1 frame and the current image of the T frame, but may have a bank configuration of more than that.

  As shown in FIG. 2 (A) (only a part of the image blocks are shown in FIG. 2), the image block of the processed image here is 80 × 60 × 1. The image blocks for the columns are not used for feature point extraction. Therefore, the image block of the processing area is 78 vertical × 58 horizontal. In order to distinguish each image block, identification numbers 1 to 4524 are sequentially set along the raster direction.

  As shown in FIG. 3, the motion information table 17 can refer to information on the position and movement of feature points on an image block using the identification number (1-4524) assigned to each image block as an index. ing. Here, the contents of each memory of the motion information table 17 corresponding to each identification number of the feature point map are referred to as a table block. The motion information table 17 has a two-bank configuration for the past image of the T-1 frame and the current image of the T frame, but may have a bank configuration of more than that.

Specifically, the motion information table 17 stores feature point coordinate data (feature point coordinates: data of x and y coordinates of pixels) as feature point data.
Further, the next tracking coordinates (that is, the end point of the movement vector one frame before: for example, the coordinates at which the feature point has moved between one or two consecutive frames) are stored.

  Further, the next tracking block ID is stored. This next tracking block ID indicates the next object (table block) for making the motion information table 17 a list structure. That is, as described later (see FIG. 6), for example, if 2 is stored as the next tracking block ID in the table block corresponding to the identification number 3, then the process proceeds to the table block corresponding to the identification number 2. Necessary computation (for example, computation for calculating the next tracking coordinate or motion vector) is performed on the feature point corresponding to the identification number 2.

  The image storage unit 19 stores image data in units of image blocks. For example, when a memory of 8 bits per pixel and 32 bits per word is used, data is stored in units of 4 pixels, and in the case of an 8 × 8 pixel image block, data is stored in units of 16 words.

  As will be described in detail later, the motion detector 21 is an image storage unit (image memory) that stores output data of the smoothing processing unit 7, the X differentiation processing unit 9, and the Y differentiation processing unit 11 in units of image blocks. 19, image data of one frame before stored in the image storage unit 19 (smooth image, X differential image, Y differential image), current image data (smooth image, X differential image, Y differential image), feature point map 15. Using the motion information table 17, perform processing for calculating the next tracking coordinate and motion vector.

c) Next, a procedure of main processing performed in the image processing apparatus 1 of the present embodiment will be described.
In the present embodiment, the movement of feature points across a plurality of frames is detected.

  (1) Here, as shown in FIG. 4, the tracking of feature points between three frames (1 to 3 frames, 3 to 5 frames,...) Will be described as an example. In addition, since it demonstrates using a some feature point map and a motion information table below, they were divided into A, B, C, and D, and were demonstrated.

<1-3 frames>
First, the feature point is extracted from the first frame image (STEP 0).
Then, the image block in which the feature point of the first frame exists is stored in the feature point map A. That is, the flag of the corresponding image block is set to 1.

Also, the coordinates of the feature point of the first frame are stored in the motion information table A. At this time, only the coordinates of the feature points are stored in the motion information table A.
Next, motion detection between 1 and 2 frames (processing for detecting feature point motion) is performed using image data of 1 and 2 frames (STEP 1 and 2).

  Specifically, the image data of the first and second frames and the data of the feature point map A and the motion information table A (here, the coordinate data of the feature point of the first frame) are used to move between the first and second frames. Processing for obtaining the coordinates of the feature points (next tracking coordinates) is performed. That is, where the image block having the feature point of the first frame has moved to the position of the second frame is obtained by a known image processing technique, thereby obtaining the next tracking coordinate that is the coordinate of the feature point after the movement. . In addition, you may perform the process which calculates | requires a movement vector simultaneously.

  For this motion detection processing, for example, a known method as described in Japanese Patent Application Laid-Open No. 2008-192060, that is, a Kanade-Lucas-Tomasi method (KLT method), which is a kind of gradient method, or block matching is used. can do.

Next, the image block where the next tracking coordinate of the second frame exists is stored in the feature point map B.
Also, the feature point coordinates (first frame) and the next tracking coordinates (obtained by the motion detection) are stored in the motion information table B. The feature point coordinates (first frame) are copied from the motion information table A.

Next, using two or three frames of image data, motion detection between two or three frames is performed (STEPs 3 and 4).
Specifically, the image data of two or three frames and the data of the feature point map B and the motion information table B (here, the data of the next tracking coordinates of the second frame) are used to move between two or three frames. Processing for obtaining the next tracking coordinate is performed.

  Next, the image block where the next tracking coordinate of the third frame exists is stored in the feature point map A. At this time, the previous feature point map A and motion information table A data are cleared.

  Further, the feature point coordinates (first frame) and the next tracking coordinates (obtained by the motion detection) are stored in the motion information table A. The feature point coordinates (first frame) are copied from the motion information table B.

  After the motion detection of the second frame and the third frame is completed, the motion vector calculation result between the first and third frames (ie, between the next tracking coordinate and the feature point coordinate of each table block of the motion information table A) Vector).

<3-5 frames>
First, the feature point is extracted from the third frame image.
Then, the image block in which the feature point of the third frame exists is stored in the feature point map C.

Further, the coordinates of the feature point of the third frame are stored in the motion information table C.
Next, motion detection between 3 and 4 frames is performed using image data of 3 and 4 frames.
Specifically, using the image data of 3 and 4 frames and the data of the feature point map C and the motion information table C, a process for obtaining the next tracking coordinates moved between 3 and 4 frames is performed.

Next, the image block where the next tracking coordinate of the fourth frame exists is stored in the feature point map D.
Further, the feature point coordinates (in the third frame) and the next tracking coordinates (obtained by the motion detection) are stored in the motion information table D. The feature point coordinates (third frame) are copied from the motion information table C.

Next, motion detection between 4, 5 frames is performed using image data of 4, 5 frames.
Specifically, a process for obtaining the next tracking coordinates moved between the fourth and fifth frames is performed using the image data of the fourth and fifth frames and the data of the feature point map D and the motion information table D. In addition, you may perform the process which calculates | requires a movement vector simultaneously.

Next, the image block where the next tracking coordinate of the fifth frame exists is stored in the feature point map C.
Further, the feature point coordinates (in the third frame) and the next tracking coordinates (obtained by the motion detection) are stored in the motion information table C. The feature point coordinates (third frame) are copied from the motion information table D.

Then, the calculation result of the motion vector between 3 and 5 frames (that is, the vector between the next tracking coordinate and the feature point coordinate of each table block of the motion information table C) is output.
(2) Next, the processing of <1-3 frames> will be described in more detail along the time series.

<STEP1>
This process is a process between one frame and two frames.
As shown in FIG. 5 (left figure), for example, a case where the feature point of the image block 2 has moved to the image block 80 is considered.

  Here, the motion vector from the feature point of the image block 2 to the feature point of the image block 80 includes the image data of 1 and 2 frames, the data of the feature point map A and the motion information table A (the feature point coordinates of the first frame). ). The next tracking point is a feature point that has moved to the image block 80.

  As shown in FIG. 5 (right figure), the presence / absence of the next feature point is stored in the feature point map B. That is, the identification number 2 of the image block (indicating the image block where the feature point exists in the first frame) is stored at the position of the image block 80 to which the feature point has moved.

  In addition, in the motion information table B, the table block 2 corresponding to the image block identification number 2 (indicating the image block in which the feature point exists in the first frame) includes the feature point coordinate X, the next tracking coordinate Y, and the next A tracking block ID Z is stored.

  That is, the feature point coordinates (that is, the feature point coordinates of the first frame) are copied from the motion information table A and stored. Further, as the next tracking coordinate, a value calculated by the motion detection process (the coordinate of the feature point moved in the second frame) is stored. Further, as described above, the next tracking block ID indicates the next object for making the motion information table 17 a list structure, that is, a table corresponding to the feature point identification number 2 obtained from the feature point map B. This indicates a table block that proceeds next to block 2. Here, if the block stored in the feature point map B is 0, it is set to NULL indicating the end of data.

<STEP2>
This processing is also processing between one frame and two frames.
As shown in FIG. 6 (left figure), for example, consider a case where the feature point of the image block 3 has moved to the same image block 80 as in STEP1.

  As shown in FIG. 6 (right figure), the next feature point is stored in the feature point map B. That is, the identification number 3 of the image block (in which another feature point in the first frame exists) is stored at the position of the image block 80 to which the feature point has moved.

In the motion information table B, the feature block coordinate X, the next tracking coordinate Y, and the next tracking block ID Z are stored in the table block 3 corresponding to the image block identification number 3.
That is, the feature point coordinates are copied from the motion information table A and stored, and the next tracking coordinates are stored as values calculated by the motion detection process. The next tracking block ID is set to 2 because the block stored in the feature point map B is already 2.

<STEP3>
This process is a process between 2 and 3 frames.
As shown in FIG. 7 (left figure), for example, consider a case where the feature point of the image block 80 is further moved. Here, there are two next tracking points, a feature point of the image block 2 and a feature point of the image block 3.

  Looking at the feature point map B (right figure in FIG. 6), it is described as 3 when searching in the raster direction. Therefore, the table block 3 is extracted from the motion information table B using 3 as an index number, and the data is used. A motion detection process is performed to calculate next tracking coordinates and motion vectors.

  As shown in FIG. 7 (right figure), the next feature point is stored in the feature point map A (after clearing). That is, the identification number 3 of the image block (indicating an image block in which another feature point exists in the first frame) is stored at the position of the image block 158 to which the feature point has moved.

  Further, in the motion information table A (after clearing), the feature block coordinate X, the next tracking coordinate Y, and the next tracking block ID Z are stored in the table block 3 corresponding to the image block identification number 3.

  That is, the feature point coordinates are copied from the motion information table B and stored, and the next tracking coordinates are stored as values calculated by the motion detection process. If the block stored in the feature point map A is 0, the next tracking block ID is NULL indicating the end of data.

<STEP4>
This process is also a process between 2 frames and 3 frames.
As shown in FIG. 8 (left figure), the next tracking block ID of the table block 3 of the motion information table B is 2. Therefore, data is extracted with the index number 2 and a motion detection process is performed using the data to calculate the next tracking coordinate and motion vector.

  As shown in FIG. 8 (right figure), the next feature point is stored in the feature point map A (after clearing). That is, the identification number 2 of the image block (indicating the image block where the feature point exists in the first frame) is stored at the position of the image block 157 where the feature point has moved.

  Further, in the motion information table A (after clearing), the feature block coordinate X, the next tracking coordinate Y, and the next tracking block IDZ are stored in the table block 2 corresponding to the identification number 2 of the image block.

  That is, the feature point coordinates are copied from the motion information table B and stored, and the next tracking coordinates are stored as values calculated by the motion detection process. If the block stored in the feature point map A is 0, the next tracking block ID is NULL indicating the end of data.

  In this way, in the present embodiment, the feature point map 15 is searched in the raster direction, and if a feature point exists there, the feature point identification number is used and the identification number of the motion information table 17 is used. Find the table block that corresponds to. Since this table block stores the next tracking block ID (that is, the number indicating the next table block to be processed), the table block data (feature point coordinates and next tracking) indicated by the next tracking block ID is next stored. Further tracking coordinates and motion vectors can be calculated using (coordinates).

  Accordingly, it is not necessary to search the motion information table, and when the motion of an image block across a plurality of frames is detected, the problem of speeding up the image processing by speeding up the calculation of the motion detection can be solved.

Next, the entire image processing including the processing in steps 1 to 4 will be described with reference to FIGS.
This process extracts feature points that are pixels that become features on the past image, and next tracking that indicates where the template image having a predetermined size including the feature points has moved on the current image. For example, processing for calculating coordinates and motion vectors.

In FIGS. 9 and 10, the flow of processing is indicated by a solid line, and the flow of data is indicated by a broken line.
(1) First, based on FIG. 9, the process etc. regarding the tracking of the feature point between 1 and 2 frames are demonstrated. This processing is performed every time image data is input.

As shown in FIG. 9, in step (S) 100, image data captured by the camera 3 is captured (image input I / F processing).
In the subsequent step 110, preprocessing of the captured image data is performed. Specifically, the above-described smoothing process, X differential process, and Y differential process are performed.

  In the following step 120, processing for extracting feature points is performed (feature point extraction processing). This feature point extraction process is a process of extracting feature points in units of image blocks from an image obtained from preprocessing, and if feature points can be extracted, a flag indicating the presence of feature points in an image block is displayed as a feature point map. This is a process for storing the position of the feature point in the motion information table.

In addition to the feature point extraction processing (in parallel), processing for storing the smoothed image data in the image storage unit 19 is performed.
In subsequent step 130, it is determined whether the calculation of the next tracking coordinates and motion vectors of all image blocks in which feature points exist has been completed (end determination process). If an affirmative determination is made here, the present process ends. If a negative determination is made, the process proceeds to step 140.

  In step 140, an image data transfer amount determination process is performed. This image data transfer amount determination processing refers to the other (past) feature point map (here, the (T-2 frame) feature point map) that is flagged when a past image is input, To process all image blocks with feature points in the raster direction, calculate the distance between the image block for which motion vectors have been calculated and the next image block with feature points on the image block line. Then, from the distance value, it is determined whether the image data that has already been transferred to the motion detection image storage unit 21b can be used to calculate a motion vector in the next image block in which a feature point exists, and is insufficient. This is a process for clarifying the amount of image data.

  Here, the feature point map of (T-2 frame) and the motion information table of (T-2 frame) correspond to the feature point map A and motion information table A of FIG. The feature point map of (first frame) and the motion information table of (T-1 frame) correspond to the feature point map B and motion information table B of FIG. 4 (in this case, T = 3 frames) Give an example when).

In the subsequent step 150, the image data amount is transferred from the image storage unit 19 to the motion detection image storage unit 21 (image data transfer process).
In the following step 160, motion detection calculation processing is performed. This motion detection calculation processing uses the motion detection image storage unit 21b to calculate next tracking coordinates and motion vectors indicating where a template image including a feature point on a past image has moved on the current image. This is a process of calculating and storing the calculation result in the motion information table. In the above description, an example in which the next tracking coordinates are stored has been described.

  Here, the processing in steps 130 to 160 is motion detection processing (processing for calculating the next tracking coordinate and motion vector) performed in the motion detection unit 21, and this motion detection processing is performed in STEP 1 of FIG. It corresponds to 2.

In addition, the process of said step 100-120 and the process of steps 130-160 are processed in parallel.
Therefore, the following tracking coordinates and motion vectors indicating where the template image having a predetermined size including the feature point has moved on the current image can be calculated by the above-described processing.

(2) Next, based on FIG. 10, the process etc. regarding the tracking of the feature point after 2 frames are demonstrated.
As shown in FIG. 10, in step 200, it is determined whether the calculation of the next tracking coordinates and motion vectors of all image blocks where feature points exist has been completed (end determination process). If an affirmative determination is made here, the present process ends. If a negative determination is made, the process proceeds to step 210.

  In step 210, an image data transfer amount determination process is performed. In this image data transfer amount determination process, the calculation of the motion vector is completed at this time in order to process all image blocks in which the feature points exist in the raster direction with reference to the feature point map of (T-2 frame). The distance between the image block and the next image block where the feature point on the image block line exists is calculated, and the image data already transferred to the motion detection image storage unit 21b from the distance value is the feature point. Is a process for determining whether or not the next image block in which the image data is present can be used to calculate a motion vector and clarifying the amount of image data that is lacking.

  Here, the feature point map of (T-2 frame) and the motion information table of (T-2 frame) correspond to the feature point map B and motion information table B of FIG. The feature point map of (first frame) and the motion information table of (T-1 frame) correspond to the feature point map A and motion information table A of FIG. 4 (in this case, T = 4 frames) Give an example when).

In the subsequent step 220, the image data amount is transferred from the image storage unit 19 to the motion detection image storage unit 21b (image data transfer process).
In subsequent step 230, motion detection calculation processing is performed. This motion detection calculation processing uses the motion detection image storage unit 21b to calculate next tracking coordinates and motion vectors indicating where a template image including a feature point on a past image has moved on the current image. This is a process of calculating and storing the calculation result in the motion information table.

  In subsequent step 240, continuation determination processing is performed. That is, it is determined whether there is another next tracking point. If an affirmative determination is made here, the process returns to step 230. On the other hand, if a negative determination is made, the process returns to step 200.

  In other words, the next tracking block ID is confirmed, and if it is not NULL, it is considered that there is another next tracking point, so the process proceeds to step 200 again using the data of the table block indicated by the next tracking block ID. Thus, motion detection calculation processing is performed.

  Here, the processing in steps 200 to 240 is motion detection processing (processing for calculating the next tracking coordinate and motion vector) performed in the motion detection unit 21, and this motion detection processing is performed in STEP 3 of FIG. Corresponds to 4.

  Therefore, the following tracking coordinates and motion vectors indicating where the template image having a predetermined size including the feature point has moved on the current image can be calculated by the above-described processing.

  Of the above-described processing, the processing of each image block line has the same contents as the processing in steps 200 to 320 in FIG. 14 of Japanese Patent Laid-Open No. 2008-192060, and the current processing block and the following feature points The distance to the image block in which is present is obtained from the feature point map, and based on the value, it is determined whether the already transferred image data can be used for the next motion detection calculation, and only the missing data is transferred. Control to minimize data transfer.

  Specifically, the distance (unit: image block) between the image block for which the current processing has been completed and the image block in which the next feature point exists is obtained. For example, if the distance value is 3 image blocks or less, the data of adjacent image blocks can be used, so only the image data (3 × 6 image data) that is insufficient for motion detection processing is transferred. (Burst transfer) If there are more than 4 image blocks, transfer 9 × 6 image data (skip transfer). Note that the number of image data is the number of image blocks, and the six pieces of image data are T-1, T smoothed images, X differential images, and Y differential images.

  Similar processing is repeated until all the image blocks having feature points on the same image block line are processed. Further, these processes are applied to all image block lines.

With the above configuration, it is possible to transfer only image data necessary for processing an image block in which a feature point exists, and to detect a motion vector at high speed.
In addition, this invention is not limited to the said Example, It is possible to implement in various aspects.

  For example, the motion information table may store not only the next tracking coordinate but also a motion vector. Further, the motion vector may be performed at a timing different from the calculation of the next tracking coordinate (as required in image processing). The motion vector can be calculated from the next tracking coordinates before and after the movement.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 3 ... Camera 5 ... Image input I / F part 7 ... Smoothing process part 9 ... X differential process part 11 ... Y differential process part 13 ... Feature point extraction part 15 ... Feature point map 17 ... Motion information table 19 ... Image storage unit 21 ... Motion detection unit

Claims (4)

Extracting feature points, which are pixels that are features on the past image, and where the template image with a predetermined size including the feature points moved on the current image across multiple frames An image processing apparatus that calculates movement information indicating
A feature point map showing where the feature points on the image captured by the image processing apparatus exist in units of image blocks, which are image units for motion detection;
A motion information table in which table blocks for storing data used for calculating the movement information are set in the order of identification numbers of the feature point maps;
With
The motion information table includes, for each table block, feature point position data extracted in units of pixels, position data of the next tracking point which is position data after the feature point has been moved, And the next tracking block identification number indicating the table block storing the data to be processed,
The table block to be processed in the motion information table is determined using the identification number of the image block stored in the feature point map as an index, and the next tracking block identification number stored in the table block is used as an index. A table block to be processed is determined. When a table block to be processed is determined in the motion information table,
2. The image according to claim 1, wherein the movement information is calculated using data of a position of the next tracking point stored in the table block and image data before and after movement after the determination of the next tracking point. Processing equipment. In the first frame of the image, movement information indicating where the template image including the feature point on the past image has moved on the current image is calculated, and the calculation result is used as the feature point coordinates of the past image. Performing a first process of storing in the stored first motion information table;
For the second and subsequent frames of the image, the data of the position of the next tracking point and the next tracking block identification number are extracted from the motion information table using the identification number of the image block stored in the feature point map as an index. The movement information is calculated based on the coordinates, and the second process of storing the calculation result in the second motion information table storing the feature point coordinates of the past image is performed. 2. The image processing apparatus according to 2.   The image processing apparatus according to claim 3, wherein the second process is repeated until the next tracking block identification number reaches a value indicating 0.