JP2008192060A - Image processing device and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device and an image processing method for increasing calculation speed of a movement vector for speedup of image processing. <P>SOLUTION: In a step 120, processing for extracting feature points is carried out. In the feature point extraction processing, the feature points are extracted by an image block unit from an image obtained from the previous processing, and if the feature points can be extracted, a flag showing presence of the feature point in the image block is raised in a feature point map and the positions of the feature points are stored in a movement information table. In a step 140, image data transfer quantity determination processing is carried out. In the image data transfer quantity determination processing, image data quantity lacking for movement vector calculation is made clear. In a step 150, transfer by the image data quantity is carried out from an image storage part 19 to a movement detection image storage part 21. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image using a motion vector.

  2. Description of the Related Art Conventionally, as an image processing device that captures and processes an image around a vehicle, a device that uses a motion vector of a moving image is known. As a method of detecting this motion vector, in order to reduce false detection of a motion vector, a feature point is determined from an image once stored in an image memory, and a template image surrounding the feature point is located anywhere on the image after one frame. There has been proposed a method for calculating a motion vector indicating whether or not it has moved (for example, see Patent Documents 1, 2, and 3).

In this conventional technique, for example, consider motion detection in units of 8 × 8 pixel images (hereinafter referred to as image blocks) as in the input image of FIG. At this time, feature point extraction processing is performed on each image block. In this feature point extraction processing, a value indicating an index of the feature point likelihood of each pixel is calculated, and the pixel having the highest value is determined. Then, when the value exceeds a certain threshold value, it is determined as a feature point, and it is determined where the template image surrounding the feature point has moved on the image one frame later.
JP-A-8-242453 JP-A-9-55943 JP-A-9-55944

  In the above-described image processing device, a motion vector detection device (motion vector detection device) can be applied and deployed to detect a target object such as a car or a person and to grasp a situation such as a moving direction and speed of the target object. It is an essential function for the recognition system.

  In in-vehicle image recognition that requires this reliability, it is conceivable to employ a gradient method such as the Kanade-Lucas-Tomasi method (hereinafter referred to as the KLT method) that can be detected with subpixel accuracy as a motion detection method.

  The KLT method will be described as an example. The size of a motion vector that can be detected with high accuracy by the KLT method is about 3 pixels in both the X and Y directions. Therefore, by shortening the image capture cycle, that is, by using a high frame rate camera, the motion vector detection device can be used in many scenes in the vehicle.

  Even when the block matching method is adopted as the motion detection method, using a high frame rate camera is effective because the detection range can be reduced and the probability of mismatching can be reduced.

However, when a high frame rate camera is used, an image capturing cycle is shortened, so that a device capable of calculating a motion vector at high speed is required.
However, in the above-described conventional motion vector detection device, in order to extract a feature point as a target for obtaining a motion vector on an image, the feature vector is obtained by accessing the image storage device storing the image data. Therefore, there is a problem in speeding up the processing.

  That is, after all images (smoothed image, X differential image, Y differential image) obtained by preprocessing the image captured by the camera are stored in the image recording apparatus, each image is calculated in order to calculate feature points from the image recording apparatus. Since the image data of the block is transferred, the feature points are calculated from the image data, and the motion vector is obtained, it takes time for the arithmetic processing.

  For example, when the Harris corner detection method is used as a feature point extraction method, an X differential image and a Y differential image are used. At this time, when motion is detected in the entire range of the input image, all image data From the image storage device storing the image data, two types of image data are transferred to the arithmetic processing unit and feature points are extracted, which is a factor that hinders speeding up.

  Furthermore, conventionally, when image data is transferred to extract not only feature points but also motion vectors, all (smoothed images of the past and current frames, X differential images, Y differential images) are all images. It is necessary to uniformly transfer the image data of the block, which also hinders the speeding up of computation.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide an image processing apparatus and an image processing method capable of speeding up calculation of motion vectors and speeding up image processing.

  (1) An image processing apparatus according to claim 1, which has been made to solve the above problems, is based on an image storage unit that stores image data captured by the image processing apparatus, and image data captured by the image processing apparatus. A feature point extraction unit that extracts feature points in units of image blocks that are image units for motion detection, a feature point map that indicates where the feature points extracted by the feature point extraction unit exist in units of image blocks, Based on the feature point data stored in the feature point map in units of image blocks and the feature point position data extracted in units of pixels, using the image data stored in the image storage unit, the And a motion detection unit for calculating a motion vector.

  Here, the image processing apparatus of the present invention will be described with reference to FIG. FIG. 1 illustrates an apparatus configuration including the image processing apparatus of the present invention, and other configurations are also described in order to help understanding of the contents of the present invention.

  As shown in the figure, in the image processing apparatus A1 of the present invention, for example, an image photographed by the camera A2 is captured by, for example, the image input I / F unit A3, and the image data is processed by, for example, the preprocessing unit A4. Is done. Accordingly, the image storage unit A5 stores, for example, preprocessed image data. Also, the feature point extraction unit A6 extracts feature points in units of image blocks, which are image units for motion detection, based on preprocessed image data, for example (in this feature point extraction unit A6, pixel units The position (coordinates) of the feature point can be extracted with The feature points extracted by the feature point extraction unit A6 are stored in a feature point map A7 indicating where the feature points exist in units of image blocks. Therefore, in the motion detection unit A8, the feature point data stored in the feature point map A7 and the feature points in pixel units (for example, stored in the motion information table A9) are stored by the well-known KLT method, for example. Based on the position data, the motion vector can be calculated using the image data stored in the image storage unit A5.

  Thus, in the present invention, when the image is input, the feature point is extracted from the image data by the feature point extraction unit without accessing the image storage unit, and the image of the feature point (in the photographed image) is obtained. A feature point map indicating the position of each block can be created. Then, using the feature point map created at the time of feature point extraction, it is possible to calculate the motion vector of the image block where the feature point exists (and hence the motion vector of the feature point).

  In this feature point map, the presence or absence of feature points can be recorded for each image block. Therefore, using this feature point arrangement, calculation processing can be reduced (for example, when motion vectors are calculated, motion detection is performed from the image storage unit). Reduction of image data transferred to the copy unit).

The feature point map includes a feature point map in the current image (latest image) and a feature point map in the past image (for example, an image one frame before the latest image).
The map may be any map as long as it can store the positions of feature points in units of image blocks (data indicating in which part an image block exists in the image). An image block is composed of a plurality of pixels.

  The position (coordinates) of feature points in pixel units can be extracted by the feature point extraction unit, but is not limited thereto. The data of the feature point positions in pixel units can be stored in, for example, a motion information table.

  For example, when feature points are extracted and motion vectors are detected using image data (for example, a smooth image, an X differential image, or a Y differential image) obtained by performing image processing on an input image, various types of data are Processing (for example, smoothing processing, X differentiation processing, Y differentiation processing) is performed, and image data is stored in the image storage unit.

  (2) In the image processing device according to claim 2, the feature point extraction unit performs motion detection based on the image data captured by the image processing device in parallel with the processing of storing the image data in the image storage unit. Feature points are extracted in units of image blocks that are units of images to be processed.

  As a result, it is possible to quickly calculate the motion vector as compared with the conventional case in which all image data is once stored in the image recording unit and then the feature points are extracted by obtaining the image data from the image recording unit. There is an effect that can be done.

  (3) An image processing device according to claim 3 is an image storage unit that stores image data captured by the image processing device, a feature point on the image data is extracted, and a motion vector is calculated. A pre-processing unit that creates necessary image data, a feature point map that indicates where the feature points on the image data exist in image block units that are motion detection image units, and pixel points extracted When a feature point is extracted in units of image blocks from a motion information table storing feature point position data and motion vector data, and image data obtained from the pre-processing unit, the feature point can be extracted. A feature point extraction unit that sets a flag indicating that a feature point exists in the block in the feature point map and stores the position of the feature point in the motion information table, and image data stored in the image storage unit A motion detection unit that calculates a motion vector based on the feature point data stored in the feature point map in units of image blocks and the pixel-based feature point position data stored in the motion information table; It is characterized by having.

  As illustrated in FIG. 1, in the image processing apparatus A1 of the present invention, for example, an image captured by the camera A2 is captured by, for example, the image input I / F unit A3, and the image data is received by the preprocessing unit A4. It is processed. Accordingly, the image storage unit A5 stores, for example, preprocessed image data. Further, the feature point extraction unit A6 extracts feature points based on the preprocessed image data, and the feature points are stored in the feature point map A7, and the positions of the feature points in pixel units (positions in the image) : Coordinate) is stored in the motion information table A9. Therefore, in the motion detection unit A8, for example, by the well-known KLT method, the feature point data stored in the feature point map A7 and the feature point position data stored in the motion information table are stored in the motion information table. Based on the above, the motion vector can be calculated using the image data stored in the image storage unit A5.

  As described above, in the present invention, the feature point is extracted by the feature point extraction unit based on the image data obtained from the preprocessing unit without accessing the image storage unit when the image is input, and the feature point It can be stored in a map and motion information table. Therefore, a motion vector can be calculated using the feature point map, and this motion vector can be stored in the motion information table.

  As a result, it is possible to quickly calculate the motion vector as compared with the conventional case in which all image data is once stored in the image recording unit and then the feature points are extracted by obtaining the image data from the image recording unit. There is an effect that can be done.

  In addition, since the feature point map can record the presence or absence of feature points for each image block, the feature point arrangement is used to reduce arithmetic processing (for example, when calculating motion vectors, the motion from the image storage unit) Reduction of image data transferred to the detection unit) can be achieved.

  As the motion information table, a table storing feature point and motion vector data in the current image (latest image) and feature point and motion vector data in the past image (for example, an image one frame before the latest image) are stored. The stored table is listed.

Further, the image recording unit can store the image data created by the preprocessing unit as necessary.
(4) In the image processing apparatus according to claim 4, as illustrated in FIG. 1, the motion detection unit A8 stores past image data and current image data necessary for motion detection, and stores the current image data. Using the storage unit A10, the image data transfer unit A11 that transfers the image data stored in the image storage unit A5 to the motion detection image storage unit A10, and the motion detection image storage unit A10, A motion vector indicating where the template image including the feature point has moved on the current image is calculated, and the calculation result is stored in the past motion information table storing the feature point coordinates when the past image is input. In order to process all image blocks in which feature points exist in the raster direction with reference to the motion detection calculation unit A12 to be stored and the past feature point map that is flagged when a past image is input. The distance between the current image block for which the motion vector has been calculated and the next image block where the feature point on the image block line is present is calculated, and the distance is transferred to the motion detection image storage unit A10. It is determined whether the current image data can be used for motion vector calculation in the next image block in which a feature point exists, and the image data transfer unit A11 is instructed to transfer only the missing image data. And a motion detection control unit A13.

  According to the present invention, the motion detection control unit can transfer only image data necessary for motion detection calculation. Further, the motion detection control unit obtains the distance from the current processing block to the image block where the next feature point exists from the feature point map, and from the value, the image data already transferred is used for the next motion detection calculation. It is determined whether it can be used, and control is performed so as to minimize the data transfer so that only the missing image data is transferred. Thereby, the processing speed of motion vector detection can be improved.

  (5) According to the image processing apparatus of the fifth aspect, the motion detection control unit waits for the motion data to be calculated while the motion detection calculation unit calculates the motion vector. The image data transfer unit is instructed to transfer image data of a predetermined number of image block sequences adjacent in the motion detection direction to the image range stored in the storage unit.

  This eliminates the waiting time of the image data transfer unit while calculating the motion vector, and transfers image data that may be necessary for calculating the motion vector of the subsequent image block first. be able to.

  (6) According to the image processing apparatus of the sixth aspect, the motion detection control unit scans the feature point map, performs raster processing in the X-axis direction, and predictive processing when raster processing is performed in the Y-axis direction. The time is calculated, and the shorter processing time is set as the processing direction.

Thereby, the processing speed of motion vector detection is improved.
(7) According to the image processing device of the seventh aspect, the feature point map stores information on the number of appearances according to the distance from the image block in which the feature point exists to the image block in which the next feature point exists. is doing. Therefore, by using such information, the prediction processing time can be calculated without scanning the feature point map, and the processing speed of motion vector detection is improved.

  (8) According to the image processing apparatus of the eighth aspect, when the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture period, the image block that can be processed Calculate up to the line and process up to the image block line.

As a result, it is possible to avoid failure of computation and to secure information within a possible range.
(9) According to the image processing apparatus of the ninth aspect, when the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture cycle, the priority order is high. Process only the detection area.

As a result, it is possible to avoid failure of computation and secure highly important information as much as possible.
(10) According to the image processing device of the tenth aspect, when the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture period, the reduced image ( For example, a reduced image with reduced image dots) is processed.

As a result, it is possible to avoid a calculation failure and to secure information necessary for image processing within a possible range.
(11) According to the image processing apparatus of the eleventh aspect, the feature point extracting unit has a predetermined number greater than the reference threshold value in addition to the reference threshold value. Threshold values that can be set can be set, feature points are extracted for auxiliary threshold values, and feature points are detected in the same number of auxiliary feature point maps as auxiliary threshold values. Write. Then, when the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture cycle, some of the auxiliary feature point maps are in time for the capture cycle. If there exists, the auxiliary feature point map is used to calculate the feature point coordinates (for example, the image data transfer unit transfers the image data and newly added, for example, the subsequent feature point extraction unit). The motion vector is calculated (by giving the coordinate value to the motion detection calculation unit).

As a result, it is possible to avoid a calculation failure and to secure information necessary for image processing within a possible range.
(12) According to the image processing apparatus of the twelfth aspect, a plurality of motion detection units are mounted, and the image region to be processed is divided into processing regions equal to the total number of motion detection units. Each motion detection unit independently calculates a motion vector of an image block.

Thereby, the processing speed of motion vector detection is improved.
(13) According to the image processing device of the thirteenth aspect, a plurality of motion detection calculation units are mounted, and each motion detection calculation unit synchronizes with adjacent image blocks in the same direction, Calculate the motion vector of the block.

Thereby, the transfer amount of image data is suppressed, and the processing speed of motion vector detection is improved by parallel processing.
(14) According to the image processing device of the fourteenth aspect, the image processing device is used as a vehicle front detection sensor in an in-vehicle image recognition system.

  In other words, in the in-vehicle image recognition system, when the usage point for the vehicle forward detection sensor is high, the motion vector detection is performed in a short time. Since it can be processed, a camera with a higher frame rate (for example, 60 to 100 fps) can be used.

  (15) According to the image processing apparatus of the fifteenth aspect, a camera with a high frame rate with a short capturing period is used as a camera for capturing an image, and a motion vector is obtained using the KLT method.

  In other words, the motion detection calculation unit can detect a high-accuracy in units of subpixels, but uses a KLT method with a narrow detectable range, and captures a camera with a high frame rate with a short capture cycle so that it falls within the detectable range. It can be applied to an in-vehicle image recognition system that requires high reliability.

Here, the KLT method is the well-known Kanade-Lucas-Tomasi method.
(16) According to the image processing device of the sixteenth aspect, the image processing device is used as a vehicle intruder detection sensor in the in-vehicle image recognition system.

(17) According to the image processing apparatus of the seventeenth aspect, since control is performed so that the motion vector is not calculated when there is no intruder, low power consumption can be realized.
Specifically, the motion detection control unit calculates the total number of feature points existing on the image from the feature point map, and controls to calculate a motion vector when the total number exceeds a predetermined threshold value. To do. As a result, power consumption can be reduced when motion detection processing is not required.

(18) According to the image processing device of the eighteenth aspect, since control is performed so that the motion vector is not calculated when there is no intruder, low power consumption can be realized.
Specifically, the motion detection control unit compares the feature point map with the past feature point map, and when the total number of image blocks that have changed in the presence or absence of feature points exceeds a predetermined threshold, Control is performed to calculate a motion vector. As a result, power consumption can be reduced when motion detection processing is not required.

  (19) According to the image processing method of the nineteenth aspect, image data is fetched (in an image processing apparatus that performs image processing) by image input I / F processing, and image data is converted into an image block by image storage processing. Stored in the image storage unit in units. Also, in order to extract feature points on the image data by preprocessing and to calculate motion vectors, necessary image data is created, and image is obtained from the image data obtained from the preprocessing by feature point extraction processing. If feature points are extracted in units of blocks and feature points can be extracted, a flag indicating that feature points exist in the image block is set in the feature point map. Then, by the motion vector detection process, the feature point data stored in the feature point map and the position of the feature point in pixel units (for example, extracted by the feature point extraction unit and stored in the motion information table) are stored. Based on the data, a motion vector is calculated using the image data recorded in the image recording unit.

  Accordingly, in the present invention, it is possible to create a feature point map by extracting feature points from the image data by the feature point extraction unit without accessing the image storage unit when an image is input. Then, the motion vector can be calculated using the feature point map created at the time of extracting the feature points and the data of the positions of the feature points in pixel units (for example, stored in the motion information table). Therefore, it is possible to reduce the processing load and increase the processing speed.

  (20) According to the image processing method of the twentieth aspect, in the motion vector detection process, it is determined whether or not the calculation of the motion vectors of all the image blocks in which the feature points exist has been completed by the end determination process. If it is determined by the end determination process that the calculation has not ended, the past feature point map that is flagged when a past image is input is referred to by the image data transfer amount determination process. In order to process all image blocks with feature points in the raster direction, the distance between the image block for which motion vector calculation has been completed at the present time and the next image block with feature points on the image block line is calculated. In the next image block in which the feature point exists, the image data that has been calculated and transferred to the motion detection image storage unit that stores the past and current image data used to calculate the motion vector from the distance value is calculated. It is determined whether it can be used for motion vector calculation, and the amount of image data that is lacking is clarified. Then, when the amount of image data that is lacking is clarified, an image storage unit that stores all image data in units of image blocks for the amount of image data that is lacking by image data transfer processing. To the motion detection image storage unit.

  That is, according to the present invention, only image data necessary for motion detection calculation can be transferred. Furthermore, the distance from the current processing block to the image block where the next feature point exists is obtained from the feature point map, and it is determined from the value whether the already transferred image data can be used for the next motion detection calculation. Then, control is performed to minimize data transfer so as to transfer only missing image data. Thereby, the processing speed of motion vector detection can be improved.

Embodiments of the present invention will be described below with reference to the drawings.
[First Embodiment]
a) FIG. 2 is a block diagram of the image processing apparatus according to this embodiment. As shown in the figure, the image processing device 1 is a device that detects a motion vector of a moving image taken by the camera 3 and processes an image based on the motion vector.

  The image processing apparatus 1 includes an image input I / F unit 5 that captures an image, a smoothing processing unit 7 that smoothes input pixels, a Y differentiation processing unit 9 that performs X differentiation on the smoothed input pixels, Similarly, a Y differentiation processing unit 11 that Y-differentiates the smoothed input pixel, and a feature point that extracts a feature point that is a pixel on the image from the X differentiated input pixel and the Y differentiated input pixel An extraction unit 13, a feature point map 15 indicating where the feature points are in image block units, which are image units for motion detection, a motion information table 17 for storing the coordinate positions and motion vectors of the feature points in pixel units, The output pixels of the smoothing processing unit 7, the X differentiation processing unit 9, and the Y differentiation processing unit 11 are stored in an image storage unit (image memory) 19 for storing image data in units of image blocks, and 1 stored in the image storage unit 19. Image data before frame (Smooth image, X differential image, Y differential image), current image data (smooth image, X differential image, Y differential image), feature point map 15, and motion detection unit that calculates a motion vector using the motion information table 17 21.

  Note that the feature point of each image block indicates whether or not the feature point exists in the image block, and therefore the feature point map is when the image is divided into a plurality of image blocks. It is a map showing the presence or absence of feature points in image block units.

  Among them, the motion detection unit 21 includes an image data transfer unit 21a that transfers image data stored in the image storage unit 19 necessary for calculating a motion vector, and a motion detection image that stores the transferred image data. Using the storage unit 21b and the motion detection image storage unit 21b, a motion vector indicating where the template image including the feature point on the image one frame before has moved on the current image is calculated, and the calculation result In the motion information table 17 and a motion detection control unit 21d for controlling the image data transfer unit 21a and the motion detection calculation unit 21c to calculate the motion vector of the feature point on the image. Is done.

  In the image processing apparatus 1, the image storage unit 19, the motion detection image storage unit 21b, the feature point map 15, and the information table 17 are memory portions for storing various data. Arithmetic processing in which the processing unit 7, the X differential processing unit 9, the Y differential processing unit 11, the feature point extraction unit 13, the image data transfer unit 21a, the motion detection calculation unit 21c, and the motion detection control unit 21d perform calculation processing using various data. Part.

b) Next, the operation of each block in FIG. 2 will be described.
The camera 3 outputs monochrome image data, and here, the image size is set to a VGA size of 640 × 480 pixels. However, the image size is not limited to this. Further, there is no problem with a color image, and in that case, a function for converting a luminance image needs to be added to the image input I / F unit 5.

  The image input I / F unit 5 extracts the luminance value, x-coordinate, and y-coordinate of each pixel from the output signal from the camera 3 and outputs them to the smoothing processing unit 7. Further, the processed image size may be reduced by thinning out the image data pixel by pixel.

  As shown in FIG. 3, the smoothing processing unit 7 holds a buffer for 4 lines therein, and when pixel data of coordinate values (x, y) is input, it is converted to a 5 × 5 smoothing filter coefficient. The smoothed pixel data of the coordinate value (x−2, y−2) is output to the X differentiation processing unit 9, the Y differentiation processing unit 11, and the image storage unit 19 by the convolution operation based on the above.

  Similar to the smoothing processing unit 7, the X differentiation processing unit 9 holds a buffer for 4 lines therein, and when pixel data of coordinate values (x, y) is input, 5 × 5 X The X differential pixel data of the coordinate value (x−2, y−2) is output to the feature point extraction unit 3 and the image recording unit 19 by convolution calculation based on the differential filter coefficient.

  Similar to the smoothing processing unit 7, the Y differentiation processing unit 11 holds a buffer for four lines therein, and when pixel data of coordinate values (x, y) is input, 5 × 5 Y The Y differential pixel data of the coordinate value (x−2, y−2) is output to the feature point extraction unit 13 and the image recording unit 19 by the convolution calculation based on the differential filter coefficient.

Needless to say, the filter sizes of the smoothing, X differentiation, and Y differentiation need not be limited to this.
When the size of an image block, which is an image unit for motion detection, is 8 × 8 pixels, the feature point extraction unit 13 has eight lines of X differential images and Y differential images as shown in FIG. Two buffers are provided. While the image data is being written into the line buffer of one X differential image and the Y differential image, the other X differential image and Y differential image line buffer is used in the image block of 8 × 8 pixels. Extract the most corner-like pixels.

  Specifically, an index indicating the likelihood of a corner of each pixel is calculated by a technique called Harris corner detection, and the pixel having the highest value in the image block that is larger than an arbitrary threshold is used as a feature point. To do.

  More specifically, as shown in FIG. 4B, an index indicating the corner-likeness of each pixel is calculated by shifting the Harris window of 4 × 4 pixels. If a feature point can be detected, the result is stored in the feature point map 15 and the motion information table 17.

  The size of the Harris window may be smaller than the size of the image block, and need not be limited to 4 × 4 pixels. Further, the number of feature points to be detected need not be limited to one in each image block. For example, all pixels having a cornerness index exceeding a certain reference value may be processed.

  The feature point map 15 is a table indicating in which image block the feature point exists as shown in FIG. For example, the storage unit can set 1 as a flag when a feature point can be detected and set 0 as a flag when the feature point cannot be detected. In addition, there are two banks for the past image of the T-1 frame and the current image of the T frame.

  As shown in FIG. 5A, the image block of the processed image here is 80 horizontal x 60 vertical, and the image block for one column in the outer peripheral portion is not used for feature point extraction. Therefore, the image block of the processing area is 78 vertical × 58 horizontal. FIG. 5 shows only a part of the image blocks.

  In the motion information table 17, as shown in FIG. 6, an identification number is assigned to each image block, and the feature point and motion vector information on the image block can be referred to using the identification number as an index. Yes. In addition, there are two banks for the past image of the T-1 frame and the current image of the T frame.

  The motion information table 17 stores feature point coordinate data (pixel x and y coordinate data) as feature point data, and vector X component and Y component data as motion vectors. ing.

  Here, the motion vector stored in the motion information table 17 for the T-1 frame is a motion vector calculated from the image data of the T-1 frame and the T frame, and is stored in the motion information table 17 for the T frame. The stored motion vector is a motion vector calculated from T frame and T + 1 frame image data.

  As shown in FIG. 7, the image storage unit 19 stores image data in units of image blocks. For example, when a memory of 8 bits per pixel and 32 bits per word is used, data is stored in units of 4 pixels, and in the case of an 8 × 8 pixel image block, data is stored in units of 16 words.

The processing so far is processing for the current input image T frame.
c) Next, the motion detection unit 21 that performs motion detection processing using the feature point map 15 and the motion information table 17 obtained in the T-1 frame and the images of the T-1 frame and the T frame will be described.

  As shown in FIG. 8, the motion detection unit 21 processes each image block in the X raster direction. In the following description, 78 × 58 image blocks excluding the image blocks at each end are processed. Note that the number of image blocks to be processed can be arbitrarily changed, and the processing direction may be the Y raster direction.

First, the motion detection calculation unit 21c will be described.
As illustrated in FIG. 9, the motion detection calculation unit 21c applies an 8 × 8 pixel template image centered on the feature point on the T-1 frame image, which is given from the motion detection control unit 21d, to the T frame image. The motion vector which shows where it moved is calculated.

  Specifically, the Kanade-Lucas-Tomasi method (hereinafter referred to as the KLT method), which is a kind of gradient method, is used. In this KLT method, a motion vector is calculated using six types of image data of T-1 frame and T frame (that is, smoothed image, X differential image, Y differential image of T-1 and T frame), The calculated identification number of the image block is used as an index number, and the result is stored in the motion information table 17.

  For example, as shown in FIG. 10, since the size of the template image is 8 × 8 pixels and the KLT method can be detected with high accuracy, if the motion vector detection range is ± 3 pixels, it may be necessary for the calculation. As the characteristic image data, the hatched image data in the left figure is required. Therefore, in order to perform motion detection processing on one image block, image data of 9 (9 image blocks centered on the processing target) × 6 (six types of image data) image blocks are required. It becomes. However, data for three pixels in the outer peripheral portion is not used.

  Although the size of the template image is 8 × 8 pixels and the motion detection range is ± 3 pixels, it need not be limited to this. In that case, the number of image blocks necessary for the motion detection calculation and the size of the image blocks themselves are appropriately changed.

Next, the motion detection control unit 21d will be described.
The motion detection control unit 21d refers to the feature point map 15 and the motion information table 17 one frame before (T-1 frame), and performs motion detection processing only on the image block in which the feature point on the image exists. The motion detection calculation unit 21c and the image data transfer unit 21a are controlled.

  First, with reference to the feature point map 15 of the T-1 frame, the first image block having a feature point on the image block line to be processed (here, a block of one line in the X raster direction) is targeted.

  As described above, the motion detection calculation unit 21c requires image data of 9 × 6 image blocks including the target image block in order to calculate a motion vector. The image data transfer unit 21a is instructed to transfer the image data of 9 × 6 image blocks from the image storage unit 19 to the detection image storage unit 21b. The transferred image data is stored in the motion detection image storage unit 21b as shown in the image data (A) of FIG.

  Then, after the image data transfer unit 21a finishes transferring the image data, it notifies the motion detection control unit 21d of the completion of the data transfer. The motion detection control unit 21d that has received the data transfer end notification passes the coordinate values of the feature points referenced from the motion information table 17 of the T-1 frame to the motion detection calculation unit 21c and instructs to calculate the motion vector. . Further, in parallel with issuing the calculation command to the motion detection calculation unit 21c, the image data transfer unit 21a is commanded to transfer the image data shown in the image data (B) of FIG. After receiving the end notification from the motion detection calculation unit 21c and the image data transfer unit 21a, the image block in which the next feature point exists is processed.

  At this time, it is inefficient to transfer 9 × 6 image data every time an image block having feature points is processed. Therefore, the motion detection control unit 21d realizes transfer of image data without waste by the following data transfer method.

  Specifically, the distance (unit: image block) between the image block for which the current processing has been completed and the image block in which the next feature point exists is obtained. Then, as shown in FIG. 12, if the distance value is 3 image blocks or less, the data of adjacent image blocks can be used, so only the image data that is insufficient for motion detection processing is transferred (burst transfer). If there are 4 image blocks or more, 9 × 6 image data are transferred.

  Similar processing is repeated until all the image blocks having feature points on the same image block line are processed. Further, these processes are applied to all image block lines.

With the above configuration, it is possible to realize an apparatus that can transfer only image data necessary for processing an image block having feature points and can detect a motion vector at high speed.
d) Next, the image processing method of this embodiment will be described.

First, the entire process will be described with reference to FIG. In the figure, the flow of processing is indicated by a solid line, and the flow of data is indicated by a broken line.
This process extracts a feature point that is a pixel serving as a feature on the past image, and a motion vector indicating where the template image having a predetermined size including the feature point has moved on the current image Is a process for calculating.

As shown in FIG. 13, in step (S) 100, image data captured by the camera 3 is captured (image input I / F processing).
In the subsequent step 110, preprocessing of the captured image data is performed. Specifically, the above-described smoothing process, X differential process, and Y differential process are performed.

  In the following step 120, processing for extracting feature points is performed (feature point extraction processing). This feature point extraction process is a process of extracting feature points in units of image blocks from an image obtained from preprocessing, and if feature points can be extracted, a flag indicating the presence of feature points in an image block is displayed as a feature point map. This is a process for storing the position of the feature point in the motion information table. In addition to the feature point extraction processing (in parallel), processing for storing the smoothed image data in the image storage unit 19 is performed.

  In subsequent step 130, it is determined whether or not the calculation of the motion vectors of all image blocks in which feature points exist has been completed (end determination process). If an affirmative determination is made here, the present process ends. If a negative determination is made, the process proceeds to step 140.

  In step 140, an image data transfer amount determination process is performed. This image data transfer amount determination process refers to the other (past) feature point map that is flagged when a past image is input, and processes all image blocks in which the feature points exist in the raster direction. To calculate the distance between the image block for which the motion vector has been calculated at this time and the next image block where the feature point on the image block line exists, the motion detection image storage unit is already calculated from the distance value. In this process, it is determined whether the image data transferred to 21b can be used for motion vector calculation in the next image block in which a feature point exists, and the amount of image data that is lacking is clarified.

In the subsequent step 150, the image data amount is transferred from the image storage unit 19 to the motion detection image storage unit 21 (image data transfer process).
In subsequent step 160, the motion detection calculation process is performed, and then the process proceeds to step 130. This motion detection calculation processing uses the motion detection image storage unit 21b to calculate a motion vector indicating where a template image including a feature point on a past image has moved on the current image. This is a process of storing the result in the other (past) motion information table storing the feature point coordinates when a past image is input.

Note that the processing of steps 130 to 160 is motion detection processing (processing for calculating a motion vector) performed by the motion detection unit 21.
Next, among the processes described above, the process of each image block line will be described with reference to FIG.

The variables used for processing are as follows.
STATE: Variable indicating state map [78]: Feature point map for one line mc: Feature point map counter L: Distance (number) to the image block where the next feature point exists
As shown in FIG. 14, in step 200, an initialization process is performed. Specifically, STATE is set to INIT indicating the initial state, and mc is set to 0.

  In the following step 210, it is determined whether or not the process for one line is completed depending on whether mc is 78 or not. If an affirmative determination is made here, it is determined that the processing has been completed for one line, and the processing proceeds to the next image block line. On the other hand, if a negative determination is made, the process proceeds to step 220.

  In step 220, the presence / absence of a feature point is determined by whether or not the flag indicating the feature point is 1. If an affirmative determination is made here, the process proceeds to step 230, while if a negative determination is made, the process proceeds to step 300.

In step 300, it is determined whether STATE is INIT. If an affirmative determination is made here, the process proceeds to step 320, while if a negative determination is made, the process proceeds to step 310.
In step 310, burst transfer (transfer of 3 × 6 image blocks) is performed, and the process proceeds to step 320.

In step 320, mc is incremented, and the process returns to step 210.
On the other hand, in step 230, where the determination is affirmative in step 220, it is determined whether STATE is INIT or SKIP (indicating skip transfer). If an affirmative determination is made here, the process proceeds to step 240, while if a negative determination is made, the process proceeds to step 250.

In step 240, step transfer (transfer of 9 × 6 image blocks) is performed.
In the following step 250, KLT calculation is performed. In parallel, burst transfer of adjacent image blocks (transfer of 3 × 6 image blocks) is performed.

  In the next step 260, the distance L to the image block where the next feature point exists is measured. When the distance L cannot be obtained (when one line is completed), the present process is temporarily terminated.

In the following step 270, it is determined whether or not the distance L is 4 or more. If an affirmative determination is made here, the process proceeds to step 290, whereas if a negative determination is made, the process proceeds to step 280.
In step 280, STATE is set to BURST (indicating burst transfer), and the process proceeds to step 320.

  On the other hand, in step 290, STATE is set to SKIP (indicating skip transfer). Further, a value obtained by adding the distance L to mc is set as the counter mc, and the process returns to step 210.

Since the image block processing for one line can be performed by the above-described processing, the processing for all lines can be completed by processing this processing for 58 lines.
FIG. 15 shows the state transition set by the variable STATE. As is apparent from this state transition diagram, STATE is switched to INIT, BURST, and SKIP according to the distance L.

  e) As described above, in the present embodiment, necessary image data can be transferred at a high speed by performing the above-described processing, so that the motion vector can be quickly processed based on the image data.

  In addition, the image processing apparatus 1 is effective for use in a vehicle front detection sensor in a vehicle-mounted image recognition system that has a high occupation ratio of roads and sky areas in which feature points are difficult to detect on a captured image.

  Furthermore, since high reliability is required when used in this vehicle forward detection sensor, a gradient method that can be detected with high accuracy in units of subpixels, such as the KLT method, may be used for the motion detection calculation unit 21c. desirable. However, since the gradient method has a narrow detectable range, it is desirable to use a high frame rate camera with a short capture period as the camera 3 so that it falls within the detectable range.

[Second Embodiment]
Next, the second embodiment will be described. However, since the second embodiment is often in common with the first embodiment, detailed description of common parts will be omitted below, and different parts will be emphasized. I will explain it.

  a) In the image processing apparatus according to the first embodiment, the processing direction of the image block is arbitrary for raster processing on the X axis or raster processing on the Y axis. On the other hand, in the image processing apparatus according to the present embodiment, the motion detection control unit has a function of determining which of the raster processing on the X axis and the raster processing on the Y axis can be performed at high speed.

That is, the motion detection control unit according to the present embodiment calculates prediction processing times for both raster processing in the X-axis direction and the Y-axis direction, and determines which is used for processing.
A specific method for calculating the predicted processing time is shown in FIG. 16A shows a case where feature points are scanned in the X-axis direction, and FIG. 16B shows a case where feature points are scanned in the Y-axis direction.

Here, for example, when feature points are arranged as shown in FIG. 16C, a case where feature points are scanned in the direction of the arrow in the figure is taken as an example.
First, as shown in the equations of Steps 1 to 5 below, at the first feature point, the processing time K of motion detection calculation is added to the processing time for initial transfer (Step 1).

  Next, the second feature point that is a distance 1 away from the first feature point has a distance of 1 and does not require transfer of image data. Therefore, during the prediction processing time S up to the previous step, the motion detection calculation is performed. The processing time K is added (Step 2).

  Next, in the third feature point that is a distance 2 away from the second feature point, the prediction processing time S up to the previous step is the processing time (3 × 6) × T corresponding to the distance 2 The detection calculation processing time K is added (Step 3).

  Next, at the fourth feature point that is a distance 3 away from the third feature point, the prediction processing time S up to the previous step is the processing time (6 × 6) × T corresponding to the distance 3 and motion detection. The calculation processing time K is added (Step 4).

  Next, at the fifth feature point that is a distance 4 away from the fourth feature point, the prediction processing time S up to the previous step is the processing time corresponding to the distance 4 (9 × 6) × T and motion detection The calculation processing time K is added (Step 5).

・ Step1: S = (9 × 6) × T + K
Step 2: S = S + K
・ Step3: S = S + (3 × 6) × T + K
・ Step4: S = S + (6 × 6) × T + K
・ Step5: S = S + (9 x 6) x T + K
However, prediction processing time: S, processing time for motion detection calculation: K cycles, transfer time for one image block: T cycles (T <K)
That is, in the present embodiment, the processing time in one line can be obtained by performing the processing of Steps 1 to 5, and therefore, the processing time in the X-axis direction and the Y-axis direction can be known by adding up all the lines. Therefore, by comparing the processing times, a raster processing direction with a short processing time can be obtained.

Thereby, it is possible to realize a higher processing speed than the first implementation.
b) Next, a modification of this embodiment will be described.
In this modification, when determining whether to perform raster processing on the X axis or raster processing on the Y axis, which can be performed at high speed, the processing time map is not predicted by scanning the feature point map as described above. Instead, as shown in FIG. 17, information on the number of appearances of the initial transfer is added to the distance value of 1 block, 2 blocks, 3 blocks, 4 blocks or more for each row and column of each image block.

  The processing time according to the distance value is 0 for 1 block, (3 × 6) × T for 3 blocks, (6 × 6) × T for 3 blocks, 4 blocks or more, or initial transfer ( 9 × 6) × T.

And in the case of the distance value shown in FIG. 17, prediction processing time is calculated | required like a following formula.
S = 1 time x {(0 x 6) x T + K} (← when distance value is 1)
+1 time x {(3 x 6) x T + K} (← distance value 2)
+1 time x {(6 x 6) x T + K} (← distance value 3)
+2 times x {(9 x 6) x T + K} (← distance value 4 and initial transfer)
Thus, in this modification, the processing time can be predicted without scanning the feature point map, so that there is an advantage that the processing time can be further reduced.

[Third Embodiment]
Next, a third embodiment will be described. Since the third embodiment is often in common with those according to the first and second embodiments, a detailed description of common parts will be omitted below, and different parts will be described. Will be explained with emphasis.

  As described in the second embodiment, the processing time required to calculate the motion vector can be predicted from the feature point map. However, if the processing time is set so as not to have a sufficient margin, the processing will fail if the motion vector calculation processing is not in time during the image capture period.

  Therefore, in the present embodiment, when the motion detection control unit 21d determines that the processing time is not in time, a range that can be processed (for example, an image block line that can be processed) is calculated, and the range is processed. To be able to calculate motion vectors continuously.

Details will be described below.
As shown in FIG. 18A, image data is stored using three frame buffers in a circulating manner (stored in a ring).

  Therefore, as shown in FIG. 18B, when the processing is not delayed, the motion vector is calculated from the T-2 frame image data and the T-1 frame image data when the T frame image data is captured. Can be calculated.

On the other hand, as shown in FIG. 18C, when the processing is delayed, for example, an image of T-2 frame is used, but the image is rewritten to an image of T + 1, so that the operation fails.
As a countermeasure against this, as shown in FIG. 19, when the frame synchronization signal Vsync is checked and Vsync changes from H → L, if the motion detection process is still performed, the next frame (in this example, it can be said). For example, when the image of T + 1 frame) is stopped and the motion detection process is completed, the operation of the entire circuit is temporarily stopped and then the operation of the circuit is resumed (in this example, the process from the T + 2 frame is performed). To resume). Thereby, failure of operation can be avoided.

(Modification 1)
If the frame synchronization signal described above is used, it is possible to avoid the failure of the operation. However, in this case as well, the processing direction is the raster direction, so there is a possibility that the processing block where the motion detection was really desired may not be reached. .

  Therefore, in the first modification, when it is determined from the feature point map 15 that the motion vector calculation process is not in time for the image capture cycle, only the detection area with a high priority set in advance is processed.

That is, before starting the motion detection, it is determined whether or not it can be completed within the capturing period. If it is not in time, only the detection range with a high priority is processed.
The detection range with high priority indicates, for example, a region around the vehicle when a preceding vehicle is detected. For example, the sky or the landscape region above has a low priority.

Details will be described below.
(1) First, the prediction processing time is estimated by the method shown in FIG. 16 or FIG.
Here, how much processing time it takes for motion detection can be accurately estimated by designing. Specifically, when the circuit is designed and simulated, the number of image data transfer cycles, the number of motion detection processing cycles, and the number of memory arbitration cycles are known. It can be calculated by using that number. In the description of FIGS. 16 and 18, the number of memory arbitration cycles is not taken into consideration. This is because, in practice, Tv described in the following (2) may be set more strictly than the actual value.

(2) Next, the prediction processing time is compared with the image capture cycle.
The image capture cycle Tv (for example, 16.7 msec for a 60 fps camera) is compared with the predicted processing time Tp obtained in (1).

If Tp> Tv, it is determined that the process cannot be completed within the capture period. If Tp ≦ Tv, it is determined that the process can be completed within the capture cycle.
Therefore, based on this determination result, if the process is not in time, only the detection range with a high priority is processed.

Thereby, there is an advantage that important processing can be surely executed.
(Modification 2)
In the second modification, when it is determined from the feature point map that the motion vector calculation process is not in time for the image capture cycle, the process is performed using image data with a reduced image size.

Details will be described below.
As shown in FIG. 20, in the image processing apparatus 31 of the second modification, as in the first embodiment, an image input I / F unit 35 (capturing an image from the camera 33), and a smoothing processing unit 37, an X differentiation processing unit 39, a Y differentiation processing unit 41, a feature point extraction unit 43, a feature point map 45, a motion information table 47, an image storage unit 49, and a motion detection unit 51. The motion detection unit 51 includes an image data transfer unit 51a, a motion detection image storage unit 51b, a motion detection calculation unit 51c, and a motion detection control unit 51d.

  Further, the image processing apparatus 31 includes a micro image feature point extracting unit 53 that extracts feature points of a micro image based on data from the smoothing processing unit 37, the X differential processing unit 39, and the Y differential processing unit 41; A minute feature point map 55 that stores feature point data of the minute image extracted by the minute image feature point extracting unit 53 and a minute motion information table 57 are provided.

  The configurations of the feature point map 45 and the motion information table 47 and the configurations of the minute feature point map 55 and the minute motion information table 57 are the same except that the stored data is different.

  In the second modification, in addition to the processing of the first embodiment, the smoothing processing unit 37, the X differentiation processing unit 39, and the Y differentiation processing unit 41 perform the minute image feature point extraction unit 53. The image data thinned out in units of one pixel is sent and stored.

  Therefore, the reduced image feature point extraction unit 53 extracts feature points with less image data than usual, and stores the feature point data of the reduced image in the minute feature point map 55 and the minute motion information table 57.

  Further, when the motion detection control unit 51d determines from the feature point map 45 that the motion vector calculation process is not in time for the image capture period, the motion detection calculation unit 51c reduces the reduced image feature point map 55 to the reduced size. The motion vector on the image is calculated using the data of the motion information table 57.

As a result, processing can be performed in a short time, so that necessary processing can be performed.
[Fourth Embodiment]
Next, since the fourth embodiment is often in common with that according to the third embodiment, a detailed description of the common parts will be omitted, and different parts will be described mainly.

  As shown in FIG. 21, in the image processing apparatus 61 of the present embodiment, an image input I / F unit 65 (capturing an image from the camera 63) and a smoothing processing unit 67, as in the first embodiment. An X differentiation processing unit 69, a Y differentiation processing unit 71, a feature point extraction unit 73, a feature point map 75, a motion information table 77, an image storage unit 79, and a motion detection unit 81, The motion detection unit 81 includes an image data transfer unit 81a, a motion detection image storage unit 81b, a motion detection calculation unit 81c, and a motion detection control unit 81d.

Further, the image processing apparatus 61 includes two auxiliary feature point maps 83 and 85, and the motion detection unit 81 includes a subsequent feature point extraction unit 81 e.
Details will be described below.

  In the feature point extraction unit 73, in addition to a reference threshold value (threshold value for extracting a feature point), an arbitrary number of auxiliary values that are larger than the reference threshold value Set the auxiliary threshold. Thereby, the feature point to be used can be reduced. Here, since two auxiliary threshold values are set, two auxiliary feature point maps are set.

  For this auxiliary threshold, only feature points are extracted, and if feature points can be extracted, a flag is set in the auxiliary feature point maps 83 and 85, but no feature point coordinate values are written in the motion information table 77.

  Then, when the motion detection control unit 81d determines from the feature point map 75 that the motion vector calculation process is not in time for the image capture cycle, the motion detection control unit 81d determines the capture cycle from the two auxiliary feature point maps 83 and 85. A search is made as to whether there is a calculation process in time.

  As a result, when there is an object in time, the auxiliary feature point map 83 or 85 is used to transfer the image data to the image data transfer unit 81a in the same manner as the image processing apparatus 1 of the first embodiment. Then, the latter-stage feature point extraction unit 81e calculates the feature point coordinates and gives the coordinate values to the motion detection calculation unit 81c to calculate the motion vector.

As described above, in this embodiment, only representative feature points are used, so that the burden of arithmetic processing can be reduced.
[Fifth Embodiment]
Next, since the fifth embodiment is often in common with that according to the first embodiment, a detailed description of common parts will be omitted, and different parts will be described mainly.

  As shown in FIG. 22, in the image processing apparatus 91 of the present embodiment, an image input I / F unit 95 (capturing an image from the camera 93), and a smoothing processing unit 97, as in the first embodiment. An X differentiation processing unit 99, a Y differentiation processing unit 101, a feature point extraction unit 103, a feature point map 105, a motion information table 107, an image storage unit 109, and two motion detection units (first Motion detection unit 111 and second motion detection unit 113). Each motion detection unit 111 and 113 includes image data transfer units 111a and 113a, motion detection image storage units 111b and 113b, and motion detection, respectively. The calculation units 111c and 113c and the motion detection control units 111d and 113d are configured.

  In this embodiment, since the motion detectors 111 and 113 which are two arithmetic chips are provided, the image area to be processed is divided into two detection areas of arbitrary size as shown in FIG. The motion detection calculation units 111 and 113 independently calculate the motion vector of the image block in each detection area. By this parallel processing, motion vectors can be calculated at high speed.

(Modification)
As shown in FIG. 24, in the image processing apparatus 121 according to the first modification, as in the first embodiment, an image input I / F unit 125 (capturing an image from the camera 123), and a smoothing processing unit 127, an X differentiation processing unit 129, a Y differentiation processing unit 131, a feature point extraction unit 133, a feature point map 135, a motion information table 137, an image storage unit 139, and a motion detection unit 141. The motion detection unit 141 includes an image data transfer unit 141a, a motion detection image storage unit 141b, two motion detection calculation units (a first motion detection calculation unit 141c1, a second motion detection calculation unit 141c2), a motion It is comprised by the detection control part 141d.

Details will be described below.
In the fifth embodiment described above, the number of motion detection units 111 and 113 is the same as the total number, in the above example, divided into two detection regions, and the motion vector of the image block in each detection region is divided into each motion detection calculation unit 111c. 113c are calculated independently. In contrast, in the first modification, the number of motion detection units 141 is one, but two motion detection calculation units 141c1 and 141c2 are provided.

  And in this modification, as shown in FIG. 25, it processes to the same direction, synchronizing an adjacent image block. If a feature point exists in one of the processing blocks, one of the motion detection calculation units 141c1 and 141c2 is stopped until the motion vector of the feature point is calculated, and the same direction (for example, X (Axial direction).

  The merit of this method is that the amount of data transferred from the image storage unit 139 can be suppressed. For example, when a feature point is found and 9 × 6 image blocks are newly transferred, in the case of the fifth embodiment, 108 image blocks that are doubled are simply transferred. However, in the first modification, since necessary image data blocks overlap, it is only necessary to transfer 12 × 6 image blocks, that is, 72 image blocks.

Therefore, the image data transfer amount can be suppressed to about 1.3 times, and high-speed motion vector calculation by parallel processing can be expected.
[Sixth Embodiment]
The sixth embodiment aims to use the image processing apparatus as a vehicle intruder detection sensor in an in-vehicle image recognition system, and controls the motion vector not to be calculated when there is no intruder. It aims at power consumption.

Specifically, the motion detection control unit calculates the total number of feature points present on the image from the feature point map, and controls to calculate a motion vector when an arbitrary threshold value is exceeded.
That is, the presence / absence of an intruder is determined based on, for example, whether or not the total number of feature points has reached 300 or more, and a motion vector is calculated only in that case.

(Modification)
In the sixth embodiment, the calculation of the motion vector is determined based on the total number of feature points. In the present modification, the current feature point map is compared with the past feature point map, and the feature points are compared. Control is performed so that a motion vector is calculated when the total number of image blocks in which the presence or absence of points exceeds a certain threshold.

  In addition, this invention is not limited to the said embodiment, It is possible to implement in various aspects.

It is a block diagram which shows the image processing apparatus concerning each claim. 1 is a block diagram illustrating an image processing apparatus according to a first embodiment. It is a figure explaining the smoothing process concerning 1st Embodiment. It is a figure explaining feature point extraction concerning a 1st embodiment. It is a figure which shows the feature point map concerning 1st Embodiment. It is a figure which shows the motion information table concerning 1st Embodiment. It is a figure explaining the storage of the image memory | storage part concerning 1st Embodiment. It is a figure which shows the direction of the motion detection process concerning 1st Embodiment. It is a figure explaining the outline | summary of the motion detection process concerning 1st Embodiment. It is a figure which shows the range of a motion detection concerning 1st Embodiment. It is a figure explaining the memory | storage part for a motion detection concerning 1st Embodiment. It is a figure explaining the data transfer system concerning 1st Embodiment. It is a flowchart explaining the motion detection process concerning 1st Embodiment. It is a flowchart explaining the process of an image block line concerning 1st Embodiment. It is a figure explaining state transition concerning a 1st embodiment. It is a figure explaining the calculation method of prediction processing time concerning a 2nd embodiment. It is a figure explaining the calculation method of prediction processing time concerning the modification of 2nd Embodiment. It is a figure explaining the utilization method of a frame buffer concerning 3rd Embodiment. It is a figure explaining the timing of a motion detection concerning 3rd Embodiment. It is a block diagram which shows the image processing apparatus concerning the modification 2 of 3rd Embodiment. It is a block diagram which shows the image processing apparatus concerning 4th Embodiment. It is a block diagram which shows the image processing apparatus concerning 5th Embodiment. It is a figure explaining the processing content concerning 5th Embodiment. It is a block diagram which shows the image processing apparatus concerning the modification of 5th Embodiment. It is a figure explaining the processing content concerning the modification of 5th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 31, 61, 91, 121, A1 ... Image processing apparatus 3, 33, 63, 93, 123, A2 ... Camera 5, 35, 65, 95, 125, A3 ... Image input I / F part 7, 37, 67, 97, 127 ... smoothing processing units 9, 39, 69, 99, 129 ... X differentiation processing units 11, 41, 71, 101, 131 ... Y differentiation processing units 13, 43, 73, 103, 133, A6 ... Feature point extraction unit 15, 45, 75, 105, 135, A7 ... Feature point map 17, 47, 77, 107, 137, A9 ... Motion information table 19, 49, 79, 109, 139, A5 ... Image storage unit 21 51, 81, 111, 113, 141, A8... Motion detectors 21a, 51a, 81a, 111a, 113a, 141a, A11... Image data transfer units 21b, 51b, 81b, 111b, 113b, 14 b, A10 ... motion detection image storage units 21c, 51c, 81c, 111c, 113c, 141c1, 141c2, A12 ... motion detection calculation units 21d, 51d, 81d, 111d, 113d, 141d, A13 ... motion detection control unit 53 ... Reduced image feature point extraction unit 55 ... minute feature point map 57 ... minute motion information tables 83, 85 ... auxiliary feature point map 81e ... latter stage feature point extraction unit A4 ... pre-processing unit

Claims (20)

An image that extracts a feature point that is a pixel that is a feature on a past image, and calculates a motion vector that indicates where the template image having the predetermined size including the feature point has moved on the current image A processing device comprising:
An image storage unit for storing image data captured by the image processing device;
A feature point extraction unit that extracts feature points in units of image blocks that are units of images for motion detection based on image data captured by the image processing device;
A feature point map indicating where the feature points extracted by the feature point extraction unit exist in units of the image blocks;
Based on the feature point data stored in the feature point map in units of image blocks and the feature point position data extracted in units of pixels, using the image data stored in the image storage unit, the A motion detector that calculates a motion vector;
An image processing apparatus comprising:   The feature point extraction unit is characterized in units of image blocks, which are image units for motion detection based on the image data captured in the image processing device in parallel with the process of storing the image data in the image storage unit. The image processing apparatus according to claim 1, wherein points are extracted. An image that extracts a feature point that is a pixel that is a feature on a past image, and calculates a motion vector that indicates where the template image having the predetermined size including the feature point has moved on the current image A processing device comprising:
An image storage unit for storing image data captured by the image processing device;
A pre-processing unit for generating necessary image data in order to extract feature points on the image data and to calculate a motion vector;
A feature point map indicating where the feature points on the image data exist in image block units, which are image units for motion detection; and
A motion information table storing feature point position data and motion vector data extracted in pixel units;
Extracting feature points in image block units from the image data obtained from the preprocessing unit, and if feature points can be extracted, set a flag indicating that feature points exist in the image block in the feature point map, And a feature point extraction unit that stores the position of the feature point in the motion information table;
Based on image data stored in the image storage unit, feature point data stored in the feature point map in units of image blocks, and feature point position data in units of pixels stored in the motion information table A motion detector for calculating a motion vector;
An image processing apparatus comprising: The motion detector is
An image storage unit for motion detection for storing past image data and current image data necessary for motion detection;
An image data transfer unit that transfers the image data stored in the image storage unit to the motion detection image storage unit;
Using the motion detection image storage unit, a motion vector indicating where the template image including the feature point on the past image has moved on the current image is calculated, and the past image is input as a result of the calculation. A motion detection calculation unit that stores in a past motion information table that stores feature point coordinates,
Referring to the past feature point map that was flagged when a past image was input, the motion vector calculation was completed at this point in order to process all image blocks with feature points in the raster direction. The distance between the image block and the next image block where the feature point on the image block line exists is calculated, and the image data already transferred to the motion detection image storage unit from the distance value is the feature point. A motion detection control unit that determines whether it can be used to calculate a motion vector in the next image block in which the image data exists, and instructs the image data transfer unit to transfer only the missing image data;
The image processing apparatus according to claim 3, further comprising:   The motion detection control unit is adjacent to the image range stored in the motion detection image storage unit in the motion detection direction during a waiting time while the motion detection calculation unit calculates the motion vector. 5. The image processing apparatus according to claim 4, further comprising a motion detection function that instructs the image data transfer unit to transfer image data of a predetermined number of image block sequences.   The motion detection control unit has a function of referring to the feature point map to determine whether the processing direction of an image block whose motion is to be detected is raster-processed in the X-axis direction or the Y-axis direction. The image processing apparatus according to claim 4, wherein:   7. The feature point map also holds information on the number of appearances according to a distance from an image block in which a feature point exists to an image block in which a next feature point exists. The image processing apparatus described.   If the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture cycle, the motion detection control unit calculates up to the processable image block line and processes the image block line. The image processing apparatus according to claim 4, further comprising a function.   The motion detection control unit has a function of processing only a detection region with a high priority when it is determined from the feature point map that a motion vector calculation process is not in time for an image capture period. The image processing apparatus according to claim 4.   The motion detection control unit has a function of processing a reduced image when it is determined from the feature point map that a motion vector calculation process is not in time for an image capture period, and the preprocessing unit includes a reduced image The image processing apparatus according to claim 4, further comprising: a function for generating the image. In addition to the reference threshold value, the feature point extraction unit can set a predetermined threshold value that can set a predetermined value larger than the reference threshold value. For feature thresholds, feature points are extracted, and feature point detection presence / absence is written in the same number of auxiliary feature point maps as auxiliary thresholds,
When the motion detection control unit determines from the feature point map that the motion vector calculation process is not in time for the image capture period, some of the auxiliary feature point maps are in time for the capture period. The image processing apparatus according to any one of claims 4 to 10, wherein the auxiliary feature point map is used instead of the feature point map when the feature point map is searched.   A plurality of motion detection units each including the motion detection control unit, the motion detection image storage unit, the motion detection calculation unit, and the image data transfer unit are mounted, and the motion detection is performed on the image area to be processed. The image processing apparatus according to claim 4, wherein the image processing device is divided into processing regions equal to the total number of copies, and each motion detection unit independently calculates a motion vector for an image block in each region. .   5. A plurality of the motion detection calculation units are mounted, and each of the motion detection calculation units calculates a motion vector of each image block while synchronizing with adjacent image blocks in the same direction. The image processing apparatus according to any one of 12.   The image processing apparatus according to claim 1, wherein the image processing apparatus is used as a vehicle front detection sensor in an in-vehicle image recognition system.   15. The image processing apparatus according to claim 14, wherein a camera having a high frame rate with a short capturing cycle is used as a camera for capturing an image, and a motion vector is obtained using the KLT method.   The image processing apparatus according to claim 1, wherein the image processing apparatus is used as a vehicle intruder detection sensor in an in-vehicle image recognition system.   The total number of feature points existing on the image is calculated from the feature point map, and control is performed so as to calculate a motion vector when the total number exceeds a predetermined threshold value. The image processing apparatus described.   Compare the current feature point map with the past feature point map, and calculate the motion vector when the total number of image blocks that have changed in the presence or absence of feature points exceeds a predetermined threshold The image processing apparatus according to claim 16, wherein the image processing apparatus is controlled as follows. An image that extracts a feature point that is a pixel that is a feature on a past image, and calculates a motion vector that indicates where the template image having the predetermined size including the feature point has moved on the current image A processing method,
Image input I / F processing for capturing image data;
Image storage processing for storing the image data in an image storage unit in units of image blocks;
Preprocessing for generating necessary image data in order to extract feature points on the image data and to calculate a motion vector;
The feature point is extracted from the image data obtained from the pre-processing in units of image blocks, and when the feature point can be extracted, a flag indicating that the feature point exists in the image block is set in the feature point map Point extraction processing,
Based on the data of feature points in units of image blocks stored in the feature point map and the data of the positions of feature points extracted in units of pixels, using the image data recorded in the image recording unit, A motion vector detection process for calculating a motion vector;
An image processing method comprising: The motion vector detection process includes:
An end determination process for determining whether the calculation of motion vectors of all image blocks in which the feature points exist is completed;
When it is determined by the end determination process that the calculation has not ended, all images in which feature points exist are referred to with reference to a past feature point map that is flagged when a past image is input. In order to process the block in the raster direction, the distance between the image block for which the motion vector has been calculated at this time and the next image block where the feature point on the image block line is present is calculated. Can image data already transferred to the motion detection image storage unit that stores past and current image data used for motion vector calculation be used for motion vector calculation in the next image block in which a feature point exists? Image data transfer amount determination processing for determining the amount of image data that is lacking, and
An image data transfer process for transferring only the amount of image data that is insufficient from an image storage unit storing all image data in units of image blocks to the motion detection image storage unit;
The image processing method according to claim 19, further comprising: