JP4710562B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image captured by a camera.

  The following vehicle periphery monitoring device is known from Patent Document 1. The vehicle periphery monitoring device detects a change in speed of a moving object based on a luminance value of a desired point in an image at the previous frame shooting and a luminance value at a point corresponding to the desired point in the image at the current frame shooting. The optical flow is calculated using an energy functional created on the condition that it is smooth. Then, based on the calculated optical flow, the part of the object present in the image is detected, and the detected part of the object is traced between frames to monitor the situation around the vehicle.

Japanese Patent Laid-Open No. 2003-67752

  However, in the conventional vehicle periphery monitoring device, the area around the image is monitored by detecting the part of the object present in the image based on the calculated optical flow and tracking the detected part of the object between frames. It was. For this reason, when a part of an object existing in an image is detected based on the optical flow, it is specified whether the part is composed of a single object or a plurality of objects. I couldn't. For this reason, when the site | part is comprised with multiple objects, the problem that the movement between the frames of the said site | part may not be tracked correctly has arisen.

The present invention detects a part (traceable part) that can be traced between frames existing in an image captured by the imaging means, calculates a movement speed between the detected frames of the traceable part, the site associated with trackable site within images of each frame is extracted as a target site, and calculates the moving velocity between the attention site frame, to estimate the error of the moving speed between frames of the calculated area of interest The movement speed between the frames of the target region taking into account the error is estimated, and the difference between the movement speeds between the frames of the calculated target region in consideration of the estimated error by comparing with a predetermined threshold value, whether or not it is necessary to correct the moving speed between frames of the calculated area of interest is determined, if the difference is smaller than the predetermined threshold value, between the frames of the calculated area of interest And correcting the moving speed of the.

  According to the present invention, it is determined whether or not it is necessary to correct the movement speed between frames of the site of interest, and the movement speed between frames of the site of interest is corrected based on the determination result. As a result, even when the attention site is composed of a plurality of objects, the accurate movement speed of the attention site can be calculated, and the movement of the attention site between frames can be accurately tracked. .

  FIG. 1 is a block diagram showing a configuration of an embodiment of an image processing apparatus according to the present embodiment. The image processing apparatus 100 includes a camera 101 and a control device 102. The camera 101 includes, for example, an image sensor such as a CCD or a CMOS, continuously captures a subject image input through a lens, and outputs the captured image data to the control device 102.

  The control device 102 includes a CPU and other peripheral circuits, processes image data of each frame continuously input from the image sensor 101, and tracks the movement of an object existing in the image between frames. For this purpose, the control device 102 functionally includes a traceable part detection unit 102a, a speed calculation unit 102b, a speed error estimation unit 102c, a speed estimation unit 102d, and a speed correction unit 102e.

  The traceable part detection unit 102a detects a part (traceable part) of an object that is a target for tracking movement between frames of an object present in the image. This process is necessary to calculate an accurate two-dimensional movement amount and movement speed in the image.

  For example, when the matching between local blocks between consecutive frames or the density gradient method, which is a well-known method, is used, tracking between frames of a part having a uniform density value cannot be performed. In addition, tracking of a region having a density gradient only in a one-dimensional direction depends on the order in which the correspondence between consecutive frames greatly scans. Movement amount and speed may not be calculated. Therefore, in order to measure an accurate movement amount and movement speed between successive images, each target object (target site) has a density gradient in at least two dimensions, and the accurate speed in the image is It is necessary to specify a part that can be measured as a traceable part.

  The traceable part detection unit 102a extracts a corner edge existing in the image using a known SUSAN operator from the image input from the camera 101, and detects the extracted corner edge as a traceable part. Specifically, the process is as follows.

First, as shown in FIG. 2A, the similarity between the density distribution in the mask 2a set in the image and the density at the center of the mask 2a is calculated. The movable range of the mask 2a is limited to a predetermined area. For example, the traceable part detection unit 102a uses the distance d in the density space to calculate the similarity c between the color of the center (x0, y0) of the mask 2a and the density distribution in the mask (x, y) as follows: ). In the following equation (1), th is a threshold value for determining whether the colors are similar.

Next, the corner edge likelihood s of the mask center is calculated by taking the sum of the similarities in the mask according to the following equation (2).

Since the corner edge likelihood s of the mask center calculated by the equation (2) takes a larger value as the mask center (x0, y0) moves away from the corner edge, inversion processing is performed by the following equation (3). In the following equation (3), g is a parameter that is set to emphasize the difference between the vicinity of the corner edge and the others.

  As a result, the corner edge likelihood R is greatly calculated in the vicinity of the corner edge in the image as shown in FIG. 2 (a), and at a point away from the corner edge in the image as shown in FIG. 2 (b). The corner edge likelihood R is calculated to be small. Next, the corner edge existing in the image can be extracted by binarizing the image by using the inverted corner edge likelihood R calculated by the equation (3). Then, the traceable part detection unit 102a detects the extracted corner edge as a traceable part.

  The speed calculation unit 102b specifies a correspondence relationship between successive frames of the traceable part detected by the traceable part detection unit 102a, and calculates a movement amount of the traceable part between frames. Then, based on the calculated movement amount and the time interval between frames, the movement speed of the traceable part between frames (movement speed of the traceable part) is calculated. Specifically, the movement speed of the traceable part is calculated by performing a known optical flow calculation process on the images of successive frames and calculating the optical flow. For example, as shown in FIG. 3, the speed calculation unit 102 b moves the corner edge 51 detected as a traceable part by the traceable part detection unit 102 a in the x direction (horizontal direction) and the y direction (vertical direction). The moving speed 51b is calculated.

  Furthermore, the speed calculation unit 102b extracts a point where the corner edge detected as the traceable part by the traceable part detection unit 102a and the density gradient are continuous from the image, and sets the extracted point as a target part. This makes it possible to detect from the image a site of interest that moves continuously with a traceable site capable of measuring a two-dimensional velocity.

Then, using a known optical flow calculation process for the set attention site, the correspondence between consecutive frames is specified, and the movement amount of the attention site between frames is calculated. Based on the calculated amount of movement between frames and the time interval between frames, a moving speed in a specific direction between the frames of the attention site (movement speed of the attention site) is calculated. For example, as shown in FIG. 3, the velocity 52a in the x direction at the site of interest 52 is calculated.

  The speed error estimation unit 102c estimates an error in the movement speed in a specific direction of the site of interest calculated by the speed calculation unit 102b. In the present embodiment, a method for estimating the error in the moving speed in the x direction of the region of interest will be described. However, the error in the moving speed in the y direction can be similarly estimated.

  Here, an error in the movement speed of the target region will be described. For example, when the edge e (t) and the edge e (t + 1) shown in FIG. 3 correspond between successive frames, the continuity between the edge including the target region 52 and the edge including the traceable region 51 is obtained. It is assumed that both points are determined to be portions that form the same edge. At this time, consider the case where the x-direction velocity 52a of the site of interest 52 is calculated when the edge e (t) at time t moves to e (t + 1) at time t + 1 (when the edge moves diagonally downward). In this case, as described above, tracking of a region having a density gradient only in the one-dimensional direction largely depends on the scanning order of correspondence between consecutive frames, and there is a possibility that miscorrespondence occurs between frames. For this reason, the x-direction velocity 52a of the target region 52 is calculated including an error.

The speed error estimating unit 102c performs the following process in order to estimate the error included in the calculated x-direction speed 52a of the target region 52. First, the speed error estimation unit 102c calculates the inclination of the attention site 52. For the inclination of the attention area 52, for example, an edge including the attention area 52 is detected using an edge detection filter such as a Sobel filter, and the intensity Iv of the vertical edge and the intensity Ih of the horizontal edge are calculated. Based on the calculated vertical edge intensity Iv and horizontal edge intensity Ih, the vertical / horizontal edge intensity ratio Iv / I is calculated, and the edge inclination θ is calculated by the following equation (4). In the following equation (4), it is assumed that θ = 0 when the edge is parallel to the x-axis.

  Next, the speed error estimating unit 102c calculates an edge lateral movement error caused by the amount of vertical movement of the edge including the target region 52. Specifically, based on the angle of the edge calculated by Equation (4), the influence of the vertical movement amount and movement speed on the horizontal movement amount and movement speed when the edge has a movement component in the vertical direction. (Error) is calculated. That is, as shown in FIG. 3, based on the vertical movement speed 51b of the traceable part 51 and the inclination of the target part 52, the speed error 53 generated in the target part 52 due to the edge moving in the vertical direction is calculated.

  The speed estimation unit 102d subtracts the speed error 53 generated in the target part 52 calculated (estimated) by the speed error estimation unit 102c from the x-direction speed 52a of the target part 52 calculated by the speed calculation unit 102b, thereby obtaining the target part. 52 x-direction velocity 54 is estimated.

  In the above description, the processing has been performed on the assumption that the attention site 52 belongs to the same edges e (t) and e (t + 1) as the traceable site 51 due to the continuity of density change. In this case, if the traceable portion 51 and the target portion 52 form a contour of the same target, the x-direction velocities 51a and 54 of both portions are substantially the same. However, when the background and the target edge in front overlap, for example, as shown in FIG. 4, when the edge detected from the background wall and the edge detected from the person overlap, the estimated x direction of the target region 52 is estimated. The speed 54 is estimated to be different from the x-direction speed 51a of the traceable part.

  Therefore, the speed correction unit 102e uses the lateral speed 51a of the traceable part 51 where the accurate movement amount and speed are observed and the estimated speed 54 of the target part, and the target part 52 is the same target as the traceable part 51. It is determined whether or not the region belongs to the region, and the x-direction speed 52a of the region of interest 52 is corrected based on the determination result.

  The speed correction unit 102e first compares the absolute value of the difference between the lateral speed 51a of the traceable part 51 and the estimated speed 54 of the target part with a predetermined threshold Th. When the absolute value of the difference between the lateral speed 51a of the traceable part 51 and the estimated speed 54 of the target part is equal to or less than a predetermined threshold Th, the target part 52 belongs to the same target as the traceable part 51 And the x-direction speed 52a of the site of interest 52 is replaced with the lateral speed 51a of the traceable part 51 and corrected.

  On the other hand, when the absolute value of the difference between the lateral velocity 51a of the traceable part 51 and the estimated speed 54 of the target part is larger than a predetermined threshold Th, the target part 52 is set to the same target as the traceable part 51. It determines with it being a site | part which does not belong, sets the attention site | part 52 as a corner edge, ie, a traceable site | part, and performs the process mentioned above again.

  In this way, the part capable of measuring the two-dimensional velocity by the traceable part detection unit 102a is identified as the traceable part, the speed of the traceable part is calculated, and the attention that is continuous to the traceable part is calculated from the calculated result. The speed of the part was estimated, and the consistency of speed between the traceable part and the target part was determined. This makes it possible to determine with high accuracy whether the traceable part and the target part belong to the same target or different parts. In addition, since the speed of the target region is corrected based on the result of the consistency determination, the accurate speed can be calculated even for a site composed of a plurality of objects.

  FIG. 5 is a flowchart showing processing of the image processing apparatus according to the present embodiment. The processing shown in FIG. 5 is executed by the control device 102 as a program that starts when image input from the camera 101 is started.

  In step S10, as described above, the traceable part detection unit 102a sets a mask in the image, calculates the corner edge likelihood s of the mask center, detects the corner edge, and tracks the detected corner edge. Set as possible part. Thereafter, the process proceeds to step S20, and the speed calculation unit 102b calculates the speed between consecutive frames of the traceable part detected by the traceable part detection unit 102a, and proceeds to step S30.

  In step S30, the speed calculation unit 102b determines whether or not a point where the corner edge detected as the traceable part by the traceable part detection unit 102a and the density gradient are continuous can be extracted from the image. If it is determined that it cannot be extracted, the process proceeds to step S110 described later. On the other hand, if it is determined that the extraction has been completed, the process proceeds to step S40. In step S40, the extracted point is set as a site of interest, and the process proceeds to step S50.

  In step S50, the speed calculation unit 102b calculates the speed between successive frames of the set target region. Thereafter, the process proceeds to step S60, the inclination of the edge including the target region is calculated, and the process proceeds to step S70. In step S70, the speed error estimating unit 102c calculates the movement error of the edge including the site of interest as described above. Thereafter, the process proceeds to step S80, and the speed estimation unit 102d subtracts the speed error generated in the target site estimated by the speed error estimation unit 102c from the speed of the target site calculated by the speed calculation unit 102b, thereby obtaining the speed of the target site. Is estimated. Thereafter, the process proceeds to step S90.

  In step S90, as described above, the speed correction unit 102e uses the speed of the traceable part where the accurate movement amount and speed are observed and the estimated speed of the target part as described above, and the target part is the same as the traceable part. It is determined whether or not the part belongs to the target, and it is determined whether or not the speed of the target part needs to be corrected. If it is determined that correction is necessary, the process proceeds to step S100. In step S100, the speed correction unit 102e corrects the speed of the site of interest by replacing it with the speed of the traceable site. Then, it progresses to step S110.

  On the other hand, if it is determined that no correction is necessary, the process proceeds to step S130. In step S130, the speed correction unit 102e sets the attention site 52 as a traceable site, and returns to step S50.

  In step S110, it is determined whether the image input from the camera 101 has stopped. If it is determined that the image input is not stopped, the process proceeds to step S120. In step S120, the next frame image input from the camera 101 is set as a processing target, and the process returns to step S10. On the other hand, if it is determined that the input of the image has been stopped, the process is terminated.

According to the present embodiment described above, the following operational effects can be obtained.
(1) A part where a two-dimensional velocity can be measured by the traceable part detection unit 102a is specified as a traceable part, the speed of the traceable part is calculated, and the traceable part moves continuously from the calculated result. The speed of the region of interest was estimated, and the consistency of the speed between the traceable region and the region of interest was determined. This makes it possible to determine with high accuracy whether the traceable part and the target part belong to the same target or different parts.

(2) Moreover, since the speed of the region of interest is corrected based on the consistency determination result, the accurate speed can be calculated even in a region composed of a plurality of objects.

(3) The traceable part detection unit 102a extracts a corner edge existing in the image by using a known SUSAN operator for the image input from the camera 101. This makes it possible to extract corner edges that exist in the image with high accuracy by utilizing conventional techniques.

-Modification-
The image processing apparatus according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the traceable part detecting unit 102a extracts a corner edge existing in the image by using a known SUSAN operator from the image input from the camera 101, and extracts it. An example in which the corner edge is detected as a traceable part has been described. However, the present invention is not limited to this, and the traceable part may be detected by other methods. For example, the following process may be performed to detect a traceable part present in the image.

  First, the traceable part detection unit 102a divides the entire image input from the camera 101 into regions having a predetermined area. For example, the entire image is divided into 5 × 5 pixel regions. Then, edge extraction processing is performed on each divided area, and edges existing in each area are extracted. Thereafter, the angle of each edge extracted in each region is calculated, and the variance of the angle of each edge included in each region is calculated. Then, the edge angle variance calculated in each region is compared with a predetermined threshold value, and a region having an edge angle variance greater than or equal to the threshold value is extracted as a traceable part.

  As a result, an area having an edge angle variance greater than or equal to a threshold means that the density gradient direction of at least two dimensions is included in the area, and in this area, the correspondence between frames Therefore, the traceable part can be detected with high accuracy, considering that the object can be tracked between frames.

(2) In the above-described embodiment, in order to calculate the speed of the attention site, a known optical flow calculation process is used to identify the correspondence between successive frames, and An example of calculating the movement amount has been described. However, the present invention is not limited to this, and when the site of interest has a concentration gradient and an edge is detected from the site of interest, the speed of the site of interest is calculated by measuring the time that the edge stays in the site of interest. May be.

(3) In the above-described embodiment, the speed correction unit 102e compares the absolute value of the difference between the speed of the traceable part and the estimated speed of the target part with a predetermined threshold Th, and according to the comparison result. An example in which the speed of the attention site is corrected has been described. However, the present invention is not limited to this. In comparing the absolute value of the difference between the speed of the traceable part and the estimated speed of the target part and the predetermined threshold Th, the traceable part and the target part are limited to a minute region, You may make it calculate the absolute value of the difference of the speed | velocity | rate of the site | part which can be tracked between micro area | regions, and the estimation speed | rate of an attention site | part. As a result, even if at least one of the traceable part and the target part is a part where the concentration gradient direction is not uniform, that is, a part having a concentration gradient in a curved line, the concentration gradient direction in each minute region is linearly approximated. Therefore, accurate speed verification can be performed.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

  The correspondence between the constituent elements of the claims and the embodiment will be described. The camera 101 corresponds to an imaging unit, the traceable part detection unit 102a corresponds to a traceable part detection unit, and the speed calculation unit 102b corresponds to a traceable part speed calculation unit, a target part extraction unit, and a target part speed calculation unit. The speed error estimation unit 102c and the speed estimation unit 102d correspond to a moving speed estimation unit, and the speed correction unit 102e corresponds to a determination unit and a correction unit. The above description is merely an example, and when interpreting the invention, there is no limitation or restriction on the correspondence between the items described in the above embodiment and the items described in the claims.

It is a block diagram which shows the structure of one Embodiment of an image processing apparatus. It is a figure which shows the specific example about calculation of a corner edge. It is a figure which shows the speed correction principle of an attention site | part. It is a figure which shows the specific example in case a background and the edge of the front object overlap. It is a flowchart figure which shows the process of an image processing apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Camera 102 Control apparatus 102a Traceable part detection part 102b Speed calculation part 102c Speed error estimation part 102d Speed estimation part 102e Speed correction part

Claims (6)

A traceable part detecting means for detecting a part (traceable part) that can be traced between frames existing in an image imaged by the imaging means;
A traceable part speed calculating means for calculating a moving speed between frames of the traceable part detected by the traceable part detecting means;
And attention site extracting means for extracting a region as a target site associated with said trackable site within images of each frame,
Attention site speed calculating means for calculating a moving speed between frames of the attention site;
A movement speed estimation unit that estimates an error of a movement speed between frames of the attention part calculated by the attention part speed calculation unit, and estimates a movement speed between frames of the attention part in consideration of the error;
A difference between a movement speed between frames of the traceable part calculated by the traceable part speed calculation unit and a movement speed between frames of the target part in consideration of an error estimated by the movement speed estimation unit A determination unit that determines whether or not it is necessary to correct the movement speed between frames of the target region calculated by the target region speed calculation unit by comparing with a threshold ;
Oite said determining means, when the difference is smaller than the predetermined threshold value, characterized in that it comprises a correcting means for correcting the moving speed between the attention area speed calculation region of interest calculated by means frame Image processing device. The image processing apparatus according to claim 1.
The traceable part detecting means calculates an angle of an edge included in a predetermined area set in the image, and detects the traceable part based on variance of the angle of the edge in the predetermined area. An image processing apparatus. The image processing apparatus according to claim 1.
The traceable part detecting means detects a corner edge extracted from an image as the traceable part. In the image processing device according to any one of claims 1 to 3,
The image processing apparatus , wherein the determination unit determines that the parts belong to the same target when the difference is smaller than the predetermined threshold . In the image processing device according to any one of claims 1 to 4,
The determination unit limits the traceable part and the attention part to a minute range, the movement speed between frames of the traceable part limited to the minute range, and the frames of the attention part limited to the minute range. An image processing apparatus that determines whether or not it is necessary to correct the moving speed between frames of the target region calculated by the target region speed calculating unit based on the moving speed at the point. An image processing method executed by an image processing unit that detects an object present in an image captured by an imaging unit and tracks the movement of the detected object,
Detecting a part (traceable part) that can be traced between frames existing in the captured image;
Calculating a moving speed between frames of the detected traceable part;
Extracting a part related to the traceable part in the image of each frame as a target part;
Calculating a moving speed between frames of the site of interest;
Estimating the movement speed error between frames of the calculated target region, and estimating the movement speed between frames of the target region taking into account the error;
By comparing the calculated movement speed between frames of the traceable part and the difference in movement speed between frames of the part of interest in consideration of the estimated error with a predetermined threshold, the calculated frame of the part of interest Determining whether or not the movement speed needs to be corrected between,
And a step of correcting the calculated movement speed of the target region between frames when the difference is smaller than the predetermined threshold value.

Images