JP2004220059A - Moving image processing device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving image processing device and method that quickly and stably detect an object independent of an observing system in movement. <P>SOLUTION: An optical flow showing a movement of each image point is detected every certain time from a moving image acquired from an image sensor such as a television camera provided in a mobile observing system. The detected optical flow is grouped in a plurality of areas according to preset criteria. An analysis is then made with a feature value such as a movement vanishing point obtained from optical flow information included in each area. This realizes processing of separately detecting a moving object moving independently of the background. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
FIELD OF THE INVENTION
The present invention relates to a moving image processing apparatus used in a moving image processing apparatus that analyzes a series of images to measure, for example, a traffic flow, detect the appearance of an illegal intruder, and control an autonomous mobile vehicle or a mobile robot. And methods.
[0002]
[Prior art]
In recent years, with the popularization of TV cameras due to the popularization of personal computers, the miniaturization and the speeding up of PCs have been progressing with the spread of PCs. Was. On the other hand, research and development of intelligent mobile work robots, amusement hobby robots, autonomous mobile vehicles, etc. are being actively conducted, and practical application is strongly desired. When constructing systems such as intelligent mobile robots and autonomous mobile vehicles, a moving image processing apparatus and method for analyzing images that change every moment and detecting other moving objects appearing while moving at high speed and stability are known. Indispensable.
[0003]
In the conventional moving object detection processing, a method is generally used in which images in a stationary observation system such as a fixed camera are subjected to difference and threshold processing to detect a newly appearing object. In addition, the technology to detect the motion vector (optical flow) of each point by processing the moving image has been actively studied as a part of the moving image processing research. This optical flow can be considered as the movement of the mobile observation system relative to the three-dimensional space projected onto the image plane. However, when actually detecting the optical flow, error flows are mixed due to brightness changes due to illumination fluctuations. However, it is not easy to detect highly accurate flows at high speed. As a method for detecting a moving object based on this optical flow, for example, a "moving object detection device and method" (Japanese Patent Application No. 7-7446, filed on January 20, 1995) has been proposed.
[0004]
[Problems to be solved by the invention]
When the observation system itself moves, such as an intelligent mobile robot or an autonomous mobile vehicle, the entire environment projected on the image changes every moment. There was a problem that it was not possible to detect the appearing moving object area separately from the background. In addition, when separating background and autonomous moving objects using optical flow, there is a need to deal with the above error flow and to develop a method to efficiently analyze optical flow information generated over the entire image. Had been an issue. In the "moving object detection apparatus and method", a vanishing point of motion (FOE) is detected using an optical flow, and an object that moves independently is detected using a change in this position or an estimated residual in a small area. However, when the flow accuracy is reduced, there is a problem that the detection performance decreases when the moving object appears in the distance and occupies a small proportion in the image. In view of the above points, an object of the present invention is to provide a moving image processing apparatus and method for quickly and stably detecting an object that moves independently of the movement of an observation system.
[0005]
[Means for Solving the Problems]
The moving image processing apparatus and method according to the present invention are characterized in that each camera provided in the moving observation system has means for detecting an optical flow.
A moving image processing apparatus and method according to the present invention include means for grouping extracted optical flows based on preset evaluation criteria. The moving image processing apparatus and method according to the present invention include means for detecting an independently moving object by analyzing optical flow information included in the grouped optical flow set. It is characterized by
[0006]
BEST MODE FOR CARRYING OUT THE INVENTION
Figure 1 shows the relationship between the claims. Hereinafter, embodiments of the present invention will be described with reference to FIGS.
[0007]
First, embodiments of the first, second, fourth, and sixth aspects will be described. FIG. 2 shows the overall configuration of the present invention. The motion vector extracting means a1 estimates a motion vector (optical flow) on the screen for one or a plurality of camera images mounted on the mobile observation system. In the motion vector integrating means a2, the extracted optical flows are grouped based on a preset evaluation criterion. In the moving object detecting means a3, for example, when a set of the grouped optical flows is included in the projection area of the independent moving object, the movement of the moving object existing on the extension of each flow is considered. By using the difference between the vanishing point (moving object FOE) and the vanishing point of the movement (background FOE) caused by the self-movement of the moving observation system, the detection of the object region that moves independently to the moving observation system can be performed. Do it.
[0008]
The specific configuration of each part is described below. FIG. 3 shows an example of a concrete configuration of the motion vector extracting means a1. The region-of-interest setting unit b1 sets a local region of M pixels × N pixels for performing a moving object detection process on an image that is momentarily input via one or more cameras mounted on the moving observation system. . The values of the region sizes M and N are appropriately set according to the application and the processing speed requirement. The similarity calculation unit b2 calculates a similarity between successive points in time of each pixel belonging to the region of interest and detects a position having this maximum, thereby detecting a motion vector (optical flow). As the degree of similarity, a correlation coefficient, a sum of absolute values of differences in lightness values, and the like can be used. In this configuration example, after analyzing the lightness distribution in the vicinity area of each pixel, each of the pixels in the vicinity area at a continuous point in time is analyzed. The case where the value obtained by multiplying the sum of the absolute values of the differences of the corresponding pixel brightness values by -1 is used as the similarity is described below.
[0009]
When obtaining an optical flow starting from a pixel position (xi (t), yi (t)) at time t, first, a K pixel × L pixel having this (xi (t), yi (t)) as a center position This sets the neighborhood of. The values of K and L are appropriately set in consideration of the processing time. Next, the brightness distribution of all the pixels KL in this neighboring area is obtained. Is selected as the target for optical flow extraction. If it is not determined that there is a sufficient amount of information, the optical flow extraction accuracy cannot be improved, so the optical flow extraction is not performed at this pixel, and the process moves to the next pixel in the region of interest and the above processing is repeatedly performed. I do. For the pixel selected as the target of the optical flow extraction, the above brightness distribution evaluation is executed for each candidate point in the predetermined search area at the time point t + 1. Candidate points that are not determined to have sufficient information are not subject to similarity calculation and move to the next candidate point. For a candidate point determined to have a sufficient amount of information, after clearing the sum memory S (k) to 0, a neighboring area centered at (xi (t), yi (t)) at time t And the absolute value sum | gi (t) -gi (t + 1) of the difference between the pixel value gi (t) of the point at and the pixel value gi (t + 1) of the corresponding point in the neighboring area centered on the candidate point at time t + 1 Is calculated and added to the sum total memory S (k) for the pixels in all the neighboring areas, and the finally obtained sum value multiplied by -1 is newly stored in the sum total memory S. deep. The above processing is executed for all the candidate points, and the candidate point (xi (t + 1), yi (t + 1)) at which the obtained total memory value is the maximum is detected as the end point position of the motion vector. In the motion vector memory storage unit b3, the pixel position (xi (t + 1), yi (t + 1)) at the time t + 1 at which the similarity calculated by the similarity calculation unit b2 becomes the maximum, and the corresponding pixel at the time t The difference vector (xi (t + 1) -xi (t), yi (t + 1) -yi (t)) of the position (xi (t), yi (t)) is converted to the pixel position (xi (t), yi (t)) is obtained as an optical flow vector, and sequentially stored in a memory together with position coordinate values (xi (t), yi (t)).
[0010]
FIG. 4 shows an example of a specific configuration of the motion vector integrating means a2. First, the local region dividing unit c1 divides the entire region of interest into local region groups having a size of A pixels × B pixels. A and B are set to be divisors of the region of interest M and N, respectively. This makes it possible to divide the entire region of interest into (M / A) × (N / B) pieces without overlapping, but in this embodiment, in order to minimize missing of a moving object, A method is used in which a new local region is set between two adjacent regions of the local region obtained as a result of the non-overlapping division. As a result, about (9 MN / 4AB) local regions are generated. Next, a local area selection unit c2 selects a local area to be used for moving object detection from the set local areas based on a predetermined criterion. As a criterion for this selection, for example, the number of optical flows included in each local area is obtained by referring to the contents of the motion vector memory storage unit b3 in the motion vector extraction unit a1, and this value is larger than a preset threshold value. It is effective to adopt a method of selecting things. In the local area memory storage unit c3, the position and size information of each local area used for detecting a moving object selected by the local area selection unit c2 is sequentially stored in a memory.
[0011]
FIG. 5 shows an example of a specific configuration of the moving object detection means a3. First, the background FOE estimating unit d1 estimates a vanishing point (background FOE) of the motion of the background region caused by the movement of the mobile observation system itself. In this method, it is possible to detect an object that is stationary in the environment before moving, and to find the intersection point of a group of straight lines obtained from a set of optical flow vectors included in this area. The local area FOE estimating unit d2 obtains each position and size information of the local area used for the moving object detection by referring to the contents of the local area memory storage unit c3 in the motion vector integrating unit a2. Is obtained by referring to the contents of the motion vector memory storage unit b3 in the motion vector extracting means a1, and the set of optical flows included in the motion vector extraction means a1 is defined as the intersection point of the line group set based on the flow vector information. Of the vanishing point (local area FOE) is estimated.
In the moving object existence determination unit d3, the vanishing point of the movement of the moving object (moving object FOE) existing on the extension of each flow when the moving object is included in the projection area of the object that moves independently of the moving observation system and the moving observation system The detection of the object region that moves independently from the mobile observation system is performed by utilizing the difference in position with the vanishing point (background FOE) of the motion due to the self-movement. Specifically, the difference vector between the position vector of the background FOE and the intersection (local region FOE) of the optical flow straight line in each local region is determined. If this length is equal to or greater than a preset threshold, the local region is determined to be independent. It can be detected as a region belonging to an object that moves to. The present invention is not limited to the above-described embodiment.
[0012]
Next, a third embodiment will be described. In the third aspect, the motion vector integrating means a2 employs a method of extracting an edge from an image and performing grouping based on an evaluation criterion of selecting an optical flow included in a region near the edge.
[0013]
Next, an embodiment of claim 5 will be described. In the fifth aspect, the background FOE estimating unit d1 of the moving object detection unit a3 uses the fact that the moving observation system goes straight in the direction of the line of sight, and the vanishing point of the movement of the background region caused by the movement of the moving observation system itself (background FOE) is substituted for the image center position.
[0014]
Next, an embodiment of claim 7 will be described. In the seventh aspect, the motion vector integrating means a2 performs grouping by using together a criterion that the average brightness or the hue value of the start pixel or the end pixel of the optical flow included in each local region is similar. Adopt the method. It is also possible to adopt a method of further grouping using the similarity of average brightness or hue value between local regions selected using the number of optical flows.
[0015]
Next, an eighth embodiment will be described. In the eighth aspect, a distance between a straight line obtained by extending each optical flow included in a local area for detecting a moving object and a vanishing point (background FOE) of the movement due to the self-movement of the moving observation system in the moving object detection means a3. Is calculated, and this distance value is compared with a preset threshold to determine whether the optical flow is caused by an independently moving object.
[0016]
Next, an embodiment of claim 9 will be described. In the ninth aspect, the representative position coordinates of the projection area of the independent moving object, which are respectively detected by the moving object detecting means a3 from the image pairs of the simultaneous points in the plurality of moving observation systems, are associated with each other in stereo. By adding a means to extract 3D position information from the moving observation system of an independent moving object, we estimate the moving speed and judge the risk of collision.
[0017]
Next, a tenth embodiment will be described. According to a tenth aspect, the center of gravity of an independent moving object is used as each representative position coordinate of a projection area of the independent moving object detected by the moving object detecting means a3 from the image pair at the same point in the plurality of moving observation systems in the ninth aspect. Use coordinates.
[0018]
As described above, the present invention can be variously modified and implemented without departing from the spirit thereof.
[0019]
【The invention's effect】
The effects of the present invention will be described below.
[0020]
In the embodiment of the present invention, by providing a motion vector extracting means, a motion vector integrating means, and a moving object detecting means, a series of images taken by one or a plurality of cameras mounted on the moving observation means is provided. Moving objects can be detected.
[0021]
According to an embodiment of the present invention, the motion vector integrating means divides the whole or a part of an image into a plurality of small areas and selects a small area in which the number of detected optical flows is larger than a specified value. By providing a means for grouping based on, it is possible to suppress the occurrence of error flows and perform grouping efficiently.
[0022]
According to an embodiment of the present invention, the motion vector integrating means includes means for extracting an edge from an image and performing grouping based on an evaluation criterion of selecting an optical flow included in a region near the edge. It is possible to limit the area where optical flows are calculated, suppress the occurrence of error flows, and perform grouping efficiently.
[0023]
In the embodiment of the present invention, the moving object detecting means extracts an optical flow group included in the still environment object region in the image, and based on the motion vector information, a vanishing point of the motion due to the self-movement of the mobile observation system. By providing a means for estimating (background FOE), it is possible to separate the vanishing point of the movement of the moving object (moving object FOE) from the background FOE, and the image is captured by one or more cameras mounted on the moving observation means. Moving objects can be detected from a series of images.
[0024]
In the fifth embodiment, the moving object detecting means is provided with means for substituting the vanishing point (background FOE) of the movement due to the self-movement of the moving observation system at the center of the image, thereby improving the processing speed. It becomes possible.
[0025]
According to an embodiment of the present invention, the moving object detecting means estimates a vanishing point (moving object FOE) of the moving object based on an optical flow vector group included in the group of small areas. By comparing the length of the difference vector between the position vector of the vanishing point (background FOE) of the movement due to the self-movement of the mobile observation system with the threshold value, the grouped small area set is included in the independently moving object. It is possible to determine whether or not it is possible.
[0026]
In the embodiment of the present invention, the motion vector integrating means includes means for performing grouping by using together a criterion that the average brightness or the hue value in each small area is similar, thereby improving the accuracy of grouping. Can be improved.
[0027]
According to an eighth aspect of the present invention, in the moving image processing apparatus and method according to the third aspect, in the moving object detecting means, a straight line extending each of the selected optical flows and a motion disappearance due to self-movement of the moving observation system. Calculating a distance to a point (background FOE) and comparing the distance value with a specified threshold to determine whether the optical flow is caused by an independently moving object; Moving objects can be detected from a series of images taken by one or more cameras mounted on the observation means.
[0028]
In the embodiment of the ninth aspect, each of the representative position coordinates of an area belonging to an independently moving object, which is detected from a pair of images at the same point in a plurality of moving observation systems, is associated with the stereo, thereby achieving the independent movement. It is possible to extract 3D position information from the moving observation system of the object.
[0029]
In the tenth embodiment, each representative position coordinate of an area belonging to an independently moving object, which is detected from an image pair at the same point in a plurality of mobile observation systems, is calculated by representing the coordinate of the center of gravity of the area. Can be made easier.
[0030]
By using the present invention, it is possible to efficiently and efficiently realize an obstacle detection function that is indispensable for controlling intelligent mobile robots and autonomous vehicles in dynamically changing indoor and outdoor environments. It is possible to provide a simple moving image processing apparatus and method, and this social and practical effect is enormous.
[Brief description of the drawings]
FIG. 1 shows the relationship between the claims.
FIG. 2 shows the overall configuration of the present invention.
FIG. 3 shows a configuration example of a motion vector extraction unit a1.
FIG. 4 shows a configuration example of a motion vector integration means a2.
FIG. 5 shows a configuration example of a moving object detection unit a3.

Claims (10)

In a moving image processing apparatus and method for recognizing an object or detecting a change in a situation by processing a series of images, each point in an image captured by a camera or a plurality of cameras mounted on a moving observation means is obtained. A motion vector extracting means for extracting a motion vector (optical flow) representing a motion, a grouping of the extracted optical flows based on a predetermined evaluation criterion, a motion vector integrating means, and a grouping by the motion vector integrating means For the set of optical flows obtained, when the set is included in the projection area of the independent moving object, the vanishing point of the motion of the moving object existing on the extension of each flow (moving object FOE: Focus of Expansion) Check that the position of the moving observation system is different from the vanishing point (background FOE) due to the self-movement. A moving image processing apparatus and method comprising moving object detection means for detecting an object that moves independently by using the moving image detection apparatus. 2. The moving image processing apparatus and method according to claim 1, wherein the motion vector integrating means divides the whole or a part of the image into a plurality of small regions, and detects the number of optical flows larger than a specified value. A moving image processing apparatus and method, comprising: means for performing grouping based on an evaluation criterion for selecting a moving image. 2. The moving image processing device and method according to claim 1, wherein the motion vector integrating means extracts edges from the image and performs grouping based on an evaluation criterion of selecting an optical flow included in a region near the edges. A moving image processing apparatus and method, comprising: 2. The moving object detecting means in the moving image processing apparatus and method according to claim 1, wherein an optical flow group included in a still environment object region in the image is extracted, and a moving observation system self-extracts based on the motion vector information. A moving image processing apparatus and method, comprising: means for estimating a vanishing point (background FOE) of movement due to movement. 2. The moving image detecting apparatus and method according to claim 1, further comprising means for using a vanishing point (background FOE) of a movement due to self-movement of the moving observation system at the center of the image. Moving image processing apparatus and method. 3. The moving object detecting means in the moving image processing apparatus and method according to claim 2, wherein a vanishing point (moving object FOE) of the moving object is determined based on an optical flow vector group included in the grouped small areas. By estimating and comparing the length of the difference vector between this position vector and the position vector of the vanishing point (background FOE) of the movement due to the self-movement of the mobile observation system with a threshold value, the grouped small region sets can be independently obtained. A moving image processing apparatus and method comprising: means for determining whether or not a moving object is included. 3. The motion image processing apparatus and method according to claim 2, wherein the motion vector integrating means uses together a criterion that conditions defined by lightness values or conditions defined by hue values in each small area are similar. Moving image processing apparatus and method, comprising: means for performing grouping according to (1). 4. The moving image processing apparatus and method according to claim 3, wherein, in the moving object detecting means, a distance between a straight line extending each of the selected optical flows and a vanishing point (background FOE) of the movement due to self-movement of the moving observation system. And a means for determining whether the optical flow is caused by an independently moving object by comparing the distance value with a prescribed threshold value. 2. The moving image processing apparatus and method according to claim 1, wherein each representative position coordinate of an area belonging to an independently moving object, which is detected from a pair of images at the same time in a plurality of moving observation systems, is associated with a stereo. And a means for extracting three-dimensional position information from the movement observation system of the independent moving object. 10. The moving image processing apparatus and method according to claim 9, wherein the barycentric coordinates are used as the representative position coordinates of the region belonging to the independently moving object detected from the image pairs at the same point in the plurality of moving observation systems. A moving image processing apparatus and method.

Images