JP3480894B2 - Image monitoring apparatus and image monitoring method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image monitoring apparatus and an image monitoring method for detecting the presence or absence of a person or a flow in a plane area such as a pedestrian crossing or its periphery.

[0002]

2. Description of the Related Art An ultrasonic sensor used for measuring the flow rate of a vehicle or detecting a vehicle waiting for a signal can detect only a hard object that reflects ultrasonic waves well, and is not suitable for detecting humans.
In addition, since the measurement range is narrow, it is difficult to detect the presence or absence of people waiting for a red traffic light around a pedestrian crossing and the flow of pedestrians crossing the green traffic light in a wide range.

Therefore, a method of monitoring a pedestrian using an image input from a TV camera has been conventionally proposed. As a pedestrian monitoring method using images, a time difference method of detecting a moving area as a pedestrian by taking a difference between images input at two different time points, a background image and an input image prepared in advance The difference between the background image and the background image is detected as a pedestrian in the area where the brightness is different from the background image, and the motion information (optical flow) of each point in the image is used to make a uniform linear motion in a certain direction. There is a method of detecting the pedestrian as a pedestrian.

[0004]

However, the above-mentioned method has the following problems. The time difference method is
For example, it cannot be applied to stationary objects such as pedestrians waiting for a signal. In addition, the background subtraction method makes it difficult to update the background image due to changes in sunshine when the monitored area is outdoors, and detects areas of fluctuation in brightness on the road caused by shadow areas of pedestrians, headlights of cars, etc. Since it is included in the area,
These need to be removed. In addition, the method using motion information requires a large calculation cost for optical flow extraction and cannot detect a stationary person.

The present invention has been made in view of the above problems, and an object of the present invention is to prevent erroneous detection of a variation area of road surface brightness caused by a pedestrian's shadow or headlights. An object of the present invention is to provide an image monitoring apparatus and an image monitoring method that do not require background updating and that can detect pedestrians even in a stopped state and that has a low calculation cost.

In the present invention, the observation directions are substantially
Arrange them so that they are orthogonal or face each other and the observation fields of view overlap.
At least two or more image input means provided, and image input
Multiple images acquired by force means at the same time in a three-dimensional space
The means for projecting onto the plane set in and the multiple images projected.
There is a threshold difference between the brightness values of the corresponding pixels when comparing the image data.
Areas larger than the value have different heights with respect to the setting plane
Means for extracting as a region, and the position of the image input means
A point determined from the relative relationship with a specific position on the setting plane
Is the pattern including multiple wedge-shaped shape regions centered on
Using a matching mask consisting of
An image monitoring device comprising means for extracting the existence position of the visual target.
Provide the storage.

Further, in the present invention, the observation directions are mutually
So that the observation fields of view overlap with each other.
Simultaneously with at least two image input means arranged in
A set of multiple images captured every second in a three-dimensional space
Project on a plane and compare the projected image data
If the difference in the brightness value of the corresponding pixel is larger than the threshold value,
Areas are extracted as areas with different heights from the set plane
However, the position of the image input means and a specific position on the setting plane
Multiple wedges centered on a point determined by the relative relationship with the position
Using a matching mask consisting of a pattern containing a bi-shaped area
The location of the monitoring target from the extracted area
An image monitoring method is provided.

[0008]

[0009]

[0010]

With such a configuration, a region in which the road surface brightness fluctuates due to a shadow to be monitored, a headlight, or the like is not erroneously detected, background updating is unnecessary, and is stopped. It is possible to efficiently detect pedestrians including their states.

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

DETAILED DESCRIPTION OF THE INVENTION The details of the present invention will be described below with reference to the drawings. FIG. 1 shows a schematic configuration of an image monitoring apparatus according to the present embodiment, in which 1: TV camera i (1 ≦ i ≦
n), 2: image memory i (1 ≦ i ≦ n), 3: image projection unit i (1 ≦ i ≦ n), 4: selection unit, 5: comparison unit j (1 ≦ i)
j ≦ m), 6: extraction unit j (1 ≦ j ≦ m), 7: integration unit.

FIG. 9 shows the flow of processing relating to the image monitoring apparatus and method. Processing is performed in the order of image input, image projection, projection data selection, projection data comparison, monitoring target position extraction, and integration.

The image input from the TV camera i (1≤i≤n) is stored in the image memory i (1≤i≤n). The image projection unit i (1 ≦ i ≦ n) has an image memory i (1 ≦ i ≦ n).
The plane P: ax based on the coordinate system Cxyz preset in the three-dimensional space for each image stored in n)
Project onto + by + cz = 0.

In this plane equation, after setting the coordinate system Cxyz in the monitoring area by the surveying instrument, the position of the sample point in this coordinate system is measured by the surveying instrument, and the plane including the sample point is obtained by the least square method. It is obtained by calculating the formula. When the monitoring area is an intersection area including a pedestrian crossing, a plurality of sample points are provided on the road, and the positions of these sample points are measured by a measuring machine. In the case of indoor passages, sample points will be provided on the floor of the passage.

A projection method suitable for the image monitoring apparatus of the present invention will be described with reference to FIG. 21: Coordinate system C of monitoring area
The origin of xyz is O, 22: the image center of the TV camera i (1≤i≤n) is Oi, the focus position of the TV camera i is Ci, and the point qi on the image plane is projected onto the plane P by 23. Qi is the coordinate system with the origin at the center of the image, xi, -yi,
A vector whose length corresponds to the pixel size in the x-axis direction of the image plane on the xi axis starting from Oi is nix, and length on the y-axis starting from Oi corresponds to the pixel size in the y-axis direction of the image plane. Let the vector be niy. The position of qi on the image plane is (kix, kiy). The position at Cxyz of the point Qi obtained by projecting the point qi on the plane P is obtained by the following formula. qi
The following relation holds for.

[0023]

[Equation 1] For the TV camera i (1 ≦ i ≦ n), the following parameters can be obtained in advance by calibration using a calibration plate and a surveying instrument.

[0024]

[Equation 2] At this time, the equation of the straight line connecting Qi and qi is

[0025]

[Equation 3] Expressed as This x, y, z is expressed by the plane formula P: ax + b
The position (Qix, Qiy, Qiz) of the projection point Qi of qi on P can be calculated by substituting for y + cz = 0 and calculating t. The image projection unit i (1 ≦ i ≦ n) is a TV
Image memory i in which images input from camera i are stored
The above processing is performed for all pixels qi = (kix, kiy) in the pixel, and the projection positions (Qix, Qiy, Qiz) and q of each qi are processed.
The projection data consisting of the brightness value of i is output to the selection unit.

The selecting section selects two sets of projection data for n pieces of projection data input from the image projecting section i (1≤i≤n), and the comparing section j (1≤j≤m). Send to. Which output of the image projection units is combined with each other is set in advance and stored in the selection unit. For example, at the intersection shown in FIG. 3, a person crossing the pedestrian crossing and a person waiting for a signal in front of the pedestrian crossing are monitored using eight TV cameras installed in the directions shown in FIG. 3, respectively. Consider the case.

Since each camera has an observation field of view as shown in FIG. 4 (in FIG. 4, only the observation fields of view of the TV cameras 2, 3, 5, and 8 are schematically shown), the region T1 is monitored. Therefore, the projection data from the image projection unit 3 and the image projection unit 8 which handle the images of the TV camera 3 and the TV camera 8 are stored in the selection unit so as to be output to the comparison unit 1. Similarly, in order to monitor the area T2, the TV camera 2 and the TV camera 3
The projection data from the image projecting unit 2 and the image projecting unit 3 that handle the image are stored in the selecting unit so as to be output to the comparing unit 2. Further, in order to monitor the area T3, the image projection unit 2 and the image projection unit 5 that handle the images of the TV camera 2 and the TV camera 5 are monitored.
The projection data from are stored in the selection unit so as to be output to the comparison unit 3.

As described above, assuming that eight monitoring areas shown in FIG. 5 are set in the intersection area shown in FIG. 3, the area T
In order to monitor 4 to T8, a set of projection data output from the image projecting unit 4 and the image projecting unit 5 to the comparing unit 4, the comparing unit 5
A pair of projection data output from the image projection unit 4 and the image projection unit 7, a pair of projection data output from the image projection unit 6 and the image projection unit 7 to the comparison unit 6, and an image projection unit 1 to the comparison unit 7. The selection unit stores the projection data sets output from the image projection unit 6 and the comparison unit 8 inputs the projection data sets output from the image projection unit 1 and the image projection unit 8, respectively. In this way, the selection unit selects the combination of the image projection unit i (1 ≦ i ≦ n) connected to each comparison unit j (1 ≦ j ≦ m) according to the combination stored in advance, and outputs the projection data. Output.

The comparison unit j (1≤j≤m) extracts regions R having different lightness at the same position on the plane P from the two sets of projection data input from the selection unit. Comparison part j
Projection data K1 from the image projection unit i1 to (1≤j≤m)
Consider the case where the projection data K2 from the image projection unit i2 is input. K1 and K2 are respectively composed of the following projection coordinate value and brightness value data.

[0030]

[Equation 4] Here, Q1 and Q2 are projection coordinate values, and I1 and I2 are brightness values of projected pixels.

With respect to the projected coordinate values included in K1 and K2, the maximum value Xmax and the minimum value Xmin of the x coordinate value are obtained. Similarly, the maximum y-coordinate value Ymax and the minimum y-coordinate value Ymin
Ask for. Then, an area memory consisting of an M × N two-dimensional array is prepared, Xmin to Xmax are sampled into M equal parts, and Ymin to Ymax are sampled into N equal parts, and each element included in the projection data K1 and K2 is sampled. The brightness value of is written to the corresponding address on this area memory. Each element (i, j) of the area memory is composed of three cells which store the brightness value of K1, the brightness value of K2 and the flag value (0 or 1). The x coordinate value Q1ix and the y coordinate value Q1iy of each element included in K1 are

[0032]

[Equation 5] If it is within the range, the brightness value of the element is written in the cell that stores the brightness value of K1 of (i, j) on the area memory. Similarly, the x coordinate value Q2ix and y coordinate value Q2iy of each element included in K2 are

[0033]

[Equation 6] If it is within the range, the brightness value of the element is written in the cell that stores the brightness value of K2 of (i, j) on the area memory.

After writing all the elements of K1 and K2 to the area memory, the area memory is scanned to find an element (i, j) in which data is not written in the cell storing the brightness value of K1 or K2. Then, those elements are linearly interpolated using the surrounding data. At this time, in each (i, j), interpolation for K1 is performed between K1 and that for K2 is performed between K2.

Then, in each element (i, j) of the area memory, the absolute value Dif (i, j) of the difference between the brightness value of K1 and the brightness value of K2 stored in the cell is calculated. And Dif (i,
If j) is larger than the preset threshold value, 1 is stored in the cell that stores the flag. If it is smaller than the set threshold value, 0 is stored in the cell that stores the flag. As shown in FIG. 6, a point S ′ on the set plane
Since the projection points from the TV camera are the same, Dif (i,
j) becomes smaller and the flag value becomes 0. In contrast,
Since the point S on the pedestrian has different projection positions from the TV camera, D f (i, j) becomes large at the projection point Vi from the TV camera 1 and the projection point Ui from the TV camera 2, and the flag The value is 1.

Thus, when a pedestrian is present,
An area having a flag value of 1 spreads in a state of falling in the opposite direction with respect to the position of the camera. This shape on the area memory usually shows a wedge-shaped shape as shown in the top view of FIG.
After setting flags for all elements (i, j) of the area memory, the area memory is output to the extraction unit j (1 ≦ j ≦ m). The absolute value of the difference between the brightness value of K1 and the brightness value of K2 is K1
A value normalized by the average value of the luminance value of K2 and the luminance value of K2 may be used instead of the absolute value of the difference. The value stored in the flag is not limited to 0 and 1, and any value that can distinguish the states may be used.

The extraction part j (1 ≦ j ≦ m) is compared with the comparison part j (1 ≦ j ≦ m).
The area memory input from (j ≦ m) is scanned to find the elements (ib, jb) in which the two wedge-shaped shapes having the flag value 1 respectively extend in the fixed direction. Then, this position is output to the integration unit as the pedestrian existing position. The direction in which the wedge shape extends on the area memory is the same as the direction of the vector extended from the position of the TV camera to the position represented by each element (i, j) of the area memory in the reference coordinate system Cxyz. Is.

For example, in FIG. 7, the area memory (i,
j) represents the position of the point P at Cxyz, and when a pedestrian is present at this point P, the wedge-shaped area for the TV camera 1 is in the direction d1 and for the TV camera 2. The wedge-shaped shaped region of is extended in the d2 direction. Cxyz
The position of the TV camera and each element (i,
Since the position of j) is already known, the direction in which the wedge-shaped shape region extends is obtained for each (i, j), and the matching mask is formed by extending the wedge-shaped shape region in that direction.

When the wedge-shaped shaped region extends in the direction shown in FIG. 7, a matching mask as shown in FIG. 8 is generated and its center is applied to the element (i, j). If a pedestrian exists in the element (i, j) portion of the area memory, many elements having the flag value 1 appear in the shaded portion of the mask. Therefore, the sum of the flag values corresponding to the shaded portions of the mask is obtained, and when this sum is greater than or equal to a certain value, the position in Cxyz corresponding to the element (i, j) is set as the pedestrian's existing position.

The integration unit integrates the pedestrian position information obtained from each extraction unit j (1 ≦ j ≦ m), and determines the position, number, moving direction, and direction of the pedestrian in all monitoring areas in the coordinate system Cxyz. Find the moving speed respectively. From each extraction unit j, the coordinate system C
Since the position of the pedestrian based on xyz is output,
The position and number of pedestrians can be obtained directly. In addition, the moving direction and the moving speed of the pedestrian are obtained by associating the pedestrians closest to the previous measurement position.

[0041]

As described in detail above, according to the present invention,
Since multiple TV cameras are used to detect objects of different heights with respect to a set plane and determine the position of the pedestrian, the brightness fluctuation area on the road surface caused by the shadow of the pedestrian, headlights, etc. is erroneously detected. Never detected. Further, since the data such as the background image is not used, there is no difficult problem of updating the background data due to the change of the sunshine. Furthermore, not only moving pedestrians but also stationary pedestrians can be detected, and since the main processing is the projection and difference of the image, the calculation cost is low.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a basic configuration of the present invention.

FIG. 2 is a diagram illustrating an image projection method.

FIG. 3 is a diagram illustrating a monitoring area and a camera arrangement.

FIG. 4 is a diagram illustrating a visual field range of a TV camera.

FIG. 5 is a diagram illustrating a monitoring area in charge of a set of TV cameras.

FIG. 6 is a diagram illustrating a projection position.

FIG. 7 is a diagram illustrating a direction of a wedge-shaped shaped area.

FIG. 8 is a diagram illustrating a matching mask.

FIG. 9 is a diagram illustrating a processing flow of the present invention.

[Explanation of symbols]

1: TV camera i (1 ≦ i ≦ n) 2: Image memory i (1 ≦ i ≦ n) 3: Image projection unit i (1 ≦ i ≦ n) 4: Selector 5: Comparison part j (1 ≦ j ≦ m) 6: Extraction part j (1 ≦ j ≦ m) 7: Integration Department

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-9-81738 (JP, A) JP-A-5-143156 (JP, A) Kazunori Onoda et al. Road area detection and information using planar projection stereo method IPSJ Research Report (95-CV-93), Japan, Information Processing Society, March 23, 1995, Vol. 95, No. 34 (CV93-7), 61-68 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 315 G06T 1/00 330 G01C 3/06 G05D 1/00 G06T 7/20 JISST file (JOIS)

Claims (2)

(57) [Claims] 1. Observation directions are substantially orthogonal to each other or face each other.
At least 2 with overlapping observation fields
Acquired at the same time with one or more image input means and image input means
Means for projecting a plurality of images in a plane which is set in a three-dimensional space, pair by comparing the plurality of image data projected
Means for extracting an area in which the difference in the luminance value of the corresponding pixel is larger than a threshold value as an area having a different height from the setting plane, and the position of the image input means and the specification on the setting plane.
Multiple points centered on the point determined by the relative relationship with the position of
Uses a matching mask consisting of a pattern that contains a rust-shaped area
There, the image monitoring apparatus and means for extracting the location of the monitoring target from the extracted region. 2. The observation directions are substantially orthogonal to each other or face each other.
At least 2 with overlapping observation fields
A plurality of images acquired at the same time by one or more image input means are projected on a plane set in a three-dimensional space, and the plurality of projected image data are compared to determine the luminance value of the corresponding pixel.
A region having a difference larger than a threshold value is extracted as a region having a different height with respect to the set plane , and the position of the image input means is extracted.
And the specific position on the setting plane
A pattern containing multiple wedge-shaped shape regions centered on a point
An image monitoring method is characterized in that the existing position of the monitoring target is extracted from the extracted area using a matching mask consisting of .