JP2000231637A - Image monitor device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve total monitor performance by preventing measurement precision from decreasing. SOLUTION: The image monitor device generates the positional relation among a three-dimensional scene model 11 showing the surface structure of a monitored space 1 and the arrangement of structures, a three-dimensional shape model 13 for an object body to be monitored, a three-dimensional motion model 14 for the object body to be monitored in the monitored space 1, an image pickup means 2, and the monitored space 1, extracts the object body in the monitored space 1 from an input image and traces it while performing mutual conversion between two-dimensional information and three-dimensional information in the monitored space 1 by using mutual converting means 8 and 8' according to those models and positional relation, and calculates and outputs the reliability of the result as a monitor result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for extracting and tracking a target object from an image of a monitored space picked up by an image pickup means such as a television camera, and measuring the presence / absence, number, and moving speed of the target object. The present invention relates to an image monitoring device for detecting an abnormal state.

[0002]

2. Description of the Related Art Conventionally, as a device for measuring and monitoring a moving object from an image in a monitoring space taken by an image pickup means such as a television camera, there is a "moving object measuring device" disclosed in Japanese Patent Application Laid-Open No. Hei 8-94320. This means that after extracting the position of the target object from the image, using the continuity of the motion of the target object in the three-dimensional space, the position and motion in the three-dimensional space obtained by the 2D / 3D conversion means are used. By integrating the predicted position obtained from the position and motion information of the past object, the influence of an error included in the object extraction result is suppressed, and a stable and accurate moving object measuring device is realized.

[0003]

In the above conventional example, when an error is included only in a part of the object extraction result in the three-dimensional space, the error can be suppressed. . However, in consideration of actual applications, in the following cases, there is a problem that errors included in the position and motion of the target object continuously increase, and the accuracy cannot be improved only by the above method.

A case where the resolution of a target object changes depending on the distance from an imaging means in a monitoring space having a depth. When a plurality of target objects are overlapped or the object can be observed only in a partially overlapping state. Rain, snow, fog, smoke, and other factors can reduce visibility,
When the whole or distant target object cannot be observed, or even if it can be observed, the observation accuracy is significantly reduced. When a part of the monitoring area includes a structure that conceals the target object, and it is not always possible to observe the exact shape of the target object in the vicinity.

[0005] In the case where all or a part of the screen is always hidden due to an obstruction such as a flying object or an insect attached to the front surface of the imaging means. When the image is transmitted and the image is distorted due to the line condition or the monitoring image is not shown. In addition, a similar problem occurs in a case where the captured image itself does not reach an image quality that can be processed for some reason. An object of the present invention is to prevent a decrease in measurement accuracy from occurring in these cases and improve overall monitoring performance.

[0006]

According to the present invention, the following models are created in advance for the monitored space, the monitored object, and the imaging means. A three-dimensional scene model showing the rough surface structure and structure layout of the monitored space. Two-dimensional / three-dimensional mutual conversion means determined by the positional relationship between the imaging means and the monitored space and the angle of view of the imaging means. 3D shape (size, etc.) and motion model of the monitored object
(Restriction conditions, speed, acceleration range, etc. of the area where the target object exists).

Next, the image of the monitored space is compared with the three-dimensional scene model, the visibility is determined, and it is determined whether subsequent monitoring target object extraction / tracking is possible. For the visibility, it is determined whether the entire monitored space is good, there is a partially defective part in a distant place, or the whole is defective, and it is determined that the entire visual field is defective. In this case, the following processing is not performed, the visibility determination is repeated, and a visibility failure is output as a monitoring result.

If it is determined that the object is partially defective or overall good, the monitored object is extracted from the image by comparing it with the three-dimensional model of the monitored object. The object extraction position (two-dimensional coordinates) on the image is converted into coordinates in a three-dimensional space by the 2D / 3D mutual conversion means. In the case of a single imaging means, for example, a constraint condition that an object is always in contact with a certain surface is required. Such a constraint condition is also included in the target object motion model. By placing the target object shape model at the object extraction position in the three-dimensional scene model and creating an image of the target object projected on the imaging surface by 2D / 3D mutual conversion means,
The feature amount extracted from the original image is compared with the feature amount on the model, and the comparison result is used as the reliability of object extraction and tracking.

Further, the calculated reliability is evaluated. As a result of the evaluation, if the result is judged to be low in reliability,
Reduce contribution to output results. In addition, when the target object can be seen only small and the resolution is reduced because the target object is located far from the image pickup means, even if the difference is one pixel on the image, the target object is characterized in the three-dimensional space. Since the result is converted into a large difference, it is possible to detect that the result is unreliable. Furthermore, even when a plurality of objects are imaged in an overlapping manner, when the object extraction result is compared with the feature amount when the 3D model is arranged at the extraction position, the feature amount is obtained except when the object is completely hidden. Can be determined based on the presence or absence of the change.

When it is determined that the visibility is partially poor, it is possible to reduce the influence of the extraction result of this portion on the entirety by lowering the reliability of the extraction result at the poor visibility portion. it can. Furthermore, when there is a structure or a tree that hides the monitoring target object in the monitoring area, by adding this to the three-dimensional scene model, the reliability of the extraction result in a specific area on the screen is reduced, The effect of the extraction result in this area on the whole can be reduced.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of the present invention. In the figure, 1 is a monitored space, 2 is an imaging unit, 3 is an A / D conversion unit, 4 is an image memory, 6 is a change feature amount extraction unit, 7 is a target object extraction and tracking unit, 8 is
2D / 3D conversion means, 8 'is 3D / 2D conversion means, 9 is reliability calculation means, and 10 is monitoring result output means. 11 is a three-dimensional scene model, 12 is a target object model, and the target object model 12
Includes a three-dimensional shape model 13 and a three-dimensional motion model 14. Reference numeral 15 denotes a three-dimensional surveillance space model.

An imaging means 2 receives an image of the monitored space 1 and monitors the presence or absence or movement of a target object within its field of view. In this embodiment, a three-dimensional scene model 11 and a target object model 12 are created in advance. The three-dimensional scene model 11 has information on the rough surface structure of the monitored space 1 and the shape, position, and relative positional relationship of the structure. Using the three-dimensional scene model 11, an image of the monitored space 1 viewed from a specific viewpoint can be reconstructed.

FIG. 2 shows an example of a three-dimensional scene model 11 in which a road is a monitored space 1. (A) of the figure shows a model in which the scene is viewed from directly above, where the inclination of the road surface, the curve, the road width,
Models include white line spacing, pedestrian bridge location, width, height, height and spacing of trees planted on the sidewalk, and the size of buildings facing the road. (B) of the figure has shown the state which looked at the model from the side.

The 2D / 3D conversion means includes a 2D / 3D conversion means 8 and a 3D / 2D conversion means 8 '.
The coordinates and size on the two-dimensional image obtained by the imaging means 2 are converted into three-dimensional coordinates and size. In other words, the 2D / 3D conversion means 8 uses the positional relationship between the monitored space 1 and the imaging means 2 and the fixed distance conditions such that the coordinates and size on the screen are always in contact with the surface on which the object is located, depending on the focal length of the lens. Is converted into coordinates and size in the three-dimensional monitored space under Also,
The 3D / 2D conversion means 8 ′ converts the three-dimensional monitoring space model 15 in which the three-dimensional shape model of the target object is arranged on the three-dimensional scene model 11 into a two-dimensional image viewed from the position of the imaging means 2. That is, the 3D / 2D conversion means 8 'projects coordinates and sizes in a three-dimensional space into a two-dimensional image.

FIG. 3 shows an example of the conversion means. The position of the object is converted from two-dimensional coordinates to three-dimensional coordinates using a condition that the target object is constrained to the ground (horizontal plane). FIG. 4 shows an example of the three-dimensional shape model 13. This three-dimensional shape model
Reference numeral 13 denotes a model representing the actual size and shape of the monitoring target object. In each of (a) and (b) of this figure, a vehicle having a width, depth, and height is created by approximating a vehicle to be monitored as a rectangular parallelepiped.

FIG. 5 shows an example of the three-dimensional motion model 14. The motion model 14 is described as a change in the representative position and angle of the target object. In the case of a vehicle running on a road as an example, a motion model is represented by a constraint on a road surface, a motion in the direction of a road lane, a range of speed and acceleration as a vehicle, and the like.

Next, the operation of this embodiment will be described with reference to FIG. First, the image of the monitored space 1 input to the imaging unit 2 is temporarily stored in the image memory 4 via the A / D conversion unit 3, and the change area is extracted by the change feature amount extraction unit 6. The target object extraction / tracking means 7 extracts the position and orientation of the target object on the image while referring to the previously created models 11 to 14 and the tracking results up to the previous time and the reliability. The 2D / 3D conversion means 8 converts the position / posture on the extracted image into a three-dimensional position / posture.

The reliability calculating means 9 calculates the degree of reliability of the extracted object information according to the position of the extracted object on the three-dimensional scene model 11 and the like.
For example, if the object is far from the imaging unit 2, the object is partially hidden by the shadow of the structure, or if a plurality of target objects overlap, the reliability is reduced. The monitoring result output means 10 outputs a monitoring result in consideration of the reliability of the object output from the reliability calculating means 9. Also, this reliability is
It is also fed back to the target object extraction / tracking means 7.

Next, the reliability calculating means according to the second aspect of the present invention will be described. In this embodiment, the reliability calculation means 9 of FIG. 1 uses the position / posture of the target object extracted on the image, and the position where the three-dimensional model of the target object is extracted at the position on the three-dimensional model. The model is projected and transformed into a two-dimensional image viewed from the imaging position and angle,
Next, the feature value of the model when it is ideally observed and the feature value of the target object extracted from the actual monitoring image are compared, and the degree of the difference is regarded as the reliability of the target object at that time. .

Here, an embodiment according to the second aspect of the present invention will be described with reference to FIG. (A) of the figure is a monitoring image by the imaging means 2. From this image, four vehicles surrounded by a square are extracted as target objects by image processing, and the position and orientation of each vehicle are detected. The two-dimensional data extracted from the image is converted by the 2D / 3D conversion means 8 by using a constraint condition on a plane included in the motion model of the target object.
Can be converted to three dimensions. (b) is a three-dimensional shape model 13 of the target object. Here, a rectangular parallelepiped having a width, height, and depth equivalent to that of an ordinary vehicle is used as a vehicle model.

(C) is a three-dimensional scene model of the monitored space 1
11 is shown. In this figure, the three-dimensional shape model 13 of (b) is arranged at the target object extraction position on the three-dimensional scene model 11. In this figure, the three-dimensional model is seen from directly above, and the 3D / 2D
By applying the converting means 8 ', it is possible to obtain a field of view substantially equivalent to the monitoring image. The result is the field of view from the imaging position of the three-dimensional model in (d). The feature amount of the target object calculated from this image is compared with the feature amount extracted from the actual monitoring image, and the reliability of the target object is calculated.

Next, the reliability calculating means according to the third aspect of the present invention will be described. This embodiment addresses a case where a plurality of target objects are close to each other and can be observed only in an overlapping state. First, an object is extracted using an individual target object three-dimensional shape model. However, if a plurality of objects overlap, the object may not be extracted by this method. However, even in such a case, it is possible to detect that there is any object in the monitored space 1.

When the method of overlapping the target objects is determined to some extent, a model in which the three-dimensional shape model 13 is arranged in an overlapping state on the three-dimensional scene model 11
Projection transformation to a two-dimensional image viewed from the posture is performed, and the feature amount when the model is ideally observed is compared with the feature amount extracted from the actual monitoring image, and these are within a certain error range If there is, the target object is extracted in an overlapping state. Further, the reliability is calculated and evaluated based on the degree of overlapping of the objects.

Here, an embodiment according to the third aspect of the present invention will be described with reference to FIG. (A) of the figure is an image in which a target object overlapping a monitoring image is extracted as an object region. The part surrounded by the quadrilateral is the object area. (b) is a three-dimensional target object model obtained by combining a plurality of target objects. (c) shows a model in which the overlapped three-dimensional model of (b) is arranged on a three-dimensional scene model.

By specifying the position and angle of the image pickup means 2 and the focal length (angle of view) of the lens and operating the 3D / 2D conversion means 8 ′, this model can obtain a field of view substantially equivalent to the surveillance image. Can be. (d) is the field of view of the three-dimensional model obtained in this way. Here, it is assumed that the overlapping method and the arrangement method of the three-dimensional model are severally set depending on the target object. (d) the feature amount of the overlapping object calculated from the image and
When the feature amount of the object region extracted from the monitoring image in (a) is within a certain error range, a duplicate object can be extracted.

Next, a description will be given of the reliability calculating means according to the fourth aspect of the present invention. In this embodiment, when the target object is hidden behind a structure or the like and only a part of the target object can be observed, the position / posture at which the target object is extracted is represented by 2D / 3D.
Convert the model in which the three-dimensional shape model is placed at the object extraction position in the three-dimensional scene model by specifying the position and orientation of the imaging means and the focal length (angle of view) of the lens, and convert the model into 3D / 2D conversion means 8 '.
To obtain a field of view almost equivalent to that of the surveillance video, determine how much occlusion is possible in the object on the model, and calculate the reliability of target object extraction based on that・
evaluate.

Here, an embodiment according to the fourth aspect of the present invention will be described with reference to FIG. (A) of the figure shows the monitoring image and the target object extracted on it. The inside of a square box indicates the extracted target object. (b) is a three-dimensional shape model. (c) is a three-dimensional scene model in which the three-dimensional shape model of (b) is arranged at the position of the object subjected to 3D conversion. (d) is (c)
3D / 2D conversion of the three-dimensional scene model according to the position / posture of the imaging means 2 and the focal length (angle of view) of the lens, to obtain a field of view equivalent to the surveillance video. Further, here, the pedestrian bridge over the road becomes an obstacle, and a part of the vehicle passing under the pedestrian bridge is hidden, so that the reliability of object extraction for that part is reduced.

Next, an embodiment of the present invention will be described. FIG. 9 is a block diagram showing the configuration of this embodiment. In the figure, 1 is a monitored space, 2 is an imaging means,
3 is A / D conversion means, 4 is image memory, 5 is visibility judgment means, 6 is change feature amount extraction means, 7 is target object extraction and tracking means, 8 is 2D / 3D conversion means, 8 'is 3D / 2D. Conversion means, 9 indicates reliability calculation means, and 10 indicates monitoring result output means. 11 is a three-dimensional scene model, 12 is a target object model, the target object model 12 is a target object shape model 13, a three-dimensional motion model
14 is included. Reference numeral 15 denotes a three-dimensional surveillance space model.

Next, the operation of this embodiment will be described with reference to FIG. The monitored space 1 is imaged by the imaging means 2, and is temporarily stored in the image memory 4 via the A / D conversion means 3. Next, the visibility determining means 5 processes the monitoring image stored in the image memory 4 to determine whether a sufficient visibility is secured. In addition, the characteristic amount extracting means 6 extracts a characteristic amount of a portion which seems to be an object region. In the target object extraction / tracking means 7, each of the models 11 to
The position and orientation of the target object on the image are extracted by referring to the object extraction results up to 14 and the previous time and the reliability. 2D / 3
The D conversion means 8 converts the position / posture on the extracted image into the position / posture in the three-dimensional scene model.

The reliability calculating means 9 calculates the degree of reliability of the extracted object information from the position of the extracted object on the scene model and the visibility at that time. For example, when the overall visibility is not ensured, an object that is far from the imaging unit 2, an object that is partially hidden by the shadow of a structure, and a case where a plurality of target objects are overlapped are reliable. Sex is reduced. The calculated reliability is sent to the monitoring result output means 10 and is fed back to the target object extracting / tracking means 7. The monitoring result output means 10 outputs monitoring information according to the evaluated reliability.

Next, the visibility determining means according to the sixth aspect of the present invention will be described with reference to FIG. Here, feature points P1 to Pn such as corners of the object are extracted from the background image when no object is present, feature points are extracted from the monitoring image in the same manner, and the visibility is determined from the ratio of the matching feature points. judge. As a feature point, a point at a position where it is not hidden even if the target object exists is selected as much as possible. FIG. 10A shows an example of a monitoring image when visibility is good and feature points extracted therefrom. In the figure, the end of the white line of the center line is detected as a feature point.

FIG. 3B shows an example of an image for viewing reference and feature points. In (a) and (b), the feature points P1 to P5 match, so that it can be determined that the visibility is good. On the other hand, (c) is an example of a monitoring image in the case where the entire screen is hazy due to fog or the like and feature points extracted therefrom. Near feature points P1 and P2 can be detected, but far feature points are not clear and cannot be detected. In such a case, visibility becomes poor at a distant place, and the reliability of the target object extracted at a distant place becomes low.

Next, the visibility determining means according to the seventh aspect of the present invention will be described with reference to FIG. (A) of the figure is an edge intensity distribution image of the monitoring image when visibility is good. A white portion indicates that the edge strength is high. In the figure, the white line of the road and the edge of the border of the tree plant appear. (b) is an edge intensity distribution for visibility reference. In (a) and (b), most of the edge portions overlap except for the vehicle portion, so that it can be determined that the visibility is good. (c) is an example of a monitoring image in the case of poor visibility.
(d) shows the edge intensity distribution of the image of (c). Here, an edge is present, but the intensity is much weaker than that of (b), indicating that the visibility is reduced as a whole. Also in such a case, the reliability of the extracted target object is reduced.

Next, the visibility determining means according to the invention of claim 8 will be described with reference to FIG. (A) of the figure shows a monitor image, some regions in the monitor image which are relatively unaffected by the presence of the target object, and their color distributions. The color distribution is R, G,
It may be expressed in the ratio of B or in H, S, V format.
The color distribution is obtained for each region surrounded by the square, and is compared with the color distribution of the same region of the visibility reference image in (b). When smoke or fog is generated and visibility is reduced as in (c), since the color in the specific area is also whitish compared to the visibility reference image, the reduction in visibility is detected. can do.

Next, the visibility determining means according to the ninth aspect of the present invention will be described with reference to FIG. (A) of the figure is a monitoring image,
(b) is an example of a visibility reference template. In the example shown, the speed sign that appears in the middle of the screen is used as a template,
Perform shading matching. Here, the surveillance image is defined as I (x, y), x = 1,
X, y = 1, 2,..., Y, and the template images are T (u, v), u = 1, 2,..., U, v = 1, 2,. ···, V , x−u, j = 0, 1,..., y−v, and the similarity S (i, j) becomes the maximum value (i, j) Is the matching position, and the value of S (i, j) at that time is the matching degree.
S (i, j) is represented by the following equation.

[0036]

(Equation 1)

When the degree of coincidence is obtained from this equation, it is shown that the degree of coincidence with the template is maximized in a portion surrounded by a square in the monitoring image of (a). On the other hand, if the visibility is reduced as in (c), the matching position is different, or even if the positions match, the result is that the degree of coincidence is low, so that a reduction in visibility can be detected.

[0038]

As described above, according to the present invention, a decrease in measurement accuracy can be prevented even when there are various causes for reducing measurement accuracy. Also, by adding reliability to the monitoring result, the overall monitoring performance is improved.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment according to the first aspect of the present invention.

FIG. 2 is a diagram illustrating an example of the three-dimensional scene model in FIG. 1;

FIG. 3 is a diagram showing a 2D / 3D conversion unit of FIG. 1;

FIG. 4 is a diagram showing an example of the three-dimensional shape model of FIG.

FIG. 5 is a diagram showing an example of the three-dimensional motion model of FIG. 1;

FIG. 6 is a diagram showing an embodiment according to the invention of claim 2;

FIG. 7 is a diagram showing an embodiment according to the third invention.

FIG. 8 is a diagram showing an embodiment according to the invention of claim 4;

FIG. 9 is a block diagram showing a configuration of an embodiment according to the invention of claim 5;

FIG. 10 is a diagram showing an embodiment according to the invention of claim 6;

FIG. 11 is a view showing an embodiment according to the invention of claim 7;

FIG. 12 is a view showing an embodiment according to the invention of claim 8;

FIG. 13 is a diagram showing an embodiment according to the ninth aspect of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Monitored space 2 Imaging means 3 A / D conversion means 4 Image memory 5 Visibility judgment means 6 Change feature amount extraction means 7 Target object extraction and tracking means 8 2D / 3D conversion means 8 '3D / 2D conversion means 9 Reliability calculation Means 10 Monitoring result output means 11 3D scene model 12 Target object model 13 3D shape model 14 3D motion model 15 3D monitoring space model

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Hideaki Uekusa 1st Fuji Town, Hino City, Tokyo Inside FFC Corporation (72) Inventor Yukiyoshi Sakakibara 1st Tanabe Nitta, Kawasaki-ku, Kawasaki City, Kanagawa Prefecture No. 1 Fuji Electric Co., Ltd. F term (reference) 2F065 AA04 AA53 BB05 FF04 JJ03 QQ24 QQ25 QQ31 RR03 5B057 AA19 BA02 BA11 CD14 DA06 DA11 DB03 DC01 DC16 DC22 DC25 5C054 EB05 FC12 FD03 GA04 GB01 HA18

Claims (9)

[Claims] 1. An imaging means for imaging a monitored space, a three-dimensional scene model showing a surface structure of a monitored space and an arrangement of a structure, a three-dimensional shape model of a monitoring target object, and The three-dimensional motion model of the monitored object, the positional relationship between the imaging means and the monitored space, and the two-dimensional information on the input image and the three-dimensional information in the monitored space using the three-dimensional motion model of the target object. Conversion means for mutually converting the object, extraction and tracking means for extracting and tracking the target object in the monitored space from the input image, and calculating the reliability of the extraction and tracking results obtained for each target object An image monitoring apparatus, comprising: a reliability calculation unit; and an output unit that outputs the calculated reliability together with a monitoring result. 2. A reliability calculation means comprising: arranging a three-dimensional shape model at a target object extraction position in a three-dimensional scene model, projecting the three-dimensional shape model into a field of view from an imaging position to obtain feature amounts on a model image, 2. The image monitoring apparatus according to claim 1, wherein reliability is calculated by comparing the characteristic amount with a characteristic amount on a captured image. 3. A method for calculating a three-dimensional image of a target object when it is difficult to individually extract the target object from a three-dimensional scene model and there is a possibility that an overlapping target object may exist. According to the motion model, a plurality of three-dimensional shape models are arranged on a three-dimensional scene model, and the three-dimensional shape models are projected and transformed into a field of view from an imaging position to be a plurality of objects on the model image, and features extracted from these objects. 2. The image monitoring apparatus according to claim 1, wherein the reliability is calculated by comparing the amount and a feature amount extracted from an actual monitoring image. 4. When a target object is hidden by a structure or the like, a three-dimensional shape model is arranged at a target object extraction position in a three-dimensional scene model as reliability calculating means, and the target object is a structure object. 2. The image monitoring apparatus according to claim 1, wherein the reliability is calculated by using a degree hidden by the user. 5. The image monitoring device according to claim 1, wherein
An image monitoring apparatus, comprising: a visibility determination unit that determines whether the visibility of a monitoring image is good. 6. The image monitoring apparatus according to claim 5, wherein a degree of coincidence of feature points on the monitoring image is used as the visibility determining means. 7. The image monitoring apparatus according to claim 5, wherein an edge intensity distribution of a specific area on the monitoring image is used as the visibility determining means. 8. The image monitoring apparatus according to claim 5, wherein a color distribution of a specific area of the monitoring image is used as the visibility determining means. 9. The image monitoring apparatus according to claim 5, wherein the visibility determining means uses a density matching between the input image of the specific area and the model image.