JP2001076156A - Device for monitoring image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image processor capable of correcting a template position so that the center of a trace object can reliably be the center of a template area all the time even when either the reliability of template search or the reliability of a correcting means is low. SOLUTION: This device is provided with an image difference extracting means 7 extracting a moving object by performing differential processing between continuous images, a template searching means 5, a tracing mode selecting means 9 which evaluates the reliabilities of the both means and selects any among a 1st tracing mode in which results of the means 5 are used in accordance with the reliability of the tracing mode, a 2nd tracing mode in which the results of the means 7 are used and a 3rd tracing mode in which the results of the both means are used, and a tracing point deciding means 10 deciding a tracing point in accordance with each of the tracing modes.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image for automatically monitoring the intrusion of a person or a suspicious object into plant equipment, a public system, a building, a dwelling, or the like using an image obtained from a camera and automatically tracking the image. It relates to a monitoring device.

[0002]

[Prior Art] Crime prevention in various places such as important facilities,
A surveillance camera is installed for the purpose of disaster prevention, and a person monitors the image, and a surveillance operation for operating the camera is performed.
Then, for the purpose of labor saving of this monitoring work, its automation has been attempted. For example, pages 13 to 18 of the 5th Image Sensing Symposium Proceedings (1999/06)
Page, Ito / Ueda "Autonomous moving object tracking surveillance camera using camera with built-in image recognition hardware" has a problem in extracting the target by performing so-called template matching processing on the image obtained from the camera and automatically tracking it. Points and how to deal with them are shown.

[0003] First, a method of tracking a moving object by a known so-called template matching will be described. 9 to 12 are screen diagrams illustrating a moving object tracking process based on template matching. Needless to say, the output image from the surveillance camera is composed of continuous images (hereinafter referred to as frames). In the following description, the image obtained at the present time is referred to as “the current frame image” 1
6. The image one frame before the image 16 of the current frame is referred to as a “previous frame image” 15. In FIG.
Is a template cut out from the image 15 of the previous frame by known difference processing or the like, and 68 is an explanatory diagram of an operation of calculating template matching on the input image of the image 16 of the current frame. In FIG. 10, reference numeral 69 denotes a planar result score in which the degree of matching of template matching represented in a plane is represented by a color density for convenience of description;
Is a three-dimensional result score in which the two-dimensional result score 69 of the template matching is displayed three-dimensionally. FIG. 11 is an explanatory diagram of an operation of performing tracking while sequentially updating the template between successive image frames.

Next, a template matching operation will be described with reference to FIGS. First, a template 6 cut out from the image 15 of the previous frame centering on the previous tracking point
7 and the image small area having the highest similarity in the image 16 of the current frame is searched. As shown by 68 in FIG. 9, the template 67 is superimposed on the image 16 of the current frame while shifting it by one pixel, and the difference of the luminance value is obtained for each point of the image small area of the same size as the template 67. The absolute values of The flat result score 69 indicating the degree of matching obtained in this way is 7 when displayed three-dimensionally.
It will be like 0. Here, the result score 69 is always zero or a positive value, and is zero when completely matched, and the score value increases as the matching degree decreases. Therefore, it should be noted that in the three-dimensional display 70 of the result score, the Z axis is shown with a negative value on the upper side to facilitate understanding. Hereinafter, in this specification, the three-dimensional display is expressed such that the upper side of the Z axis becomes negative. A tracking point is determined as a point where the result score is maximum is a point having the highest matching degree. When tracking is performed to sequentially check the degree of coincidence while sequentially updating the template between successive images, the result is as shown by 71 in FIG. The template 67 is replaced with a small area having the highest similarity every frame as it is 67, 67 ′, 67 ″.
Will be updated as follows.

In the above-mentioned proceedings, problems when using the conventional template matching technique as described above are as follows. (1) When the direction of a deep target object (for example, a vehicle) changes, the template obtained is Gradually deviates from the center of the target object, and eventually comes off the target. (2) When another object crosses the target object, the target is replaced. The following two points are shown, and a countermeasure against such a phenomenon is disclosed.

[0006] Although the structure of the apparatus is not shown in the above-mentioned proceedings, the contents described (for the positional displacement between the target object and the template, the edge cumulative distribution calculating means is used.
In addition, it is presumed that the expected configuration of the apparatus will be as shown in FIG. 12 based on the comparison of the degree of coincidence between a plurality of templates. In the figure, 1 is a monitoring camera, 2 is an image input unit,
3 is an image storage means, 4 is a template setting means, 5 is a template search means, 6 is a closest matching point extracting means, 52 is an edge cumulative distribution calculating means, 8 is a center of gravity extracting means, 10 is a tracking point determining means, and 11 is a tracking point. Point information storage means, 12 is a camera control determination means, and 13 is a pan head control means.

Next, the operation of FIG. 12 will be described. The monitoring camera 1 captures an image of a monitoring target. The image input unit 2 inputs an image captured by the camera 1 into a memory (not shown) that can be randomly accessed, and sets the image as a current frame image 16. The image storage means 3 stores the image 16 of the current frame.
Is transmitted to a randomly accessible memory with a delay corresponding to the set number of frame intervals, and is set as the image 15 of the previous frame.

The template setting means 4 reads a small image area around the previous tracking point from the image 15 of the previous frame as a template 67 and registers it in a template memory (not shown). The template searching means 5 calculates the similarity between each of the image regions centered on each point in the image 16 of the current frame and the above-described template 67, and randomly calculates a result score 69 representing the similarity (or the degree of coincidence). Write to accessible memory. The closest matching point extracting means 6 extracts a point having the highest matching degree around the previous tracking point from among the result scores 69 obtained from the template search result. Hereinafter, for convenience of explanation, this processing is referred to as template matching.

Next, the edge cumulative distribution calculating means 52 calculates X, X around the tracking point obtained by the template matching.
The cumulative distribution of the edge, which is the luminance differential value in each of the Y directions, is calculated, and the center-of-gravity extracting means 8 obtains the center of gravity of the tracking target using the edge cumulative distribution in the X, Y directions. In this case, since the center of gravity of the target is obtained by using the edge cumulative distribution without using the template matching, there is less possibility that the position of the center of gravity is erroneously obtained even when the direction of the target changes.

The tracking point determining means 10 corrects the tracking point obtained by template matching using the center of gravity of the tracking target obtained by using the edge cumulative distribution, and determines a new tracking point.

[0011] The tracking point information storage means 11 temporarily stores the determined latest tracking point information. This tracking point information is input to the template setting means 4 and the best matching point extracting means 6 in the next frame.

The camera control determining means 12 executes camera displacement control (camera swing operation, zoom control, etc.) based on the latest tracking point information.

The camera head control means 13 controls the direction and the moving speed of the camera camera head based on the result of the judgment by the camera control judgment means 12.

As described above, the target is tracked while sequentially updating the template 67 by the template matching.
It is intended to enable stable tracking even when the tracking target is deformed. In addition, by correcting the tracking point obtained by template matching using the cumulative distribution of luminance obtained from the surroundings, the center of the tracking target is always set as the center of the template, and stable tracking is performed. It is.

To explain the problem of the image monitoring apparatus using the conventional template matching technique described above, FIG. 13 shows a background portion other than the tracking target inside the template 67, for example, a white line drawn on the ground. A case where the contrast is higher than that of the tracking target and the bias of the luminance distribution is large will be described.

In FIG. 13, reference numerals 72 to 79 denote frames that change in the order of the reference numerals, and it is assumed that the images change in the order of the reference numbers. Reference numerals 721 to 791 denote templates extracted in the frames 72 to 79, respectively. 80 is each frame 72-79
A background having a very high contrast, such as a white line on the floor surface or a dividing line between the floor and the wall, is displayed in the image. Since the contrast of the background 80 having a very high contrast is higher than the contrast of the object to be monitored (here, a human image), in the first template 721, the center of luminance moves slightly outward from the center of the human image. I do. Then, the same occurs again in the template 731 so that the luminance center moves further outward. By repeating such an operation, the template 791 finally falls out of the monitoring target.

Even when the tracking target itself has a biased luminance distribution, for example, the upper body has a color similar to the background and the lower body has a color contrasting with the background, the luminance is changed every time the template is updated. Tends to move little by little and finally come off.

[0018]

As described above, although the conventional image monitoring apparatus is considerably improved compared to the previous image monitoring apparatus, the tracking target itself has a bias in the luminance distribution. In some cases, or when there is a background part whose contrast is significantly higher than that of the tracking target and the bias of the luminance distribution is large, the area to be cut out as a template is shifted from the center of the target, and the tracking processing is performed while updating the template sequentially. If it continues, a phenomenon that the tracking target is eventually removed often occurs.

That is, the conventional image monitoring apparatus repeatedly performs a method of always finding a tracking point by searching for a template and correcting the luminance using the cumulative distribution of the X and Y axes of the luminance around the tracking point. X of the luminance around the tracking point which is the reliability of the search or the correction means.
When one of the reliability of the cumulative distribution on the axis and the Y axis is low, there is a problem that an error associated with the lower reliability is accumulated and eventually an erroneous tracking point is tracked.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. Even when one of the reliability of the template search and the reliability of the correction means is low, the center of the tracking target is always stabilized. It is an object of the present invention to obtain an image processing apparatus that can correct the position of a template so that the position of the template is located at the center of the area of the template.

[0021]

According to the present invention, there is provided an image monitoring apparatus comprising: a camera capable of continuously capturing and displacing an image; an image processing unit for processing the image to find a moving object; An image monitoring device having camera control means for displacing a camera in accordance with movement, wherein the image processing unit includes a template setting means for storing, as a template, an image region including a moving object in an image of a previous frame of a continuous image. A template search means for searching for an image area similar to the template in the image of the current frame and updating and storing the updated image as a new template instead of the template, and an image for extracting a moving object by performing a difference process between successive images The reliability of the search result of the difference extracting means, the search result of the template searching means, and the difference result of the image difference extracting means is evaluated. One of a first tracking mode using the result of the template searching means, a second tracking mode using the result of the image difference extracting means, and a third tracking mode using both results according to the reliability. And a tracking point determining means for determining a tracking point in accordance with the selected tracking mode.

Further, the camera control means controls the camera such that the tracking point determined by the tracking point determination means is located in a camera control unnecessary area provided in the image.

When the tracking mode selecting means selects the third tracking mode, the tracking point determining means sets the tracking point obtained by the template searching means at the position of the tracking point obtained from the result of the image difference extracting means. A position to which a vector of half the movement vector is added is set as a new tracking point.

Further, the tracking mode selecting means, if the screen occupancy of the edge portion of the difference output image of the image difference extracting means is larger than a predetermined level, the image obtained during the displacement operation of the camera. And the first
The tracking mode is selected.

The tracking mode selection means selects the second tracking mode when the score of the template search means is larger than a predetermined level, assuming that the reliability of the template search is low.

Further, the camera control means sets a camera control required area in a range in contact with the outside of the camera screen,
When the tracking point determined by the tracking point determination means is within the camera control required area, the camera is controlled so that the tracking point is substantially at the center of the screen.

Further, when the tracking mode selecting means loses sight of the tracking object, the tracking mode selecting means focuses on the direction of the tracking object immediately before losing it.
The camera is provided with a discovery mode in which the camera is displaced left and right or up and down at a predetermined angular width.

[0028]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Before describing the embodiments of the present invention, various states of images obtained by the image monitoring apparatus will be described with reference to FIG. In the figure, 58 is a previously captured background image (corresponding to the image 15 of the previous frame), 59 is an input image (corresponding to the image 16 of the current frame), and 60 is a difference between the background image 58 and the input image 59. The extracted intruding objects, 61, 62, and 63 are scenes in which the discovered intruding object is being tracked with the camera fixed, and 64, 65, and 66 are cameras 1 (or pan head) so that the tracking target falls within the field of view. This is a scene in which the player is tracking while turning (displacing). As described above, the change in the screen may be caused by the movement of the target object to be monitored, may be caused by a change in the shooting conditions of the camera 1 such as displacement and zoom control, or may occur simultaneously. The image monitoring device needs to operate stably in any of these cases.

Embodiment 1 The image monitoring device of the present invention will be described with reference to FIG. 1. FIG. 1 is a configuration diagram of the image monitoring device of the present invention. In the drawings, the same or corresponding parts as those in the related art are denoted by the same reference numerals, and detailed description thereof will be omitted. In FIG. 1, reference numeral 1 denotes a surveillance television camera (hereinafter simply referred to as a camera) having a swinging or zooming function capable of continuously capturing and displacing an image, and 2 an input of an output image of the camera 1. Image input means 3 for the image 16 of the current frame, 3 is image storage means for delaying the image 16 of the current frame by a time corresponding to the set number of frames and sending it as the image 15 of the previous frame, and 4 is the image 15 of the previous frame. A template setting means for registering an image small area around a tracking point as a template 17; a template searching means 5 for calculating the similarity between the template 17 and each image area in the image 16 of the current frame; This is the best matching point extracting means for extracting the point with the highest matching degree around the previous tracking point from the output result of the searching means 5. The above is the same as that of the conventional FIG.

Reference numeral 7 denotes an image difference extracting means for calculating a difference between the image 16 of the current frame and the image 15 of the previous frame. 8 denotes a moment around the previous tracking point for the difference result obtained by the image difference extracting means 7. A center-of-gravity extracting means 9 for calculating the position of the center of gravity by calculation, a template search result obtained by the template searching means 5, an image difference result obtained by the image difference extracting means 7, and a tracking point information storage means 11 are stored. The tracking mode selecting means 10 for selecting the tracking mode from the information of the previous tracking point which has been set, sets the result of the best matching point extracting means 6 and the center of gravity extracting means 8 to the tracking mode based on the result of the tracking mode selecting means 9. It is a tracking point determining means for determining the latest tracking point by using it alone or combining (using a specific method of fusion will be described later). The configuration from the image input unit 2 to the tracking point determination unit 10 constitutes an image processing unit. Reference numeral 11 denotes tracking point information storage means for temporarily storing the determined latest tracking point information, 12 denotes camera control means for determining a camera control method based on the latest tracking point information, and 13 denotes camera control means 1.
A pan head control unit 14 for controlling the camera pan head based on the result of 2 and a zoom unit 14 for controlling the zoom ratio based on the result of the camera control unit 12.

Next, the operation of FIG. 1 will be described with reference to FIG. In FIG. 2, 15 is the image of the previous frame, 16 is the image of the current frame, 17 is a template cut out from the image 15 of the previous frame centering on the previous tracking point position, 18
Is a tracking point extracted by changing the position of the template 17 on the image 16 of the current frame and finding a matching position (that is, extracted by template matching), and 19 is a tracking point of the previous tracking point (not shown) and the template. A motion vector between the tracking point 18 extracted by the matching, 20 indicates a difference result between the image 15 of the previous frame and the image 16 of the current frame, and 21 indicates a center of gravity calculated from the difference result 20. Here, it should be noted that the tracking point 18 is located at the center of the new template, but the center of gravity 21 is located at an intermediate point between the old template and the new template.

The camera 1 is mounted on a camera platform (not shown) and captures an image of the direction of the monitored object. The image input unit 2 inputs an image captured by the camera 1 into a random accessible memory (not shown) to obtain an image 16 of the current frame. The image storage means 3 delays the image 16 of the current frame by a time corresponding to the set number of frame intervals and sends it to a memory that can be randomly accessed, to obtain the image 15 of the previous frame.

The template setting means 4 reads a small image area around the previous tracking point (not shown, but near the human image in the image 15 of the previous frame) from the image 15 of the previous frame as a template 17 and reads it into the template memory. register. The template search means 5 calculates the similarity between each of the image areas centered on each point in the image 16 of the current frame and the above-described template 17, and writes the similarity as a result score representing the similarity in a randomly accessible memory. The closest matching point extracting means 6 extracts a point having the highest matching degree around the previous tracking point as the tracking point 18 from the result scores obtained as a result of the template searching means 5.

Next, the image difference extracting means 7 calculates a difference 20 between the image 16 of the current frame and the image 15 of the previous frame. The center-of-gravity extracting means 8 obtains a center-of-gravity position 21 by calculating a moment around the previous tracking point for the difference result 20 obtained by the image difference extracting means 7. Hereinafter, this processing is referred to as extraction of the center of gravity of the difference between frames.

Then, the tracking mode selection means 9 calculates the template search result obtained by the template search means 5 described above, the image difference result obtained by the image difference extraction means 7 described above, and the information of the previous tracking point. One of the following three tracking modes is selected (a specific selection method will be described later). That is, the first tracking mode “the tracking point obtained by template matching is used as it is.” The second tracking mode “the tracking point (centroid point obtained by difference processing) is used as it is.” The third tracking mode “template matching. The tracking point obtained by the above is used in combination with the tracking point (centroid point) obtained by the difference processing. ”The tracking point determining means 10 extracts the result of the best match point extracting means 6 based on the result of the tracking mode selecting means 9. The latest tracking point is determined by fusing the extraction result 18 with the extraction result 21 of the center-of-gravity extraction means 8 (a specific fusing method will be described in the following embodiments).

The tracking point information storage means 11 temporarily stores the determined latest tracking point information. This tracking point information is input to the template setting means 4, the best matching point extracting means 6, the center of gravity extracting means 8, and the tracking mode selecting means 9 in the next frame.

The camera control means 12 determines a camera control method based on the latest tracking point information. Head control means 1
Reference numeral 3 controls the camera platform based on the result of the camera control means 12. The zoom unit 14 controls the zoom ratio based on the result of the camera control unit 12. In the above description, the movement of the pan head swinging is sometimes referred to as the displacement of the camera, but this has the same meaning.

Embodiment 2 FIG. 3 is a diagram of an image for explaining the operation of the image monitoring apparatus according to the second embodiment of the present invention, particularly the details of the third tracking mode described above. In FIG. 3, 15 is the image of the previous frame, 16 is the image of the current frame, 17 is a template cut out from the image of the previous frame centering on the previous tracking point position, 18 is the tracking point extracted by template matching, and 19 is the previous tracking point. Tracking point and tracking point 18 extracted by template matching
20 is the difference result between the image 15 of the previous frame and the image 16 of the current frame, 21 is the position of the center of gravity obtained from the difference result 20 between the image 15 of the previous frame and the image 16 of the current frame, 22 Is a vector obtained by halving the size of the movement vector 19, and 23 is a vector 22 obtained by halving the movement vector 19 at the barycentric position 21 obtained from the difference result 20 between the image 15 of the previous frame and the image 16 of the current frame.
Is added.

Next, the operation will be described. From the image 15 of the previous frame, a template 17 cut out centering on the position of the previous tracking point is extracted from the image 16 of the current frame by template matching to obtain a tracking point 18. on the other hand,
In the difference result 20 between the image 15 of the previous frame and the image 16 of the current frame, moment calculation is performed around the tracking point 18 extracted by template matching, and the position of the center of gravity 21 is obtained. A vector 22 obtained by halving the movement vector 19 between the previous tracking point and the tracking point 18 extracted by the template matching is calculated as the center of gravity 2 obtained from the difference between the image 15 of the previous frame and the image 16 of the current frame.
A tracking point 23 is obtained in addition to 1.

Here, the centroid extraction is a method of calculating the zero-order and first-order moments for the binarized image difference result 20 with a certain threshold value within a limited area around the tracking point. Good.

Alternatively, the center of gravity may be extracted by calculating the zero-order and first-order moments using the absolute value of the image difference result as a weighting value within a limited area around the tracking point.

The tracking point 18 obtained by the template matching, the motion vector 19 obtained by the template matching, and the tracking point obtained by the center of gravity 21 of the inter-frame difference are compared. If the difference between the components is less than a certain threshold, the latter is used as it is as the corrected tracking point. If the difference between the X and Y components of the two is larger than a threshold, each difference is limited by the threshold. Then, a method in which a threshold value of the difference is added to the tracking point of the template matching search result may be used as the corrected tracking point.

Embodiment 3 4 and 5 are diagrams for explaining the operation of the image monitoring device according to the third embodiment of the present invention.
In FIG. 4, reference numeral 24 denotes an image of the previous frame when there is no camera shake (the camera is not displaced), 25 denotes an image of the current frame when there is no camera shake, and 26 denotes an image of the previous frame when there is no camera shake. 24 and current frame image 2
The result 26A of the difference from the data 5 indicates data obtained by performing edge extraction in the Y-axis direction of the result 26 of the difference. Here, the edge extraction is preferably performed for both the X and Y axes, but for convenience of explanation, only the Y axis is shown in the figure. Reference numeral 27 denotes an image of the previous frame when the camera is shaken (during movement of the camera), 28 denotes an image of the current frame when the camera is shaken, and 29 denotes an image of the previous frame when the camera is shaken and the current image. Difference result from frame image 28, 2
9A shows edge extraction data in the Y-axis direction of the difference result 29. Here, the edge extraction data is shown only for the Y axis for convenience of explanation.

In FIG. 5, reference numeral 30 denotes a template cut out from the image of the previous frame when the tracking target is clear;
1 is the image of the current frame when the tracking target is clear, 32 is the score of the template matching result when the tracking target is clear, and 33 is the score from the highest matching point of the score data for verifying the effectiveness of the template matching. Release amount, 34
Is the difference between the score value at the closest match point of the score data and the score value at a point at a certain distance from the best match point to evaluate the effectiveness of template matching, 35
Is a template cut out from the previous frame image when the tracking target is unclear, 36 is the current frame image when the tracking target is unclear, 37 is the template matching result score when the tracking target is unclear, 38 Is the amount of separation from the most matching point of the score data for verifying the effectiveness of the template matching, and 39 is the distance from the score value and the most matching point at the most matching point of the score data for evaluating the effectiveness of the template matching. Is a difference from the score value at a point at a certain distance apart from the score value.

Next, the operation will be described. First, when the camera pan or the mechanism for displacing is stationary and the zoom ratio has not changed, the edge extraction data 26A has a low occupancy and the difference is concentrated in one place, so that its reliability is low. It is high, and it is determined that the position of the center of gravity obtained from the result 26 of the image difference extracting means 7 is the position of the center of gravity before and after the movement of the tracking target, and there is no mistake. Therefore, half of the movement vector from the previous tracking point obtained by template matching to the current tracking point is added to the center of gravity of the image difference, and the center of the tracking target is corrected to be the center of the template. A tracking point can be determined.

Next, when the result of the image difference extracting means 7 becomes zero, the object is stationary and the camera platform is also stationary. Therefore, the tracking point obtained by the template matching is used as it is, and the tracking point is not corrected.

Alternatively, when the result of the image difference extracting means 7 becomes zero, it is determined that there is no movement of the target, and
A method in which the tracking point obtained by template matching is discarded and the previous tracking point is used as it is may be used.

On the other hand, when the camera or the camera platform is being displaced, or the zoom ratio is moving (fluctuating) or is not stable, the edge extraction result 29A has a high occupation ratio and a large value everywhere in the image. Therefore, the reliability 29 is low, and the result 29 of the difference between images also appears everywhere in the image, and the center of gravity position before and after the movement of the tracking target cannot be accurately obtained from the result of the image difference extracting means 7. In such a case, tracking may be performed only by template matching. After the movement of the camera stops, tracking is performed again using both the template matching and the inter-frame difference.

When the object to be tracked is clear, the score 32 as a result of the template matching is steep in the vicinity of the closest matching point, and the score value at the closest matching point of the score data is a certain distance 33 from the closest matching point. The difference 34 from the score value at the point is a large value.

On the other hand, when the tracking target is ambiguous, the template matching result score 37 is gentle near the closest match point, and the score value at the closest match point of the score data is a certain distance 33 from the best match point. The difference 39 from the score value at that point is a small value.

As described above, the effectiveness of template matching is evaluated based on the template matching score results 34 and 39. For example, the template matching score results are almost the same value (smooth) near the closest matching point. If the matching is not valid, the target motion vector is estimated from the previous motion vector,
Half of that is added to the position of the center of gravity of the inter-frame difference result as a new tracking point, tracking is performed only by the image difference extracting means 7 while updating the template using the image small area centered on the tracking point, and template matching is performed. After the validation, the tracking using the template matching and the image difference extracting means 7 together is performed again.

Furthermore, when the state of the difference result of 29A continues or the state of the template matching of 39 continues, it is determined that the object to be tracked is lost, and the direction of the camera immediately before losing the object is determined. Control is performed in a discovery mode in which the camera is displaced left and right or up and down at a predetermined angular width at the center. It is also possible to determine whether template update is necessary or not, and whether camera control is possible or not.

Embodiment 4 6 and 7 are explanatory diagrams of the operation of the image monitoring device according to the fourth embodiment of the present invention. FIG.
, 40A indicates a camera control unnecessary area (inside the dotted line) provided in the screen, 40B indicates a camera control unnecessary area,
Reference numeral 40 denotes a camera image when the tracking target exists in the camera control unnecessary area 40A. Reference numeral 41 denotes a camera image when the tracking target goes out of the camera control unnecessary area 40A. There is always a camera control required area 40B outside the camera control unnecessary area 40A. Reference numeral 42 denotes an image obtained by performing a pan and tilt operation of the camera when the tracking target enters the camera control required area 40B and moving the image to the camera control unnecessary area 40A or almost to the center of the screen. In FIG. 7, reference numeral 43 denotes a surveillance camera image when the zoom of the camera 1 is at the wide end. Reference numeral 44 denotes a surveillance camera image when the zoom of the camera is moved from the wide end to the telephoto side. Reference numeral 45 denotes a surveillance camera image when the zoom of the camera is further moved to the telephoto side.

Next, the operation will be described. The camera control unnecessary area 40A and the camera control unnecessary area 40B outside the camera control unnecessary area are set inside the camera field of view. If the tracking target is within the camera control unnecessary area 40A, the camera pan head is not controlled and the camera is fixed. Tracking. Also 41
When the tracking target enters the camera control required area 40B, the camera platform of the surveillance camera is controlled so that the tracking target is in the camera control unnecessary area 40A, preferably, the target is located substantially at the center of the screen. An image like 42 in FIG. 6 is obtained.

When the camera is controlled so as to move the object to the center of the screen, the control command to the camera platform is intermittently performed instead of continuously. In addition, hysteresis is provided on the increasing side and the decreasing side of the judgment threshold value of the control command to reduce the load on the mechanism section of the camera platform and increase the number of frames used for tracking the information of the image difference extracting means 7. To enable stable tracking of the object. When the camera is controlled so that the tracking point is located within the camera control unnecessary area 40B, the movement of the camera 1 is restricted to a necessary minimum, and unnecessary movement is reduced.

When the zoom of the camera 1 is at the wide end, as shown at 43 in FIG. 7, it is difficult to identify the tracking target.
At this time, zoom control is performed as indicated by 44 and 45 in FIG. 7 until the camera control unit 12 has a resolution that allows the tracking target to be identified.

Here, when controlling the camera zoom ratio in order to obtain a resolution at which the details of the object to be tracked can be identified, the control command to the zoom control means 14 is intermittently executed instead of continuously. Hysteresis is provided on the increase and decrease sides of the command determination threshold to reduce the load on the mechanism of the zoom control means 14 and increase the number of frames that can be effectively used by the image difference extraction means 7 to provide a stable target. To enable tracking.

[0058]

As described above, the image monitoring apparatus according to the present invention evaluates the reliability of the search result of the template searching means and the result of the image difference extracting means, and, depending on the reliability, the camera of the camera. The first tracking mode uses the result of the template search means as the tracking mode, the second tracking mode uses the result of the inter-frame processing unit, and the third tracking mode uses both results.
Tracking mode selection means for selecting which one of the following tracking modes is used, there is an effect that stable tracking of the tracking target can be performed.

Further, since the camera control means controls the camera so that the determined tracking point is located within the camera control unnecessary area provided in the image, the movement of the camera is restricted to a necessary minimum, and the unnecessary movement is restricted. This has the effect of reducing

When the tracking mode selecting means selects the third tracking mode, the tracking point determining means sets the tracking point obtained from the result of the image difference extracting means at the position of the tracking point obtained by the template searching means. Since a position to which a vector having a size half of the movement vector is added is set as a new tracking point, an effect is obtained in that stable tracking can be performed without being affected by changes in imaging conditions.

When the screen occupancy of the edge portion of the difference output image of the image difference extracting means is larger than a predetermined level, the tracking mode selecting means is an image obtained while the camera is moving. Is determined and the first tracking mode in which the output of the image difference extracting means is ignored is selected, so that an effect is obtained in which stable tracking can be performed without being affected by changes in imaging conditions.

When the score of the template searching means is higher than a predetermined level, the tracking mode selecting means determines that the reliability of the template search is low and selects a second tracking mode in which the output of the template searching means is ignored. As a result, it is possible to obtain an effect that stable tracking can be performed without being affected by changes in imaging conditions.

Further, the camera control means sets a camera control required area in a range in contact with the outside of the camera screen,
When the tracking point determined by the tracking point determination means is within the camera control required area, the camera is controlled to control the tracking point to be substantially at the center position of the screen. An effect is obtained that stable tracking that is not affected by is possible.

Further, when the tracking mode selection means loses track of the tracking target, the tracking mode selecting means focuses on the direction of the tracking target immediately before the tracking target is lost.
Since the camera is provided with the discovery mode in which the camera is displaced left and right or up and down with a predetermined angle width, it is possible to obtain an effect that stable tracking can be performed without being affected by changes in imaging conditions.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an image monitoring device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating the operation of the image monitoring device of FIG. 1;

FIG. 3 is a diagram illustrating an operation of the image monitoring device according to the second embodiment of the present invention.

FIG. 4 is a diagram illustrating an operation of an image monitoring device according to a third embodiment of the present invention.

FIG. 5 is a diagram illustrating an operation of the image monitoring device according to the third embodiment of the present invention.

FIG. 6 is a diagram illustrating an operation of the image monitoring device according to the fourth embodiment of the present invention.

FIG. 7 is a diagram illustrating an operation of the image monitoring device according to the fourth embodiment of the present invention.

FIG. 8 is a diagram for explaining a state of an image obtained by the image monitoring device.

FIG. 9 is a diagram for explaining conventional template matching.

FIG. 10 is an explanatory diagram of the matching score of FIG. 9;

FIG. 11 is a diagram illustrating an operation of image monitoring by conventional template matching.

FIG. 12 is a diagram illustrating a configuration of a conventional image monitoring device.

FIG. 13 is an operation explanatory diagram illustrating a failure of a conventional template matching operation.

[Explanation of symbols]

1 surveillance camera, 2 image input means,
3 image storage means, 4 template setting means,
5 template searching means, 6 best matching point extracting means,
7 image difference extracting means, 8 center of gravity extracting means, 9 tracking mode selecting means, 10
Tracking point determination means, 11 Tracking point information storage means, 12 Camera control means, 15 Image of previous frame, 16 Image of current frame, 17 Template, 19 Movement vector, 20 Difference result, 21 Center of gravity position, 22 Movement vector halved 23, a vector obtained by adding half of the movement vector of the tracking point to the position of the center of gravity, 30 a template when the target is clear, 32 a score result of template matching, 35 a template when the target is unclear,
37 Template matching score result, 40A
Camera control unnecessary area, 40B Camera control required area,
43 Image at the zoom wide end, 45 Image at the telephoto end of the zoom, 58 Background image previously captured, 59 Current image, 60 Difference image, 61 to 63: Tracking image fixed to the camera, 64 to 66: Tracking image by camera displacement.

 ────────────────────────────────────────────────── ─── Continuing from the front page (72) Inventor Kazuhiko Sumi 2-3-2 Marunouchi, Chiyoda-ku, Tokyo F-term in Mitsubishi Electric Corporation (reference) 5B057 AA19 BA02 CH18 DA07 DC06 DC32 5C054 CF06 CG01 CG04 CG06 FC01 FC12 FC13 FC14 GB01 GB12 HA18 HA31 5L096 BA02 CA02 FA60 GA08 HA05 JA09

Claims (7)

[Claims] A camera capable of continuously capturing and displacing an image, an image processing unit for processing the image to find a moving object, and displacing the camera in accordance with the movement of the moving object An image monitoring device comprising: a camera control unit; wherein the image processing unit includes: a template setting unit configured to store, as a template, an image area including the moving object in an image of a previous frame of the continuous image; and an image of a current frame. A template searching means for searching for an image area similar to the template in the image area and updating and storing the image area as a new template instead of the template; and performing a difference processing between the continuous images to extract a moving object. Means for evaluating the reliability of the search result of the template searching means and the difference result of the image difference extracting means The first tracking mode using the result of the template searching means, the second tracking mode using the result of the image difference extracting means, and the second tracking mode using the result of the image difference extracting means are performed according to the reliability. 3. An image monitoring apparatus comprising: a tracking mode selection unit that selects any one of the three modes; and a tracking point determination unit that determines a tracking point according to the selected tracking mode. 2. The camera control means for controlling the camera such that the tracking point determined by the tracking point determination means is located within a camera control unnecessary area provided in the image. 2. The image monitoring device according to 1. 3. When the tracking mode selecting means selects the third tracking mode, the tracking point determining means sets the tracking point obtained by the template searching means at the position of the tracking point obtained from the result of the image difference extracting means. 2. The image monitoring apparatus according to claim 1, wherein a position to which a vector having a half size of the movement vector is added is set as a new tracking point. 4. The tracking mode selecting means determines that the image is obtained during the displacement of the camera when the screen occupancy of the edge portion of the difference output image of the image difference extracting means is larger than a predetermined level. 2. The image monitoring device according to claim 1, wherein the first tracking mode is selected. 5. The tracking mode selecting means determines that the reliability of the template search is low when the score of the template search means is larger than a predetermined level.
The image monitoring apparatus according to claim 1, wherein a tracking mode is selected. 6. A camera control means sets a camera control required area in a range in contact with the outside of a screen of the camera, and when the tracking point determined by the tracking point determination means is within the camera control required area, the camera control means sets the camera control required area. 2. The image monitoring apparatus according to claim 1, wherein the tracking point is controlled so that the tracking point is substantially at the center of the screen. 7. A tracking mode selection means includes a discovery mode for displacing a camera left and right or up and down at a predetermined angular width around a direction of the tracking target immediately before the tracking target is lost when the tracking target is lost. 2. The method according to claim 1, wherein
The image monitoring device according to any one of claims 1 to 4,