JP2000105835A - Object recognizing method and object tracking and monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To discriminatingly recognize an object of detection and a moving object other than the object of detection and to improve the reliability by deciding whether or not the moving object is a specific moving object from a series of featured physical quantities which are stored according to a criterion. SOLUTION: Differences by pixels between an input image from a camera 601 and a reference background image which does not include the object to be detected are found, areas having large difference are detected sequentially as detection objects, and changes in detection positions are found as the sequentially detected detection objects by tracing back up to detection objects which are several frames precedent to determine the track of the object. Then featured physical quantities of the respective objects included in the determined track are put into one information series and featured physical quantities of the detection time and position change of the detection objects and the sizes of the detection objects are stored sequentially in a work memory 604. The continuity and variance of the track and the size of the detection area are decided from those information series which are stored and the object of detection and a moving object other than the object are discriminatingly recognized according to a combination of those decisions.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a monitoring apparatus using an image input means such as a camera, and more particularly to an apparatus for automatically detecting an object entering a field of view of an image from an acquired video signal. The present invention relates to an object recognition method of an object tracking and monitoring device that automatically tracks the movement of a damaged object.

[0002]

2. Description of the Related Art An object tracking and monitoring apparatus using an image input device such as a camera has been widely used. But,
In recent years, in such a surveillance system, in manned surveillance which performs detection and tracking of a moving object such as a human or a car entering a field of view of the surveillance or a ship in offshore surveillance while looking at an image displayed on a monitor, In addition, a system that automatically detects and tracks an intruding object from an image signal input from an image input device such as a camera and obtains a predetermined notification or an alarm is required.

In order to realize such a system,
First, an input image obtained from an image input device such as a camera is compared with a reference background image (that is, an image in which an object to be detected is not captured), and a difference in luminance value is obtained for each pixel. There is a method of detecting a large area as an object. This method is called a difference method and has been widely used conventionally.

The processing of the difference method will be described with reference to FIG. FIG. 7 is a diagram for explaining the principle of object detection in the difference method. 701 is an input image L, 702 is a reference background image R, 703 is a difference image, 704 is a binarized image, 705 is an image, and 721 is subtraction. It is a vessel. In FIG. 7, a subtractor 721 calculates a difference for each pixel between an input image L701 obtained by an image input device such as a camera and a reference background image R702 prepared in advance, and outputs the difference as a difference image 703. Next, the pixel value of the difference image 703 whose pixel value is less than the predetermined threshold value is “0”, and the pixel value that is equal to or greater than the threshold value is “255” (1
Pixels are calculated with 8 bits) to obtain a binarized image 704. Thus, the human-shaped object shown in the input image 701 is detected as the image 705 in the binarized image 704.

The automatic tracking of the detected object is performed by sequentially detecting the object by the difference method, and obtaining the movement of the target object based on a change in the position of the detected object on the image at each time, for example, a change in the position of the center of gravity. . FIG. 8 is a flowchart showing a basic processing flow of the detection method of the detected object by the difference method.
And a difference processing step 802 for obtaining a difference image 703 by a pixel-by-pixel difference between the input image 701 and a previously prepared reference background image 702 where no object exists.
A pixel whose pixel value of the difference image 703 is less than a predetermined threshold value is “0”, and a pixel value that is equal to or greater than the threshold value is “255”.
And a labeling process is performed on the binarized image 704 to obtain the object 70 that has entered the imaging field of view.
5 comprises an object detection step 804 for detecting the number 5, and a position change calculation step 806 for obtaining a change in the detection position between the detected object and the detected object in the object detection step one frame before when the detected object is present. . Here, “one frame before” means the time of time t ₀ -1 when time t ₀ is the current time. The flow of each step will be described with reference to FIG.
FIG. 9 is a diagram for explaining the principle of tracking a detected object using the difference method, in which 901, 902, 903, and 904 are images, 905, 906, and 907 are detected objects, 908, 909, and 910 are centroid positions, and 911 is an image. , 912,913
Is the displacement. First, by the above-described processing shown in FIG. 8, an object in the field of view is detected by the image input step 801, the difference processing step 802, the binarization processing step 803, and the object detection step 804. In FIG. 9, the image 901 is at time t
₀ binarized image in the -1, the image 902 is binarized image at time t _0, the image 903 represents a binarized image at time t ₀ +1. An image 904 simultaneously shows all the detected objects 905, 906, and 907 of the binarized images 901, 902, and 903 and the respective centers of gravity 908, 909, and 910 of the detected objects for explanation. . In this example, the representative position of the detected object is set as the center of gravity. However, other values may be used as long as they represent the position of the target object, such as the upper end, lower end, and the center of a circumscribed polygon of the detection area. Here, the center of gravity can be expressed by equation (1). The center of gravity C is
Let f (x, y) be the binary image of the difference (“0” below the threshold and “255” above)

[0006]

(Equation 1)

Is defined as Here, [B] is the number of pixels that become B. Image 911 shows the centroid positions 908 and 90 at each time.
9,910 are simultaneously shown for explanation. In the position change calculation step 806, the motion of the detected object obtained at each of the times t ₀ −1, t ₀ , and t ₀ +1 is continuously calculated for the barycentric positions 908, 909, and 910 at each time in the image 911. The displacement at the time t, ie, the movement between times t ₀ -1 and t ₀ calculates the displacement 912 of the centroids 908 and 909, and the movement between t ₀ and t ₀ +1 calculates the displacement 913 of the centroids 909 and 910. I do. Therefore, objects 905, 906, and 907 detected as in image 904 are at time t ₀
-1, it can be determined to have moved the centroid 908,909,910 at t _0, and t ₀ +1.

As an application example of the tracking method using the difference method, there is an invention described in Japanese Patent Application No. 10-130540. In the tracking method using the difference method, a splitting phenomenon in which one detected object is temporarily observed as a plurality of objects and is tracked as a plurality of objects despite being one object, When there is an object,
There is a problem such as a connection phenomenon in which a plurality of objects are temporarily observed as one object and tracked as one object despite being a plurality of objects. These phenomena occur when detecting an object having a luminance value close to the background image. Japanese Patent Application No. 10-130 mentioned above as an application example
In the invention of No. 540, the state change of the detected object between consecutive frames is classified into five state changes of appearance, single connection, coupling, disappearance, and separation, and the state change is detected based on the position of the center of gravity of the detected object and the detection range. , Size, and other information. Further, in the present invention, based on the stored state change and information on the detected object in each frame, a defect of a splitting phenomenon or a connection phenomenon is corrected, and accurate object tracking is realized.

FIG. 6 shows an example of the hardware configuration of the object tracking / monitoring device in the above application example. FIG. 6 is a block diagram showing the configuration of the object tracking and monitoring device. 601 is a television camera (hereinafter referred to as TV camera), 602 is image input I
/ F, 609 is data bus, 603 is image memory, 604 is work memory, 605 is CPU, 606 is program memory, 607 is output
I / F, 608 is an image output I / F, 610 is a warning light, and 611 is a monitoring monitor. The TV camera 601 is connected to the image input I / F 602,
The warning light 610 is connected to the output I / F 607, and the monitor 611 is connected to the image output I / F 608. The image input I / F 602, the image memory 603, the work memory 604, the CPU 605, the program memory 606, the output I / F 607, and the image output I / F 608 are connected to the data bus 6.
Connected to 09. In FIG. 6, a TV camera 601 captures an image in the field of view including the monitoring target area. TV camera 60
1 converts a captured image into a video signal, and inputs the converted video signal to an image input I / F 602. The image input I / F 602 converts the input video signal into a format handled by the object tracking device and sends it to the image memory 603 via the data bus 609. The image memory 603 stores the sent image data. CPU
Reference numeral 605 denotes an image memory 6 in the work memory 604 in accordance with a program stored in the program memory 606 in advance.
Analyze the images stored in 03. As a result of the above analysis, a target area to be monitored within the imaging field of view of the TV camera 601 (for example, the imaging field of view is on the road in front of the gate and the gate, and the predetermined area to be monitored is near the entrance of the gate) Obtain information such as the intrusion of an object. The CPU 605 turns on the warning light 610 from the data bus 609 via the output I / F 607 according to the processing result, and displays, for example, a processing result image on the monitor 611 via the image output I / F 608. The output I / F 607 converts the signal from the CPU 605 into a format that can be used by the warning light 610 and sends the signal to the warning light 610. The image output I / F 608 converts a signal from the CPU 605 into a format that can be used by the monitoring monitor 611, and sends the signal to the warning light 610. Monitoring monitor 611 displays an intruding object detection result image.

As described above, the difference method detects an object based on a pixel-by-pixel difference between an input image from an image input device such as a camera and a previously prepared background image. That is, no preference is given to the object to be detected. For example, an object having a motion other than the detection target, such as the motion of trees in the field of view and the motion of an ocean wave (object unnecessary for detection) is also detected. Therefore, even if the invention described in Japanese Patent Application No. 10-130540 described above is used, an object having such a motion other than the detection target is erroneously detected, and only the target object is reliably tracked. Can not.

[0011]

The above-mentioned prior art includes the following:
A moving object is detected even if it is not a detection target. For example, on land, mainly plants and paper and cloth shake due to the wind, etc., fallen leaves and small animals invade, and in the sea and coastal areas, movement mainly due to wind waves on the water surface, waves, driftwood, small animals invade the quay, etc. In many cases, it is not necessary to detect the instantaneous change in luminance due to entering / exiting between the clouds of the sun, light from a lighthouse or a car. Therefore, there is a disadvantage that accurate object tracking cannot be performed in a scene where such an object exists only by the conventional difference method.

An object of the present invention is to provide a highly reliable object recognizing method and object tracking and monitoring apparatus which eliminates the above-mentioned drawbacks and distinguishes a detection target from a moving object other than the detection target. To provide.

[0013]

In order to achieve the above object, an object recognition method according to the present invention is a method for recognizing a specific moving object in a sequentially input image signal by distinguishing the moving object from other objects. One or more moving objects are sequentially detected from a plurality of sequentially input image signals by a difference method, and at least one or more characteristic physical quantities of the detected moving objects are sequentially stored in a memory for each image signal, Based on the characteristic physical quantity, at least one trajectory of the moving object detected by the difference method is detected, and the physical quantity of the detected trajectory is stored in a memory in association with the moving object as a series of characteristic physical quantities of the moving object. In addition, a criterion for determining whether the moving object belongs to the specific moving object or to an object other than the specific moving object is set in advance, and based on the criterion, a criterion is set. From distinctive physical quantity of sequences that are,
It is determined whether or not the moving object is a specific moving object.

According to another object recognition method of the present invention, a branching process is performed by a decision tree in which a plurality of determination criteria of a series of characteristic physical quantities are combined, and a branching process is performed for each determination criterion. The moving object is classified as a plurality of specific target objects.

Further, a specific target object is extracted from an input image signal by using the object recognition method of the present invention, and the extracted specific moving object is tracked and monitored.

Still another object recognition method of the present invention is as follows.
One or more moving objects are sequentially detected from the plurality of sequentially input image signals by a difference method, and at least one or more characteristic physical quantities of the detected moving objects are sequentially stored in a memory for each image signal. Based on the stored characteristic physical quantities, at least one or more trajectories of the moving object detected by the difference method are detected, and the physical quantities of the detected trajectories are
A sequence of physical quantities characteristic of the moving object is stored in the memory in association with the moving object, and from the series of characteristic physical quantities of the moving object stored in association with each other, a time during which the trajectory of the detected object is continuous is calculated and calculated. A moving object having a continuous trajectory time within a predetermined time range is classified as a specific moving object, and a moving object having a calculated trajectory continuous time outside the predetermined time range is classified into an object that does not belong to the specific moving object. Classify,
The classified specific moving object is tracked and monitored as an intruding object to be tracked and monitored.

According to another object recognition method of the present invention, one or more moving objects are sequentially detected by a difference method from a plurality of sequentially input image signals, and at least one or more features of the detected moving objects are detected. A certain physical quantity is sequentially stored in the memory for each image signal, and based on the stored characteristic physical quantity, at least one trajectory of the moving object detected by the difference method is detected, and the physical quantity of the detected trajectory is moved. A series of characteristic physical quantities of the object are stored in the memory in association with the moving object, and from the series of characteristic physical quantities of the moving object stored in association with each other, the variation in the detected object position change is calculated, and the calculated position is calculated. Classifying a moving object whose variation is within a predetermined variation range as a specific moving object,
The calculated position change is classified into a moving object that does not belong to the specific moving object and the tracking of the classified specific moving object as an intruding object to be tracked and monitored is performed. is there.

In another object recognition method of the present invention, one or more moving objects are sequentially detected by a difference method from a plurality of image signals sequentially input, and at least one or more moving objects of the detected moving objects are detected. The characteristic physical quantity is sequentially stored in the memory for each image signal, and based on the stored characteristic physical quantity, at least one trajectory of the moving object detected by the difference method is detected, and the physical quantity of the detected trajectory is A series of characteristic physical quantities of the moving object is stored in the memory in association with the moving object, and from the series of characteristic physical quantities of the moving object stored in association with each other, the maximum size of the detected object is calculated. A moving object whose size is within a predetermined size range is classified as a specific moving object, and a moving object whose calculated maximum size is out of the predetermined size range does not belong to the specific moving object. Classifying the object, the classification identified moving object was, it is to monitor tracked as intruding object to be tracked monitoring.

In still another object recognition method of the present invention, one or more moving objects are sequentially detected from a plurality of sequentially input image signals by a difference method, and at least one or more moving objects included in the detected moving object are detected. The characteristic physical quantity is sequentially stored in the memory for each image signal, and based on the stored characteristic physical quantity, at least one trajectory of the moving object detected by the difference method is detected, and the physical quantity of the detected trajectory is The minimum size of the detected object is calculated from the sequence of the characteristic physical quantities of the moving object stored in association with the memory as the sequence of the characteristic physical quantities of the moving object, and the calculated minimum size is calculated as Classifying a moving object within a predetermined size range as a specific moving object, classifying a moving object outside the predetermined size range as an object that does not belong to the specific moving object, The animal body is for monitoring tracked as intruding object to be tracked monitoring.

Further, another object recognition method according to the present invention determines whether or not a detected moving object belongs to a specific moving object based on at least the above two object recognition methods.

Further, in the object recognition method of the present invention, the calculation of the time during which the trajectory continues is further performed by using the trajectory of the detected object calculated over a plurality of frames of the sequentially input image signal.

Further, in the object recognition method of the present invention, a circumscribed polygon of the detected object area is further used as the size of the detected object.

Still another object recognition method of the present invention uses an area of a detected object area as a size of a detected object.

Therefore, the object tracking and monitoring apparatus of the present invention comprises an image input means, an image input I / F for inputting an image picked up by the image input means, and an image for storing an image input from the image input I / F. A memory, and a program memory storing a program of an operation of the object tracking monitoring device that performs object recognition,
CPU for operating the object tracking and monitoring device according to the program stored in the program memory, work memory for analyzing the image stored in the image memory, and warning display means for generating a signal that can be detected by a human or an auxiliary animal And monitoring monitor and work memory analysis results
An output I / F that transmits a signal that causes the warning means to display a warning according to the instruction from the CPU, and an image output I / F that sends a monitor image according to the CPU instruction according to the analysis result of the work memory
A program memory held in the program memory, a means for sequentially detecting an object moving the image in the image memory by the difference method, and a characteristic physical quantity of the object detected by the means for sequentially detecting the object. First storage means for sequentially storing in a memory, means for detecting the trajectory of the detected object based on the characteristic physical quantity, and characteristic physical quantity detected by the means for detecting the trajectory. As a series of characteristic physical quantities, a second storage unit that stores in the work memory in association with a moving object having a trajectory, based on a series of characteristic physical quantities of the detected object stored by the second storage unit, Means for classifying objects, and classifying the detected objects into monitored objects and non-monitored objects by the classified means. And which performs a tracking and monitoring of the object.

The classification in another object tracking / monitoring device of the present invention further comprises: means for determining the continuity of the trajectory of the object;
It has means for determining the variation in the change in the position of the object and means for determining the size of the object.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the object tracking and monitoring method of the present invention,
Detection time, center of gravity position, area, size of detected object stored in the course of tracking processing (for example, width and height of circumscribed rectangle)
Based on the information sequence of the temporal and spatial physical quantities such as, for example, the detection target and the object other than the detection target are distinguished into at least one or more types and recognized.

That is, according to the present invention, an input image from a camera
Obtain a difference for each pixel from a reference background image in which an object to be detected is not captured, sequentially detect a region having a large difference value as a detected object, and detect one or more frames before each detected object detected sequentially. A change in the detection position is obtained by going back to the object, and the trajectory of the object is determined. The characteristic physical quantities of each object included in the determined trajectory are sequentially stored as one information series (sequence of characteristic physical quantities), and characteristic physical quantities such as the detection time and position change of the detected object and the size of the detected object are sequentially stored. I do. Then, the continuity of the trajectory and the variability of the trajectory are determined from the stored detection time and position change information series, and the size of the detection area is determined from the stored size information series. And moving objects other than the detection target are distinguished and recognized.

In particular, the movement of trees, the swaying of trees and leaves caused by the wind of plants and paper cloth, the movement and reflection of waves on the ocean, the invasion of deciduous leaves, driftwood and small animals, the instantaneous brightness of lighthouses and car lights, etc. In addition to recognizing things that have random movements, those that occur instantaneously, those that have a small detection area, etc. that are mainly caused by the natural environment, such as changes, as unnecessary detection objects,
Sunlight changes due to the ingress and egress between the clouds of the sun, camera shake,
It can recognize earthquakes and the like.

FIG. 1 is an example of a flowchart for calculating a change in the position of an object according to the first embodiment of the present invention. Hereinafter, the processing content of steps 101 to 105 of the flowchart of FIG. 1 will be described with reference to FIGS. 7 and 9. Time t ₀
, An input image 701 shown in FIG. 7 corresponding to, for example, 320 × 240 pixels is obtained from the TV camera 601 (image input step 101). Next, a difference for each pixel between the input image 701 and the reference background image 702 stored in the image memory 603 is calculated by the subtractor 721 to obtain a difference image 703 (difference processing step 102). The difference image 703 is subjected to threshold processing,
The luminance value of a pixel that is equal to or greater than a preset threshold value is converted to “255” as a portion where a detected object is present, and the luminance value of a pixel less than the threshold value is “0” as a portion where no detected object is present.
To obtain a binarized image 704 (binary processing step 103). Further, one block area 705 having a luminance value of “255”
Is detected, for example, by a labeling method and detected as a detected object (object detection processing step 104). Now, in FIG. 9, the detected object extracted by the labeling method
The detected object 905 is at t ₀ . At this time, the center of gravity 90 of the detected object 905
8 is the position of the detection object. This object detection processing step 1
If there is no detected object in 04, the process returns to the image input step 101, and if there is a detected object, characteristic physical quantities of the detected object are sequentially stored in the work memory 604 (physical quantity storage step 110), and then a position change calculation step Proceed to 106 (detected object presence / absence branch step 105).

The characteristic physical quantities of the detection object stored in the work memory 604 are the detection time of the detection object, the detection position represented by the position of the center of gravity of the detection object, and the area of the detection object, but are not limited thereto. Needless to say, other physical quantities may be included. For example, it may include a positional displacement amount or an area change amount of the detected object at each time.

In the program executed by the object tracking device, the criterion for determining whether or not the detected object is a tracking target based on these characteristic physical quantities is determined in advance by the behavior of the tracking target object and other detected objects. The optimum value is set in advance by calculation and experiment according to the characteristics such as the above and related factors such as the environment of the monitoring area.
These criteria are changed as needed by the object tracking device.

Next, steps 106 and 1 in the flowchart of FIG. 1 will be described with reference to FIGS.
The processing contents of 07 will be described. FIG. 14 is a diagram for explaining an example of the trajectory determination method according to the present invention, where 1401 and 1402 are images, 1403 and 14
04, 1405, 1406, 1403 ', 1404', 1405 ', 1406' are detection objects, 1407, 1408, 1409, 1410, 1407 ', 1408', 14
10 'is the position of the center of gravity, and 1411, 1412, 1413, 1411' and 1413 'are displacements. FIG. 15 is a diagram illustrating a list structure according to an embodiment of the present invention. 1501 image, D1 is the object detected by the frame at time t ₀ -5, D2, D3 is an object detected in the frame at time t ₀ -4, D4, E1 is detected by the frame at the time t ₀ -3 Objects D5 and E2 are objects detected in the frame at time t ₀ -2, D6 and E3 are objects detected in the frame at time t ₀ -1 and E4
Is a object detected by the frame at time t _0. FIG. 16 is a diagram illustrating the connection structure of FIG. FIG. 16 shows the time on the horizontal axis and the detected object situation on the vertical axis. D1 'is the object detected in the frame at time t ₀ -5, and D2' and D3 'are detected in the frame at time t ₀ -4. Objects, D4 'and E1' are objects detected in the frame at time t ₀ -3, D5 'and E2' are objects detected in the frame at time t ₀ -2, and D6 'and E3' are times t _0- object detected in one frame, E4 'is a object detected by the frame at time t _0. The above D1, D2, D3, D4, D5, D6, E1, E2, E3, E4, D indicated by black circles in FIGS.
1 ', D2', D3 ', D4', D5 ', D6', E1 ', E2', E
3 ′ and E4 ′ only indicate the detected object, and do not indicate the area of the detected binary image.

The object tracking and monitoring apparatus sequentially receives images from a TV camera and detects objects from the input images. Image 1401 shown in FIG. 14, sequentially detected by the binary image, all of the detection object 1403,1404,1405,1406 each detected object in the frame up to time t ₀ -5 to time t ₀ at step 104 Of the center of gravity 1407, 1408, 1409, 1410 of
They are shown at the same time for explanation. Image 1402 is
An object that should be detected over all times, such as the image 1401, cannot be detected when the luminance difference from the background image is equal to or less than the threshold (when another object, for example, is hidden behind a tree or a building) In the case where the object 1405 ′ to be detected is not detected, the code corresponding to the code of the image 1401 with respect to the detected object and its center of gravity is indicated by adding “′ (dash)” to each code. It was done.
Here, one black circle corresponds to one detected object. In the conventional object tracking and monitoring method, the connection relationship is lost before and after the detected object 1405 '.

However, in the present embodiment, as described below, the processing is performed not only based on the connection relationship with the immediately preceding frame but also as a determination criterion. The tracking of an object that has not been detected in (N-1 frames) is also possible. That is, the position change calculation step 106, at time t _0, the detected object in the frame from time t ₀ -N until time t ₀ -1, the distance of these to calculate the distance of the center of gravity position among the detected object The detected object in the frame from time t ₀ -N to time t ₀ -1 under the condition of being equal to or less than the predetermined value is defined as a connectable object (adjacent object). Here, the predetermined value is a distance at which objects detected in consecutive frames are considered to be possibly the same object (connectable), and is, for example, larger than the apparent speed of the monitoring target (for example, About twice). The reason for setting a large value is that the representative position of the detected object used for tracking is set as the center of gravity, and does not necessarily indicate a certain part of the detected object. If this value is large, the same object can be reliably connected between consecutive frames, but a detection part due to noise or the like may also be connected, and it is difficult to perform subsequent trajectory correction (division determination, division correction) and trajectory determination become. On the other hand, if the size is too small, the same object cannot be connected in consecutive frames. The actual set value depends on the shooting conditions and the moving speed of the target object. In the experimental system, 40 p
set to ix. For example, a 1 / 2-inch CCD (element size 6.5 m
The target object (moving speed 5 km / h (1.4 m / sec) is assumed to move horizontally with respect to the imaging plane 50 m away, using a TV camera with a lens focal length of 25 mm and a lens focal length of 25 mm. ) To 320x2
When monitoring with an image of 40 pix, 0.3 sec / frame, the horizontal field of view: 6.5 (mm) / 25 (mm) x 50 (m) = 13.
0 (m) Apparent speed: 1.4 (m / sec) × 0.3 (sec / frame) / 1
3.0 (m) × 320 (pix) = 10.3 (pix / frame), and the distance considered to be connectable is about 20 pix. The displacement of the detected position of the detected object in the connectable frame of the object and the time t ₀ the displacement of the detection object at time t ₀ (the detected position change). That is, the change in the detected position becomes the trajectory of the detected object. Note that N (N: natural number) is the number of frames that are targets of connectable objects. For example, when N = 1, a connectable object is detected only by consecutive frames.

This example will be described with reference to FIGS. FIG. 11 is a diagram for explaining the list structure, and FIG.
FIG. 2 is a diagram illustrating a connection structure of FIG. 1101 image, the object B1 is detected in a frame at time t _0, B2, B3 at time t ₀ +1
B4, B5, and C1 are objects detected in the frame at time t ₀ +2, B6 and C2 are objects detected in the frame at time t ₀ +3, and B7 and C3 are times t. ₀ object detected by +4 of the frame, C4 is a object detected by the frame at time t ₀ +5. Figure 12 is a time on the horizontal axis, representing the number of detected objects on the vertical axis Figure, B1 'was detected in the frame at the time t ₀ the object,
B2 ′ and B3 ′ are objects detected in the frame at time t ₀ +1;
4 ′, B5 ′, C1 ′ are objects detected in the frame at time t ₀ +2, B6 ′, C2 ′ are objects detected in the frame at time t ₀ +3, and B7 ′, C3 ′ are times t ₀ The object detected in the frame at +4, C4 ′ is the object detected in the frame at time t ₀ +5. In FIG. 11, an image 1101 simultaneously represents a detected object in a frame from time t ₀ to time t ₀ +5 for explanation,
One black circle corresponds to one detection object. Also, FIG.
The list structure of the object information stored in the work memory 604 is shown, and one black circle corresponds to one object information storage area.
The detected object B1 in the image 1101 is the object information storage area in FIG.
Corresponds to B1 '. The lines connecting the black circles in FIG. 12 represent pointer connections in the list structure of each object information storage area. Therefore, the information sequence of the object detected as the image 1101 is stored in the work memory 604 in the list structure of FIG. The above, which is represented by a black circle as above
B1, B2, B3, B4, B5, B6, B7, C1, C2, C3, C4, B1 ',
B2 ', B3', B4 ', B5', B6 ', B7', C1 ', C2', C
3 'and C4' only indicate the detected object and do not indicate the area of the area of the detected binarized image.

In the examples described with reference to FIGS. 11 and 12, the connectable object can be correctly detected even when the number of target frames of the connectable object is N = 1 (only continuous frames). However, in the examples shown in FIGS. 14 to 16 described below, if the number of target frames of a connectable object is N = 1 (only continuous frames), it is difficult to perform correct detection, so N> 1.

FIG. 15 is a view for explaining another example of the position change calculating step 106. In Figure 15, the detection object E4 for the frame at time t _0, the detected object in the frame at time t ₀ -3
E1, the detected object E2 in the frame at time t ₀ -2, and the connection with the detected object E3 in the frame at time t ₀ -1 are confirmed over successive frames, so that even if N = 1, E1 to E4 Can be confirmed to be connectable. However, in N = 1, the connection and the detection object D3 and Time in the frame at the time t ₀ -4 between the detected object D2 of the detection object D1 and time t ₀ -4 frames for the frame at time t ₀ -5 Detected object D4 and time at frame t ₀ -3
Although it can be confirmed that the detection object D5 in the frame at t ₀ -2 and the detection object D6 in the frame at time t ₀ -1 are connectable,
No object is detected in the frame at time t ₀ -3,
It cannot be confirmed that the detected object D2 in the frame at t ₀ -4 and the object D5 in the frame at time t ₀ -2 are connectable.

In the embodiment of FIG. 15, N = 5 (for example, 0.2 f
(equivalent to 2.5 seconds for a processing speed of rame / sec)
The number of frames to be detected for a connectable object has an allowable width to prevent a temporary oversight that occurs when an object having a luminance value close to the reference background image is detected. For this reason, since the connection between the detected object D2 in the frame at the time t ₀ -4 and the object D5 in the frame at the time t ₀ -2 can be confirmed, the trajectory of the moving detected object, that is, the confirmation of the connection in the related art is performed. Can be done more accurately. FIG. 16 is a diagram in which the connection state of FIG. 15 is aligned with the time axis. Regarding black circles indicating the center of gravity of the detected object, those corresponding to the reference numerals in FIG. 15 are denoted by “′ (dash)”. It is shown with the attached.

Returning to FIG. 1, at the next step 107, time t
Ensuring ₀ -N from time t ₀ at -1 frame detection object object information storage area for storing the detection position change detection time of the detection object in the work memory 604 with respect to (not shown), the object detection The detected position information is stored together with the time and the detected position change. FIG. 10 is a diagram for explaining an example of information contents (characteristic physical quantities) of the detected object stored in the object information storage area.
A detection position, area and the circumscribed rectangle, stored in the pointer (work memory 604 of the object information storage region of the detection object in the frame from time t ₀ -N connectable to the detected object to the time t ₀ -1 Address: Pointer refers to address (link)
(Having a function). By thus expressing the information sequence detected object at each time in a list structure of the object information, the time t ₀ for storing the detected position change of the detected object in the previous frame (detection time, detection position change storage step 107 ). Of the information content of the detected object stored in the object information storage area, the area and the circumscribed rectangle of the object detected in the frame at each time will be described later.

[0040] For detecting a moving object as described above Figure 15 and Figure 16, the list structure for storing the detected object E4 in the frame at the time t ₀ of the data storage area in FIG. 10, the object information of the detected object However, FIG. 16 shows the connection structure of FIG. 15 according to the time axis. In the embodiment of FIGS. 15 and 16, the object in the frame at time t ₀ -3 that was not detected was not displayed. However, for example, the position of the object at time t ₀ -3 trajectory from time t ₀ -3 frame detection object before and after the frame to estimate, also depicted in FIG. 15 and FIG. 16, in the information storage area It is also possible to store. As a method of estimating the position of the object in the frame at time t ₀ -3, for example,
Let the position of the detected object in the frame at time t ₀ -4 be (X _t0-4 , Y
_t0-4), the position of the detected object in the frame at the time t ₀ -2 (X
_t0-2, Y and the at _t0-2), the position of the detected object in the frame at the time _{t 0} -3 (X t0-3, the _{Y t0-3), X t0-3 = (} X t0-4 + X
_{t0-2) / 2, Y t0-3 =} (Y t0 -4 + Y t0-2) / 2 and may be.
Similarly, the area S of the detected object in the frame at time t ₀ -3
_t0-3 from time t ₀ the area of the detection object -4 frame S _T0-4 and time t ₀ of -2 frames detected object area S _t0-2, S
_It can be estimated by _{setting t0-3} = ( _St0-4 + _St0-2 ) / 2.

The information storage area does not need to be in the work memory 604, but may be in another memory of the object tracking and monitoring device, or a separate storage device may be prepared.
As the storage device, an MO disk storage device or the like, a remote file via a network is used, or not one of them. A plurality of combinations may be used, and all or some of the information may be compressed and stored when stored in these devices.

Next, step 10 in the flowchart of FIG.
The processing contents of step 8 and step 109 will be described. First, if the stored detection position of the detected object is, for example, in the monitoring area in the imaging field of view, it is determined that there is a target object (target object presence / absence determination step 108). Whether or not the detected position of the detected object is outside the monitoring area is the first embodiment of the present invention, which is set before the device is delivered to the user. ing. Then output I /
Send command to F607 and image output I / F608. In response to this, the output I / F 607 causes the warning light 610 to emit light indicating warning, and the image output I / F 608 causes the monitor 611 to display, for example, a display indicating warning (alarm / monitor display step 109). After the non-target object classification step 204 and the alarm / monitor display step 109, both return to the image input step 101, and the sequence shown in FIG.
Are repeated. In addition, this alarm
The means for displaying in the monitor display step 109 is to notify the observer (directly or including an auxiliary creature that has a role of transmitting information to the observer, and in some cases, the auxiliary creature can be the observer) whether or not an intruder exists. It is a means of transmitting an alarm,
Means transmitted from outside the body of the observer through sensory organs such as auditory, visual, tactile, etc., due to electromagnetic waves, static electricity, sound, vibration, pressure, etc., and means for stimulating the inside of the observer's body And at least one that generates a signal that can be sensed by a human or assisting animal can be applied.

As described above, according to the first embodiment of the present invention, the detected object is tracked based on the sequence of the stored detection time and detection position, and the detected object is tracked within the monitoring area (for example, the entire port is determined to be within the imaging field of view). If there is a detected object near the pier at the time of the detection (for example, near the entrance of the gate when the imaging field of view is on the road in front of the gate), the object is tracked while sequentially determining this as the target object. be able to.

A second embodiment of the present invention will be described with reference to FIG. In the second embodiment, the tracking elapsed time of the detected object is calculated based on the information sequence of the detection time and the detection position in the information content of the detected object stored in the object information storage area, and based on the tracking elapsed time. Is classified into a target object and a target object outside (not a target object).

FIG. 2 is a flowchart for judging the continuity of the trajectory of the detected object and classifying whether or not the detected object is a monitored object. The flowchart of FIG. 2 is different from the flowchart of FIG. 1 in that a continuity determination step 201 for determining the continuity of the trajectory of the detected object is added after the detection time / position change storage step 107, instead of the detected object existence branching step 108. A continuity branching step 202 for branching based on the determination result of the continuity determination step 201, and furthermore, a target object classification step 203 for classifying a detected object determined to have a continuous trajectory as a target object, A non-target object classification step 204 of classifying a detected object determined to have no trajectory as a non-target object is added.

[0046] In FIG. 2, through the same processing steps as described in FIG. 1, when it proceeds to the continuity determination step 201, the continuity determination step 201, in the form of a list structure of the detection object object information of the time t ₀ From the time t ₀ -1 to the time t ₀ -N, use a pointer to connect an object that can be connected to the stored detected object.
The continuity of the trajectory of the detected object is obtained by going back to the detected object information. That is, when the value of the number of continuously tracked frames is larger than the predetermined value, it is determined that there is continuity of the trajectory. When the value of the number of frames is smaller than the predetermined value, it is determined that there is no continuity of the trajectory. For example, if the number of continuously tracked frames is equal to or greater than a predetermined value (eg, 10 frames at a processing speed of 0.2 frames / second when an object tracked for 5 seconds or more is determined to have continuity), The trajectory is determined to be continuous. That is, when the trajectory of the object exists continuously for a predetermined time or longer, it is determined that the trajectory of the object has continuity of the trajectory.

Next, in the continuity determination step 201, when the trajectory of the detected object is determined to be continuous, the process proceeds to the target object classification step 203. When it is determined that the trajectory is not continuous, the non-target object classification step 204 is performed. (Continuity branch step 202). The detected object having the trajectory determined to be continuous in the continuity branching step 202 is classified as a target object (target object classification step 203). On the other hand, a detected object determined to have a non-continuous trajectory is classified as outside the target object (outside the target object classification step 204). If there is a detected object having a continuous trajectory in the target object classification step 203, the process proceeds to the alarm / monitor display step 109. After the non-target object classification step 204 and the alarm / monitor display step 109, both of the image input step 1
Returning to 01, the processing of the flowchart of FIG. 2 is sequentially repeated.

According to the second embodiment, the continuity of the trajectory of the detected object, which is determined based on the sequence of the stored detection time and the detected position, enables the movement of trees, the leaves of trees, and other plants and paper cloths. Classifying objects that are detected instantaneously, such as swaying due to the wind, or the movement or reflection of waves on the ocean, as outside the target object, and tracking the object while determining a moving object other than the target object in the imaging field of view Can be.

FIGS. 18 and 19 are diagrams for explaining the imaging field of view and the area to be monitored. FIG. 18 illustrates that the surroundings of a private house are set as the imaging field of view and a person approaching the entrance is monitored. 1800 is an image, 1801 is a monitored area, 1802 is a private house, 1803 is an entrance, 1804 is people, 1805 is trees,
1806 is weeds and 1807 is defoliation. FIG. 19 illustrates that a port jetty is set in a visual field range to monitor a person approaching a lighthouse at an end of the jetty, and 1900 is an input image (imaging visual field range) obtained by imaging the vicinity of the jetty. The image of a part of, 1901 is a monitored area, 1902 is a lighthouse, 1903 and 1904 are people, 1905 is a jetty, 1906 is a wave, 1907 is a wave hitting a jetty, 1908 is a driftwood, and 1909 is a bird. In FIG.
An image 1800 is an image of a part of an input image (imaging visual field range) obtained by imaging the vicinity of a private house. Entrance 1803 of a private house in image 1800
The area to be monitored 1801 where a warning or a monitor is displayed when a person 1804 enters (when there is a target object) is shown by a broken line frame. Objects detected in this image 1800 are people 1804, trees
1805, weeds 1806, and fallen leaves 1807. The person 1804 is approaching on foot to the entrance 1803, leaves 1805 trees, weeds 180
6. Assume that the wind is blowing and moving on the fallen leaves 1807 from time to time. Here, by determining whether the trajectory is continuous or intermittent in the trajectory continuity determination step 201, the trees 1805
The leaves and weeds 1806 are determined to be intermittent trajectories and recognized as being outside the target object to be monitored, and are therefore excluded from the monitoring target. Also, since the person 1804 and the fallen leaves 1807 are recognized as target objects, when entering the monitoring target area 1801, an alarm or a monitor display will be performed. In the case of FIG. 18, since the person 1804 has not yet entered the monitored area 1801, no alarm or monitor display is made, but the person 1804 walks further to monitor the monitored area 1801.
When entering, an alarm and a monitor display are performed. In addition, since the fallen leaves 1807 are present in the monitoring target area 1801, an alarm or a monitor display is performed. However, the leaves 1807 can be excluded from the monitoring target by another embodiment described later. Next, FIG.
In 9, an image 1900 is an image of a part of an input image (imaging visual field range) obtained by imaging the vicinity of the jetty. When a person 1903 invades the lighthouse 1902 of the image 1900 (when there is a target object), a monitoring target area 1901 for performing an alarm or a monitor display is indicated by a broken line frame. The object detected in this image 1900 is human 1903
And 1904, waves 1906 and 1907, driftwood 1908, and bird 1909. Person19
04 is far from the monitored area of jetty 1905. Also, the person 1903 is walking on the jetty 1905 and approaching the lighthouse 1902, and the wave 1906 is relatively calm and the wave that hits the jetty
Suppose that 1907 is not so intense and that driftwood 1908 is floating on the sea. Here, trajectory continuity determination step 201
By determining whether the trajectory is continuous or intermittent, the waves 1906 and 1907 and the driftwood 1908 are determined to be intermittent trajectories and are recognized as being outside the target object to be monitored, and are therefore excluded from the monitoring target. In addition, since the people 1903 and 1904 and the bird 1909 are recognized as target objects, when entering the monitoring target area 1901, an alarm or a monitor display is performed. In the case of FIG. 19, since the person 1904 is not in the monitoring target area 1901, no alarm or monitor display is performed, but the person 1903 is in the monitoring target area.
Since it is in 1901, an alarm and a monitor display will be made. Further, since the bird 1909 exists in the monitoring target area 1901, an alarm or a monitor display is performed. However, the bird 1909 can be excluded from the monitoring target by another embodiment described later. A third embodiment of the present invention will be described with reference to FIG. In the third embodiment, the variation in the position change of the trajectory of the detected object is calculated based on the sequence of the object detection time and the detection position of the list structure stored in the object information storage area. This is a method of determining that the object is outside the target object.

FIG. 3 shows the change in position (displacement) of the detected object.
9 is a flowchart for judging the variation of the object and classifying whether or not the detected object is a monitoring target object. The flowchart of FIG. 3 is different from the flowchart of FIG. 1 in that a variation determination step 301 for determining variation in the change in the position of the trajectory of the detected object is provided after the detection time / position change storage step 107. Instead, there is provided a variation branching step 302 for branching based on the determination result of the variation determination step 301, and a detected object determined to have a trajectory that changes smoothly in position (no variation) is classified as a target object. Step 203,
A non-target object classification step 204 for classifying a detected object determined to have no trajectory that smoothly changes position (variable, that is, having a trajectory that changes randomly) outside the target object is added. Things.

[0051] In FIG. 3, through the same processing steps as described in FIG. 1, when it proceeds to the variation determination step 301, the variation determination step 301, and stored in the form of a list structure of the detection object object information of the time t ₀ The presence / absence of variation (degree of variation) is determined, for example, from the variance of the amount of position change from the position change information of the trajectory of the detected object, which is calculated by tracing an object connectable to the detected object using a pointer. The variance of the amount of position change at time t ₀ is given by equation (2)
Can be obtained by

[0052]

(Equation 2)

[0053] However, the detection position in the frame at time t ₀ and (X _t0, Y _t0), and calculates a variance of the change in position of the object back to N1 frames from time t _0. The variance of the position change amount becomes σ ² = 0 when a constant position change is made from N1 frames, and becomes a large value as the position change amount varies for each frame.

Next, if the degree of variation obtained in the variation determining step 301 is less than a predetermined value, it is determined that the change in the position of the locus of the detected object is smooth, and the process proceeds to the target object classification step 203. In the above case, it is determined that the trajectory is not a smooth trajectory, and the process proceeds to the non-target object classification step 204 (variation branching step 302). The detected object determined to have a smooth trajectory in the variation branching step 302 is classified as a target object (the target classification step 20).
3). The detected object determined not to have a smooth trajectory (having a trajectory that changes randomly) in the variation branching step 302 is classified as outside the target object (target object non-classification step 204). If there is a detected object having a continuous trajectory in the target object classification step 203, the process proceeds to the alarm / monitor display step 109. After the non-target object classification step 204 and the alarm / monitor display step 109, both return to the image input step 101, and the processing of the flowchart in FIG. 3 is sequentially repeated.

According to the third embodiment, for example, the movement of trees and the leaves and leaves of plants and paper are determined by the variation in the position change of the trajectory of the detected object which is determined based on the stored detection time and series of detected positions. Classify objects detected by random position changes, such as shaking of cloth or the like, movement or reflection of waves on the ocean, outside of the target object, and determine objects other than the target object in the imaging field of view while moving. Can be tracked.

The third embodiment will be described with reference to FIGS. 18 and 19 again. In FIG. 18, it is assumed that the wind is blowing violently, and the leaves of trees 1805, weeds 1806, and fallen leaves 1807 are constantly moving. Here, the variation determination step 30
By judging whether the trajectory changes smoothly or randomly according to 1, leaves and weeds of trees 1805
Since 1806 is determined to be a random position change and is recognized as being outside the target object to be monitored, it is excluded from the monitoring target. In addition, since the fallen leaves 1807 are present in the monitoring target area 1801, an alarm or a monitor display is performed. However, the leaves 1807 can be excluded from the monitoring target by another embodiment described later. Next, in FIG. 19, the wind and the waves are violent,
04 and 1908 are always moving. Here, by determining whether the locus is a smooth position change or a random position change by the variation determination step 301, the wave 1906 and
The drift tree 1907 and the driftwood 1908 are determined to be intermittent trajectories and are recognized as being outside the target object to be monitored, and are therefore excluded from the monitoring target. In addition, since the people 1903 and 1904 and the bird 1909 are recognized as target objects, when entering the monitoring target area 1901, an alarm or a monitor display is performed. Bird 1909 is a monitored area
Since it exists in 1901, an alarm or a monitor display is performed, but it can be excluded from monitoring targets by another embodiment described later.

A fourth embodiment of the present invention will be described with reference to FIG. In the fourth embodiment, the detected size of a detected object is evaluated based on a sequence of the size of the detected object having a stored list structure, and a target object and a non-target object are determined based on the detected size. How to

FIG. 4 shows the size of the detected object (for example,
9 is a flowchart for determining the width and height of a circumscribed rectangle) and classifying whether a detected object is a monitoring target object.
This flowchart is different from the detection time / detection position change storage step 107, the continuity determination step 201, and the continuity branching step 202 in the embodiment of FIG. 2 in that a size storage step 401 for storing a series of sizes of detected objects is used. And a size determining step 402 of a detected object, and a size branching step 403 for branching based on the determined size.

In FIG. 4, the process proceeds to step 105 in the same manner as described with reference to FIG. Next, in step 105, if there is no detected object in the object detection processing step 104, the process returns to the image input step 101, but if there is a detected object, the process proceeds to the physical quantity storage step 110 and then to the size storage step 401. The size storage step 401, to ensure the object information storage area for storing the size of the detection object in the work memory 604 to the detection object at time t ₀ (not shown), stores information of the object size Then, the process proceeds to the size determination step 402. The size of the detected object by going back with the magnitude determination pointer object can be connected to the detection object are stored in the form of a list structure of the detection object object information for the frame at time t ₀ In step 402, for example, Is evaluated using the circumscribed rectangle of the detection area, and the maximum value and the minimum value are obtained. Here, the circumscribed rectangle may be a polygon instead of a rectangle. In addition to the circumscribed rectangle, the area of the detection area (the number of pixels) can also be used. When evaluating the size based on the vertical and horizontal widths of the area, evaluate the size of the target object using a circumscribed rectangle, and evaluate the size based on the area of the detection area in a scene where the detected object is detected sufficiently large If so, use the area. If the size obtained in the size determination step 402 is within a predetermined range (for example, set from the assumed minimum value of the size of the target object to the maximum value), it is determined that the detected object is the target object and the target object is determined. The process proceeds to the object classification step 203. If the size is out of the predetermined range, the detected object is determined to be outside the target object, and the process proceeds to the non-target object classification step 204 (size branching step 403).
The size determining step 402 and the size branching step 403, the magnitude determination step 402, which can be connected to a detected object that is stored in the form of a list structure of the object information of the detected object in the frame at the time t ₀ Evaluate the size of each connectable object by going back using the object pointer,
From the number N _{S of} objects whose size is within a predetermined range and the number N _{L of} objects outside the range, a ratio R _N whose size is within a predetermined range
(R _N = N _S / (N _S + N _L )). Next, the size branching step 403, the target determines that the percentage ratio R _N is a predetermined object size is within a predetermined range (e.g., 0.8) detecting the object in the case of equal to or more than is the object The process may proceed to the object classification step 203, and if the ratio is within the predetermined ratio, the detected object may be determined to be outside the target object, and the process may proceed to the non-target object classification step 204. Here, the predetermined ratio is an index indicating whether a large detection object is temporarily allowed to be observed to a degree.

Next, the detected object for which the size of the detected object is determined to be within the predetermined range in the size branching step 403 is classified as a target object (target classification step 20).
3). The detected object whose size is determined to be outside the predetermined range in the size branching step 403 is classified as outside the target object (target object non-classification step 204). If there is a detected object whose size is within the predetermined range in the target object classification step 203, the process proceeds to the alarm / monitor display step 109. After the non-target object classification step 204 and the alarm / monitor display step 109, both return to the image input step 101, and the processing of the flowchart of FIG. 4 is sequentially repeated.

According to the fourth embodiment, based on the series of stored detection times and detection positions, for example, the movement of trees, the swaying of plants and paper cloths such as leaves, and the movement of ocean waves An object detected as a small object such as an object or a reflection can be classified as outside the target object, and the object can be tracked while determining a moving object other than the target object in the imaging field of view.

The fourth embodiment will be described with reference to FIGS. 18 and 19 again. In FIG. 18, it is assumed that the wind is blowing violently, and the leaves of trees 1805, weeds 1806, and fallen leaves 1807 are constantly moving. Here, size determination step 401
By judging whether the size of the object detected by the method is within a predetermined size set in advance as a criterion, the leaves and weeds of trees 1805 and weeds 1806 and fallen leaves
Since 1807 is determined to be outside the range of the predetermined size and recognized as being outside the target object to be monitored, it is excluded from the monitoring target. Next, in FIG. 19, the wind and the waves are violent,
Assume that 1903, 1904 and driftwood 1908 are always moving. Here, by determining whether or not the size of the object detected in the size determination step 301 is within a predetermined size range, the waves 1906 and 1907, the driftwood 1908, and the bird 1909 are determined to be outside the predetermined size range. Since it is determined and recognized as being outside the target object to be monitored, it is excluded from the monitoring target. In addition, person 1903,
Since 1904 is recognized as a target object, the monitored area 19
When entering 01, an alarm and monitor display will be performed.

FIG. 5 shows a fifth embodiment of the present invention.
This will be described with reference to FIG. In the fifth embodiment, the second embodiment, the third embodiment, and the fourth embodiment of the present invention described above are processed together to determine whether the detected object is a target object or not. Is what you do. In this flowchart, a detection time / position change / size storage step 501 is provided instead of the detection time / detection position change storage step 107 in the embodiment of FIG. Branch step 302, maximum circumscribed rectangle determination step 502, maximum circumscribed rectangle branch step 503, and
This is obtained by inserting a small area determination step 504 and a minimum area branching step 505.

In FIG. 5, the process proceeds to step 105 in the same manner as described with reference to FIG. Next, in step 105, if there is no detected object in the object detection processing step 104, the process returns to the image input step 101, but if there is a detected object, it passes through the physical quantity storage step 110 and the position change calculation step 106, and The process proceeds to a position change / size storage step 501. In the detection time / position change / size storage step 501, an object information storage area as shown in FIG. 10 is secured, a necessary detection information sequence of the detected object is stored, and the process proceeds to the next step. Here, in the fifth embodiment, the necessary detection information sequence of the detected object is, for example, the detection time, the detection position, the area, the circumscribed rectangle, and the detection of the object detected at each time. Pointer (work memory 60) of the object information storage area of the detected object from time t ₀ -N to time t ₀ -1 connectable to the object
4), and if necessary, another characteristic spatiotemporal physical quantity, for example, a vector,
It may be a displacement, a speed, an acceleration, a circumscribed shape, or the like.

In the subsequent steps, the same steps as those described above perform the above-described processing, but are as follows.
First, the continuity determination step 201 of the trajectory of the detected object is performed, and then the variation in the detected position of the trajectory is determined in the determination step 301 with respect to the detected object whose trajectory is determined to be continuous by the continuity branching step 202. move on. If it is determined that the trajectory is not continuous, the process branches to a non-target object classification step 204 for classifying the detected object as a non-target object. Since a detected object having a non-continuous trajectory has a small number of detected objects connected to the object, the continuity is determined first, and the variation in the detected position of the trajectory can be reduced using a sufficient number of position change information. Can be determined. Next, when it is determined in the variation determination step 301 that the trajectory has a smooth trajectory, the process proceeds to the maximum circumscribed rectangle determination step 502 in the variation branching step 302. Variation determination step 301
If it is determined that the position changes randomly, the process branches to a non-target object classification step 204 for classifying the detected object as a non-target object.

Next, in the maximum circumscribed rectangle determination step 502 for the detected object determined to have a smooth trajectory, the maximum circumscribed rectangle detected in a predetermined frame from the stored circumscribed rectangle series of the detected object is determined. calculated, the maximum circumscribed rectangle branching step 503 on the basis of the maximum circumscribed rectangle, when the maximum circumscribed rectangle is equal to or larger than a predetermined size S _a, object classification step 20 for classifying the detected object as a target object
Proceeding to 3, if the maximum circumscribed rectangle is less than the predetermined size S _A , proceed to a non-target object classification step 204 of classifying the detected object as a non-target object. This is because the relative relationship between the imaging field-of-view range, the size of the monitoring target area, and the size of the monitoring target object is known in advance, and the largest circumscribed rectangle in the information sequence in which the connection of the detection object can be confirmed is the monitoring target area. size can be estimated that the object minimum, namely because it can be said that not the object is smaller than a predetermined size S _a. Here, by using the circumscribed rectangle instead of the area of the detected object, it is possible to reduce an oversight of an object that is partially missing and detected in the object detection by the difference method. Of course, the area may be used in a situation where no dropout occurs. FIG. 17 is a diagram illustrating an example of determining the size of a target object based on a circumscribed rectangle. Reference numeral 704 denotes the binarized image described in FIG.
05 is an image obtained by binarization, 706, 707, 708, 70
9, 710 and 711 are circumscribed rectangles. In FIG. 17, an image 705 is detected as a human-shaped object from the binarized image 704. The image 706 detected as this humanoid object is
Taking a circumscribed rectangle in the Y direction results in a circumscribed rectangle 706. next,
The circumscribed rectangle 707 has a predetermined size S _A. The length of the predetermined size S _{A in} the X direction is X ₀ , and the length in the Y direction is Y ₀ . At this time, the circumscribed rectangle 708 has a length in the X direction X ₁ , the length in the Y direction Y ₁ , and the circumscribed rectangle 709 has a length in the X direction X ₂ and a length in the Y direction Y
₂ , the circumscribed rectangle 710 has a length in the X direction X ₃ , a length in the Y direction Y ₃ ,
The circumscribed rectangle 711 has a length in the X direction as X ₄ and a length in the Y direction as Y _4, and the relationship between the lengths in each direction is X ₂ = X ₃ <
If X ₀ <X ₄ <X ₁ and Y ₁ = Y ₃ <Y ₀ <Y ₄ <Y ₂ , the circumscribed rectangle 70
Size of 8 S _1, the length X ₁ in the X direction is predetermined magnitude of S _A X
Longer than the direction of the length X _0, but the length Y ₁ of the Y-direction is shorter than the Y direction length Y ₀ of a predetermined size S _A, a predetermined size S _A
Judge as smaller than. Also, the size S ₂ of the circumscribed rectangle 709
Is the length Y ₀ in the Y direction of the predetermined size S _A with the length Y _{2 in} the Y direction.
Longer but the length X ₂ in the X direction is shorter than the X-direction length X ₀ of a predetermined size S _A, determines that the predetermined size S _A smaller. In addition, the size S ₃ of the circumscribed rectangle 710 is the X size of the predetermined size S _A for both the length X ₃ in the X direction and the length Y _{3 in} the Y direction.
Is shorter than the direction of the length X ₀ and Y direction length Y _0, it determines that the predetermined size S _A smaller. However, the size S ₄ of the circumscribed rectangle 711, the length X ₄ also length X ₀ and the length of the Y direction of the X direction Y direction both length Y ₄ of a predetermined size S _A of the X-direction is longer than Y _0, determines that greater than a predetermined size S _a.
Further, in the minimum area determination step 504, the minimum area detected in the predetermined frame is calculated from the stored series of areas of the detected object. Then, in the minimum area branching step 505 based on the area of its minimum, if the minimum area was estimated less than the maximum area S _B of the monitored object, the process proceeds to the object classification step 203 of classifying the detected object as a target object, If the minimum area is equal to or _larger than SB, a non-target object classification step 204 for classifying the detected object as a non-target object
Proceed to. This is because the relative relationship between the imaging visual field range, the size of the monitoring target area, and the size of the monitoring target object is known in advance, and the object with the smallest area in the information sequence that can confirm the connection of the detection object is monitored. area can be estimated as the maximum of the target object, that is, because it can be said not to be the target object is larger than a predetermined area S _B. Further, in the fifth embodiment, the area is used instead of the circumscribed rectangle of the detected object. Therefore, even if the circumscribed rectangle cannot be specified, for example, a shake of the TV camera is detected, And the outside of the target object. Of course, a circumscribed rectangle may be used in a situation where the circumscribed rectangle can be specified in advance.

The processing of the fifth embodiment described above is performed by
An example of a method of classifying detected objects using a branching process using a decision tree as described above will be described. FIG. 13 is a diagram illustrating a state of determining whether a detected intruding object is a target object or a non-target object by using the classification method of the fifth embodiment in a tree structure. First, an object detection step 104
Object 1201 detected in the first determined the continuity of the trajectory by the continuous determination step 201 and the continuity branching step 202, intermittently detected object 1203 is determined to be the "are instantaneously detected The outside object classification step 204 classifies the object as the outside object 1214 as a “object outside”. The detected object 1202 determined to have a continuous trajectory by the continuity determination step 201 determines the variation of the position change of the trajectory by the variation determination step 301 and the variation branching step 302, and performs a random position change. The detected object 1205 that is determined to be present is classified as an “object outside the target object” in the non-target object classification step 204 as “an object having random motion”. In addition, the detected object 12 determined to have changed the position of the smooth trajectory in the variation determination step 301
04 is the detected object 12 whose circumscribed rectangle is determined by the maximum circumscribed rectangle determination step 502 and the maximum circumscribed rectangle branching step 503, and whose circumscribed rectangle is determined to be a predetermined size or less.
07 is a “small object” as a classification step outside the target object
204 is classified into 1212 outside the target object. The detected object 1206 whose maximum circumscribed rectangle is determined to be equal to or larger than the predetermined size by the maximum circumscribed rectangle determination step 502 is determined by the minimum area determination step 504 and the minimum area branching step 505, and the area is determined to be equal to or larger than the predetermined area. The determined detected object 1209 is
The non-target object classification step 204 classifies the object as a non-detection target 1211 as “detection abnormal or undetectable”. The detected object 1208 determined to be smaller than the predetermined area by the minimum area determination step 504 is the target object classification step 20.
3 is classified as a target object.

According to this embodiment, based on the sequence of the stored detection time and detection position, for example, the movement of the trees, the swaying of the plants such as the leaves, the movement of the ocean waves, the reflection of the waves, etc. Small animals such as fallen leaves and garbage, insects, small birds and cats, moving toys such as radio-controlled cars and paper airplanes, driftwood between waves, flags and cloths swaying in the wind, rotation and reciprocation of windmills, wind gauges, and motors Moving object which is not a monitoring target such as an instantaneous change in luminance due to a lighthouse, a car light, etc.
Or object 1213 or small object with random movement
1212, a large detected object caused by a change in the sun due to the ingress and egress of the sun's clouds, a camera shake caused by an earthquake, etc. The object can be tracked while determining the moving object.

As in the above-described embodiment, the determination method may be large or small with respect to a predetermined value, or may be pass / fail with respect to a predetermined value range. It may have a judgment range and a plurality of classification items.

The above description is all of the embodiments in which the object is classified into the target object and the target object. However, it is obvious that, depending on the classification method, it is possible to specify a moving object that has entered or occurred in the monitoring target area or the imaging field of view. Therefore, it goes without saying that the present invention can be widely applied to an object recognition technique for not only classifying a target object into a target object and outside the target object but also specifying a moving object in a monitoring visual field.

Further, when the size of the object to be tracked and monitored is, for example, only the size as a criterion, the criterion is relaxed in order to reduce detection omission (determination size SA).
It is necessary to increase the range of FIG. 17), but as a result, the rate of recognizing even the detection object which is not the monitoring target as the tracking monitoring target object increases. Conversely, if the criterion is strict, detection omissions increase, and an object to be tracked may be missed. However, in the present invention, the determination is not made on the characteristic physical quantity between a set of frames, but on the characteristic physical quantity between a plurality of continuous frames. You can reduce your awareness.

[0072]

Therefore, according to the present invention, when a moving object other than the target object exists in the field of view of the camera, this object can be excluded as outside the target object, and only the target object can be accurately tracked. can do. The application range of the object tracking and monitoring device can be greatly expanded.

In addition, it is possible to exclude erroneous detection due to fluctuations in the camera and the imaging visual field range due to an earthquake, ground noise, wind, water current, or the like.

Further, as another effect of the present invention, it is necessary to loosen the criterion in order to reduce the omission of detection of the object to be monitored and tracked. The percentage of people who recognize it as "" increases. Conversely, if the criterion is strict, detection omissions increase, and an object to be tracked may be missed. However, in the present invention, the determination is not made on the characteristic physical quantity between a set of frames, but on the characteristic physical quantity between a plurality of continuous frames. You can reduce your awareness.

Further, as another effect of the present invention, an intruding object is not temporarily lost due to a temporary change in luminance at the time of imaging or a shadow of another object.
A highly reliable intruder tracking and monitoring device can be provided.

Further, as other effects of the present invention, items and ranges for determining detected physical quantities can be selected and combined, and classification for specifying the type of detected object can be performed. It is highly reliable and can perform more detailed object recognition automatically.

[Brief description of the drawings]

FIG. 1 is a flowchart for explaining the operation of a first embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the second embodiment of the present invention.

FIG. 3 is a flowchart for explaining the operation of the third embodiment of the present invention.

FIG. 4 is a flowchart for explaining the operation of the fourth embodiment of the present invention.

FIG. 5 is a flowchart for explaining the operation of the fifth embodiment of the present invention.

FIG. 6 is a block diagram showing an example of a configuration of an object tracking monitoring device.

FIG. 7 is a view for explaining the principle of object detection by the difference method.

FIG. 8 is a flowchart for explaining a basic principle of tracking a detected object using a difference method.

FIG. 9 is a diagram illustrating a conventional principle of tracking a detected object using a difference method.

FIG. 10 is a view for explaining the contents of an information sequence stored in an object information storage area of a detected object according to the present invention.

FIG. 11 illustrates a list structure.

FIG. 12 is a diagram illustrating the connection structure of FIG. 11;

FIG. 13 is a diagram illustrating the embodiment of FIG. 5 of the present invention using a decision tree.

FIG. 14 is a view for explaining the principle of an example of object tracking according to the present invention.

FIG. 15 illustrates a list structure.

FIG. 16 is a diagram illustrating the connection structure of FIG.

FIG. 17 is a diagram illustrating an example of determining the size of a target object based on a circumscribed rectangle.

FIG. 18 is a diagram illustrating an example of a monitoring target area and a monitoring target object.

FIG. 19 is a diagram illustrating an example of a monitoring target area and a monitoring target object.

[Explanation of symbols]

101: image input step, 102: difference processing step,
103: binarization processing step, 104: object detection step,
105: detected object presence / absence branching step, 106: position change calculation step, 107: detection time / position change storage step, 108: target object presence / absence determination step, 109: alarm /
Monitor display step, 110: Physical quantity storage step, 2
01: continuity determination step, 202: continuity branch step, 203: target object classification step, 204: non-target object classification step, 301: variation determination step, 302:
Variation branch step, 401: size storage step,
402: size determination step, 501: detection time / position change / size storage step, 502: maximum circumscribed rectangle determination step, 503: maximum circumscribed rectangle branch step, 504: minimum area determination step, 505: minimum area branch step,
601: TV camera, 602: Image input I / F, 603: Image memory, 604: Work memory, 605: CPU, 606: Program memory, 607: Output I / F, 608: Image output I / F,
609: Data bus, 610: Warning light, 611: Monitoring monitor, 701: Input image, 702: Reference background image, 703:
Difference image, 704: Binary image, 705: Image, 706, 7
07, 708, 709, 710, 711: bounding rectangle, 721: subtractor,
901, 902, 903, 904: image, 905, 906, 907: detected object, 908, 909, 910: center of gravity position, 911: image, 91
2,913: displacement, 1101: image, B1, B1 ': the object detected in the frame at time _{t 0, B2, B2',} B3, B3 ': Time
t ₀ +1 object detected in the frame of, B4, B4 ', B5, B
5 ', C1, C1': time t ₀ +2 object detected in the frame of, B6, B6 ', C2, C2': time t ₀ +3 detected objects in, B7, B7 ', C3, C3 ': Object detected in frame at time t ₀ +4, C4, C4': object detected in frame at time t ₀ +5, 1401, 1402: image, 1403, 1404, 1405,
1406, 1403 ', 1404', 1405 ', 1406': Detected object,
1407, 1408, 1409, 1410, 1407 ', 1408', 1410 ': Center of gravity position, 1411, 1412, 1413, 1411', 1413 ': Displacement,
1501: Image, D1, D1 ': time t ₀ -5 object detected in the frame of, D2, D3, D2', D3 ': time t ₀ the object is detected -4 frames, D4, D4', E1, E1 ': time t ₀ -3
Objects detected in the frame of D5, D5 ', E2, E
2 ′: Object detected in frame at time t ₀ -2, D6, D
6 ′, E3, E3 ′: objects detected in the frame at time t ₀ −1, E4, E4 ′: objects detected in the frame at time t ₀ ,
1800: Input image taken near the private house (imaging field of view)
Part of the image, 1801: monitored area, 1802 is a private house,
1803: Entrance, 1804: People, 1805: Trees, 1806: Weeds, 1807: Fallen leaves, 1900: Partial image of the input image (imaging range) near the jetty, 1901: Monitoring area, 1902: Lighthouse, 1903, 1904: People, 1905: Jetty,
1906, 1907: Wave, 1908: Driftwood, 1909: Bird,

Claims (13)

[Claims] 1. An object recognition method for recognizing a specific moving object in a sequentially input image signal by distinguishing it from other objects, wherein one or more moving objects are sequentially detected by a difference method from a plurality of the sequentially input image signals. At least one characteristic physical quantity of the detected moving object is sequentially stored in a memory for each image signal, and based on the stored characteristic physical quantity, the moving object detected by the difference method is detected. Detecting at least one trajectory, storing the physical quantity of the detected trajectory in the memory as a series of characteristic physical quantities of the moving object in association with the moving object, A criterion for determining whether the moving object belongs to the object or the moving object belongs to an object other than the specific moving object is set in advance. An object recognition method, comprising: determining whether the moving object is the specific moving object from a sequence of characteristic physical quantities obtained. 2. The object recognition method according to claim 1, wherein a branching process is performed by a decision tree in which a plurality of determination criteria of each of the characteristic physical quantity sequences are combined, and a branch process is performed for each of the determination criteria. And classifying the moving object as a plurality of specific target objects. 3. An object tracking and monitoring apparatus using the object recognition method according to claim 1 to track and monitor the specific target object recognized from the input image signal. 4. An object recognition method for recognizing a specific moving object in a sequentially input image signal by distinguishing it from other objects, wherein one or more moving objects are sequentially detected by a difference method from a plurality of the sequentially input image signals. At least one characteristic physical quantity of the detected moving object is sequentially stored in a memory for each image signal, and based on the stored characteristic physical quantity, the moving object detected by the difference method is detected. Detecting at least one trajectory, storing a physical quantity of the detected trajectory as a series of characteristic physical quantities of the moving object in the memory in association with the moving object, and storing the moving object stored in the association; From the series of characteristic physical quantities, the time during which the trajectory of the detected object continues is calculated, and the time during which the calculated trajectory continues is within a predetermined time range. Is classified as the specific moving object, the calculated trajectory is classified as a moving object that does not belong to the specific moving object, the moving object outside the predetermined time range, and the classified specific moving object is An object recognition method characterized by tracking and monitoring an intruding object to be tracked and monitored. 5. An object recognition method for recognizing a specific moving object in a sequentially input image signal by distinguishing it from other objects, wherein one or more moving objects are sequentially detected by a difference method from a plurality of the sequentially input image signals. At least one characteristic physical quantity of the detected moving object is sequentially stored in a memory for each image signal, and based on the stored characteristic physical quantity, the moving object detected by the difference method is detected. Detecting at least one trajectory, storing a physical quantity of the detected trajectory as a series of characteristic physical quantities of the moving object in the memory in association with the moving object, and storing the moving object stored in the association; The variation of the position change of the detected object is calculated from the series of characteristic physical quantities of the movement of the detected object, and the variation of the calculated position change moves within a predetermined variation range. Classifying an object as the specific moving object, classifying a moving object whose variation in the calculated position change is outside the predetermined variation range into an object that does not belong to the specific moving object, and classifying the classified specific moving object. An object recognition method characterized by tracking and monitoring an intruding object to be tracked and monitored. 6. An object recognition method for recognizing a specific moving object in a sequentially input image signal by distinguishing it from other objects, wherein one or more moving objects are sequentially detected from a plurality of the sequentially input image signals by a difference method. At least one characteristic physical quantity of the detected moving object is sequentially stored in a memory for each image signal, and based on the stored characteristic physical quantity, the moving object detected by the difference method is detected. Detecting at least one trajectory, storing a physical quantity of the detected trajectory as a series of characteristic physical quantities of the moving object in the memory in association with the moving object, and storing the moving object stored in the association; Calculating a maximum size of the detected object from the series of characteristic physical quantities of the moving object, and moving the moving object whose calculated maximum size is within a predetermined size range to the specific movement. Classifying moving objects whose calculated maximum size is out of a range of a predetermined size as objects that do not belong to the specific moving object, and classifying the classified specific moving object for tracking and monitoring; An object recognition method characterized by tracking and monitoring a target intruding object. 7. An object recognition method for recognizing a specific moving object in a sequentially input image signal by distinguishing it from other objects, wherein one or more moving objects are sequentially detected by a difference method from a plurality of the sequentially input image signals. At least one characteristic physical quantity of the detected moving object is sequentially stored in a memory for each image signal, and based on the stored characteristic physical quantity, the moving object detected by the difference method is detected. Detecting at least one trajectory, storing a physical quantity of the detected trajectory as a series of characteristic physical quantities of the moving object in the memory in association with the moving object, and storing the moving object stored in the association; Calculating the minimum size of the detected object from the series of characteristic physical quantities of the above, and identifying the moving object whose calculated minimum size is within a predetermined size range. Classifying a moving object outside a predetermined size range into an object that does not belong to the specific moving object; and tracking and monitoring the classified specific moving object as an intruding object to be tracked and monitored. An object recognition method characterized by: 8. A method for determining whether the detected moving object belongs to the specific moving object, based on at least two of the object recognizing methods according to claim 4. Object recognition method. 9. The object recognizing method according to claim 4, wherein the calculation of the continuous time of the trajectory uses a trajectory of the detected object calculated over a plurality of frames of the sequentially input image signal. An object recognition method characterized in that: 10. The object recognition method according to claim 6, wherein the size of the detected object is a polygon circumscribing a detected object area. Object recognition method. 11. The object recognition method according to claim 6, wherein an area of a detected object area is used as the size of the detected object. Object recognition method. 12. An image input unit, an image input I / F for inputting an image captured by the image input unit, an image memory for storing an image input from the image input I / F, and performing object recognition. A program memory for storing a program for operating the object tracking and monitoring device, a CPU for operating the object tracking and monitoring device according to the program held in the program memory, and analysis of an image stored in the image memory A work memory, a warning display means for generating a signal that can be sensed by a human or an auxiliary animal, a monitoring monitor, and displaying a warning on the warning means in response to an analysis result of the work memory by an instruction of the CPU. An output I / F for transmitting a signal, and an image output I / F for transmitting the monitoring monitor image in accordance with an instruction of the CPU in accordance with an analysis result of the work memory, A program memory held in a gram memory, a means for sequentially detecting an object moving the image of the image memory by a difference method, and a characteristic physical quantity of the object detected by the sequential detection means is stored in the work memory. A first storage unit for sequentially storing, a unit for detecting a trajectory of the detected object based on the characteristic physical amount, and the characteristic physical amount of the trajectory detected by the unit for detecting the trajectory, Second storage means for storing in the work memory in association with the moving object as a sequence of characteristic physical quantities of the detected object; characteristic physical quantities of the detected object stored by the second storage means Means for classifying an object based on the sequence of the object. The means for classifying the detected object converts the detected object into a monitored object and a non-monitored object. An object tracking / monitoring device characterized in that a moving object is tracked and monitored by classifying the object into a moving object. 13. The object tracking / monitoring device according to claim 12, wherein the means for performing classification includes means for determining the continuity of the trajectory of the object, means for determining a variation in the change in the position of the object, and size of the object. An object tracking / monitoring device, comprising: