JPH1132325A - Object tracing method, and object tracking and supervising system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object tracing and supervising system by which a position of an object is accurately estimated even when the object is detected in a disruption state, positions of plural objects are accurately estimated even hen plural objects are detected in line and tracing paths of the respective objects are detected. SOLUTION: An object tracing method detects a difference between an input image signal from a television(TV) camera and a reference background image signal as an object, classifies a detected state change based on a change in the number of objects detected in a continuous input image signal, stores a position of the detected object, an area of a detected area and an image pattern of the detected object for each time as connection information to correct the detected position of the object detected in a distribution state and to correct loci of plural objects detected in line.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a monitoring apparatus using a television camera, and automatically detects a plurality of objects that have entered the field of view of the television camera from video signals of the television camera. The present invention relates to an object tracking method and an object tracking monitoring device that detect and automatically track the movement of the detected object.

[0002]

2. Description of the Related Art Video surveillance apparatuses using television cameras (hereinafter referred to as TV cameras) have been widely used. However, in recent years, in such a surveillance system, detection and tracking of a moving object such as a human being or a car entering the surveillance field of view are not automatically performed by manned monitoring on an image monitor surface, but automatically from an image signal. There is an increasing demand for a system that can detect and provide a predetermined notification or alarm.

In order to realize such a system,
First, an input image signal obtained from a TV camera is compared with a reference background image, that is, an image signal not including an object to be detected, and a difference in luminance value is obtained for each pixel. Detect as an object. This method is called a difference method and has been widely used conventionally. An application example of this method is described in, for example, JP-A-09-73541. The processing of the difference method will be described with reference to FIG. First, an input image 701 obtained by a TV camera
Then, a difference for each pixel from the reference background image 702 registered in advance is calculated to obtain a difference image 703. Next, a pixel whose pixel value of the difference image is smaller than a predetermined threshold value is “0”, and a pixel value that is equal to or larger than the threshold value is “255” (assuming that one pixel has 8 bits).
To obtain a binarized image 704. This allows input image 7
01 is the image D01 in the binarized image 704.
Is detected as

The automatic tracking of the detected object is performed by sequentially detecting the object by the difference method, and obtaining the motion of the target object based on the position of the detected object on the image at each time.
The principle of automatic tracking will be described with reference to FIG. In FIG.
Reference numeral 801 denotes a binarized image at time t0, 802 denotes a binarized image at time t0 + 1, and 803 denotes a binarized image at time t0 + 2. Also
Numeral 804 indicates all the detected objects and the positions of the centers of gravity of the binarized images 801, 802, and 803 at the same time for explanation, and 805 indicates the positions of the centers of gravity of the detected objects of the image 804 as points. Things. In the binarized images 801, 802, 803 obtained at the respective times t0, t0 + 1, t0 + 2 in FIG. 8, the objects can be detected at the positions of the detected objects D02, D03, D04 at the respective times. When the object moves, image 80
4 detected objects D02, D03, D04 at respective times
Arrow V01 connecting the respective centers of gravity C01, C02, C03
V02.

Here, the center of gravity can be obtained from equation (1). The center of gravity C is defined as follows: f (x, y) is a binarized image of the difference (“255” is greater than or equal to the threshold, and “0” is less than the threshold).

[0006]

(Equation 1)

Is defined as However, [B] is [B] =
{(X, y) | f (x, y) = 255}.

In the difference method, when an object having a luminance value close to that of a background image is detected, a part of the object may be missing or split and detected as a plurality of objects. Therefore, in the object tracking method using the difference method, when one object in the field of view is detected as a plurality of objects, it is determined that there are objects by the detected number, and even if the object is one object Regardless, it tracks as if there are multiple objects. Also,
In the difference method, objects that are apparently connected and detected are detected as one object. Therefore, when there are a plurality of objects in the field of view, if a plurality of objects are connected and photographed at a certain time, it is determined that there is only one object at that time. It tracks as a single object despite its existence. Further, since tracking is not performed by comparing the shapes and image patterns of images, it is not possible to reliably guarantee that the same target is being tracked when there are a plurality of detected objects.

An example of an application of the tracking method based on a change in the position of a detected object is described in, for example, “Television Society Journal”, Vol. 43, No. 12, pp. 1370-1374, published in December 1989. There is a paper entitled "Image processing analysis of skier motion". In this paper, when finding the arrows V01 and V02,
The moving direction of the object is estimated from a change in the detected position several frames in the past, and the selection range of the center of gravity to be connected is limited so that tracking can be performed even in a scene where a plurality of detected objects exist. However, this method cannot accurately track an object that has been temporarily split or a plurality of objects running in parallel.

[0010]

As described above, in the difference method which has been widely used in the prior art, if the detected object has a luminance value close to the background image in the camera field of view, the object is split and detected. In addition, there is a problem that accurate object tracking cannot be performed when a plurality of objects exist and they are connected and photographed.

A first object of the present invention is to eliminate the above-mentioned drawbacks, and to judge that one object has been split and judge that a plurality of objects exist and they are connected. An object of the present invention is to provide a highly reliable object tracking and monitoring device.

A second object of the present invention is to estimate an accurate position of an object even when one object is divided and detected, and to estimate the position of each object even when a plurality of detected objects are connected and detected. It is an object of the present invention to provide an object tracking and monitoring device capable of estimating an accurate position and detecting a tracking path of each object.

Further, a third object of the present invention is to correct each tracking path from the detection state before and after the time, even if one object is divided and detected or a plurality of objects are connected and detected. Another object of the present invention is to provide an object tracking / monitoring device capable of determining whether to accurately track an object.

A fourth object of the present invention is to provide an object tracking / monitoring device which guarantees that the same object is being tracked even when a plurality of objects exist.

[0015]

According to the present invention, in order to attain the above object, a change in the number of detected objects is obtained, and splitting or connection of the object is detected from the change in the detection state. is there.

Therefore, the present invention provides a method for detecting an object in an input image signal, a step for detecting a change in the number of detected objects in successive frames, and a method for detecting an object from the detected change in the number of objects. And a detection state change classification step of classifying the state change, and determining whether the object is split or connected by classifying the state change of the object.

Further, a connection information creating step of creating connection information representing a change in the position of the object detected at each time and a change in the area of the detected area from the detection state change, and within a variable time of indefinite length. Trajectory estimation step of estimating the trajectory of the object by evaluating the change in the detection position of the object, and tracking the detected object to evaluate the change in the detection position of the object within a long time, thereby more accurately tracking the object. Is realized.

Furthermore, a division determining step of determining that one object is divided into a plurality of parts upon detection of the object from the connection information, and combining a plurality of detection areas of the object determined to be divided into one. By performing the tracking of the detected object by providing a split correction step to correct, even if one object is temporarily split and detected as a plurality of objects, the object is corrected to be one object. is there.

Further, a connection determining step of determining from the connection information that a plurality of objects are connected and detected at the time of object detection, and correcting the object determined to be connected as a plurality of objects from connection information before and after the connection. Providing a connection correction step and tracking the detected object, even if a plurality of objects are temporarily connected and detected as one object,
The object has been corrected so as to be a plurality of objects according to the front-back relationship.

Furthermore, when the division correction is performed in the division correction step, a division correction recursive determination step for determining that the division correction has been performed is provided, and the division correction is performed recursively, and one object is divided. Thus, even if the object is detected as a plurality of objects, the fact that the object is a single object is corrected more accurately.

Further, when the connection correction is performed in the connection correction step, a connection correction recursive determination step for determining that the connection correction has been performed is provided, and the connection correction is performed recursively to connect a plurality of objects. Thus, even if a single object is detected, it can be corrected more accurately that the object is a plurality of objects.

Further, the position of the object detected as the connection information and the area of the detected area as well as the pixel pattern of the detected area are stored as connection information in the connection information creating step, and the pixel pattern stored in the connection determination step is stored. In this case, the connection is determined by matching with the object, the detected object is tracked, and even if a plurality of objects exist, it can be guaranteed that the same object is being tracked.

That is, the present invention provides, for example, a TV camera for capturing an image of a field of view to be monitored, image input interface means for converting a signal from the TV camera into an image signal, at least a CPU, an image memory, a work memory, and a program. An object tracking / monitoring device having processing means for processing the image signal by a memory, wherein a difference between an input image signal from the TV camera and a reference background image signal in which an object to be detected is not reflected is determined by a luminance for each pixel. A difference between the detected values is determined, an area having a large difference value is detected as an object, a detected state change is classified based on a change in the number of detected objects in a continuous input image signal, and a detected object is detected at each time from the detected state change. The position and area of the detection area, the image pattern of the detected object is stored as connection information, and the detection position of the divided and detected object is corrected, In which it was corrected trajectory multiple object detected by binding.

[0024]

FIG. 9 shows an example of a hardware configuration of an object tracking device common to each embodiment of the present invention.

In FIG. 9, reference numeral 901 denotes a TV camera; 902, an image input I / F; 903, a data bus; 904, an image memory;
5 is a work memory, 906 is a CPU, 907 is a program memory, 908 is an output I / F, 909 is an image output I / F, 910 is a warning light, and 911 is a monitoring monitor. The TV camera 901 is connected to the image input I / F 902, the warning light 910 is connected to the output I / F 908, and the monitoring monitor 911 is connected to the image output I / F 909. Image input I / F 902, image memory 904, work memory 905, CPU 906, program memory 907, output I
/ F908 and the image output I / F909 are connected to the data bus 903. In FIG. 9, a TV camera 901 captures an image of a target area (viewing range). The captured video signal is transferred from the image input I / F 902 to the image memory 904 via the data bus 903.
Is accumulated in The CPU 906 analyzes the image stored in the image memory 904 in the work memory 905 according to the program stored in the program memory 907. The CPU 906 turns on the warning lamp 910 via the output I / F 908 according to the analysis result, and displays an image on the monitor 911 via the image output I / F 909. All the flowcharts described below are described using an example of the hardware configuration of the above-described object tracking device.

FIG. 1 is an example of a basic flowchart of the present invention. At time t0, in the image input step 101, for example, the number of pixels is 320 × 2
An input image signal 701 (FIG. 7) corresponding to 40 is obtained, and a difference is calculated for each pixel from a reference background image 702 (FIG. 7) in which an object to be detected is not registered in advance in a difference processing step 102, and a difference image is calculated. 703 (FIG. 7) is obtained. Next, in the binarization processing step 103, the threshold value processing of the difference image 703 (FIG. 7) is performed, and the luminance value of a portion equal to or greater than a preset threshold value is converted into “255” as an object to be detected. Then, the luminance value of a portion less than the threshold value is converted to “0” as a portion where no object to be detected exists, and the binary image 704 (FIG. 7)
Get. At this time, an area D01 having a luminance value of "255" is extracted by a method such as labeling, and is detected as a detection object at time t0, and the center of gravity of the detection area D01 is determined as the position of the detection object. The “preset threshold value” is a value for determining the presence or absence of an object with respect to a difference value between an input image and a background image, and is set to an empirical value depending on a monitoring target object. . For example, one condition for monitoring the intrusion of an offshore ship is
It is about 20-40. Work memory for the detected object
In 905 (FIG. 9), an object information storage area is secured, and a detected object position, a detection area area, a state change, and a connection pointer set are stored. FIG. 17 is a diagram showing an example of information contents (list structure) stored in the object information storage area according to the present invention.
Object (object) information includes detection time, detection state (appearance, single connection, combination, disappearance, separation), detection position, detection area area, other data (pixel pattern, etc.), determined trajectory, pointer to connected object Stores sets. The state change and the connection pointer set will be described later. Next, in the detected object number change calculation step 104, the position (center of gravity) of a certain detected object detected at the time t0 and the time t0−
The distance between 1 and the position of each detected object at time t0 + 1 is obtained. The number of objects (connectable objects) whose distance is equal to or less than a predetermined value is counted, and the number of connectable objects counted at time t0-1 with respect to the detected object at time t0 is counted at Np and at time t0 + 1. Let Nn be the number of connected connectable objects. This processing is obtained for each of the detected objects detected at time t0. Note that the “predetermined value” is a distance at which objects detected in consecutive frames are considered to be likely to be the same object (can be connected as the same object). "Is set to be larger (about twice) than the apparent speed of the monitoring target object. The reason for setting the size larger is that when the representative position of the detected object used for tracking is set as the center of gravity, it does not necessarily indicate a fixed part of the detected object. Also, if this "predetermined value" is large, the probability that an object can be correctly connected between consecutive frames increases, but the probability of connecting a detection portion due to noise or the like also increases. Division determination, division correction) and trajectory estimation become difficult. Conversely, if the "predetermined value" is too small, the same object cannot be connected in successive frames. The actual set value depends on the imaging condition and the moving speed of the monitoring target object. For example, in one example of the present invention, "predetermined value"
Was set to "20". At this time, using a 1 / 2-inch CCD (element size 6.5 mm x 4.8 mm) and a focal length of the lens of 25 mm, the monitoring target (moving speed 5 km / h) that moves horizontally to the imaging plane 50 m away (Assuming 1.4m / sec) 320x
When monitoring with 240pix, 0.3frames / sec image, horizontal field of view: 6.5mm / 25mm × 50m = 13.0m Apparent speed: 1.4m / sec × 0.3sec / frame / 13.0m × 320
pix = 10.3pix / frame, and the distance considered to be connectable is about 20pix. The time t0-1, time t0, and time t0 + 1 are times (time intervals) at which an image to be processed is input, and are at least about five frames from when the moving object to be tracked enters the field of view until it comes out. Set the time interval so that it appears. If the value of the time interval is low, the number of frames in which the target object can be tracked decreases, and high-precision tracking cannot be performed. The value of this time interval was set to 2-3 frames / sec, for example, under one condition for monitoring the intrusion of an offshore ship described above.

In the detection state change classification step 105, five detection states are assigned according to the values of Np and Nn of the target object. FIG. 10 is a diagram illustrating these five detection states for a certain detection object at time t0. In FIG.
The black circle represents the position of the detected object, and the white circle represents that there was no connectable object at time t0-1 or time t0 + 1. First, an example 1001 in which Np = 0 indicates that a certain object C04 detected at time t0 was not present in the visual field at time t0-1. Hereinafter, this state change is referred to as "appearance". Np
= 1, an example 1002 is a certain object C0 detected at time t0.
5 indicates that the object was the object C06 at the time t0-1, and this state change is hereinafter referred to as "simple connection". Example where Np> 1 1003
Indicates that an object C07 detected at time t0 may have been an object C08 and an object C09 at time t0-1.
This example 1003 is an example where Np = 2, and can be applied to the case where Np is 2 or more. Hereinafter, this state change is referred to as “connection”.
An example 1004 in which Nn = 0 indicates that a certain object C10 detected at time t0 has disappeared from the visual field at time t0 + 1, and this state change is hereinafter referred to as "extinction". Example 100 where Nn> 1
5 indicates that an object C11 detected at time t0 may have become an object C12 and an object C13 at time t0 + 1.
This example 1005 is an example where Nn = 2, and can be applied to the case where Nn is 2 or more. Hereinafter, this state change is referred to as “separation”.
Note that two or more of these state changes may occur simultaneously. Next, in the connection information creation step 106, the connection pointer set of the object information storage area in the work memory 905 (FIG. 9) is changed in response to the state change assigned in the detection state change classification step 105. First, for the object C05 determined to be “simple connection” as in Example 1002,
The detected position of the object C06 is added to the area where the determined trajectory of the object C05 is stored, and the object C05 is added to the connection pointer set of the object C05.
While adding the address where the object information storage area of 06 is,
The address at which the object information storage area of the object C05 in the work memory 905 is located is added to the set of connection pointers of the object C06, and designated as a pointer to a connectable object of the object C06. Next, for the object C07 determined to be “combined” as in Example 1003, the detection positions of the objects C08 and C09 are added to the area storing the determined trajectory of the object C07, and the connection of the object C07 is added. The addresses of the object information storage areas of the object C08 and the object C09 in the work memory 905 are added to the pointer set, and the object C08 and the object C09 are added.
The address at which the object information storage area in the work memory 905 of the object C07 is located is added to each set of connection pointers of the object C07, and designated as the next pointer of the object C08 and the object C09. Further, for the object C11 determined to be "separated" as in Example 1005, the address of the object information storage area in the work memory 905 of the object C12 and the object C13 is added to the connection pointer set of the object C11, The detected position of the object C11 is added to the area for storing the determined trajectories of the object C12 and the object C13, and the address of the object information storage area of the object C11 is added. In other words, by performing the connection information creation step 106, the object information storage areas in the work memory 905 of each object detected so far are connected in a list structure via a set of connection pointers, and the detection that can be connected to each detected object is performed. Objects can be stored.

Next, the list structure of the object information storage area will be described with reference to FIGS. FIG. 11 is a diagram illustrating the state of the object detected between time t0 and time t0 + 4.
FIG. 12 illustrates a list structure of the detected objects in FIG. 11 as a connection structure diagram. In FIGS. 11 and 12,
The black circles only indicate the position of the detected object or the detected object, but do not indicate the area of the detected binary image. In FIG. 11, when the position of the object detected between time t0 and time t0 + 4 is indicated by a black circle, the above-described detected object number change calculation step 10 is performed.
The object information storage area of each object has a connection structure as shown in FIG. 12 by the detection state change classification step 105 and the connection information creation step 106. For example, FIG.
Paying attention to the object C19 at time t0 + 2 of No. 1, the objects detected at time t0 + 3 immediately after that are C16, C20, and C21.
Here, when the distance between the three objects C16, C20, C21 and the object C19 is obtained, the distance between the object C19 and the object C16 becomes a predetermined value or more and is excluded. 104, the objects C20 and C21
Is determined as a connectable object of the object C19, and the object C19 and the objects C20 and C21 at the time t0 + 2 are classified into a state change of "separation" in the detection state change classification step 105. On the other hand, the object C15 at the same time t0 + 2 is also classified into a state change of "simple connection" with the object C16 whose distance is within the predetermined value by the same method as described above. Next, in trajectory estimation step 113,
The change of the detected position of the object stored in the object information storage area is performed by going back to the connection pointer, and the trajectory of the object is estimated. For example, the object C22 in FIG.
The trace of the object C22 can be traced back in the order of C18 and C17.
It can be estimated that the object has passed the detection positions of the objects C17, C18, C19, C20, and C22. Next, abnormality judgment step 10
In step 8, the trajectory of the object obtained in the trajectory estimation step 107 is evaluated based on time and moving distance. For example, when determining the presence or absence of a stopped vehicle, it is evaluated whether or not the trajectory of the object within a predetermined time falls within a predetermined range. If the trajectory exceeds the predetermined range (the moving distance is large), the image input step 101 is performed. Back to
If it is within the predetermined range (the moving distance is short), it is determined that the vehicle is stopped and the process proceeds to the alarm / monitor display step 109. In the alarm / monitor display step 109, a command is sent to the output I / F 908 (FIG. 9) and the image output I / F 909 (FIG. 9). In response, the output I / F 908 (FIG. 9) causes the warning light 910 (FIG. 9) to emit light indicating an abnormality, and the image output I / F 909 (FIG. 9).
(FIG. 9) displays an abnormal state on the monitor 911. According to this embodiment, the object can be tracked while detecting the fragmentation or coupling of the object from the change in the detection state.

If the time (predetermined time) for determining that the stopped vehicle has stopped due to a failure is set to, for example, 5 seconds (5 seconds), if the vehicle has stopped for 5 seconds, the stopped vehicle is determined to be a failed vehicle. In addition, the above-described predetermined range in which the moving distance is small (the maximum moving distance regarded as stopping) is set to, for example, the apparent size of the target object. This is because the center of gravity of the target object is used as the representative position in the detection process, and the representative position may be located at the front or rear of the vehicle depending on the detection state, and the position of the target object always indicates a fixed position. Because there is no.

The actual set value depends on the imaging conditions and the size of the monitored object. For example, in one example of the present invention, the “predetermined range” is set to a rectangular range of “40 × 25 pix”, and when the position change of the trajectory falls within this range, it is determined that the vehicle is stopped.
At this time, 1/2 inch CCD (element size 6.5mm × 4.8mm),
Using a lens focal length of 25mm, the monitored object (horizontal size 6.0mm ×
4.8 mm) is monitored by 320 × 240 pix, 0.3 frames / sec image, the apparent size of the target object in the horizontal direction is the horizontal field of view: 6.5 mm / 25 mm × 200 m = 52.0 m The apparent size So: 6.0m / 52.0m × 320pix = 36.9pix.

The objects C14, C15, and C1 shown in FIG.
Reference numeral 6 denotes one connection pointer set, which indicates a state change of "single connection" according to the classification of FIG. The method of determining the trajectory will be described with reference to FIG. FIG. 18 is a diagram showing an example of a list structure after trajectory determination and correction of an object having a single connection state change according to the present invention.
Reference numeral 14 denotes an example in which a single connection state change has occurred. The detection position of the object C15, which has been connected in a list structure, is added to the area for storing the determined trajectory of the object C14.
The pointer of the object C14 pointing to the object C15 is changed to point to the object C16, and finally the connection information of the object C15 is deleted.

Here, light emission by a warning light has been described as an example of a display indicating an abnormality. However, not only light but also means such as sound, vibration, or a combination thereof that can be sensed by an assisting organism such as an operator or a dog. It is obvious that anything is possible.

FIGS. 2 and 1 show another embodiment of the present invention.
3, and will be described with reference to FIG. In this embodiment, even when one object captured by the TV camera is divided and detected as a plurality of objects, the tracking is performed while correcting the correction. FIG. 2 is a flowchart of this embodiment. This flowchart is obtained by adding a division determination step 110 and a division correction step 111 to the embodiment of FIG. In the division determination step 110, it is determined from the state change of the list structure obtained in the connection information creation step 106 whether or not the detected object has appeared to be divided. Figure 1
3 and FIG. FIG. 13 is a diagram showing, as a connection structure diagram, a list structure of the detected objects before performing the division correction of the objects detected between time t0 and time t0 + 1, ‥‥, and time t1-1 and time t1. 14 shows a connection structure diagram after the splitting correction of the detected object in FIG. 13 is performed. In FIGS. 13 and 14, the black circles indicate only the position of the detected object or the detected object, and do not indicate the area of the detected binary image. In FIG. 13, at time t0, the object C23 undergoes a state change of "separation", which is one of the state changes described above, and the object C23
The state change is repeated from 23 to time t1, and at time t1, the object C28 changes the state of "combined". In the division determination step 110, the distance between the object C24 and the object C25 at the time t0 + 1 connected to the object C23 at the time t0 and the object C24 at the time t0 + 1 is equal to or smaller than the maximum size of the predetermined object. And the sum of the area areas of the object C24 and the object C25 is equal to or less than the area area of the object C23, and the objects C26 and C27 at the time t1-1 connected to the object C28 at the time t1 by the list structure. About the object C26
When the distance between the object C27 and the object C27 is equal to or less than the maximum size of the predetermined object and the sum of the area areas of the object C26 and the object C27 is equal to or less than the area area of the object C28, the objects C23 and C2
4, C25, .DELTA., C26, C27, C28 are determined to be "split". That is, one object C23 at the time t0 is separated into two objects between the time t0 + 1 and the time t1-1 due to some influence, and returns to the one object as the object C28 at the time t1. Presumably detected. Note that the “maximum size of the predetermined object” may be set to the apparent size of the target object, similarly to the setting of the “predetermined range” described in relation to the determination of the stopped vehicle. The set value depends on the photographing conditions and the size of the target object in the same manner as described above.

Next, in the division correction step 111, object information storage areas of the objects C29 and C30 are newly created in the work memory 905 as objects detected at time t0 + 1,
The detection position of 29 is a weighted average based on the area of the detection positions of the object C24 and the object C25, and the area area of the object C29 is
Let the detection position of object C30 be the sum of the area areas of C25 and object C26
And the area of the object C27, and the area area of the object C30 is the sum of the area areas of the object C26 and the object C27.
The state change of the object C23 is set to “simple connection”, the address where the object information storage area in the work memory 905 of the object C24 and the object C25 is deleted from the connection pointer set of the object C23, and the connection pointer set of the object C23 is deleted from the connection pointer set of the object C23. An address where the object information storage area in the work memory 905 is located is added. Similarly, an object information storage area is newly created in the work memory 905 for each time from time t0 + 1 to time t1-1. Then, the detection state of the objects C23 and C28 is changed to "simple connection". As a result, the object information storage area and the link structure of the object determined to be divided from time t0 + 1 to time t1-1 can be accurately corrected. According to this embodiment, even when one object is split and detected as a plurality of objects, this can be corrected and accurate object tracking can be performed.

Next, another embodiment of the present invention will be described with reference to FIGS.
This will be described with reference to FIGS. In this embodiment, even when a plurality of objects captured by a TV camera are connected and detected as one object, tracking is performed while correcting this. FIG. 3 is a flowchart of this embodiment. The flowchart of FIG. 3 is obtained by adding a connection determination step 112 and a connection correction step 113 to the embodiment of FIG. In the connection determination step 112, it is determined from the state change of the list structure obtained in the connection information creation step 106 whether or not the detected objects are viewed as being connected. FIG.
Are time t0-1 to time t0, ‥‥, and time t1 to time t1 + 1
FIG. 16 is a connection structure diagram showing a list structure of detected objects before performing connection correction of the objects detected during the period, and FIG. 16 is a connection structure diagram after connection correction of the detected objects of FIG. 15 is performed. Things. In FIGS. 15 and 16,
The black circles only indicate the position of the detected object or the detected object, but do not indicate the area of the detected binary image. In FIG. 15, at time t0, the object C33 changes the state of “combination”, which is one of the state changes described above, and repeats the state change from the object C33 to time t1. The state is changing. In the connection determination step 112, for the object C31 and the object C32 at the time t0-1 connected to the object C33 at the time t0 by the list structure, the distance between the object C31 and the object C32 is equal to or smaller than the maximum size of the predetermined object. When the area of the object C33 is equal to or less than the sum of the areas of the objects C31 and C32, and the objects C35 and C36 at time t1 + 1 connected to the object C34 at time t1 by a list structure. When the distance between the object C35 and the object C36 is equal to or less than the maximum size of the predetermined object and the area of the object C34 is equal to or less than the sum of the area areas of the objects C35 and C36, the object C
31, C32, C33,..., C34, C35, C36 are determined to be "connected". That is, the two objects C31 and C32 at the time t1-1 are combined with one object between the times t0 and t1 due to some influence, and the object C28 is detected at the time t1 + 1.
Is assumed to have been detected by returning to two objects.

Next, in the connection correction step 113, the object C31
Of the trajectory from time t0-N to time t0-1 of the object C32, the average trajectory vector V04 of the trajectory of the object C32 from time t0-N to time t0-1, and the time t1 + 1 of the object C35 An average movement vector V05 of the trajectory from time t1 + N to time t1 and an average movement vector V06 of the trajectory of the object C36 from time t1 + 1 to time t1 + N are obtained. The movement vector v has a locus (x
1, y1), (x2, y2), ... (xN, yN)

[0037]

(Equation 2)

Is calculated. Note that N designates the number of frames used when obtaining the average vector, and N is set to 5 in one example of the present invention. From the calculated average movement vectors, the angles formed by the vectors V03 and V05, V03 and V06, V04 and V05, and V04 and V06 are calculated, and the combination of the smallest angle among them is set as a candidate for connection. Furthermore, the combination of the smallest angle among the angles formed by the vectors excluding the combination as the connection candidate is further set as the connection candidate. This process is repeated until there are no more moving vector sets. Moving vector v1 = (x1, y1) and v2 = (x2, y2)
Angle θ is

[0039]

(Equation 3)

Is calculated. In the example of FIG. 16, it is assumed that a combination of the average movement vectors V03 and V05 and V04 and V06 is obtained. In the connection correction step 113, the time t0
The object information storage areas of the objects C37, C38,..., C39, and C40 are newly created as the objects detected at each time from the time t1 to the time t1, and the detection positions of the objects C37 and C38 are set to the object C33.
And the detection positions of the objects C39 and C40 are set as the detection positions of the object C34. Further, the area of the object C37 and the area of the object C38 are set to half the area of the area of the object C33.
And the area area of the object C40 is set to half the area area of the object C34. The state change of the object C31, the object C37, ‥‥‥, and the object C39 and the state change of the object C32, the object C38, ‥‥‥, and the object C40 are “simple connection”. As a result, the object information storage area and the list structure of the object determined to be connected from the time t0 to the time t1 can be accurately corrected. According to this embodiment, even when a plurality of objects are linked and detected as one object, this can be corrected and accurate object tracking can be performed.

Next, another embodiment of the present invention will be described with reference to the flowchart of FIG. The embodiment shown in FIG. 4 is obtained by adding a division correction recursive determination step 114 to the embodiment shown in FIG. This division correction recursion determination step 114
Determines whether the splitting correction has been performed, and when the splitting correction has been performed, returns to the splitting determination step 110, and performs the splitting correction again based on the new connection information after the splitting correction. If the division correction has not been performed, the process proceeds to the trajectory estimation step 107. According to this embodiment, when a plurality of objects are split and double-triple and detected as a plurality of objects, correction can be made as long as split correction can be performed, so that more accurate object tracking can be performed. .

Next, another embodiment of the present invention will be described with reference to the flowchart of FIG. The embodiment of FIG. 5 is obtained by adding a connection correction recursion determination step 115 to the embodiment of FIG. This connection correction recursion determination step 115
Determines whether or not the connection has been made, and performs the connection correction again based on the new connection information after the connection correction. If the connection correction has not been performed, the trajectory estimation step 107
Proceed to. According to this embodiment, even when a plurality of objects are detected due to double and triple connection, correction can be made as long as connection correction can be performed, so that more accurate object tracking can be performed.

Next, another embodiment of the present invention will be described with reference to FIG. This embodiment is different from the embodiment shown in FIGS. 4 and 5 in the division determination step 110, the division correction step 111,
Split correction recursion determination step 114, connection determination step 112,
Connection correction step 113 and connection correction recursion determination step 115
Are combined, and the split correction recursion determination step 114 and the connection correction recursion determination step 115 are combined into a correction recursion determination step 116. In the flowchart of FIG. 6, it is determined whether at least either the splitting correction or the connection correction has been performed in the correction recursion determination step 116, and if the correction has been performed, the process returns to the split determination step 110, and the corrected new connection information The division correction or the concatenation correction is performed again based on. When the division correction and the connection correction are not performed, the process proceeds to the trajectory estimation step 107. According to this embodiment, even when phenomena such as splitting and connection occur simultaneously and double and triple, these can be corrected, so that more accurate object tracking can be performed.

An example in which the correction recursion determination step 116 is required will be described with reference to the connection structure diagrams of FIGS. 19, 20 and 21. In FIGS. 19 to 21, black circles only indicate the position of the detected object or the detected object, and do not indicate the area of the area of the detected binarized image. In FIG. 19, an object C50 is separated into an object C51 and an object C52 and connected by an object C59, a connection from another object C71 occurs at an intermediate object C54, and is separated into objects C58 and C74 by a next object C56. ing. FIG. 20 is a diagram showing the connection structure after performing the connection correction for the connection structure of FIG. 19, and the object C54 is converted to the object C84 from the calculated movement vector.
And the object C72, and the object C56 is the object C85 and the object C73.
Is corrected to In this case, next, it is necessary to perform split correction on the connection from the object C49 to the object C60. Therefore,
It is necessary to return to the division determination step 110 by the correction recursion determination step 116 of FIG. 6 and repeat the correction. FIG. 21 shows a division determination step 110 after passing through the correction recursion determination step 116.
Returning to FIG. 7, the connection structure after the division correction in the division correction step 113 is shown. In the division correction step 113, the objects C51, C52, C53, C84, C55, C85, C57, and C58 are corrected to the objects C80, C81, C82, and C83.

Next, another embodiment of the present invention will be described. This embodiment is different from the invention shown in FIG. 3, FIG. 5 or FIG. 6 in that an image pattern of a detected object is also stored as information of an object in the connection information step 106, and a connection correction step 113 is performed. The correction of the object detected by the connection based on the similarity by the pattern matching of the image pattern is performed. The pattern matching method is described in, for example, “Introduction to Computer Image Processing” supervised by Hideyuki Tamura, Sogo Shuppan, 1987, pp. 148-1.
53. In the connection correction step 113, FIG.
Objects C31 and C35, Objects C31 and C36, Objects C32 and C3 in 5
5. Match the objects C32 and C36, calculate the similarity between the respective image patterns, and select the combination having the highest similarity among them as a connection candidate. This process is performed for all the image patterns. As in the previous embodiment, in the example of FIG. 16, the combination of the objects C31 and C35 and the combination of the objects C32 and C36 are set as connection candidates, and in the subsequent connection correction step 113, detection is performed at each time from time t0 to time t1. As an object, a new object C
Create object information storage areas of 37, C38, ..., C39, C40,
The detection positions of the objects C37 and C38 are set as the detection positions of the object C33, and the detection positions of the objects C39 and C40 are set as the detection positions of the object C34. Further, the area of the area between the objects C37 and C38 is changed to the object C33.
And the area area of the object C39 and the object C40 is half the area area of the object C34. As a result, the object information storage area and the list structure of the object determined to be connected from the time t0 to the time t1 can be accurately corrected. According to this embodiment, even when a plurality of objects are linked and detected as one object, this can be corrected and accurate object tracking can be performed.

As described above, according to the above embodiment, a change in the detection state is classified based on a change in the number of detected objects with respect to a continuous input image, connection information of the change in the detection state is held, and one connection is obtained from the connection information. By tracking the detected objects while correcting for the division of the objects and the connection of the plurality of objects, more accurate tracking can be performed.

[0047]

As described above, according to the present invention, when a plurality of objects are present in the field of view of a TV camera, when a temporarily divided object or a plurality of objects intersect, the individual objects can be accurately tracked. This makes it possible to solve the difficulties and to accurately track the object, thereby greatly expanding the applicable range of the object tracking and monitoring apparatus.

[Brief description of the drawings]

FIG. 1 is a flowchart for explaining the operation of an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 3 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 4 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 5 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 6 is a flowchart for explaining the operation of the embodiment of the present invention;

FIG. 7 is a block diagram illustrating the principle of object detection in the difference method.

FIG. 8 is a block diagram illustrating the principle of conventional tracking and tracking of a detected object.

FIG. 9 is a configuration block diagram illustrating an example of hardware according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a detection state change according to the present invention.

FIG. 11 is a diagram illustrating connection information of a detected object according to the present invention.

FIG. 12 is a connection structure diagram illustrating the list structure of FIG. 11;

FIG. 13 is a diagram for explaining division determination according to the present invention.

FIG. 14 is a diagram illustrating a connection state of an object after division correction according to the present invention;

FIG. 15 is a diagram illustrating a connection determination according to the present invention.

FIG. 16 is a diagram illustrating a connection state of objects after connection correction according to the present invention.

FIG. 17 is a diagram showing an example of information content (list structure) stored in an object information storage area according to the present invention;

FIG. 18 is a diagram showing an example of a list structure after trajectory determination and correction of an object causing a single connection state change according to the present invention;

FIG. 19 is a view for explaining a correction recursion step of the present invention.

20 is a view for explaining the connection state of the object after the connection correction in FIG. 19;

21 is a view for explaining the connection state of the objects after the division correction in FIG. 20;

[Explanation of symbols]

101: image input step, 102: difference processing step,
103: binarization processing step, 104: detection object number change detection step, 105: detection state change classification step, 106:
Connection information creation step, 107: trajectory estimation step, 1
08: Abnormality determination step, 109: Alarm / monitor display step, 110: Division determination step, 111: Division correction step, 112: Connection determination step, 113: Connection correction step, 114: Division correction recursive determination step, 115: Connection correction Recursion determination step, 116: Correction recursion determination step, 901: TV camera, 902: Image input I / F, 90
3: Data bus, 904: Image memory, 905: Work memory, 906: CPU, 907: Program memory, 90
8: Output I / F, 909: Image output I / F, 910: Warning light, 911: Monitor, 1001: Detection state change indicating appearance, 1002: Detection state change indicating single connection, 1003: Detection state indicating coupling Change, 1004: Detection state change indicating disappearance, 1005: Detection state change indicating separation

Claims (11)

[Claims] An object tracking method comprising a camera and sequentially detecting an object that has entered a field of view of the camera, wherein an object detection method detects an image of a monitored object from an image signal obtained from the camera. A step of detecting a change in the number of objects detected by the step of detecting the number of objects; a step of classifying the change detected by the step of detecting change of the object into a plurality of states; An object tracking method, characterized in that the state change classification is performed sequentially. 2. The object tracking method according to claim 1, wherein
The classification in the detection state change classification step includes appearance,
An object tracking method characterized by five state changes: simple connection, combination, disappearance, and separation. 3. The object tracking method according to claim 1, wherein the position of the detected object and the area of the detected area at each time are determined from the five state changes classified in the detection state change classification step. A connection information creation step of creating connection information representing a change in the object, and a trajectory estimation step of estimating a trajectory of an object from a change in the position of the object detected by the object detection step from the connection information created by the connection information step. And an object tracking method, wherein the object is tracked. 4. The object tracking method according to claim 3, wherein a division is performed by the connection information to determine that one object has been divided and detected at the time of detecting the object. A division correction step of correcting a plurality of objects determined to be detected into one object, and tracking the object, wherein the object is tracked. 5. The object tracking method according to claim 3, wherein from the connection information, a connection determination step of determining that a plurality of objects are connected and detected as one object, and it is determined that the plurality of objects are connected and detected. A connection correction step of correcting one object as a plurality of objects from connection information before and after the object, and tracking the object. 6. The object tracking method according to claim 4, wherein
An object tracking method, comprising a division correction recursion determination step, and repeating the division determination step and the division correction step until division correction cannot be performed. 7. The object tracking method according to claim 5, wherein
An object tracking method, comprising a connection correction recursion determination step, wherein the connection determination step and the connection correction step are repeated until the connection correction cannot be performed. 8. The object tracking method according to claim 3, further comprising a division determination step, a division correction step, a connection determination step, and a connection correction step, wherein the division determination step is performed when one of the objects is detected from the connection information. It is determined that the object has been split into a plurality of pieces, and the split correction step corrects the plurality of objects determined to be split by the split determination step into one object, and the connection determination step is based on the connection information. It is determined that a plurality of objects are connected and detected as one object, and the connection correcting step corrects one object determined to be connected by the connection determining step as a plurality of objects. Tracking method. 9. The object tracking method according to claim 3, further comprising a split determination step, a split correction step, a split correction recursion determination step, a connection determination step, a connection correction step, and a connection correction recursion determination step. An object tracking method comprising: repeating a split determination step and a split correction step until split correction cannot be performed; and repeating a link determination step and a link correction step until link correction cannot be performed. 10. The object tracking method according to claim 7, 8 or 9, wherein a pixel pattern of the area detected together with the position of the object detected as connection information and the area of the detected area in the connection information creating step is also included. An object tracking method, characterized in that connection is held as connection information and connection is determined by pixel pattern matching in a connection determination step. 11. A camera, an image input I / F for inputting an image captured by the camera, an image memory for storing an image input from the image input I / F, and a program for operating the object tracking monitoring device And a CPU for operating the object tracking / monitoring device according to the program stored in the program memory
A work memory for analyzing an image stored in the image memory, a warning display means for displaying at least one of sound, visible light, and vibration to generate a signal that can be detected by a human body or an auxiliary animal; And an output I / F for transmitting a signal for displaying a warning on the warning display means in accordance with the instruction of the CPU in accordance with the analysis result of the work memory.
And the CPU corresponding to the analysis result of the work memory.
And an image output I / F for sending an image to be displayed on the monitoring monitor in accordance with the instruction of (1), and a data bus connecting the above-described components, and performs object tracking monitoring.