JP2019003334A - Object tracking method and object tracking program and object tracking system - Google Patents

Abstract

To provide an object tracking method capable of reducing a calculation load of object tracking.SOLUTION: An object tracking method has a step of classifying an object whose elapsed time from a time when the object is detected in a moving image where a plurality of frames is lined up along a time sequence continue for a prescribed time as a dynamic body, a step of classifying an object whose elapsed time is below the prescribed time as a candidate of a dynamic body, a step of checking a correspondence of an unrecognized object detected in a current frame to the dynamic body (S1 to S6), a step of mapping the unrecognized object as the dynamic body when it is judged that the unrecognized object corresponds to the dynamic body (S7:YES, S30), a step of checking a correspondence of the unrecognized object to the candidate of a dynamic body when it is judged that the unrecognized object does not correspond to the dynamic body (S7:NO, S8 to S10), and a step of mapping the unrecognized object with the candidate of a dynamic body when it is judged that the unrecognized object corresponds to the candidate of a dynamic body (S10:YES, S31).SELECTED DRAWING: Figure 4

Description

  The present invention relates to an object tracking method, an object tracking program, and an object tracking system.

  Object tracking is a technique for identifying and tracking an object moving in a plurality of images (frames) arranged in time series.

  As a conventional object tracking technique, for example, there is a technique for detecting a preceding vehicle. In this technique, among measurement points detected by a millimeter wave radar, measurement points that are close to each other and have the same relative velocity are grouped to generate a measurement point group. Among the generated measurement point groups, measurement point groups that satisfy the conditions of the preceding vehicle are extracted, and a measurement point group that is extracted a predetermined number of times or more is detected as the preceding vehicle as time passes (Patent Document 1). .

JP-A-2005-173806

  However, in the prior art, in determining whether the measurement points detected at the current time point are the same as the measurement points of the measurement point group recognized as the preceding vehicle, all the measurement point groups are Investigating the degree of association with measurement points. For this reason, the load at the time of calculating the correspondence between the preceding vehicle (object) recognized in the past time passage and the current measurement point is large.

  Therefore, an object of the present invention is to provide an object tracking method that can reduce the calculation load of object tracking. Another object of the present invention is to provide an object tracking program capable of reducing the object tracking calculation load. Furthermore, the other object of this invention is to provide the object tracking system which can reduce the calculation load of object tracking.

  The above object is achieved by the following means.

(1) An object that is detected in a moving image in which a plurality of frames are arranged in time series and that has elapsed since detection is continued for a predetermined time or more. Classifying as an object of type 1 and classifying an object of less than the predetermined time as an object of type 2 (a);
(B) examining the correspondence between the unrecognized object and the first type object, using the object detected in the current frame as an unrecognized object;
A step (c) of associating the unrecognized object and the first type object when it is determined in the step (b) that the unrecognized object and the first type object correspond to each other;
In the step (b), when it is determined that the unrecognized object and the first type object do not correspond to each other, a step (d) of checking the correspondence between the unrecognized object and the second type object;
In the step (d), when it is determined that the unrecognized object and the second type object correspond to each other, the step (e) of associating the unrecognized object with the second type object;
An object tracking method.

  (2) In the step (e), if the elapsed time after the second type object associated with the unrecognized object is detected as an object has passed the predetermined time, the second type object is The object tracking method according to (1), further including a step (e1) of classifying the first type object.

(3) In the step (a), the second type object is associated with an elapsed time since it was detected as an object,
The object according to (1) or (2), wherein in the step (d), the second type object is checked for correspondence with the unrecognized object in order from the longer elapsed time after being detected as an object. Tracking method.

(4) In the step (a), the first type object is associated with an elapsed time since it was detected as an object,
In the step (b), any one of the above (1) to (3), wherein the first type object is checked for correspondence with the unrecognized object in order from the longest elapsed time after being detected as an object. The object tracking method described in one.

(5) In the step (a), the first type object that does not correspond to the unrecognized object in the step (b) is classified as a third type object,
In the step (d), when it is determined that the unrecognized object and the second type object do not correspond to each other, a step (f) of checking the correspondence between the unrecognized object and the third type object;
In the step (f), when it is determined that the unrecognized object and the third type object correspond to each other, the unrecognized object and the third type object are associated with each other, and the third type Reclassifying an object into the first type object (g);
In the step (f), when it is determined that the unrecognized object and the third type object do not correspond to each other, the step (h) of classifying the unrecognized object as the second type object;
The object tracking method according to any one of (1) to (4) above, including:

  (6) In the step (d), when it is determined that the unrecognized object does not correspond to any second-type object, the unrecognized object is classified as the second-type object (d1) The object tracking method according to any one of (1) to (4) above.

  (7) The object tracking method according to any one of (1) to (6), wherein the predetermined time is 0.2 seconds or more and less than 1 second.

  (8) The object tracking method according to any one of (1) to (7), wherein the time is counted in units of the frame.

  (9) The object tracking method according to any one of (1) to (8), including a step (i) of displaying the first type object together with information indicating that it is a moving object.

  (10) An object tracking program for causing a computer to execute the object tracking method according to any one of (1) to (9).

(11) an image acquisition sensor that acquires frames in time series;
A computer that executes the program according to (10) above;
An object tracking system.

  According to the present invention, it is checked whether the first type object that has been detected continuously for a predetermined time or more until the previous frame corresponds to the unrecognized object first. Only when the recognition object is associated with the first type object and does not correspond, the correspondence with the second type object having a shorter elapsed time than the first type object is examined. Thus, if the first type object and the unrecognized object are associated with each other, the second type object need not be examined. For this reason, the calculation load can be reduced accordingly.

It is explanatory drawing for demonstrating the whole structure of an object tracking system. It is a block diagram for demonstrating the structure of an object tracking apparatus. It is a list figure showing an example of a list which classified an object. It is a flowchart for demonstrating the process sequence of the object tracking method of this embodiment. It is a flowchart for demonstrating the process sequence of the object tracking method of this embodiment. It is a flowchart for demonstrating the process sequence of the object tracking method of this embodiment. It is explanatory drawing explaining the example of the moving body recognized by the previous frame, a 1st moving body candidate, and a 2nd moving body candidate. It is a list figure which shows each list at the time of FIG. It is explanatory drawing explaining the example which detected the unrecognized object by the present flame | frame. FIG. 10 is an explanatory diagram in which the previous frame shown in FIG. 7 and the current frame shown in FIG. 9 are superimposed. It is explanatory drawing explaining the state with which the moving body M1 and the unrecognized object Ob1 corresponded. As a comparative example, it is explanatory drawing explaining the example which investigates the unrecognized object detected by the present flame | frame, and all the objects recognized by the previous frame by brute force. It is explanatory drawing explaining the state with which unrecognized object Ob2 and 1st moving body candidate C1 corresponded. It is a list figure which shows each list | wrist when a process progresses to the last. It is explanatory drawing explaining the present frame at the time of a process end. It is a figure which shows the example of the three-dimensional distance image obtained by a laser radar. It is sectional drawing explaining the structure of a laser radar. It is explanatory drawing explaining the state which scans the inside of the monitoring space of the monitoring apparatus MD with the laser spot light SB (it shows by hatching) radiate | emitted according to rotation of the mirror unit MU.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The present invention is not limited to the following embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios. In the following description, the current frame is referred to as the current frame among the frames arranged in time series, the frame before the current frame is referred to as the past frame, and the frame immediately before the current frame is referred to as the previous frame. .

(Object tracking system)
FIG. 1 is an explanatory diagram for explaining the overall configuration of the object tracking system. FIG. 2 is a block diagram for explaining the configuration of the object tracking apparatus.

  The object tracking system includes a laser radar 101 that acquires frames arranged in time series, and an object tracking device 102 that tracks a moving object in the frames arranged in time series along the time series. The frames arranged in time series are reproduced (displayed) in time series to become a moving image.

  The laser radar 101 is an image acquisition sensor. Although details of the laser radar 101 will be described later, the laser is scanned to measure the distance from the reflected light to the object in real time. For this reason, the distance and size (height, width, etc.) from the installation position of the laser radar 101 to the unrecognized object can be obtained. This also makes it possible to obtain an image called a distance image. This distance image is arranged in chronological order as one frame per scan (that is, displayed continuously) so that a moving image is obtained. A distance image (frame) from the laser radar 101 is transmitted to the object tracking device 102.

  The object tracking device 102 is a computer, and may be a general personal computer or a dedicated computer. As shown in FIG. 2, the configuration of the object tracking device 102 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and an HDD (Hard Disk Drive) 14.

  The RAM 12 is a storage unit that the CPU 11 uses as a work area, and temporarily stores programs and data necessary for processing, for example.

  The ROM 13 stores a basic program necessary for starting the object tracking device 102 and basic operations as a computer.

  The CPU 11 reads out a program corresponding to the processing content from the HDD 14 and expands it in the RAM 12 and executes the expanded program. Here, a moving image is mainly configured based on a frame obtained from the laser radar 101, and an object in the moving image is tracked.

  The HDD 14 stores programs and data necessary for each process. A nonvolatile semiconductor memory (for example, a flash memory) may be used instead of the HDD.

  The object tracking device 102 includes an input device 15 such as a mouse and a keyboard, a display 16 for displaying an image (moving image), and a network interface 17 for connecting an external device.

  The network interface 17 is, for example, Ethernet (registered trademark), and is used for communication with the laser radar 101. The network interface 17 is also connected to a LAN (Local Area Network) or the like, and is also used for communication with a server (described later).

  The object tracking device 102 is achieved by such a computer executing a processing procedure of an object tracking method described later. Note that the object tracking device 102 may acquire only the measurement values from the laser radar 101 and configure the distance image and its moving image.

  For example, the server 103 is connected to the object tracking device 102 via a network. The server 103 receives a moving image composed of frames arranged in time series from the object tracking device 102 and stores it. In addition, the server 103 stores other moving images and provides them in response to reading from the object tracking device 102. Other moving images are not limited to moving images acquired by the laser radar 101, but include moving images of ordinary movie cameras.

(Object tracking method)
The object tracking method of this embodiment will be described.

  In the present embodiment, for object tracking, objects other than the background and stationary objects in the frame acquired by the laser radar 101 are classified as follows.

  (1) “Moving object” (first type object): an object that has been recognized continuously for a predetermined first predetermined time or more.

  (2) “First moving object candidate” (second type object): an object that is continuously recognized for a second predetermined time or more and less than a first predetermined time.

  (3) “Second moving object candidate” (second-type object): an object that is continuously recognized for a time period equal to or longer than a third predetermined time and shorter than a second predetermined time.

  (4) “Unrecognized object”: an object detected in the current frame (current time).

  The length of each predetermined time is first predetermined time> second predetermined time> third predetermined time.

  Time can be measured by counting a system clock of a computer constituting the object tracking apparatus 102. For example, the first predetermined time is preferably 0.2 seconds or more and less than 1 second, and more preferably 0.3 seconds or more and less than 0.5 seconds. By setting the time as described above, it is possible to prevent the noise that appears and disappears for a very short time, the flowers and trees swaying in the wind from being recognized as moving objects. Of course, the time set as the first predetermined time is arbitrary, and may be set in accordance with, for example, the width and depth of the measurement range by the sensor, the monitoring target, and the like.

  Here, the distance image, that is, the frame acquired by the laser radar 101 is, for example, 10 frames per second. In this case, the time for one frame is 1/10 second. Therefore, in this embodiment, instead of directly counting the system clock, the time is counted in units of frames. In other words, the number of frames is counted instead of time.

  Therefore, redefining the predetermined time in units of frames is as follows. Here, the case is 10 frames per second.

  (1) “Moving object”: an object that is continuously recognized in the previous three frames or more among the past frames.

  (2) “First moving object candidate”: an object that is continuously recognized in the immediately preceding two frames among the past frames.

  (3) “Second moving object candidate”: an object recognized only in the immediately preceding frame of the past frames.

  (4) “Unrecognized object”: an object detected in the current frame.

  The moving body, the first moving body candidate, and the second moving body candidate defined as described above are classified as a list and stored in the RAM 12. A storage device other than the RAM may be used, but a storage device capable of reading and writing at high speed is preferable so that at least object tracking processing can be performed.

  The present invention is not limited to such a number of frames and time. If the number of frames per second changes, the time for one frame also changes. For example, 5 frames per second used in a surveillance camera or the like is 1/5 second per frame. Conversely, for a moving image of 30 frames per second, it is 1/30 second.

  FIG. 3 is a list diagram illustrating an example of a list in which objects are classified.

  As shown in FIG. 3, the moving object, the first moving object candidate, and the second moving object candidate are each classified as a list. That is, the moving object is classified into a moving object list (continuing 3 frames or more), the first moving object candidate is classified into a first moving object candidate list (continuing 2 frames), and the second moving object candidate is classified into a second moving object candidate list (only 1 frame).

  Furthermore, in this embodiment, a lost moving object list is provided. The lost moving object list is a list provided when a moving object is temporarily invisible. That is, the moving object may disappear from the frame that is the measurement range of the sensor, but may disappear temporarily in the frame. For example, when the moving object is hidden behind a stationary object when viewed from the sensor installation position, there are multiple moving objects and one moving object is hidden behind the other moving object. Etc. In such a case, an unrecognized object is not detected in the current frame, or the moving object recognized in the past frame cannot be associated. Therefore, a lost moving object list is provided, and moving objects that do not have such correspondence are registered as lost moving objects (type 3 objects). If an unrecognized object is detected and there is no correspondence with the moving object or the moving object candidate, the correspondence with the lost moving object in the lost moving object list is checked to determine whether the object is the same as the previous moving object. be able to. The lost moving body registered in the lost moving body list may be continuously stored until the system is stopped, but may be deleted when a predetermined time elapses or a predetermined time is reached. For example, it is deleted after several seconds to several minutes or hours, or deleted every day at 0:00.

  Note that the lost moving object list may not be provided, and in that case, a moving object that does not correspond to an unrecognized object need only be deleted from the moving object list. Even in this case, when the same object as the object deleted from the moving object list appears again, the object appears as a new object, but object tracking is performed.

  These lists are stored in the RAM 12. In FIG. 3, each list is grouped together. However, the present invention is not limited to this. For example, each list may be stored independently.

The objects classified in each list are managed by attaching an identification number (ID), for example. In FIG. 3, moving objects are M1, M2, M3,..., Moving object candidates are C1, C2, C3,... (Without distinction between the first and second), and unrecognized objects are Ob1, Ob2, Ob3,. Here, for convenience of explanation, the moving object M, the moving object candidate C, the unrecognized object Ob, and the like are shown separately. However, in actual processing, the IDs and tags assigned to the moving object, the moving object candidate, and the unrecognized object are These may be serial numbers without distinction, or may be numbers or codes based on time, and are not limited.
A processing procedure of the object tracking method will be described.

  4 to 6 are flowcharts for explaining the processing procedure of the object tracking method of the present embodiment.

  This processing procedure is performed by the object tracking device 102. In the following description, as already defined, an unrecognized object is an object that is detected in the current frame and is not classified in any way.

  First, measurement data, that is, the current frame is acquired from the laser radar 101 (S1). In S <b> 1, it is assumed that the moving body recognized in the past frame, the first moving body candidate, and the second moving body candidate are listed according to the processing procedure described here.

  Subsequently, an unrecognized object is detected from the acquired current frame (S2). In the detection of the unrecognized object, for example, a difference between a predetermined reference frame and the current frame is obtained, and an object that exists only in the current frame is determined as an unrecognized object. The reference frame may be, for example, any of the past frames (for example, the previous frame), or a frame having only a background without a moving object may be prepared (a frame without a moving object that is found and selected by the user visually) Such). Note that such a reference frame is stored in the HDD 14 and is expanded in the RAM 12 during operation and used for comparison with the current frame.

  In S2, if there are a plurality of unrecognized objects, all of them are detected. The detected unrecognized object is given an ID or a label necessary for processing (for example, unrecognized objects Ob1, Ob2, Ob3,...).

  In S2, the detected objects may be clustered to select whether they are unrecognized objects that can be moving objects. For example, the objects in the measurement data by the laser radar 101 are clustered to a certain size (number of pixels). If there are multiple objects, clustering is performed for each object. And if it is more than the predetermined size threshold value, it will be set as an unrecognized object. The size threshold may be arbitrarily determined depending on the monitoring location and the monitoring target. For example, if a person is tracked and monitored, the normal minimum size of the person may be set as a size threshold for clustering. On the contrary, if all objects are tracked, the size threshold may be a small value, for example, about 1 to several pixels, or the size threshold may not be provided.

  Subsequently, as a result of S2 (or after returning from a step described later), if there is no unrecognized object (S3: NO), the process proceeds to S40 (FIG. 6), whereas if there is an unrecognized object (S3: YES). Subsequently, one of the unrecognized objects is selected (S4). This is for checking the unrecognized objects one by one when a plurality of unrecognized objects are detected in S2. The order of selection is not particularly limited. For example, the order of the IDs assigned in S2 is set, or the coordinate system is set in the frame and the order is closer to the origin (the one with the smaller coordinate value).

  Subsequently, it is confirmed whether or not there is an unsupported moving object in the moving object list (S5). An unsupported moving object is a moving object that is not associated with an unrecognized object in this processing procedure. If there is no unsupported moving object in the moving object list (S5: NO), the process proceeds to S8. On the other hand, if there is an unsupported moving object in the moving object list (S5: YES), then the unsupported moving object in the moving object list (hereinafter simply referred to as a moving object) and the unrecognized object selected in S4 Is checked (S6).

  The process of checking the correspondence between the moving object and the unrecognized object is performed by checking the degree of association between them. The relevance will be described.

  The degree of association can be examined using, for example, the distance, the moving speed, the distance from the sensor, the size, the number of pixels, the color, and the like as an index.

  When the image acquisition sensor is the laser radar 101, the distance is the degree of relevance between the moving object in the previous frame and the unrecognized object among the moving objects. In the case of the laser radar 101, the distance between the moving body of the previous frame and the unrecognized object can be obtained from the measurement data, so that the position of the object in three dimensions can be determined. The position of the moving object and the position of the unrecognized object are known as coordinate values. Therefore, the distance between them is obtained as a difference in coordinate values. In the case of a movie camera, the distance between the moving object of the previous frame and the unrecognized object in the frame is compared, and the value is used as the relevance. In the case of a movie camera image, a two-dimensional coordinate system is set for the frame, and the difference between coordinate values can be obtained in the same manner as described above. In any case, the closer the distance (the smaller the value), the higher the degree of association. In the following description, “position” is also a coordinate value.

  The moving speed compares the moving speed of the moving object and the moving speed of the unrecognized object, and uses the value as the relevance level. The moving speed of the moving body is obtained by dividing the moving distance of the moving body by the time (time of one frame), for example, because the difference between the position of the moving body in the previous frame and the position of the moving body in the previous frame is the moving distance of the moving body. As for the moving speed of the unrecognized object, assuming that the moving object whose correspondence is being examined is the same object as the unrecognized object, the difference between the position of the previous frame of the moving object and the position of the unrecognized object is expressed as time (one frame). Divide by (time). The closer the moving speed, the higher the relevance.

  For the distance from the sensor, the distance from the laser radar 101 to the moving object in the previous frame is compared with the distance from the laser radar 101 to the unrecognized object, and the value is used as the relevance. The closer the distance values are, the higher the degree of association.

  For the size, the size of the moving object in the previous frame measured by the laser radar 101 and the size of the unrecognized object are compared, and the value is used as the relevance. The closer the size value, the higher the relevance.

  For the number of pixels, the number of pixels occupied by the moving object and the unrecognized object in the previous frame captured by the movie camera are compared, and the value is used as the relevance level. The closer the number of pixels, the higher the degree of association.

  For the color, the color of the moving object in the previous frame shot by the color movie camera is compared with the color of the unrecognized object, and the value is used as the relevance. For color, for example, a color difference based on color information such as a Lab value may be compared, or a difference in RGB gradation values may be simply compared. The smaller the color difference, the higher the relevance. Alternatively, the most number of colors (number of pixels), an average value of colors, or the like may be used.

  These indicators may be used alone or in combination. When a plurality of indices are used in combination, for example, in the case of a color movie camera (two-dimensional image), the degree of association is examined using distance and color. In this case, the degree of association is examined by weighting each distance and color. For example, “distance × 0.6 + color difference × 0.4”. The weighting constant used here is a value obtained through experience values or experiments. In addition, when using a movie camera, the distance and the number of pixels, the moving speed and the color, the number of the pixels and the color, etc. are combined in various ways, and the degree of relevance is examined by weighting each with an experience value or an experimental value.

  In the case of the laser radar 101, for example, the degree of association is checked using the distance and the size. In the case of the laser radar 101, the distance from the sensor and the size may be combined. Further, the moving speed and size may be used. By investigating the degree of association using a plurality of indices, a more accurate degree of association can be obtained.

  Here, when there is one moving object and one unrecognized object in the previous frame, it may be determined that they are related without checking the degree of association. This is because if there is one moving object and one unrecognized object, it is very likely that the moving object in the previous frame has been detected as an unrecognized object in the current frame. However, there is a possibility that the unrecognized object is noise or appears suddenly. Therefore, it is preferable to check the degree of association even if there is one moving object and one unrecognized object in the previous frame. Then, when the relevance is equal to or greater than a predetermined threshold, they are associated with each other as being the same. In this case, the predetermined threshold value is different for each index, but it may be obtained in advance as an empirical value or an experimental value.

  On the other hand, when there are two or more moving objects in the moving object list, the degree of relevance between them is examined and a combination with a high degree of association is associated.

  For example, when there are two moving objects M1 and M2 in the moving object list at S6, the degree of association between the moving object M1 and the unrecognized object Ob1, and the degree of association between the moving object M2 and the unrecognized object Ob1, are examined, respectively. The higher combination is associated.

  Note that when there are a plurality of moving objects, the correspondence with which moving object is checked first, for example, the degree of association between the old moving object and the unrecognized object is checked first in time series. When first checking the degree of association between an old moving object and an unrecognized object in time series, the time (time) when the moving object was detected as an object or the elapsed time since the detection was given as a tag, etc. Keep it. The given time (time) is, for example, real time. The elapsed time may be the time itself, or the number of frames after detection may be counted. Note that “detected as an object” is a time point when an object (ie, an unrecognized object) is detected in the current frame.

  Alternatively, for example, a coordinate system may be set for the frame (a three-dimensional coordinate system for the laser radar 101 frame and a two-dimensional coordinate system for the movie camera frame), and the first closer to the origin may be examined first. Furthermore, you may investigate not only in these but arbitrary orders.

  Returning to the flowchart, the description will be continued. As a result of examining the correspondence between the moving object and the unrecognized object in S6, if it is determined that they correspond (S7: YES), the moving object is associated with the unrecognized object, and the moving object is associated in the moving object list. It records and updates with a tag etc. (S30). Then, it returns to S3 and continues a process.

  On the other hand, if it is determined in S7 that the moving object does not correspond to the unrecognized object (S7: NO), then it is confirmed whether or not there is a first moving object candidate in the first moving object candidate list (S8), If there is no 1st moving body candidate (S8: NO), it will progress to S11 (refer FIG. 5). On the other hand, if there is a first moving object candidate in the first moving object candidate list (S8: YES), the correspondence between the first moving object candidate and the unrecognized object selected in S4 is examined (S9). Here, as in S6, the degree of association between the first moving object candidate and the unrecognized object is obtained to check whether it corresponds. Since the method of checking the relevance is merely the change of the moving object to the first moving object candidate in S6, the description is omitted.

  As a result of S9, if it is determined that the first moving object candidate corresponds to the unrecognized object (S10: YES), the associated first moving object candidate is changed to a moving object and moved to the moving object list (S31). ). That is, at this stage, since the first moving object candidate recognized for two consecutive frames is recognized as the same as the unrecognized object, it is recognized for three consecutive frames. Therefore, the first moving object candidate is changed to a moving object. Then, it returns to S3 and continues a process.

  On the other hand, if it is determined as a result of S9 that the first moving object candidate and the unrecognized object do not correspond (S10: NO), then (see FIG. 5 hereinafter), the second moving object candidate is in the second moving object candidate list. If there is no second moving object candidate (S11: NO), the process proceeds to S15. On the other hand, if there is a second moving object candidate in the second moving object candidate list (S11: YES), the correspondence between the second moving object candidate and the unrecognized object selected in S4 is examined (S12). Here, as in S6, the degree of association between the second moving object candidate and the unrecognized object is obtained to check whether it corresponds. Since the method of checking the degree of association, etc. is merely the change of the moving object in S6 to the second moving object candidate, the description thereof is omitted.

  As a result of S12, if it is determined that the second moving object candidate corresponds to the unrecognized object (S13: YES), the second moving object candidate is changed to the first moving object candidate and moved to the first moving object candidate list. (S32). That is, at this stage, since the second moving object candidate that has been recognized for only one frame is recognized as the same as the unrecognized object, it is recognized continuously for two frames. Therefore, the second moving object candidate is changed to the first moving object candidate.

  On the other hand, if it is determined as a result of S12 that the second moving object candidate and the unrecognized object do not correspond (S13: NO), then it is confirmed whether or not there is a lost moving object in the lost moving object list (S15). If there is no lost moving object in the lost moving object list (S15: NO), the unrecognized object selected in S4 is entered as the second moving object candidate in the second moving object candidate list (S33). On the other hand, if there is a lost moving object in the lost moving object list (S15: YES), the correspondence between the lost moving object and the unrecognized object is examined (S16). Here again, as in S6, the degree of association between the lost moving object and the unrecognized object is obtained to check whether it corresponds. Since the method of checking the relevance and the like is the same as in S6, the description is omitted.

  If it is determined as a result of S16 that the lost moving object corresponds to the unrecognized object (S17: YES), the lost moving object moves from the lost moving object list to the moving object list as a moving object (S18).

  As a result of S16, if it is determined that the moving object does not correspond to the unrecognized object (S17: NO), the unrecognized object selected in S4 is put in the second moving object candidate list as the second moving object candidate (S33). Then, it returns to S3 and continues a process.

  A case where there is no unrecognized object in S3 (S3: NO) will be described (see FIG. 6). If there is no unrecognized object, the correspondence between the unrecognized object and the moving object, the first and second moving object candidates, and the lost moving object is determined according to the case where the unrecognized object is not detected in the current frame. This is a case where no unrecognized object exists after the examination.

  If there is no unrecognized object in S3 (S3: NO), it is subsequently confirmed whether or not there is an unsupported moving object in the moving object list (S40).

  If there is an unsupported moving object in the moving object list (S40: YES), all the unsupported moving objects in the moving object list are moved to the lost moving object list as the lost moving object (S41). On the other hand, if there is no unsupported moving object in the moving object list (S40: NO), the process proceeds to S42 as it is.

  Then, the 1st moving body candidate and the 2nd moving body candidate in a 1st moving body candidate list | wrist and a 2nd moving body candidate list | wrist are deleted (S42), and a process is complete | finished. After the process is completed, the process returns to S1 and the next process is continued.

  In the processing procedure described above, when there are a plurality of unrecognized objects, unrecognized objects to be checked for correspondence are selected one by one (the process of S4). Instead, for a plurality of unrecognized objects at the same time (even if there is a time difference, as the same step), the correspondence is examined and processing corresponding to each correspondence, that is, moving object, moving object candidate, lost moving object, etc. These processes may be executed in parallel, or processes that do not correspond to any of them may be executed in parallel.

  Next, the object tracking method will be described with examples of moving objects, first moving object candidates, second moving object candidates, and unrecognized objects.

  FIG. 7 is an explanatory diagram illustrating examples of moving objects, first moving object candidates, and second moving object candidates recognized up to the previous frame. The state shown in the figure shows the previous frame at the time when the current frame is acquired in S1 of the above processing procedure. At this time, the moving body M1, the first moving body candidate C1, and the second moving body candidate C2 are recognized. FIG. 8 is a list diagram showing each list in FIG. The moving body M1, the first moving body candidate C1, and the second moving body candidate C2 are classified and registered. Here, there is no lost moving object in the lost moving object list.

  FIG. 9 is an explanatory diagram illustrating an example in which an unrecognized object is detected in the current frame. In the state shown in the figure, unrecognized objects Ob1 and Ob2 are detected in S2 in the current frame acquired in S1.

  FIG. 10 is an explanatory diagram in which the previous frame shown in FIG. 7 and the current frame shown in FIG. 9 are overlapped. In actual processing, such superposition (image synthesis) of the previous frame and the current frame is not necessary.

  In this example, in S6, the correspondence between the moving object M1 and the unrecognized object Ob1 is examined.

  As a result of the process of S6, it is assumed here that the moving object M1 corresponds to the unrecognized object Ob1. FIG. 11 is an explanatory diagram illustrating a state where the moving object M1 and the unrecognized object Ob1 correspond to each other. In FIG. 11, the moving body M1 and the unrecognized object Ob1 correspond (Ob1 = C2). In the figure, an arrow indicates that the moving body M1 has moved to the unrecognized object Ob1.

  At this stage, the unrecognized object Ob1 is associated with the moving object M1. Therefore, it is not necessary to examine the correspondence with the first moving object candidate C1, the second moving object candidate C2, and the like existing in the previous frame. Therefore, the calculation load for object tracking is reduced accordingly.

  In this regard, in the conventional method, an unrecognized object detected in the current frame and all objects recognized up to the previous frame (in this example, moving object M1, first moving object candidate C1, and second moving object candidate C2) We examine correspondence with brute force. FIG. 12 is an explanatory diagram illustrating an example in which a non-recognized object detected in the current frame and all objects recognized up to the previous frame are examined in a brute force manner as a comparative example.

  As shown in FIG. 12, in the comparative example using the conventional method, all correspondences between the unrecognized object Ob1 and the moving object M1, the first moving object candidate C1, and the second moving object candidate C2 are examined. As a result, when the second moving object candidate C2 is located closer to the moving object M1 relative to the unrecognized object Ob1, the second moving object candidate C2 that is near is erroneously recognized as the same as the moving object from the previous frame. In some cases, it is determined that the unrecognized object Ob1 and the second moving object candidate C2 correspond to each other (Ob1 = C2). In this case, the second moving object candidate C2 has moved to the unrecognized object Ob1 as indicated by the arrow in the figure. In addition, it takes a lot of time to check the correspondence between the unrecognized object and all the objects in the past frame. In the description of the comparative example, for the sake of clarity, the moving object M1, the first moving object candidate C1, the second moving object candidate C2, and the like are classified and shown. However, the object recognized in the past frame in the conventional method There is no classification for.

  On the other hand, in this embodiment, since the correspondence with the moving object recognized continuously for a predetermined time (first predetermined time) or longer in the past frame is checked first, there is no erroneous recognition as in the comparative example. In the present embodiment, since correspondence with an object that has already been recognized as a moving object from the past frame (that is, an object that is classified as a moving object here) is first examined, there is a possibility that an unrecognized object may correspond. Get higher. Then, it is not necessary to check the correspondence with other objects. In particular, since it is not necessary to examine the correspondence with an object that appears only for a short time, such as one frame or two frames, the calculation load is reduced accordingly, and the processing time is also shortened.

  Returning to the case of FIG. After the correspondence result of the unrecognized object Ob1 is obtained, the correspondence of the unrecognized object Ob2 is continuously examined. At this stage, since the moving object M1 has already been associated, the process proceeds to S8. In S8, the correspondence between the unrecognized object Ob2 and the first moving object candidate C1 is examined. As a result, it is assumed here that the unrecognized object Ob2 corresponds to the first moving object candidate C1. FIG. 13 is an explanatory diagram illustrating a state where the unrecognized object Ob2 and the first moving object candidate C1 correspond to each other. In FIG. 13, the first moving object candidate C1 that has been recognized continuously for two frames is further recognized to be the same as the unrecognized object Ob2 here, so that it is recognized continuously for three frames. As a result, the first moving object candidate C1 becomes the moving object M2. As indicated by the arrows, the first moving object candidate C1 has moved to the moving object M2.

  Thereafter, when the processing further proceeds, since there is no corresponding object in the second moving object candidate C2, it is secondly deleted from the moving object candidate list. At this time, the second moving object candidate C2 is, for example, noise.

  FIG. 14 is a list diagram showing each list when the processing has progressed to the end. As illustrated, the moving bodies M1 and M2 are registered in the moving body list, and the first moving body candidate list and the second moving body candidate list are not registered. The lost moving body list is not originally registered.

  FIG. 15 is an explanatory diagram illustrating the current frame at the end of processing. As shown in FIG. 15, at the end of the process, moving objects M1 and M2 are recognized in the current frame.

(Laser radar 101)
Next, the laser radar 101 will be described.

  The laser radar 101 is also referred to as a lidar (LiDAR (Light Detection And Ranging)), and measures the time (TOF (Time of Flight)) until the laser is launched and reflected back to the object. Find the distance. By performing this while scanning within a certain range, a three-dimensional distance image can be obtained. FIG. 16 is a diagram illustrating an example of a three-dimensional distance image obtained by the laser radar 101. Each such distance image becomes a frame and becomes a moving image by being continuous in time series.

  In the distance image, a three-dimensional image in which the distance from the installation position of the laser radar 101 is found is obtained as illustrated. Here, a plurality of moving bodies M1, M2, M3, etc. are shown. In addition, stationary objects St1, St2, and St3 are also shown. In these, the measurement result by the laser radar 101 is drawn and imaged. In this image example, information representing a moving object is displayed superimposed on an object recognized as a moving object. Here, the object (moving object) is surrounded by a rectangular parallelepiped frame (frame of white line in the figure) as information indicating that it is a moving object. This makes it easier to understand an object that is recognized as a moving object when the user views the screen (moving image). Such a frame line may not be provided, and it may be added or deleted according to a command from the user. Further, the information indicating that the object is a moving object is not limited to a frame line, and may be an arbitrary mark such as a triangle or an arrow, or character information.

  FIG. 17 is a cross-sectional view illustrating the configuration of the laser radar 101.

  A laser radar 101 shown in FIG. 17 is an example used as a monitoring device MD. Although the monitoring device MD is shown in a state where it is attached to the inclined wall surface WL, the shape and length of the components may differ from the actual ones. In FIG. 17, it is assumed that the monitoring device MD is installed with the top and bottom reversed.

  The monitoring device MD includes, for example, a pulsed semiconductor laser LD that emits a laser beam, a collimating lens CL that converts divergent light from the semiconductor laser LD into parallel light, and laser light that is collimated by the collimating lens CL. A mirror unit MU that scans and projects light toward the monitoring space by a rotating mirror surface and reflects reflected light from the unrecognized object, and a lens that collects reflected light from the unrecognized object reflected by the mirror unit MU. LS, a photodiode PD that receives light collected by the lens LS, and a processing circuit (processing unit) PROC that obtains distance information according to the time difference between the emission timing of the semiconductor laser LD and the light reception timing of the photodiode PD; , A motor MT that rotationally drives the mirror unit MU, and a housing CS that houses these motors To. The photodiode PD has a plurality of pixels arranged in the Y direction.

  The semiconductor laser LD and the collimating lens CL constitute an emission part LPS, the lens LS and the photodiode PD constitute a light receiving part RPS, the mirror unit MU constitutes a scanning part, and these constitute a light projecting / receiving unit. .

  The optical axes of the emission part LPS and the light receiving part RPS are preferably orthogonal to the rotation axis RO of the mirror unit MU.

  The box-shaped casing CS fixed to the wall surface WL or the like includes an upper wall CSa, a lower wall CSb facing the upper wall CSa, and a side wall CSc connecting the upper wall CSa and the lower wall CSb. An opening CSd is formed in a part of the side wall CSc, and a transparent plate TR is attached to the opening CSd.

  The mirror unit MU has a shape in which two quadrangular pyramids are joined in opposite directions, that is, four pairs of mirror surfaces MR1 and MR2 tilted in a direction facing each other in pairs (but not limited to four pairs). ) The mirror surfaces MR1 and MR2 are preferably formed by depositing a reflective film on the surface of a resin material (for example, PC (Polycarbonate)) in the shape of a mirror unit.

  The mirror unit MU is connected to a shaft MTa of a motor MT fixed to the casing CS and is driven to rotate. Here, the axis (rotation axis) of the axis MTa extends in the Y direction inclined with respect to the vertical direction, and the ZX plane perpendicular to the Y direction and the ZX plane formed by the X direction are inclined with respect to the horizontal plane. However, the axis of the axis MTa may coincide with the vertical direction.

  An unrecognized object detection principle of the monitoring device MD will be described. In FIG. 17, divergent light intermittently emitted in a pulse form from the semiconductor laser LD is converted into a parallel light beam by the collimator lens CL, enters the first mirror surface MR1 of the rotating mirror unit MU, and is reflected there. Further, after being reflected by the second mirror surface MR2, it is scanned and projected as a laser spot light having, for example, a vertically long rectangular cross section through the transparent plate TR and directed to the external monitoring space. The direction in which the emitted laser spot light is reflected by an unrecognized object and returned as reflected light is referred to as a light projecting / receiving direction. Laser spot light beams traveling in the same light projecting / receiving direction are detected by the same pixel.

  FIG. 18 is an explanatory diagram for explaining a state in which the monitoring space of the monitoring device MD is scanned with the laser spot light SB (shown by hatching) emitted according to the rotation of the mirror unit MU.

  Here, in the combination of the first mirror surface MR1 and the second mirror surface MR2 of the mirror unit MU, the crossing angles are different.

  The laser light is sequentially reflected by the rotating first mirror surface MR1 and second mirror surface MR2. First, the laser light reflected by the first pair of the first mirror surface MR1 and the second mirror surface MR2 moves from the left to the right in the horizontal direction in the uppermost area Ln1 in accordance with the rotation of the mirror unit MU. Is scanned.

  Next, the laser light reflected by the second pair of the first mirror surface MR1 and the second mirror surface MR2 moves from the left in the horizontal direction to the second region Ln2 from the top of the monitoring space according to the rotation of the mirror unit MU. Scan to the right.

  Next, the laser light reflected by the third pair of the first mirror surface MR1 and the second mirror surface MR2 passes through the third region Ln3 from the top of the monitoring space in the horizontal direction from the left in accordance with the rotation of the mirror unit MU. Scan to the right.

  Next, the laser light reflected by the fourth pair of the first mirror surface MR1 and the second mirror surface MR2 moves from the left to the right in the horizontal direction in the lowest region Ln4 in the monitoring space according to the rotation of the mirror unit MU. Is scanned.

  Thus, one scan of the entire monitoring space that can be monitored by the monitoring device MD is completed. As described above, the laser spot light beam is scanned two-dimensionally without any gap (when the trajectories of the scanned laser spot light beam are adjacent to each other (for example, the region Ln1 and the region Ln2), the adjacent locus is in contact with no gap. , Including a case where a part of the images overlaps each other), a distance image that allows the user to intuitively grasp the space intuitively is preferable when setting the monitoring device MD.

  One frame FL is obtained by combining images obtained by scanning the regions Ln1 to Ln4. If the first pair of the first mirror surface MR1 and the second mirror surface MR2 return after one rotation of the mirror unit MU, the region from the uppermost region Ln1 to the lowermost region Ln4 is again displayed. Scanning is repeated to obtain the next frame FL.

  Referring to FIG. 17, a part of the laser beam reflected by the unrecognized object among the light beams projected and transmitted again passes through the transparent plate TR and the second mirror surface MR2 of the mirror unit MU in the housing CS. , And is reflected by the first mirror surface MR1, collected by the lens LS, and detected for each pixel on the light receiving surface of the photodiode PD.

  Further, a processing circuit (not shown) obtains distance information according to the time difference between the emission timing of the semiconductor laser LD and the light reception timing of the photodiode PD. As a result, an unrecognized object is detected in the entire area in the monitoring space, and a distance image (also referred to as a measurement point marker group) having distance information (three-dimensional measurement value) for each pixel (measurement point). A frame FL (see FIG. 18) can be obtained. The shape of the distance image is the same as the shape of the spot light beam SB that is actually scanned. The distance image is sent from the processing circuit to the object tracking device 102 to perform object tracking. The distance image is sent to the server 103 and stored as necessary.

  According to the embodiment described above, the following effects can be obtained.

  (1) In this embodiment, an object that has been detected for a predetermined time or more until the previous frame is classified as a first type object, and an object that is less than a predetermined time is classified as a second type object. In the present embodiment, the first type object is referred to as a moving object, and the second type object is referred to as a moving object candidate. Then, it is checked whether the unrecognized object detected in the current frame corresponds to the moving object first. When an unrecognized object corresponds to a moving object, the unrecognized object is recognized as a moving object by associating with the moving object, and if the unrecognized object does not correspond to the moving object, the correspondence with the moving object candidate is examined. As a result, if the moving object is associated with the unrecognized object, the moving object candidate need not be examined. For this reason, the calculation load can be reduced correspondingly, and the processing time can be shortened. Also, noise such as one frame or two frames that appears only for a very short time, or shaking of trees is evaluated later as a moving object candidate. For this reason, such noise and the like are not mixed into the entire object tracking data.

  (2) A moving object candidate associated with an unrecognized object is classified as a moving object if the elapsed time from the detection as an object has passed a predetermined time or more. Thereby, when the same object is recognized over time, the moving object candidate can be classified as a moving object and the object can be tracked.

  (3) The moving object candidates are classified into a plurality of stages, such as a first moving object candidate, a second moving object candidate, and the like in order from the longer elapsed time (the one with the larger number of frames) after the detection as an object. Then, the correspondence with unrecognized objects is examined in order from the longer elapsed time (the one with the larger number of frames). Thereby, the amount of calculation can be reduced more finely and the influence of noise can be reduced.

  (4) The moving object is also classified into a plurality of stages in order from the longest elapsed time after detection as an object (the one with the largest number of frames), and the correspondence with the unrecognized object in the order from the longest elapsed time. I decided to investigate. Thereby, it becomes possible to track in order from an object recognized as a moving object for a longer time.

  (5) Furthermore, in this embodiment, moving objects that do not correspond to unrecognized objects are classified as third type objects. Again, in the present embodiment, the third type object is referred to as a lost moving object. When an unrecognized object corresponds to a lost moving object, it is classified as a moving object. Thereby, the object temporarily hidden in the frame can be recognized as the same moving object when it reappears.

  (6) If it is determined that the unrecognized object does not correspond to any moving object candidate, the unrecognized object is classified as a moving object candidate as newly appearing in the current frame. As a result, an object that appears only in the current frame is classified as a moving object candidate, and if the same object is not recognized over time, it can be eliminated without being a moving object.

  (7) By setting the predetermined time to not less than 0.2 seconds and less than 1 second, noise that appears and disappears for a very short time, and flowers and trees that live by shaking with the wind can be prevented from being recognized as moving objects.

  (8) Time is counted in units of frames. Thereby, the elapsed time can be understood by counting the frames. In addition, since the recognition of the object in the frame is performed in units of frames, the time interval for recognizing the object can be counted with a good time.

  (9) The recognized moving object is displayed together with information indicating that it is a moving object. This makes it easier to understand the recognized moving object when the user looks at the screen.

  Although the embodiments to which the present invention is applied have been described above, the present invention is not limited to these embodiments.

  As the image acquisition sensor, an ordinary movie camera can be used instead of the laser radar 101. When a movie camera is used, three-dimensional distance information cannot be obtained, but it is possible to recognize how long (number of frames) an object exists in a frame. Accordingly, object tracking can be performed from a moving image (frames arranged in time series) using a movie camera, as in the above-described embodiment (except for three-dimensional measurement).

  In the embodiment, the moving object candidate (second type object) is in two stages: the first moving object candidate and the second moving object candidate. That is, it is associated with two elapsed times (the number of frames) after detection as an object. However, this may be one stage associated with one elapsed time, or may be n stages (nth moving body candidates (where n is a natural number)) associated with a plurality of elapsed times.

  Also, moving objects may be managed in multiple stages depending on the elapsed time (number of frames). For example, the first to nth moving bodies (where n is a natural number) are set for each elapsed time (number of frames) since the object is detected. By doing so, it is possible to examine the correspondence with unrecognized objects in order from the moving object that has been recognized in the past frame for the longer time (the number of frames is larger).

  In addition, the present invention can be variously modified based on the configurations described in the claims, and these are also within the scope of the present invention.

11 CPU,
12 RAM,
14 HDD,
101 laser radar,
102 Object tracking device.

Claims (11)

A first-type object that is detected in a moving image in which a plurality of frames are arranged in time series and has elapsed for a predetermined time or more after being detected. And classifying the object of less than the predetermined time as the second type object (a),
(B) examining the correspondence between the unrecognized object and the first type object, using the object detected in the current frame as an unrecognized object;
A step (c) of associating the unrecognized object and the first type object when it is determined in the step (b) that the unrecognized object and the first type object correspond to each other;
In the step (b), when it is determined that the unrecognized object and the first type object do not correspond to each other, a step (d) of checking the correspondence between the unrecognized object and the second type object;
In the step (d), when it is determined that the unrecognized object and the second type object correspond to each other, the step (e) of associating the unrecognized object with the second type object;
An object tracking method.   In the step (e), if the elapsed time after the second type object associated with the unrecognized object is detected as an object has passed the predetermined time, the second type object is changed to the first type. The object tracking method according to claim 1, further comprising a step (e1) of classifying the seed object. In the step (a), the second type object is associated with an elapsed time since it was detected as an object,
3. The object tracking method according to claim 1, wherein, in the step (d), the correspondence between the second type object and the unrecognized object is examined in order from the longest elapsed time after being detected as an object. In the step (a), the first type object is associated with an elapsed time since it was detected as an object,
The said 1st type object investigates a response | compatibility with the said unrecognized object in an order from the longest elapsed time after the said 1st type object is detected as an object in the said step (b). Object tracking method. In the step (a), the first type object that does not correspond to the unrecognized object in the step (b) is classified as a third type object,
In the step (d), when it is determined that the unrecognized object and the second type object do not correspond to each other, a step (f) of checking the correspondence between the unrecognized object and the third type object;
In the step (f), when it is determined that the unrecognized object and the third type object correspond to each other, the unrecognized object and the third type object are associated with each other, and the third type Reclassifying an object into the first type object (g);
In the step (f), when it is determined that the unrecognized object and the third type object do not correspond to each other, the step (h) of classifying the unrecognized object as the second type object;
The object tracking method according to claim 1, comprising:   In the step (d), when it is determined that the unrecognized object does not correspond to any of the second type objects, the method includes a step (d1) of classifying the unrecognized object into the second type object. The object tracking method according to any one of claims 1 to 4.   The object tracking method according to claim 1, wherein the predetermined time is not less than 0.2 seconds and less than 1 second.   The object tracking method according to claim 1, wherein the time is counted in units of the frame.   The object tracking method according to claim 1, further comprising a step (i) of displaying the first type object together with information indicating that the object is a moving object.   An object tracking program for causing a computer to execute the object tracking method according to claim 1. An image acquisition sensor that acquires frames in chronological order;
A computer that executes the program according to claim 10;
An object tracking system.