JP7161981B2 - Object tracking program, device and method capable of switching object tracking means - Google Patents

Description

The present invention relates to techniques for tracking an object through an image that may contain the object.

At present, with the widespread use of surveillance cameras, in-vehicle cameras, and the like, attention is focused on techniques for collecting video data captured and generated by such cameras and creating new information therefrom. For example, the development of technology for generating information related to the movement trajectory of a predetermined target from surveillance camera video data, and for generating driving support information and control management information for implementing automatic driving from in-vehicle camera video data, is progressing actively. It is

Here, in particular, with regard to the technology for identifying and tracking a predetermined target, it is important to solve problems such as improving the recognition accuracy and continuously grasping the predetermined target without mistaking it.

In this respect, for example, in Patent Document 1, positional information data of the preceding vehicle is acquired using a radar sensor, camera, etc., and based on the error of this data, the preceding vehicle is stored and used to calculate the traveling trajectory of the preceding vehicle. The distance range (buffering range D) of the position information data of the preceding vehicle is determined, and the position of the preceding vehicle is stored at the storage time (buffering timing) of the position information data of the preceding vehicle determined based on the buffering range D. A preceding vehicle travel trajectory calculation device is disclosed that stores information data and calculates the travel trajectory of a preceding vehicle using position information data within a buffering range D of the stored position information data of the preceding vehicle. .

Here, according to this preceding vehicle travel locus calculation device, by appropriately setting the buffering range D and the timing of the position information data of the preceding vehicle, the calculation load is not excessively increased, and the accuracy of the travel locus calculation is improved. It is said that it is possible to prevent the decline of

In addition, in Patent Document 2, a vehicle that is about to pass a forward point is detected from detection data by a vehicle detection sensor such as a camera, and vehicle identification information is added, and the vehicle passes from the forward point to the rear point. During this time, the vehicle is tracked while the vehicle identification information is attached, and the front license plate information acquired from the image data of the front camera when the vehicle enters a point ahead is associated with the vehicle identification information. Similarly, when the vehicle enters the rear point, the rear license plate information and the vehicle identification information are stored in association with each other, the vehicle identification information is compared, and the same vehicle identification information is assigned. A license plate information acquisition device for associating front license plate information and rear license plate information is disclosed.

Furthermore, in Patent Document 3, the coordinates within the image of a plurality of persons in the image are determined, a group area is specified as an area in which the persons in the image are densely packed, and the coordinates within the image of a certain person are tracked. Also disclosed is a person tracking device that determines the trajectory of coordinates in a certain person's group area based on the coordinates in the person's group area and the numbers written on the clothes. Here, this person tracking device is said to be effective in tracking a person with high accuracy.

JP 2016-206976 A WO2008/010308 WO2016/139906

As described above, conventionally, in the technology for recognizing and tracking a predetermined target, predetermined target detection means such as a radar sensor, a camera, etc., such as the preceding vehicle travel locus calculation device described in Patent Document 1, is used. It is fundamental to analyze the data obtained by

In addition, conventionally, for example, like the technology described in Patent Documents 2 and 3, the license plate attached to the vehicle and the number (for example, uniform number) written on the clothes of the person are recognized, and the target (automobile or human) is recognized. There are also techniques for tracking

However, certain parts of the tracked object, such as license plates and jersey numbers, cannot be "seen" by predetermined means such as radar sensors and cameras, depending on the relative positional relationship with the tracked object and the orientation and posture of the tracked object. There are quite a few situations where it is not possible. For example, if the target vehicle is a vehicle that is driving in the overtaking lane and is about to overtake, even if a tracking camera is installed on the side of the vehicle being tracked, the camera will detect the license plate of the vehicle. becomes very difficult to recognize.

Also, if the overtaking vehicle continues to move away in front, it will be difficult to recognize the license plate, resulting in a problem of tracking accuracy.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an object tracking program, apparatus, and method capable of performing more appropriate tracking according to the situation of the object to be tracked.

According to the present invention, a program for operating a computer that tracks a movable object by means of an image that may contain the object, comprising:
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . a first object tracking means capable of tracking the object using the discriminator of
a second object tracking means capable of tracking the object using a second discriminator constituted by a machine learning algorithm for image recognition, capable of identifying the object from pixel information related to the object in the image; ,
Object tracking switching means for causing second object tracking means to track the object when tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short period of time An object tracking program is provided that causes a computer to function as a

Furthermore, according to the present invention, a program for operating a computer that tracks a movable object by means of an image that may contain the object, comprising:
The image is generated by one of a plurality of cameras with different shooting ranges,
a first object tracking means capable of tracking the object using a first classifier capable of identifying a specific portion of the object in the image;
a second object tracking means capable of tracking the object using a second classifier capable of identifying the object from pixel information related to the object in the image;
object tracking switching means for causing a second object tracking means to track the object when the first object tracking means becomes unable to track the object or is predicted to become impossible after a predetermined short period of time; ,
An image that enables or is suitable for tracking an object when the object cannot be tracked by the image produced by a single camera or is predicted to become incapable after a predetermined short period of time . and an image acquisition means for acquiring the image generated by another camera
to make the computer work ,
When the object tracking switching means predicts that the object tracking by the first object tracking means based on the image generated by the different camera is impossible or will be impossible after a predetermined short time, the second subject tracking means to perform tracking of the subject based on the image produced by the separate camera
There is provided an object tracking program characterized by :
Here, as an embodiment of the object tracking program according to the present invention, the object tracking switching means is configured such that when the second object tracking means has tracked the object based on the image generated by a certain camera, or when it is predicted that the first object tracking means will be able to track the object in the image generated by a camera different from the one camera, or that it will be possible after a predetermined short period of time; When the image generated by the camera indicates that the tracking of the object by the second object tracking means has become impossible or is predicted to become impossible after a predetermined short period of time, the camera for the first object tracking means It is also preferred to have the tracking of the object performed on the basis of the image produced by a camera separate from the camera.

Further, in the above-described embodiment, the cameras are four cameras provided on one moving body that can respectively photograph the front, right side, left side, and rear, and the specific part of the target is the target A display containing at least one of letters, numbers, symbols, and graphics associated with another moving body that is a display for specifying information related to this other moving body preferable.

Furthermore, in the embodiment using the above four cameras, the first object tracking means can detect the position of the display object placed in front of the other moving object in the image generated by the camera capable of photographing the rear. , tracks this other moving object approaching from behind, and
The second object tracking means approaches and overtakes the other moving object from the right side or the left side based on the pixel information related to the other moving object in the image generated by the camera capable of photographing the right side or the left side. track the moving objects of
The first object tracking means detects the position of the display object installed behind the other moving object in the image generated by the camera capable of photographing the front, and the other moving object appearing in front after overtaking. Body tracking is also preferred.

Further , according to the present invention, a program that causes a computer to track a movable object with an image that may include the object,
a first object tracking means capable of tracking the object using a first classifier capable of identifying a specific portion of the object in the image;
a second object tracking means capable of tracking the object using a second classifier capable of identifying the object from pixel information related to the object in the image;
third object tracking means capable of tracking the object in the image based on the position in the image of the bounding box of the object detected in the image ;
When the tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short time, the second object tracking means is caused to track the object, and the second object is detected. When the object has been tracked by the tracking means, and the tracking of the object by the second object tracking means has become impossible because the area related to the pixel information of the object has become smaller than a predetermined size. , object tracking switching means for causing the third object tracking means to track the object;
An object tracking program is provided, characterized in that it causes a computer to function by :

Here, in the object tracking program including the third object tracking means, the image is generated by a camera capable of photographing the front provided on one moving body, and the third object tracking means It is also preferable to track other moving bodies moving away in front of and after passing this one moving body based on the position within the image of the bounding box associated with the other moving body detected in the image.
Further, in the object tracking program according to the present invention described above, the object tracking switching means causes the object to be tracked by the first object tracking means when the object has been tracked by the second object tracking means. has become possible or is predicted to be possible after a predetermined short time, or tracking of the object by the second object tracking means has become impossible or is predicted to be impossible after a predetermined short time, It is also preferred to cause the first object tracking means to perform the tracking of the object.

According to the invention, there is also a device for tracking a movable object by means of an image, which may contain a movable object, comprising:
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . a first object tracking means capable of tracking the object using the discriminator of
a second object tracking means capable of tracking the object using a second discriminator constituted by a machine learning algorithm for image recognition, capable of identifying the object from pixel information related to the object in the image; ,
Object tracking switching means for causing second object tracking means to track the object when tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short period of time There is provided an object tracking device comprising:

According to the present invention, there is further provided a method of object tracking in a computer for tracking an object through an image that may include a movable object, comprising:
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . tracking the object using the discriminator of
For image recognition that can identify the target from the pixel information of the target in the image when the tracking of the target by the first discriminator becomes impossible or is predicted to become impossible after a predetermined short time . and tracking the object using a second discriminator configured by the machine learning algorithm of .

According to the object tracking program, device and method of the present invention, more appropriate tracking can be performed according to the situation of the object to be tracked.

1 is a schematic diagram and a functional block diagram for explaining an embodiment of an object tracking system provided with an object tracking server and an image data providing client according to the present invention; FIG. FIG. 4 is a schematic diagram for explaining an embodiment of object tracking processing by the first object tracking unit according to the present invention; FIG. 10 is a schematic diagram for explaining an embodiment of object tracking processing by a third object tracking unit according to the present invention; 1 is a schematic diagram for explaining an embodiment of an object tracking method according to the present invention; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Object tracking system]
FIG. 1 is a schematic diagram and a functional block diagram for explaining an embodiment of an object tracking system provided with an object tracking server and an image data providing client according to the present invention.

The object tracking system of this embodiment shown in FIG.
(a) at least one terminal 20 which is a mobile client;
(b) It has a cloud server 1 which is a server capable of acquiring image data from the terminal 20, and in this cloud server 1, a predetermined target is identified and tracked from the acquired image data.

Here, the terminal 20 of (a) above is a drive recorder having a communication function and is installed in the automobile 2 in this embodiment. In addition, the terminal 20 includes, in this embodiment, four cameras installed in the automobile 2 and having different photographing ranges.
A front camera 202c capable of photographing the front of the automobile 2 (direction of travel when the automobile 2 is traveling straight ahead),
a rear camera 202a capable of photographing the rear of the automobile 2,
A right side camera 202b capable of photographing the right side of the automobile 2, and a left side camera 202d capable of photographing the left side of the automobile 2
For example, these cameras can capture the 360-degree surroundings of the automobile 2, generate image (video) data, and store them in their own memory or storage. It is a thing.

Furthermore, the terminal 20 is capable of wireless communication connection with the cloud server 1 via, for example, a mobile phone communication network, the Internet, etc., and a part of the image (video) data generated and stored by each camera is captured. Alternatively, all of them can be transmitted to the cloud server 1 as appropriate or according to instructions.

On the other hand, the cloud server 1 in (b) above acquires an image (data) that may include a movable tracked object (for example, the overtaking car 3 in FIG. 1) from the terminal 20, and uses the image to track It is a device that tracks the target (car 3),
(A) Using a first discriminator capable of identifying a "specific part" (for example, a license plate) of the tracked target (car 3) in the image, a first discriminator capable of tracking the tracked target (car 3) in the image a target tracking unit 114 of
(B) Using a second classifier capable of identifying the tracked object (vehicle 3) from pixel information related to the tracked object (vehicle 3) in the image, a second discriminator capable of tracking the tracked object (vehicle 3) in the image 2 object tracking unit 115,
(C) Object tracking that causes the second object tracking unit 115 to track the tracked object (car 3) when the first object tracking unit 114 cannot track the tracked object (car 3) or becomes inappropriate. It is characterized by having a switching unit 113 .

Here, the "specific part" (eg, license plate) in (A) above may be used for tracking depending on the relative positional relationship with the tracked target (eg, car 3) and the orientation and attitude of the tracked target (car 3). There are quite a few situations in which it is not possible to "see" a camera that uses the captured image as an image.

For example, if the vehicle 3 that is driving in the overtaking lane and is about to overtake is the object to be tracked, and the specific part is the license plate attached to the front and rear of the vehicle, the camera used for tracking is likened to: Even with the right side camera 202b of the automobile 2 in which the terminal 20 is mounted, it is very difficult to recognize the license plate by the camera. Furthermore, if the vehicle 3 that has finished overtaking moves away in front of the vehicle 2, even if the camera used for tracking is the front camera 202c, it is still difficult to recognize the license plate. becomes.

On the other hand, as described in (C) above, when the first object tracking unit 114 cannot track a tracked object (for example, the car 3) or becomes unsuitable for tracking, the cloud server 1 causes the second object tracking unit 115 can be made to track the tracked object (car 3).

Here, the object tracking processing by the second object tracking unit 115 uses a second classifier capable of identifying the tracked object (automobile 3) from the pixel information related to the tracked object (automobile 3). This is typically more processing intensive compared to the tracking process by identification of "parts" (eg license plates). However, the robustness of the processing against changes in the relative positional relationship with the tracked object (automobile 3) and changes in the orientation/posture of the tracked object (automobile 3) is higher.

For example, even in a situation where the license plate as the "specific part" cannot be "seen", there are many cases where the vehicle 3 as the tracking target can be "seen". The second object tracking unit 115 can perform object tracking processing as long as a predetermined number of pixel value data or more can be obtained.

As described above, the cloud server 1 can perform more appropriate tracking according to the situation of the tracked target (for example, the automobile 3). The object tracking process can continue even if the identification of the license plate becomes impossible or difficult.

In addition, although described later in detail, "tracking is impossible or unsuitable" in the above (C) includes a situation in which identification processing of a specific part of the tracked object is actually impossible or difficult.

In addition, the "specific part" in (A) above is not limited to the license plate. For example, if the tracked object is a car, the emblem or logo mark attached to the vehicle body can be the specific part. is. Furthermore, when the object to be tracked is a player (person), the part where the number (for example, uniform number) attached to the clothes can be used as the "specific part". In this regard, for example, the "specific part" of the tracked object is a part containing at least one of letters, numbers, symbols and graphics, and the letters, numbers, It is also preferable to adopt at least one of symbols and graphics that can be identified. Furthermore, various objects can be set as objects to be tracked as long as they have such a "specific part".

Also, the cameras connected to the terminal 20 are not limited to the four cameras described above, and the number of cameras other than four may be used. Also, for example, it is possible to employ one or more wide-angle cameras equipped with wide-angle lenses, fish-eye lenses, or the like.

Furthermore, the terminal 20 is not, of course, limited to an in-vehicle device (drive recorder) installed in the automobile 2. It may be an on-board device. Also, it can be a wearable terminal such as an HMD (Head Mounted Display) or a glass type terminal equipped with one or more cameras. In this case, for example, the image data generated by the user photographing while walking is uploaded to the cloud server 1 . Furthermore, it may be a non-movable terminal, for example a fixed camera device with a communication function, for example a traffic monitoring camera installed along the road.

[Functional configuration of target tracking server]
According to the functional block diagram shown in FIG. 1, the cloud server 1, which is an object tracking server, has a communication interface 101 and a processor memory. Here, the processor memory stores an object tracking program according to an embodiment of the present invention, and has a computer function to execute the object tracking process by executing the object tracking program. do.

Further, from this, as the object tracking server according to the present invention, instead of the cloud server 1, the object tracking program according to the present invention is installed, for example, a non-cloud server device, personal computer (PC), notebook type or tablet type It is also possible to adopt a computer, a smart phone, or the like.

Further, for example, the object tracking program according to the present invention can be installed in the terminal 20, and the terminal 20 can be used as the object tracking server according to the present invention. Furthermore, an embodiment in which the object tracking server according to the present invention is installed in the automobile 2 together with the terminal 20 is also possible.

Further, the above processor memory includes an image acquisition unit 111 including a camera designation unit 111a, an object tracking switching unit 113, a first object tracking unit 114, a second object tracking unit 115, a third object It has a tracking unit 116 , a tracking result synthesizing unit 117 , and a transmission/reception control unit 118 . Furthermore, the object detection unit 112 may be provided. It should be noted that these functional components can be regarded as the functions of the object tracking program stored in the processor memory. Further, the flow of processing in which the functional components of the cloud server 1 are connected by arrows in FIG. 1 can also be understood as an embodiment of the object tracking method according to the present invention.

Also in the functional block diagram of FIG. 1, the image acquisition unit 111 acquires and manages video (image) data received from the terminal 20 via the communication interface 101 and the transmission/reception control unit 118 . Here, the image acquisition unit 111 may receive the video data in the form of a video stream compression-encoded based on standards such as MPEG. In this case, the video stream is decompressed (decoded) and managed as an image data group forming the original video stream.

Further, when the camera designation unit 111a of the image acquisition unit 111 receives a switching notification (tracking means change notification) from the target tracking switching unit 113, which camera (or which camera) is selected based on this notification, which will be described later in detail. image data) is generated, and the camera designation information is transmitted to the terminal 20. FIG. Here, the camera designation unit 111a obtains in advance information about the shooting direction of each camera connected to the terminal 20 (each of the four cameras 202a to 202d in FIG. 1) and the mutual positional relationship. It is also preferable to generate camera designation information based on the information.

By notifying the terminal 20 of such camera designation information, the image acquisition unit 111 can track the tracked target when it becomes impossible or inappropriate to track the tracked target using image data generated by one camera. It is also possible to obtain image data that is (or is suitable for tracking) generated by another camera.

Here, the image acquisition unit 111 can acquire all the image data of each camera connected to the terminal 20, but instead, the image data generated by the camera specified by the camera specification information is acquired each time. It is also preferable to obtain only As a result, the data transmission load from the terminal 20 can be reduced, and the image data storage/management load on the cloud server 1 side can also be reduced.

The target detection unit 112 detects a tracked target in the image data acquired from the image acquisition unit 111 . Specifically, a tracking target detection process can be performed using a well-known object detection technique using machine learning. For example, for each small image region in the image data, an object detector trained for tracking target detection is used to calculate a score that indicates “likeness to be tracked”, and the small image region with the highest score is the tracked target. area may be determined.

Also, the tracked area determined here is, for example, a circumscribing rectangular area, that is, a bounding box, each side of which is circumscribing the image portion of the tracked object detected by the object detector. is also preferred. Of course, it is also possible to set target regions of other shapes.

Non-Patent Documents: Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: single shot multibox detector", European Conference on Computer Vision, Computer Vision-ECCV 2016, 2016, pp. 21-37 can be used.

As a modification, the terminal 20 has a tracking target detection processing function (the target detection unit 213 in FIG. 1), and information on the tracking target detection result (for example, information on the size of the bounding box and the position in the image) may be transmitted to the cloud server 1 together with the image data. In this case, the cloud server 1 can omit the target detection unit 112 described above.

Similarly, in the functional block diagram of FIG. 1, the object tracking switching unit 113
(a) When the tracking of the tracking target by the first target tracking unit 114 becomes impossible or inappropriate, the second target tracking unit 115 is caused to track the tracking target,
(b) When the second object tracking unit 115 has tracked the object and it becomes possible or appropriate to track the object by the first object tracking unit 114, the first object tracking unit 114 may be caused to perform tracking of the tracked object.

In the present embodiment, the object tracking switching unit 113 determines that the first object tracking unit 114 is unable or unsuitable for tracking the object (for example, the overtaking car 3) in the case of (a) above. From the camera that generated the image data (e.g., the rear camera 202a), another camera (e.g., the right side camera 202b) that can generate an image that enables or is suitable for tracking the tracked object. A switch notification (tracking means change notification) designating switching to is output to the camera designation unit 111a.

As a result, the object tracking switching unit 113 allows the first object tracking unit 114 to track the object based on the image generated by the switched camera (right side camera 202b). It is possible to cause the second object tracking unit 115 to track the object to be tracked.

Furthermore, in the present embodiment, the object tracking switching unit 113, in the case of (b) above, uses the camera (for example, the right A switch notification that specifies switching from the front camera 202b) to another camera (e.g., the front camera 202c) that can generate image data that makes it possible or appropriate for the first object tracking unit 114 to track the object. (tracking means change notification) is output to the camera designation unit 111a.

As a result, when the second object tracking unit 115 tracks the object to be tracked, the object tracking switching unit 113 switches the image generated by the other switched camera (the front camera 202c) to the first When it becomes possible or appropriate for the object tracking unit 114 to track the tracked object, the first object tracking unit 114 can be caused to track the tracked object.

In addition, "tracking is impossible or unsuitable" in the above (a) refers to a situation in which the identification process of a specific part of the tracked target is actually impossible or difficult, for example, a predetermined or higher identification accuracy cannot be obtained, or It can also be a situation where more than a predetermined percentage of the area disappears from the image (protrudes outside the image), or a tracked object (for example, a car 3) From the tracking situation (the position in the image and the moving speed/direction), it is predicted that it will immediately (after a predetermined short time) move toward the image related to another camera (for example, the right side camera 202b). It may be a situation where

Incidentally, if the specific partial area disappears (protrudes) toward the right side of the image of the rear camera 202a, for example, the target tracking switching unit 113 issues a switching notification (tracking means change notification) can be output to the camera designation unit 111a. On the other hand, when it disappears (protrudes) toward the left side, for example, a switch notification (tracking means change notification) specifying switching to the left side camera 202d may be output.

Furthermore, "tracking is possible or appropriate" in the above (a) can be a situation in which identification processing of a specific part of the tracked object is possible in the image currently used for tracking, or a new state that can be switched. It is possible (or expected) to identify a specific part of the tracked object in the image captured by the camera (for example, the front camera 202c).

Also in the functional block diagram of FIG. 1, the first object tracking unit 114 is object tracking means for tracking the tracked object using a first discriminator capable of identifying a specific portion of the tracked object. Here, the specific portion is a portion including at least one of letters, numbers, symbols and graphics, and the first discriminator can identify at least one of the letters, numbers, symbols and graphics. It can be.

More specifically, the specific part is the characters, numbers, symbols, and graphics associated with the tracked moving object (automobile 3) (different from the tracking source moving object (automobile 2)). It is also preferable that the display includes at least one and is a display for specifying information related to the moving object (automobile 3), such as a license plate. The first object tracking unit 114, for example, even when there are many moving objects of the same type around, identifies and distinguishes each object by using such a display object as a specific part to identify the object to be tracked. This makes it possible to more reliably identify the target and suppress the occurrence of tracking failures such as mistaking the target.

FIG. 2 is a schematic diagram for explaining an embodiment of object tracking processing by the first object tracking unit 114. As shown in FIG.

According to the embodiment shown in FIG. 2, the first object tracking unit 114 uses the rear license plate 31 provided at the rear of the automobile 3 running in front as a specific part to be tracked,
(a) Extracting an image area related to the rear license plate 31 from the tracking target area corresponding to the vehicle 3 in the image related to the front camera 202c acquired by the image acquisition unit 111 by a pattern recognition method or the like,
(b) inputting the extracted image area related to the rear license plate 31 to the first discriminator, and causing the first discriminator to output the license plate information written on the rear license plate 31 as a discrimination result, One tracking target ID (identifier) is linked to this license plate information,
(c) After that, if the license plate information output in the images acquired moment by moment matches the license plate information linked with the tracked object ID at first, the license plate information is the same tracked object ( The position of the image area of the rear license plate 31 in the image acquired every moment is stored as the position of the car 3 every moment, and tracking is performed. Do it.

Here, the first discriminator performs processing by a known text (character data) recognition technology such as OCR (Optical Character Reader) technology and character reading technology on an image region received as an input, and the image region includes may output information (eg, a character data string) related to the text to be displayed. Incidentally, the first object tracking unit 114 calculates the distance between the object to be tracked (automobile 3) and its own vehicle (automobile 2) and the distance from the size of the image area related to the specific part (license plate) and the speed of change in the size. , the function of estimating the approach/separation speed of the tracked object (vehicle 3). As a result, it is possible to identify dangerously traveling vehicles, for example, which are rapidly approaching or close to each other.

The object tracking processing by the first object tracking unit 114 as described above has a smaller processing load than the processing by the second object tracking unit 115, which will be described later. more suitable to implement. Therefore, in this embodiment, the object tracking switching unit 113 (FIG. 1) defaults to adopting object tracking processing by the first object tracking unit 114 as long as the tracking is possible or appropriate.

Returning to the functional block diagram shown in FIG. 1, the second object tracking unit 115 converts the pixel information of the tracking object region of the tracking object (for example, the automobile 3) in the image acquired by the image acquisition unit 111 into the tracking object ( It is an object tracking means for tracking a tracking object (automobile 3) in an acquired image using a second discriminator capable of identifying the automobile 3).

Here, the second classifier is a deep neural network (DNN, Deep Neural Network) widely used for image recognition, SVM (Support Vector machine), random forest, etc., image data can be configured by various types of machine learning algorithms into which is input and output the identification results (e.g., an inferred "class" and its score (likelihood)).

A second object tracking unit 115 uses a second discriminator to determine a tracked object region estimated to be a tracked object (automobile 3) with a certainty or more of a certainty in images acquired every moment, The thus-determined intra-image position of the tracking target area (for example, the midpoint position of the lower side of the tracking target area) is saved as the position of the automobile 3 every moment, and tracking is performed.

By the way, it is also preferable that the second object tracking unit 115 has a learning function of the second discriminator. Specifically, learning data for constructing a learning model that constitutes the second discriminator is generated and managed, and then a predetermined tracking target ( For example, to generate a learning model to identify a car). Note that the learning data is specifically generated by assigning a correct label (for example, class "automobile") to the tracking target area that is input.

Here, learning data for each tracked target is generated in order to correspond to multiple tracked targets (e.g., automobiles, human-powered vehicles, humans, etc.), and a learning model for each tracked target is generated based on these learning data. It is also preferred that In this case, the second object tracking unit 115 has a plurality of second classifiers capable of identifying each of the plurality of tracked objects, and can output identification results for each tracked object in parallel. It becomes.

Each of these tracked objects also preferably has a "particular portion" that is identifiable by the first discriminator. In this case, the first object tracking unit 114 possesses a plurality of first classifiers capable of identifying each of the "specific parts" of these tracked objects, and outputs the identification results for each tracked object in parallel. You may Furthermore, in this case, the object tracking switching unit 113 appropriately determines which of the first object tracking unit 114 and the second object tracking unit 115 is to be used for each of the tracking objects, and It is also preferred to perform continuous and parallel tracking processes.

In any case, the object tracking processing by the second object tracking unit 115 as described above, when compared with the processing by the first object tracking unit 114, changes in the relative positional relationship with the tracked object, The processing is more robust against changes in the orientation and attitude of the tracked target.

For example, even in a situation where it is not possible to "see" the license plate as a specific part of the tracked object, there are many cases where the vehicle 3 as the tracked object can be "seen". The second object tracking unit 115 can perform object tracking processing only if a predetermined number of (pixel value data) or more can be obtained.

Also in the functional block diagram shown in FIG. This is object tracking means for tracking the object based on the position in the image of the object area).

Here, as a preferred embodiment, the object tracking switching unit 113 switches an area related to the pixel information of the tracked object (for example, the When the second object tracking unit 115 cannot track the tracked object (car 3) because the bounding box has become smaller than a predetermined size, the third object tracking unit 116 sends the tracked object (car 3) ) will continue to be tracked.

In any case, the object tracking processing by the third object tracking unit 116 is inferior to the processing by the second object tracking unit 115 in the identification accuracy of the tracked object, but has less pixel information, that is, a smaller amount of information. With the bounding box, the tracking process can be implemented.

FIG. 3 is a schematic diagram for explaining an embodiment of object tracking processing by the third object tracking unit 116. As shown in FIG.

In the embodiment shown in FIG. 3, the second object tracking unit 115 initially uses the image of the front camera 202c to track the car 3 that has finished overtaking and moves away in front. However, as the car 3 moves further away, the bounding box (tracking target area) of the car 3 becomes smaller, and finally the second discriminator discriminates the car 3 with a certainty (score) higher than a predetermined level. I can't do it.

Here, the object tracking switching unit 113, which catches such a state, switches the subject of the object tracking processing from the second object tracking unit 115 to the third object tracking unit 116, and in response to this, the third object tracking unit 113 switches. The tracking unit 116 continues to track the vehicle 3 using the bounding box of the vehicle 3, as shown in FIG.

Specifically, the third object tracking unit 116 calculates the in-image position of the bounding box B3 of the automobile 3 at each time, in this embodiment, the in-image position coordinate values (u, v) of the midpoint p of the lower side of the box. A line segment drawn by a series of midpoints p is managed as the trajectory of the vehicle 3 (Fig. 3 shows the midpoint p0 at time t0 and the midpoint p1 at time t1 (>t0). It is shown).

In this manner, the third object tracking unit 116 can continue to track even the object that cannot be tracked by the second object tracking unit 115 .

Returning to the functional block diagram shown in FIG. 1, the tracking result synthesizing unit 117 summarizes the tracking processing results output from each of the first to third object tracking units 114, 115, and 116, for example, for each tracking object ID. In addition, target tracking result information, which is information that associates the tracked target class with the tracked target's in-image position information (or position information converted into a value in real space) at each time within the tracking period Generate.

Here, it is also preferable that the generated target tracking result information is transmitted to an external information processing device, for example, the terminal 20 via the transmission/reception control unit 118 and the communication interface 101 . Further, it may be used by a predetermined application program within the cloud server 1 .

[Functional Configuration of Image Data Providing Client]
Similarly, according to the functional block diagram shown in FIG. , and a processor memory. Note that these cameras may be external devices connected to the terminal 20 . Here, the processor memory stores an image data providing program according to an embodiment of the present invention, and has a computer function. By executing this image data providing program, the image data Execute the provision process.

For this reason, as an image data providing client according to the present invention, in place of the terminal 20, which is a drive recorder, another in-vehicle information processing device equipped with the image data providing program according to the present invention, or a camera is provided. It is also possible to adopt a smart phone, a notebook or tablet computer, or a personal computer (PC). Also, a terminal, such as a smartphone, which is connected to the drive recorder via Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like, may be used as the main image data providing client.

Further, the processor memory has a video generation unit 211 , a camera image selection unit 212 , a presentation information generation unit 214 and a transmission/reception control unit 215 . Moreover, the object detection part 213 may further be included. These functional components can be regarded as functions of the image data providing program stored in the processor memory. Further, the flow of processing in which the functional components of the terminal 20 are connected by arrows in FIG. 1 can also be understood as an embodiment of the image data providing method according to the present invention.

Similarly, in the functional block diagram of FIG. 1, the video generation unit 211 is connected to a rear camera 202a, a right side camera 202b, a front camera 202c, and a left side camera 202d installed in the automobile 2 in this embodiment. Image (video) data is generated based on the shooting data output from the four cameras, and stored and managed. In this embodiment, the terminal 20 is a drive recorder, and the image generation unit 213 acquires image data of the situation outside the vehicle from these four cameras as a default setting, at least while the vehicle 2 is running, and generates an image. (Video) data is being generated.

The camera image selection unit 212
(a) receiving camera designation information from the cloud server 1 via the communication interface 201 and the transmission/reception control unit 215;
(b) selecting image (video) data related to the camera specified by the camera specification information from among the image (video) data generated by the video generation unit 211;
(c) Send the selected image (video) data to the cloud server 1 via the transmission/reception control unit 215 and the communication interface 201 .

Here, the selected image (video) data may be transmitted as a video stream compression-encoded based on standards such as MPEG.

The target detection unit 213 is a functional unit that performs the same target detection processing as the target detection unit 112 of the cloud server 1, but can be omitted when the cloud server 1 includes the target detection unit 112. ing.

The presentation information generation unit 214 acquires the object tracking result information distributed from the cloud server 1 via the communication interface 201 and the transmission/reception control unit 215, and for example, imports it into the installed peripheral situation monitoring program to generate the presentation information. This presentation information can be displayed on the display 203 .

For example, the presentation information generating unit 214 may highlight the trajectory of the vehicle 3 approaching from behind or overtaking on the road map displayed on the display 205 to notify the user. . In this case, it is also preferable to display the distance between the vehicle 3 and the own vehicle (the vehicle 2). Furthermore, when the cloud server 1 distributes the license plate information related to the tracked target together with the target tracking result information, the presentation information generation unit 214 registers the vehicle 3 at the position of the highlighted trajectory of the vehicle 3. A number may be displayed, and this information may be recorded, saved, and managed.

[Object tracking method]
FIG. 4 is a schematic diagram for explaining an embodiment of the object tracking method according to the present invention. Here, in the present embodiment, the vehicle 3 approaches the running vehicle 2 equipped with the terminal 20 from behind and overtakes it, and after this overtaking, the vehicle 3 enters the same lane as the vehicle 2. , and then move away in front of the automobile 2 .

According to FIG. 4, initially, the first object tracking unit 114 performs tracking processing of the vehicle 3 based on the image portion of the front license plate of the vehicle 3 included in the image of the rear camera 202a. Here, the first target tracking unit 114 performs tracking after associating one tracked target ID with license plate information generated from the image portion.

Next, the object switching unit 113 receives from the first object tracking unit 114 “information indicating that the image portion has disappeared from the image toward the right side (goes out of the image)” or “a predetermined short "Information that it is predicted that it will disappear (go out of the image) after a while" is received, and based on this, the execution body of object tracking is changed from the first object tracking unit 114 to the second object tracking unit 115. switch.

The target switching unit 113 further outputs a switching notification (tracking means change notification) to the camera designating unit 111a to switch the camera related to the image used for tracking from the rear camera 202a to the right side camera 202b. As a result, after this, the second object tracking unit 115 performs tracking processing of the vehicle 3 based on the pixel information related to the vehicle 3 included in the image related to the right side camera 202b.

Here, as a preferred mode of modification, the target switching unit 113 starts tracking processing by the second target tracking unit 115 based on the above-described "information that it is predicted that the target will disappear after a predetermined short time." Next, it is also preferable to stop the tracking process by the first object tracking unit 114 based on the above-mentioned "information that the image portion has disappeared from the image toward the right side". By providing a period in which the first object tracking unit 114 and the second object tracking unit 115 are operated in parallel in this manner, the second object tracking unit 115 initially operates in this period. Object tracking processing is performed based on the pixel information of the image, and the tracking object ID used by the first object tracking unit 114 can be inherited as it is. Furthermore, when starting the object tracking process based on the pixel information of the image related to the right side camera 202b after this, by matching the identification results of the pixel information related to both cameras, the tracked object ID inherited at the beginning can be further improved. It also becomes possible to continue using it smoothly.

Note that the second object tracking unit 115, based on the timing at which the image portion of the front license plate of the automobile 3 disappears from the screen (goes out of the image) and the position in the image, Predict the time when the car 3 appears (or exists) and its position in the image, then It is also preferable to continue tracking the automobile 3 after assigning the tracking object ID inherited from the first object tracking unit 114 .

After that, the object switching unit 113 receives information from the second object tracking unit 114 that the image area related to the automobile 3 has disappeared (goes out of the image) from the image toward the left side thereof, or a predetermined short distance. Receive information that it is predicted that it will disappear (go out of the image) after time, and based on this,
(a) Output a switch notification (tracking means change notification) to the camera designation unit 111a to switch the camera related to the image used for tracking from the right side camera 202b to the front camera 202c, and
(b) Switching the execution subject of object tracking from the second object tracking unit 115 to the first object tracking unit 114 .

As a result, after this, the first object tracking unit 114 tracks the vehicle 3 based on the image portion of the rear license plate (included in the image related to the front camera 202c) of the vehicle 3 that appears in front after overtaking. processing will be carried out. Here, the first object tracking unit 114 receives the generation of license plate information that matches the license plate information linked to one tracked object ID (generated and stored in the past), It is also preferable to continue tracking the automobile 3 by associating the same tracked object ID that was previously associated with this license plate information.

Next, while the car 3 advances in front of the car 2 and pulls away from the car 2, the object switching unit 113 changes the image portion of the rear license plate from the first object tracking unit 114 to a smaller image portion. receive information indicating that object tracking processing (with a predetermined accuracy or more) based on the above is no longer possible, and based on this, switch the execution body of object tracking from the first object tracking unit 114 to the second object tracking unit 115. . As a result, after this, the second object tracking unit 115 performs tracking processing of the vehicle 3 based on the pixel information related to the vehicle 3 included in the image related to the front camera 202c.

Next, as the automobile 3 moves further away in front of the automobile 2, the object switching unit 113 reduces the bounding box of the automobile 3 from the second object tracking unit 115, and based on the pixel information related to the automobile 3 (predetermined After receiving information indicating that the object tracking process (with accuracy or higher) is no longer possible, the subject tracking execution body is switched from the second object tracking unit 115 to the third object tracking unit 116 based on this information. As a result, after this, the third object tracking unit 116 performs tracking processing of the vehicle 3 based on the information regarding the in-image position of the banding box of the vehicle 3 included in the image related to the front camera 202c. .

Here, the third object tracking unit 116, in the image acquired every moment, if the amount of change in the in-image position of the bounding box is within a predetermined range, the third object tracking unit 116 sets those bounding boxes to the same object to be tracked. It is also preferable to perform tracking by judging that.

As described above, the cloud server 1 can consistently and reliably track the vehicle 3 that is moving forward after overtaking the vehicle 2 by selecting an appropriate target tracking means according to changes in the situation. You can.

As described in detail above, according to the present invention, when a tracked target is tracked using an image that may include the tracked target, it is possible to perform more appropriate tracking according to the situation of the tracked target. Thus, for example, the object tracking process can be continued even if it becomes impossible or difficult to identify a specific part of the tracked object (for example, a license plate).

By the way, the configuration and method of the present invention, for example, uses 5G (fifth generation mobile communication system), which enables transmission of a huge amount of video data, and a large number of self-driving cars, autonomous drones, autonomous robots, etc. By uploading high-resolution video data from the camera to the server and grasping each surrounding situation more reliably and accurately, it is possible to support those automatic/autonomous driving It is thought that it will greatly contribute to the creation and distribution of services.

A person skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and aspect of the present invention for the various embodiments of the present invention described above. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and the equivalents thereof.

1 Cloud server (image data acquisition server)
101, 201 communication interface 111 image acquisition unit 111a camera designation unit 112, 213 object detection unit 113 object tracking switching unit 114 first object tracking unit 115 second object tracking unit 116 third object tracking unit 117 tracking result synthesis unit 118, 215 transmission/reception control unit 2, 3 automobile 20 terminal 202a rear camera 202b right side camera 202c front camera 202d left side camera 203 display (DP)
211 video generation unit 212 camera image selection unit 214 presentation information generation unit

Claims (10)

A program that operates a computer that tracks a movable object through an image that may include the object,
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . a first object tracking means capable of tracking the object using the discriminator of
a second object tracking means capable of tracking the object using a second discriminator constituted by a machine learning algorithm for image recognition, capable of identifying the object from pixel information related to the object in the image; ,
Object tracking switching means for causing second object tracking means to track the object when tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short period of time An object tracking program characterized by causing a computer to function as a A program that operates a computer that tracks a movable object through an image that may include the object,
The image is generated by one of a plurality of cameras with different shooting ranges,
a first object tracking means capable of tracking the object using a first classifier capable of identifying a specific portion of the object in the image;
a second object tracking means capable of tracking the object using a second classifier capable of identifying the object from pixel information related to the object in the image;
object tracking switching means for causing a second object tracking means to track the object when the first object tracking means becomes unable to track the object or is predicted to become impossible after a predetermined short period of time; ,
An image that enables or is suitable for tracking an object when the object cannot be tracked by the image produced by a single camera or is predicted to become incapable after a predetermined short period of time . and an image acquisition means for acquiring the image generated by another camera
to make the computer work ,
When the object tracking switching means predicts that the first object tracking means can not track the object based on the image generated by the other camera or will become impossible after a predetermined short time, the second cause the object tracking means of 2 to perform tracking of the object based on the image produced by the other camera ;
An object tracking program characterized by: When the second object tracking means has tracked the object based on the image generated by a certain camera, the object tracking switching means changes the image generated by a camera other than the certain camera. when tracking of the object by the first object tracking means in the image becomes possible or is predicted to become possible after a predetermined short period of time, or by the second object tracking means in the image produced by the certain camera When the tracking of the object becomes impossible or is predicted to become impossible after a predetermined short period of time, to the first object tracking means, the object based on the image generated by the camera other than the one camera. 3. The object tracking program according to claim 2, wherein the tracking of the object is performed. The cameras are four cameras provided on one moving object, each of which can photograph the front, right side, left side, and rear, and the specific part of the target corresponds to another moving body that is the target. 4. The display body according to claim 3 , wherein the display body includes at least one of attached letters, numbers, symbols, and graphics, and is a display body for specifying information related to the other moving body. subject tracking program. The first object tracking means tracks the other moving object approaching from behind from the position of the display object installed in front of the other moving object in the image generated by the camera capable of photographing the rear. death,
The second object tracking means approaches and overtakes the moving object from the right side or the left side based on the pixel information of the other moving object in the image generated by the camera capable of photographing the right side or the left side. track other moving objects,
The first object tracking means moves from the position of the display body installed behind the other moving body in the image generated by the camera capable of photographing the front to the other moving body that appears in front after overtaking. 5. The object tracking program according to claim 4 , which tracks a moving object. A program that operates a computer that tracks a movable object through an image that may include the object,
a first object tracking means capable of tracking the object using a first classifier capable of identifying a specific portion of the object in the image;
a second object tracking means capable of tracking the object using a second classifier capable of identifying the object from pixel information related to the object in the image;
third object tracking means capable of tracking the object in the image based on the position in the image of the bounding box of the object detected in the image ;
When the tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short time, the second object tracking means is caused to track the object, and the second object is detected. When the object has been tracked by the tracking means, and the tracking of the object by the second object tracking means has become impossible because the area related to the pixel information of the object has become smaller than a predetermined size. , object tracking switching means for causing the third object tracking means to track the object;
An object tracking program characterized in that it causes a computer to function by The image is generated by a camera capable of photographing the front provided in one moving body,
The third object tracking means moves away in front of the one moving body after passing the one moving body based on the position within the image of the bounding box associated with the other moving body detected in the image. 7. The object tracking program according to claim 6 , which tracks a moving object of . The object tracking switching means is configured such that when the object has been tracked by the second object tracking means, the object can be tracked by the first object tracking means or after a predetermined short period of time. or when it is predicted that the tracking of the object by the second object tracking means will become impossible or will become impossible after a predetermined short period of time, the tracking of the object will be directed to the first object tracking means 8. The object tracking program according to any one of claims 1 to 7, characterized in that it causes the execution of An apparatus for tracking a movable object with an image that may include the object, comprising:
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . a first object tracking means capable of tracking the object using the discriminator of
a second object tracking means capable of tracking the object using a second discriminator constituted by a machine learning algorithm for image recognition, capable of identifying the object from pixel information related to the object in the image; ,
Object tracking switching means for causing second object tracking means to track the object when tracking of the object by the first object tracking means becomes impossible or is predicted to become impossible after a predetermined short period of time and an object tracking device. A method of object tracking in a computer for tracking an object through an image that may include a movable object, the method comprising:
A first identifiable portion of the object in the image that includes at least one of letters, numbers and symbols by reading at least one of the letters, numbers and symbols . tracking the object using the discriminator of
For image recognition that can identify the target from the pixel information of the target in the image when the tracking of the target by the first discriminator becomes impossible or is predicted to become impossible after a predetermined short time . and tracking the object using a second discriminator configured by the machine learning algorithm of .

Images