CN112037259A - System and method for tracking dynamic target - Google Patents
System and method for tracking dynamic target Download PDFInfo
- Publication number
- CN112037259A CN112037259A CN202010879875.9A CN202010879875A CN112037259A CN 112037259 A CN112037259 A CN 112037259A CN 202010879875 A CN202010879875 A CN 202010879875A CN 112037259 A CN112037259 A CN 112037259A
- Authority
- CN
- China
- Prior art keywords
- target
- tracking
- tracking result
- dynamic
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000001514 detection method Methods 0.000 claims abstract description 74
- 230000003287 optical effect Effects 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Abstract
The embodiment of the application discloses a system and a method for tracking a dynamic target. The system comprises: an image acquisition module configured to acquire an RGBD image of the dynamic target; the dynamic target detection module is connected with the image acquisition module and is configured to perform target detection on the RGBD image to obtain an initial position of a target component; and the component position tracking module is connected with the dynamic target detection module and is configured to determine the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images so as to determine the component tracking result of the dynamic target. By executing the technical scheme, the accuracy and robustness of dynamic target tracking can be improved.
Description
Technical Field
The embodiment of the application relates to the technical field of image processing, in particular to a system and a method for tracking a dynamic target.
Background
With the rapid development of the technology level, transportation equipment is widely applied in the field of transportation of goods. In the order-to-person scenario, there is a transportation scenario where a pedestrian is tracked by a transportation device and a corresponding item is transported for the pedestrian. In this case, the transportation device needs to continuously acquire images of pedestrians and determine its own tracking target from the images. However, the pedestrian is often in motion and may be accompanied by bending over from the side, raising the arm, and the like, and may also be blocked by other objects, or the pedestrian moves out of the image range, thereby causing a situation where the pedestrian cannot track normally. This not only makes the unable normal motion of haulage equipment, still influences pedestrian's use experience simultaneously, causes pedestrian's economic loss even easily.
Disclosure of Invention
The embodiment of the application provides a system and a method for tracking a dynamic target, which can improve the accuracy and robustness of tracking the dynamic target.
In a first aspect, an embodiment of the present application provides a tracking system for a dynamic target, where the system includes:
an image acquisition module configured to acquire an RGBD image of the dynamic target;
the dynamic target detection module is connected with the image acquisition module and is configured to perform target detection on the RGBD image to obtain an initial position of a target component;
and the component position tracking module is connected with the dynamic target detection module and is configured to determine the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images so as to determine the component tracking result of the dynamic target.
Further, the dynamic target detection module is further configured to: carrying out target detection on the RGBD image to obtain the overall position of a target;
the system also includes an overall position tracking module, coupled to the dynamic target detection module, configured to:
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
Further, the system further includes a tracking fusion module, connected to the component position tracking module and to the global position tracking module, configured to:
and determining a final tracking result of the dynamic target according to the component tracking result and the at least one integral tracking result.
Further, the tracking fusion module is configured to:
if at least two integral tracking results exist, selecting the integral tracking result with the maximum superposition coefficient with the component tracking result as a final tracking result;
and if an integral tracking result exists, taking the integral tracking result as a final tracking result when the superposition coefficient of the integral tracking result and the part tracking result accords with a set threshold value.
Further, the tracking fusion module is further configured to:
and if the integral tracking result does not exist, taking the part tracking result as a final tracking result.
Further, the dynamic target detection module is configured to:
carrying out target detection by adopting a preset detection network model to obtain the initial position of a target component; wherein the components include one or more of a head, a torso, arms, and legs.
Further, the component location tracking module is configured to:
determining target motion information between at least two frames of RGBD images by adopting an optical flow method; and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information so as to determine a component tracking result of the dynamic target.
Further, an overall location tracking module configured to:
determining a target detection result in a target frame RGBD image, and determining a matching confidence of a detected target and a tracking target template of the whole position;
and determining at least one overall tracking result according to the matching confidence.
In a second aspect, an embodiment of the present application provides a method for tracking a dynamic target, where the method includes:
obtaining an RGBD image of a dynamic target;
carrying out target detection on the RGBD image to obtain an initial position of a part of a target;
and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two frames of RGBD images so as to determine the part tracking result of the dynamic target.
Further, the method further comprises:
carrying out target detection on the RGBD image to obtain the overall position of a target;
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
Further, the method further comprises:
and determining a final tracking result of the dynamic target according to the component tracking result and the at least one integral tracking result.
Further, determining a final tracking result of the dynamic target according to the component tracking result and the at least one overall tracking result, including:
if at least two integral tracking results exist, selecting the integral tracking result with the maximum superposition coefficient with the component tracking result as a final tracking result;
and if an integral tracking result exists, taking the integral tracking result as a final tracking result when the superposition coefficient of the integral tracking result and the part tracking result accords with a set threshold value.
Further, determining a final tracking result of the dynamic target according to the component tracking result and the at least one overall tracking result, further comprising:
and if the integral tracking result does not exist, taking the part tracking result as a final tracking result.
Further, performing target detection on the RGBD image to obtain an initial position of a component of a target, including:
carrying out target detection by adopting a preset detection network model to obtain the initial position of a target component; wherein the components include one or more of a head, a torso, arms, and legs.
Further, determining an estimated position of the target in the RGBD image of the target frame according to the target motion information between the at least two RGBD images to determine a component tracking result of the dynamic target, includes:
determining target motion information between at least two frames of RGBD images by adopting an optical flow method; and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information so as to determine a component tracking result of the dynamic target.
Further, determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame comprises the following steps:
determining a target detection result in a target frame RGBD image, and determining a matching confidence of a detected target and a tracking target template of the whole position;
and determining at least one overall tracking result according to the matching confidence.
The technical scheme provided by the embodiment of the application comprises the following steps: an image acquisition module configured to acquire an RGBD image of the dynamic target; the dynamic target detection module is connected with the image acquisition module and is configured to perform target detection on the RGBD image to obtain an initial position of a target component; and the component position tracking module is connected with the dynamic target detection module and is configured to determine the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images so as to determine the component tracking result of the dynamic target. By adopting the technical scheme provided by the application, the accuracy and robustness of dynamic target tracking can be improved.
Drawings
FIG. 1 is a schematic illustration of an order-to-person operation process provided in an embodiment of the present invention;
FIG. 2 is a schematic structural view of a shelf with one-way openings according to an embodiment of the present invention;
FIG. 3 is a schematic illustration of an order-to-person control process provided in an embodiment of the present invention;
FIG. 4 is a schematic diagram of a dynamic target tracking system provided by an embodiment of the present application;
FIG. 5 is a schematic diagram of a dynamic target tracking method provided in an embodiment of the present application;
fig. 6 is a schematic diagram of a tracking method for a dynamic target provided in an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
FIG. 1 is a schematic diagram of an order-to-person process provided in an embodiment of the invention. Referring to fig. 1, the memory area 110 includes: the shelf 111 is provided with a plurality of shelves 111, various articles are placed on the shelves 111, and the shelves 111 are arranged in an array form, for example, like shelves in supermarkets where various commodities are placed. The shelf 111 may be a container having a compartment through which items can be accessed, such as a storage box, wherein the shelf 111 includes a plurality of compartments and four floor-standing support columns, at least one compartment is disposed on the compartments of the shelf 111, and one or more items may be placed in the compartment. In addition, the shelf 111 may be one-way open, for example, fig. 2 is a schematic structural diagram of a one-way open shelf provided in an embodiment of the present invention, such as the one-way open shelf 111 shown in fig. 2, and may also be a two-way open shelf. The shelf 111 may be a movable shelf or a fixed shelf.
In an order-to-person scenario, the control system may send a corresponding order to the transport 120. The staff determines the order that needs to be completed by the transportation device 120 through the display of the transportation device 120 or an intelligent terminal carried by the staff. It can be understood that, if the staff uses the intelligent terminal carried in the hand to view the order or views the order through the display of the transportation device 120, before picking the goods contained in the order, the staff needs to be connected to the transportation device 120, that is, the staff is bound to the corresponding transportation device 120, specifically, the staff may be connected to the transportation device 120 by scanning a code, inputting a serial number, or connecting to a card.
After the shipping equipment 120 receives the order, the shipping equipment 120 needs to track the staff's completion of picking the items in the order. The transportation device 120 needs to determine the staff as a tracking target to track the staff in real time during walking. In tracking the staff, the staff may pick the goods and place the goods on the transportation device 120. In addition, the staff member can interact with the transportation device 120 according to the display of the transportation device 120, for example, to know the specific content of the order and so on.
FIG. 3 is a schematic diagram of an order to person control process provided in an embodiment of the present invention. As shown in fig. 3, the control system 130 is in wireless communication with the transportation device 120, the customer generates an order through the console, the order is transmitted to the control system 130, the control system 130 responds to the order and issues the order to the transportation device 120, and the transportation device 120 performs the task of the order under the control of the control system 130 and with the assistance of the staff. For example, the transportation device 120 receives the order task, binds with a worker in the warehouse, that is, the bound worker is responsible for picking the order task received by the transportation device, and the transportation device 120 may travel along the aisle space in the middle of the rack array, move to the position of the rack 111, receive the item picked by the worker, and track the worker for real-time transportation, and finally transport the picked item to the designated position.
It can be seen that after a worker removes an item from the shelf 111, the worker may be placed directly on the transport 120 and without control of the transport 120, the transport 120 may track the worker until the order assigned by the transport 120 is completed. It can be understood that, similar to the above process, when the goods are restocked, the goods restocking here means that after the number of a certain commodity in the warehouse is lower than a set certain threshold, the commodity needs to be restocked, a certain amount of the commodity is loaded by the transportation device 120, and the commodity is carried to the target storage shelf, and the worker puts the commodity on the transportation device 120 onto the target storage shelf to complete the restocking of the commodity. Compared with the traditional manual goods taking or replenishing mode, the scheme from the order to the person is more convenient to realize, and the goods shelf is not required to be improved while the manpower is saved, so that the goods taking and replenishing method can be suitable for the goods storage and retrieval of various storage warehouses.
The control system 130 may be a software system with data storage and information processing capabilities running on a control server, and may be connected to the transport equipment 120, a hardware input system, or other software systems via wireless or wired connections. The control system 130 may include one or more control servers, which may be a centralized control architecture or a distributed computing architecture. The control server has a processor 1301 and a memory 1302, and may have an order pool 1303 in the memory 1302.
The transport apparatus 120 may have only a tracking function, or both a tracking function and a navigation function. Additionally, the transport device 120 may have other desired and/or necessary functions, such as obstacle avoidance functionality, and the like.
For the tracking function, the logistics robot may include: a main controller; a movable base; an information acquisition device adapted to acquire image information and depth information of an object present within its field of view; the information acquisition device may comprise an RGBD camera adapted to acquire image information and depth information of objects present in its field of view, resulting in an RGBD image.
Alternatively or additionally, the information acquisition device may comprise a monocular camera adapted to acquire image information of objects present within its field of view and a lidar adapted to acquire depth information of objects present within its field of view.
The main controller may acquire image information and depth information of the tracking target from the RGBD camera. For example, in the case that the recognition target is beyond the view range of the RGBD camera, the master controller cannot acquire valid information from the RGBD camera for a period of time, for example, 1 second, to recognize the tracking target, and it is determined that the tracking target is lost. A plurality of RGBD cameras may be provided as necessary. In addition, the number of the monocular cameras and the number of the laser radars may be one or more, as necessary.
It is contemplated that the information acquisition device may acquire visual information, including image information and depth information, of objects present within its field of view in real time; the main controller can acquire the information acquired by the information acquisition device in real time, identify the tracked target in real time, determine the position information of the tracked target in real time and plan the tracking route in real time.
It is contemplated that the master controller may obtain the identity information of the tracking target in a variety of ways, wired or wirelessly, for example, the master controller may obtain the identity information of the tracking target from any device to which the master controller may be coupled or in communication with, such as a server, a mobile device, such as a mobile device of an operator, and the like. The master controller can be connected and communicate with such devices via cables, or via Bluetooth, as desiredTM、WiFi、ZigBeeTMOr other wireless communication protocol to communicate with such devices.
Typically, the tracking targets are personnel authorized to use the transport apparatus 120, such as the operator of the transport apparatus 120. The master controller may be configured to authorize the tracking target based on the acquired identity information of the tracking target. The identity information may be any information that can uniquely identify the identity of the tracking target, such as but not limited to: a username and password of the tracked target, a fingerprint of the tracked target, a facial image of the tracked target, etc.
The identifying characteristics of the tracking target may be associated with identity information of the tracking target. The identifying feature may be any feature worn or carried by the tracked object itself or on its body suitable for identifying the tracked object. In the case where the tracking target is an operator, the identification features may be, for example, but not limited to: the body type, the work number carried on the back of the garment, the two-dimensional code carried on the back of the garment, the color of the garment, and the like.
Fig. 4 is a schematic diagram of a system for tracking a dynamic target provided in an embodiment of the present application, where the embodiment is applicable to a situation where a transportation device tracks the dynamic target, the system may perform the method for tracking a dynamic target provided in an embodiment of the present application, and the system may be implemented by software and/or hardware.
As shown in fig. 4, the tracking system of the dynamic target includes:
an image acquisition module 410 configured to acquire an RGBD image of the dynamic target;
a dynamic target detection module 420, connected to the image acquisition module 410, configured to perform target detection on the RGBD image, so as to obtain an initial position of a target component;
and the component position tracking module 430 is connected to the dynamic target detection module 420, and configured to determine an estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images, so as to determine a component tracking result of the dynamic target.
The present solution may be executed by a robot having a transportation function, and specifically, may be a mobile device having a full-automatic driving function, a semi-automatic driving function, or an auxiliary driving function, such as an automobile having an automatic driving mode, an AGV (Automated Guided Vehicle), a robot, and the like. In this case, an AGV is taken as an example for explanation.
In scenarios where AGVs are utilized to perform tasks, multiple AGVs within the same scenario are typically scheduled by a unified management platform to enable the entire scenario to proceed in order. For example, when the AGVs are used to pick goods, all AGVs performing tasks or being ready to perform tasks in the storage area need to be scheduled and controlled. The task to be executed can be automatically imported by the management platform, and can also be input by a worker through an input device. When the management platform receives one or more tasks to be executed, the management platform determines corresponding AGVs to complete each task according to a scheduling algorithm or rule of the management platform, and of course, the management platform can also control the corresponding AGVs to execute new tasks according to the AGVs designated by a worker. This requires the routing of the AGVs that received the task (i.e., the AGVs that are to perform the task). The path planning for the AGV to execute the task may be performed by a path planning method commonly used in the art. Specifically, the selectable path is determined according to the starting point and the destination of the AGV, and then an optimal path is selected as the planned path after weighing is carried out based on time, path factors and the busyness degree of the path.
The image obtaining module 410 may be a depth camera disposed on the AGV, and the camera may obtain not only color information of the image but also depth information of each pixel point in the image. Therefore, an RGBD image of the dynamic target can be acquired by the camera. Wherein it is understood that the dynamic object may be a pedestrian. The dynamic target may be an adult or a child.
The dynamic target detection module 420 may be configured to perform target detection on the acquired RGBD image to obtain an initial position of a component of the target. It is understood that a certain algorithm may be adopted to implement the target detection on the RGBD image, for example, a human form in the image is recognized through a human form recognition technology, and if a plurality of human forms are included in the RGBD image, each human form may be marked for subsequent confirmation whether a dynamic target is included therein.
In this embodiment, optionally, the dynamic target detection module 420 is configured to:
carrying out target detection by adopting a preset detection network model to obtain the initial position of a target component; wherein the components include one or more of a head, a torso, arms, and legs.
According to the scheme, the pedestrian detection is carried out by utilizing the improved CenterNet detection network, and the positions of pedestrians and parts of the pedestrians are obtained. The method is mainly used for obtaining the positions of the pedestrians and the parts thereof in the initial frame and the positions of the pedestrians and the parts thereof in the subsequent new image frame.
The component position tracking module 430 may determine an estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images to determine a component tracking result of the dynamic target.
The motion information may be the amount of motion of each part. Specifically, the position variation amount of the pixel point at each position from the previous RGBD image array to the next RGBD image frame may be used. Because each part is a part of a human body, the motion conditions between two adjacent frames of RGBD images are often relatively close, and therefore, the estimated positions of each part in the RGBD image of the target frame can be determined according to the motion information seen by at least two frames of RGBD images.
It is understood that at least two frames of RGBD images may be two consecutive frames or may be non-consecutive, for example, the 2 nd frame RGBD image and the 4 th frame RGBD image in the video stream, and the determined target frame RGBD image may be the 6 th frame RGBD image. Specifically, the calculation of the action amount may be performed by taking two non-consecutive frames if the movement of the staff is relatively slow in the scene, which may be determined according to the need or the actual scene. Thus, the calculation amount can be reduced, and the calculation load can be reduced.
In this embodiment, optionally, the component position tracking module 430 is configured to:
determining target motion information between at least two frames of RGBD images by adopting an optical flow method; and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information so as to determine a component tracking result of the dynamic target.
Optical flow is the motion of an object, scene, or camera as it moves between two consecutive frames of images. The method is a two-dimensional vector field of an image in the process of translation, is a speed field for representing three-dimensional motion of an object point through a two-dimensional image, and reflects image change formed by motion in a tiny time interval so as to determine the motion direction and the motion rate of the image point.
By adopting an optical flow method, the estimated position of the target in the RGBD image of the target frame can be determined more accurately.
It will be appreciated that where the components include a head, torso, arms, and legs, the central position of each component may be determined as the position of the dynamic target.
According to the technical scheme, by means of component detection, the problem that the dynamic target cannot be tracked due to the fact that the body state of the dynamic target changes and the image range moves under the condition that the whole dynamic target range is tracked can be solved. The robustness of tracking the dynamic target is improved.
In a possible embodiment, optionally, the dynamic object detection module is further configured to: carrying out target detection on the RGBD image to obtain the overall position of a target;
the system also includes an overall position tracking module, coupled to the dynamic target detection module, configured to:
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
Wherein the global position may be determined based on a global tracking target template of the dynamic target. The overall position may be a supplement to the position of the component, or may be used alone for performing dynamic target tracking, and in some cases, if it is difficult to perform dynamic target tracking based on the position of the component, the overall position tracking method provided in the present solution may be adopted.
In this embodiment, specifically, the overall position tracking module is configured to:
determining a target detection result in a target frame RGBD image, and determining a matching confidence of a detected target and a tracking target template of the whole position;
and determining at least one overall tracking result according to the matching confidence.
It can be understood that, in the RGBD image of the target frame, there may be an overall position of the dynamic target, and there may also be an overall position of another dynamic target, so that, in the overall position detection process, the pedestrian detection module in the previous step may be used to obtain a pedestrian detection result, each detection result is compared with the tracking target template through a matching network, a matching confidence is obtained, the first three largest matching results are selected, and the position and the confidence are stored.
On the basis of the above technical solution, optionally, the system further includes a tracking fusion module, connected to the component position tracking module, and connected to the overall position tracking module, and configured to:
and determining a final tracking result of the dynamic target according to the component tracking result and the at least one integral tracking result.
Under the condition of simultaneously adopting the whole position and the component position for detection, the final tracking result of the dynamic target can be determined according to the two detection results. For example, when one of the detections is successful and the other detection is failed, the position obtained by the successful detection may be used as the final tracking result.
Specifically, the tracking fusion module is configured to:
if at least two integral tracking results exist, selecting the integral tracking result with the maximum superposition coefficient with the component tracking result as a final tracking result;
and if an integral tracking result exists, taking the integral tracking result as a final tracking result when the superposition coefficient of the integral tracking result and the part tracking result accords with a set threshold value.
When two or more overall tracking results exist, which one of the overall tracking results is the real tracking target can be determined according to the component tracking results. When only one overall tracking result exists, the overall tracking result can be used as a final tracking result when the superposition coefficient of the overall tracking result and the part tracking result meets a set threshold value. For example, if the coincidence coefficient of the two is 20%, it is highly likely that the overall tracking result and the part tracking result are not the same target. Only when the coincidence coefficient of the two reaches 50% or higher, the final tracking result of the dynamic target can be determined. It is to be understood that if the overall tracking result and the part tracking result are represented by point locations, the coincidence coefficient of the two may be determined by determining the distance between the two point locations.
In another possible embodiment, the tracking fusion module is further configured to:
and if the integral tracking result does not exist, taking the part tracking result as a final tracking result.
If the overall tracking result is not detected, which indicates that the dynamic target is partially or completely out of the image range at the moment, the final tracking result of the dynamic target can be determined only through the component tracking result. If the dynamic tracking result can not be detected, the current frame image can be skipped, the images of the subsequent preset number of frames can be detected, and if the overall position or the part position can not be detected, the lost dynamic tracking target can be determined.
Fig. 5 is a schematic diagram of a tracking method for a dynamic target provided in an embodiment of the present application. As shown in fig. 5, the tracking method of the dynamic target is executed by a management platform, and includes:
and S510, acquiring an RGBD image of the dynamic target.
S520, carrying out target detection on the RGBD image to obtain the initial position of a target component.
S530, according to the target motion information between at least two frames of RGBD images, determining the estimated position of the target in the RGBD image of the target frame so as to determine the part tracking result of the dynamic target.
On the basis of the above technical solution, optionally, the method further includes:
carrying out target detection on the RGBD image to obtain the overall position of a target;
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
On the basis of the above technical solution, optionally, the method further includes:
and determining a final tracking result of the dynamic target according to the component tracking result and the at least one integral tracking result.
On the basis of the above technical solution, optionally, determining a final tracking result of the dynamic target according to the component tracking result and the at least one overall tracking result, includes:
if at least two integral tracking results exist, selecting the integral tracking result with the maximum superposition coefficient with the component tracking result as a final tracking result;
and if an integral tracking result exists, taking the integral tracking result as a final tracking result when the superposition coefficient of the integral tracking result and the part tracking result accords with a set threshold value.
On the basis of the above technical solution, optionally, determining a final tracking result of the dynamic target according to the component tracking result and the at least one overall tracking result, further comprising:
and if the integral tracking result does not exist, taking the part tracking result as a final tracking result.
On the basis of the foregoing technical solution, optionally, performing target detection on the RGBD image to obtain an initial position of a component of a target, including:
carrying out target detection by adopting a preset detection network model to obtain the initial position of a target component; wherein the components include one or more of a head, a torso, arms, and legs.
On the basis of the above technical solution, optionally, determining an estimated position of a target in the RGBD image of the target frame according to target motion information between at least two RGBD images to determine a component tracking result of the dynamic target, includes:
determining target motion information between at least two frames of RGBD images by adopting an optical flow method; and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information so as to determine a component tracking result of the dynamic target.
On the basis of the foregoing technical solution, optionally, determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame includes:
determining a target detection result in a target frame RGBD image, and determining a matching confidence of a detected target and a tracking target template of the whole position;
and determining at least one overall tracking result according to the matching confidence.
The method can be executed by the system provided by any embodiment of the application, and has the corresponding execution steps and beneficial effects.
Fig. 6 is a schematic diagram of a tracking method for a dynamic target provided in an embodiment of the present application. As shown in fig. 6, the method for tracking a dynamic target includes:
the method utilizes appearance information, spatial information and motion consistency information. The person as a whole divides the body of the person into different parts and combines the whole and part tracking results for tracking. The target and the components thereof are taken as tracking objects, the target and the components thereof are described through depth features, and the consistency of motion among the components is judged by utilizing optical flow. The tracking accuracy and the tracking robustness can be guaranteed.
The pedestrian is divided into different parts, such as: head, torso, arms, legs.
The pedestrian detection system is composed of a pedestrian detection module, a pedestrian integral tracking module, a tracking module based on component motion and a tracking fusion module.
1. A pedestrian detection module to: and carrying out pedestrian detection based on the RGBD image to obtain the positions of pedestrians and parts thereof in the image. The method utilizes the improved CenterNet detection network to detect the pedestrian and obtain the positions of the pedestrian and the parts thereof. The method is mainly used for obtaining the positions of the pedestrians and the parts thereof (tracking target templates) in the initial frame and the positions of the pedestrians and the parts thereof in the subsequent new image frame.
2. The integral pedestrian tracking module is used for: and in a new frame of image, obtaining a pedestrian detection result by using the pedestrian detection module in the previous step, comparing each detection result with the tracking target template through a matching network to obtain a matching confidence coefficient, selecting the first three largest matching results and storing the position and the confidence coefficient. This step may yield a pedestrian location result that is most similar to the target template. The step can process the situation that the pedestrian disappears due to the appearance of the picture or the occlusion.
3. A component motion based tracking module to: and obtaining the position of the target frame part according to the relative motion condition between two continuous frame images. The method is realized by an optical flow method. The optical flow is a movement amount of a pixel point representing the same object (object) in one frame of a video image to a target frame, and is expressed by a two-dimensional vector. From a priori knowledge, it is known that the optical flow behavior of the same part of the human body in the previous and subsequent frames should be the same. Therefore, the position of the previous frame component in the target frame can be obtained through the position of the previous frame component and the optical flow operation information between two frames, and the component tracking result can be obtained. The center of all the parts is the overall tracking result of the pedestrian.
4. A tracking fusion module to: and combining the tracking results of the step 2 and the step 3 to obtain a final pedestrian tracking result:
(1) when the tracking result exists in the step 2, the result closest to the tracking result in the step 3 is selected as the final tracking result. The step can filter the false detection in the step 2, and improve the stability and accuracy of tracking.
(2) And when the matching of the target in the step 2 fails due to the deformation or the shielding of the human body, the result in the step 3 is used as a tracking result, and the robustness of the tracking result is improved.
The invention comprehensively utilizes the appearance information, the space information and the motion consistency information of the target, can be suitable for tracking the pedestrian appearing after disappearance, the pedestrian deforming, shielding and the like under various conditions, and simultaneously considers both the accuracy and the robustness.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (10)
1. A system for tracking a dynamic target, comprising:
an image acquisition module configured to acquire an RGBD image of the dynamic target;
the dynamic target detection module is connected with the image acquisition module and is configured to perform target detection on the RGBD image to obtain an initial position of a target component;
and the component position tracking module is connected with the dynamic target detection module and is configured to determine the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two RGBD images so as to determine the component tracking result of the dynamic target.
2. The system of claim 1, wherein the dynamic object detection module is further configured to: carrying out target detection on the RGBD image to obtain the overall position of a target;
the system also includes an overall position tracking module, coupled to the dynamic target detection module, configured to:
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
3. The system of claim 2, further comprising a tracking fusion module coupled to the component position tracking module and coupled to the global position tracking module, configured to:
and determining a final tracking result of the dynamic target according to the component tracking result and the at least one integral tracking result.
4. The system of claim 3, wherein the trace fusion module is configured to:
if at least two integral tracking results exist, selecting the integral tracking result with the maximum superposition coefficient with the component tracking result as a final tracking result;
and if an integral tracking result exists, taking the integral tracking result as a final tracking result when the superposition coefficient of the integral tracking result and the part tracking result accords with a set threshold value.
5. The system of claim 4, wherein the trace fusion module is further configured to:
and if the integral tracking result does not exist, taking the part tracking result as a final tracking result.
6. The system of claim 1, wherein the dynamic object detection module is configured to:
carrying out target detection by adopting a preset detection network model to obtain the initial position of a target component; wherein the components include one or more of a head, a torso, arms, and legs.
7. The system of claim 1, wherein the component location tracking module is configured to:
determining target motion information between at least two frames of RGBD images by adopting an optical flow method; and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information so as to determine a component tracking result of the dynamic target.
8. The system of claim 2, wherein the global position tracking module is configured to:
determining a target detection result in a target frame RGBD image, and determining a matching confidence of a detected target and a tracking target template of the whole position;
and determining at least one overall tracking result according to the matching confidence.
9. A method for tracking a dynamic target, the method comprising:
obtaining an RGBD image of a dynamic target;
carrying out target detection on the RGBD image to obtain an initial position of a part of a target;
and determining the estimated position of the target in the RGBD image of the target frame according to the target motion information between at least two frames of RGBD images so as to determine the part tracking result of the dynamic target.
10. The method of claim 9, further comprising:
carrying out target detection on the RGBD image to obtain the overall position of a target;
and determining at least one overall tracking result matched with the overall position in the RGBD image of the target frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879875.9A CN112037259A (en) | 2020-08-27 | 2020-08-27 | System and method for tracking dynamic target |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879875.9A CN112037259A (en) | 2020-08-27 | 2020-08-27 | System and method for tracking dynamic target |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112037259A true CN112037259A (en) | 2020-12-04 |
Family
ID=73586692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010879875.9A Pending CN112037259A (en) | 2020-08-27 | 2020-08-27 | System and method for tracking dynamic target |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112037259A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116883915A (en) * | 2023-09-06 | 2023-10-13 | 常州星宇车灯股份有限公司 | Target detection method and system based on front and rear frame image association |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104835182A (en) * | 2015-06-03 | 2015-08-12 | 上海建炜信息技术有限公司 | Method for realizing dynamic object real-time tracking by using camera |
| WO2015161697A1 (en) * | 2014-04-25 | 2015-10-29 | 深港产学研基地 | Method and system for tracking moving object applied to man-machine interaction |
| CN108269269A (en) * | 2016-12-30 | 2018-07-10 | 纳恩博（北京）科技有限公司 | Method for tracking target and device |
| CN110287855A (en) * | 2019-06-21 | 2019-09-27 | 上海闪马智能科技有限公司 | A kind of pedestrian track acquisition methods and system |
| CN110516620A (en) * | 2019-08-29 | 2019-11-29 | 腾讯科技（深圳）有限公司 | Method for tracking target, device, storage medium and electronic equipment |
| WO2020107524A1 (en) * | 2018-11-27 | 2020-06-04 | 上海芯仑光电科技有限公司 | Target tracking method and computing device |
-
2020
- 2020-08-27 CN CN202010879875.9A patent/CN112037259A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015161697A1 (en) * | 2014-04-25 | 2015-10-29 | 深港产学研基地 | Method and system for tracking moving object applied to man-machine interaction |
| CN104835182A (en) * | 2015-06-03 | 2015-08-12 | 上海建炜信息技术有限公司 | Method for realizing dynamic object real-time tracking by using camera |
| CN108269269A (en) * | 2016-12-30 | 2018-07-10 | 纳恩博（北京）科技有限公司 | Method for tracking target and device |
| WO2020107524A1 (en) * | 2018-11-27 | 2020-06-04 | 上海芯仑光电科技有限公司 | Target tracking method and computing device |
| CN110287855A (en) * | 2019-06-21 | 2019-09-27 | 上海闪马智能科技有限公司 | A kind of pedestrian track acquisition methods and system |
| CN110516620A (en) * | 2019-08-29 | 2019-11-29 | 腾讯科技（深圳）有限公司 | Method for tracking target, device, storage medium and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 方明等: "《视频图像光流场估计技术及应用》", 31 May 2019, 国防工业出版社, pages: 86 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116883915A (en) * | 2023-09-06 | 2023-10-13 | 常州星宇车灯股份有限公司 | Target detection method and system based on front and rear frame image association |
| CN116883915B (en) * | 2023-09-06 | 2023-11-21 | 常州星宇车灯股份有限公司 | Target detection method and system based on front and rear frame image association |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7341652B2 (en) | Information processing device, information processing method, program, and system | |
| US10810544B2 (en) | Distributed autonomous robot systems and methods | |
| US20170270579A1 (en) | Robotic equipment for the location of items in a shop and operating process thereof | |
| US11915192B2 (en) | Systems, devices, and methods for scanning a shopping space | |
| CN106347919A (en) | Automatic warehousing system | |
| US20170097232A1 (en) | Using sensor-based observations of agents in an environment to estimate the pose of an object in the environment and to estimate an uncertainty measure for the pose | |
| US11961303B1 (en) | Agent re-verification and resolution using imaging | |
| JP6722348B2 (en) | Integrated obstacle detection and payload centering sensor system | |
| CN110945449B (en) | Real-time supervision type machine learning system and method for field environment | |
| Novoselov et al. | Development of the method local navigation of mobile robot a based on the tags with QR code and wireless sensor network | |
| CN110816522B (en) | Vehicle attitude control method, apparatus, and computer-readable storage medium | |
| CN112037259A (en) | System and method for tracking dynamic target | |
| KR20190078828A (en) | Apparatus for controlling flight of unmanned aerial vehicle and method thereof | |
| CN111975776A (en) | Robot movement tracking system and method based on deep learning and Kalman filtering | |
| US20230211987A1 (en) | Pathfinding using centerline heuristics for an autonomous mobile robot | |
| US20230161356A1 (en) | Method of updating map in fusion slam and robot implementing same | |
| EP4348544A1 (en) | Using slam 3d information to optimize training and use of deep neural networks for recognition and tracking of 3d object | |
| US20230039143A1 (en) | Own-position estimating device, moving body, own-position estimating method, and own-position estimating program | |
| US20180299901A1 (en) | Hybrid Remote Retrieval System | |
| CN113632029A (en) | Information processing device, program, and information processing method | |
| RU2658092C2 (en) | Method and navigation system of the mobile object using three-dimensional sensors | |
| Aquino et al. | Image Processing Based Human Following Cart Using 360° Camera | |
| US20230266770A1 (en) | Movable platform for taking inventory and/or performing other actions on objects | |
| WO2022193762A1 (en) | Method and device for controlling transfer robot | |
| Mronga et al. | MARLIN: A cloud integrated robotic solution to support intralogistics in retail |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100020 Room 101, 1st floor, building 1, No.36 courtyard, Chuangyuan Road, Chaoyang District, Beijing Applicant after: Beijing jizhijia Technology Co.,Ltd. Address before: 100020 Room 101, 1st floor, building 1, No.36 courtyard, Chuangyuan Road, Chaoyang District, Beijing Applicant before: Beijing Geekplus Technology Co.,Ltd. |