CN117975368A - Path tracking method and warehouse information recording method thereof - Google Patents
Path tracking method and warehouse information recording method thereof Download PDFInfo
- Publication number
- CN117975368A CN117975368A CN202410178872.0A CN202410178872A CN117975368A CN 117975368 A CN117975368 A CN 117975368A CN 202410178872 A CN202410178872 A CN 202410178872A CN 117975368 A CN117975368 A CN 117975368A
- Authority
- CN
- China
- Prior art keywords
- target
- frame
- value
- warehouse
- new
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 15
- 238000001514 detection method Methods 0.000 claims description 37
- 239000011159 matrix material Substances 0.000 claims description 21
- 238000013139 quantization Methods 0.000 claims description 19
- 230000003068 static effect Effects 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 8
- 238000013135 deep learning Methods 0.000 claims description 7
- 230000000903 blocking effect Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 claims description 3
- 230000007704 transition Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 241000531116 Blitum bonus-henricus Species 0.000 claims description 2
- 235000008645 Chenopodium bonus henricus Nutrition 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims description 2
- 230000003287 optical effect Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000013138 pruning Methods 0.000 claims description 2
- 238000011002 quantification Methods 0.000 claims 3
- 239000000463 material Substances 0.000 abstract description 3
- 238000012549 training Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 238000002360 preparation method Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 231100000614 poison Toxicity 0.000 description 2
- 231100000572 poisoning Toxicity 0.000 description 2
- 230000000607 poisoning effect Effects 0.000 description 2
- 239000003440 toxic substance Substances 0.000 description 2
- 102000008297 Nuclear Matrix-Associated Proteins Human genes 0.000 description 1
- 108010035916 Nuclear Matrix-Associated Proteins Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 210000000299 nuclear matrix Anatomy 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to a cargo tracking method, in particular to a path tracking method and a warehouse information recording method thereof. According to the method, the state of the target entering and exiting is determined through the u and v changes of the target, when a new target appears, the u and v of the new target fall into the edge range marked in advance by an algorithm, and for some new targets which suddenly appear on the non-edge, the new targets are matched with the blocked targets by a Hungarian method, and if the new targets are matched, the targets which are not blocked and recorded in the completed position are treated the same way, and tracking of the track is continued. The method is particularly suitable for positioning and tracking the material target, and is convenient for generating and updating warehouse information.
Description
Technical Field
The invention relates to a cargo tracking method, in particular to a path tracking method and a warehouse information recording method thereof.
Background
The related wireless information supervision system still has high manual requirements on a plurality of steps such as labeling and checking, and the intelligent degree is not enough.
The existing technical schemes for researching positioning and recording of cargo information mainly comprise RFID and BDS; there are cases of using imaging object detection tracking to manage goods, but using machine learning, such as "a logistic warehouse monitoring method based on computer vision" in chinese patent application No. CN201510320414.7, which aims at the defect that the correlation similarity comparison is easily affected by illumination change only by using pixels in the kernel circulation tracking. Firstly, calculating a local sensitive histogram of an input image, extracting an illumination invariant feature, and then rapidly calculating a response confidence map between a target and a template in a frequency domain by using a nuclear matrix circulation structure to obtain an accurate position of the tracked target. But does not recognize the processes of target movement, occlusion, etc., the performance and coverage of the trial range are different from deep learning to complete the related tasks,
Meanwhile, the existing deep learning image target detection and tracking application in the warehouse field mainly comprises personnel tracking, such as China patent application No. 202011530890.9, intelligent warehouse personnel tracking algorithm and video monitoring system No. 202110756825.6 based on machine learning, which is a multi-target tracking of a cross camera for a warehouse scene, and the type of patent tracking personnel but not tracking goods monitoring.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a path tracking method and a warehouse information recording method thereof. According to the method, the state of the target entering and exiting is determined through the u and v changes of the target, when a new target appears, the u and v of the new target fall into the edge range marked in advance by an algorithm, and for some new targets which suddenly appear on the non-edge, the new targets are matched with the blocked targets by a Hungarian method, and if the new targets are matched, the targets which are not blocked and recorded in the completed position are treated the same way, and tracking of the track is continued. The method is particularly suitable for positioning and tracking the material target, is convenient for generating and updating warehouse information, and provides a better technical method for warehouse material dispatching.
The technical scheme of the invention is as follows: a path tracking method comprising tracking of unregistered goods and registered goods, characterized by:
The path tracking method of the unregistered goods comprises the following steps: first, modeling the state of each tracked object after completing the target detection algorithm:
Where u, v represents the two-dimensional pixel position of the center of the object, s represents the scale of the bounding box of the object, r is the aspect ratio, Representing the motion speed of the two-dimensional pixel position of the center of the object; /(I)Rate of change of aspect ratio;
Sending the modeling result as the observed value y k to the predicted boundary box position in the Kalman filter; let K be the kalman gain parameter, For outputting an estimated value, P is a covariance matrix of an input value, F is a state transition matrix, B is a control matrix, u is a control input, C is a relationship matrix for uniformly estimating input and actual observation, and R and Q are super parameters; the right subscript indicates the time of output, "K" indicates the final output at the current time, and "K-1" indicates the last output; the upper right "-" indicates that the value is an a priori estimate, the upper right "T" indicates the matrix transpose, and the prediction method is:
the predicted position in the Kalman filter is matched with the actual detection result in the next frame of image by using a Hungarian algorithm, and the specific process is as follows: n prediction frames are distributed to n actual frames, the object overlapping degree IoU of each input prediction frame and the actual frame is calculated to serve as a cost matrix, meanwhile, a threshold value of 0.3 is given to the overlapping degree IoU to define error matching, if the value of the overlapping degree IoU is smaller than 0.3, two targets are considered to be not matched, and otherwise, two targets are considered to be matched;
Setting a life frame number A for each target; the detection frame of the target is correctly matched with the prediction frame, the target is considered to be kept, the center coordinates (u, v) and (u ', v') of the target detection frame connecting the front frame and the rear frame of real images are used as the moving path of the target between the two frames, and the life frame number of the target is returned to 0; if the predicted frame of the old target is not matched with the predicted frame, the life frame number corresponding to the target is increased, the matching is tried again when a new frame of image arrives, and if the matching is repeated until the life frame number of the old target exceeds a life threshold A', the old target is determined to leave the scene to stop tracking; if a new detection frame exists, when the new detection frame is matched with an old target prediction frame within a life threshold through a Hungarian algorithm, the new target is regarded as an old target which reappears after occlusion, and the path of the new target is updated; if the new detection frame still cannot be matched with any old target prediction frame after the number of frames with the life threshold is passed, the new detection frame is considered as a new target, and a path is newly added;
Setting a static frame number B for each target, calculating the moving length L of the track according to the central coordinates (u, v) and (u ', v') of the target detection frame while updating the track for each unregistered target, and considering that the target object is moving when L is not 0, returning the static frame number of the target to 0, otherwise, increasing the value by 1, and continuing to detect; when the static frame number of a target does not move until the static threshold B' is finished, converting the u and v information corresponding to the target into actual size;
The path tracking method of the registered goods comprises the following steps: when the object moves, tracking is continued, after the life frame number reaches a life threshold A 'because of being blocked, recording the blocking frame number C is started, when the object is correctly matched in the next frame, the C value is reset to 0, otherwise, the C value is increased by 1, when the C is increased to the blocking threshold C', whether a new object exists in the position is checked, if the new object exists, the position information of the object is kept unchanged, otherwise, the object is seen to disappear in the visual field area.
The beneficial effects of the invention are as follows: by setting the number of life frames, the problem of continuous and correct tracking of the target in the moving process is solved; the real storage of the articles in the warehouse and the recording of the position information can be determined through the setting of the static frame number; the detection of the size targets and track tracking are completed through deep learning so as to record or assist in checking the article access information; the camera matrix and the target detection frame form a multi-dimensional label to realize article positioning; the system not only provides an independent management scheme, but also can comprehensively judge the system information based on the RFID or navigation in a combined way, and can give an alarm when the system information is not matched with the system information; the intelligent security scheme is provided, related pictures and 3D modeling information can be timely returned when dangerous situations are found, and a method for reducing or solving dangerous situations is provided correctly according to related security information of database objects.
Drawings
Fig. 1 is a functional diagram of a system of the present invention.
Fig. 2 is a hardware architecture diagram.
FIG. 3 is a block diagram of a model of object detection YOLOv.
Fig. 4 is a schematic structural diagram of the PTQ-QAT scheme.
Name interpretation: PTQ is called Post-Training Quantization, which is a quantization strategy for quantization after neural network training is completed, and the size and computational complexity of the model are reduced by converting the weight in the trained model from floating point number to low bit integer.
QAT is called Quantization-AWARE TRAINING, which refers to a Quantization strategy of the neural network in the process of simulating Quantization, partial nodes are replaced by pseudo Quantization nodes in the training process, subtrees with the replaced nodes are trained, and finally the summarized result is the quantized result.
The PTQ-QAT combines two modes through the characteristic of edge quantization, after the central server collects data to carry out PTQ quantization, real edge forward reasoning data is used for replacing the original QAT simulation result by using a loss function, and the quantized perception operation is carried out on the host side quantized model, so that the real quantized model running condition and accuracy can be reflected, and the time cost and the calculation cost of forward prediction of the central server host side are greatly saved.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings.
As shown in fig. 1 to 3, the implementation of the present invention uses modules such as a K210 vision chip, a 200W high-definition camera, and ESP8285WiFi on hardware. Considering that the combination of deep learning and embedded is quite common, other embedded hardware platforms with camera shooting and wired or wireless networking capabilities can be used as the case may be. Fig. 2 shows a hardware architecture diagram, which is divided into a hardware layer, a communication layer and a service layer, wherein the hardware layer comprises a camera and a singlechip, the singlechip transmits collected field signals to the communication layer, the communication layer transmits signals to the service layer in real time through a wireless wired network, the service layer exchanges data through a cloud service database, a plurality of users can read real-time data according to requirements, and the hardware part of the invention can adopt a hierarchical architecture design and has the characteristics of good compatibility, convenience for expansion and the like.
The invention can be divided in time into two phases, a preparation phase and a use phase.
The main work done in the preparation stage is the system is initialization, the specific content includes-the position arrangement and information binding of the hardware can be completed; image data acquisition of the main stored goods in the warehouse can be completed; training the deep learning network according to the acquired data; after the training is completed, the network can carry out PTQ quantization, then the model data is transmitted into the embedded equipment in the form of the network and is loaded, and a formed classification result table can be transmitted into a safety technical specification of goods related to the cloud and the warehouse to automatically construct a safety measure knowledge graph; finally, after the manual confirmation system adopts an independent operation mode or a combined operation mode and whether an unknown article selected by uncertain safety measures exists, the system formally enters a use stage.
In the using stage, the information management system can operate according to the following logic, if the system operates independently, the conditions of entering, moving out and position change in the warehouse of the warehouse target are judged by using a target detection method and a path tracking method, if the target position does not change after a set period of time, the warehouse-in and warehouse-out state is uploaded to the cloud server, and the actual position of the article is calculated according to the relative position of the target in the picture and the position and proportion information bound by hardware and uploaded to the cloud server. If the system is used in combination with other data management systems, when items in other management system databases stored in the cloud are increased, a signal is sent to hardware of a warehouse corresponding to the items, after a period of preset window time, if items in a region which is not captured by the hardware in the warehouse are newly increased, a warning prompt of 'suspected loss of warehoused goods' is sent to the cloud to remind an administrator of follow-up inspection; if the new addition of the article is detected in the window time, after the position information of the article is determined and uploaded, the position information is bound with records in other databases at the cloud; when the embedded equipment in the warehouse finds that an article moves out of the warehouse, after the cloud is uploaded, the cloud repeatedly scans other databases in a period of preset window time, and when the state of the article in the other databases is found to be unchanged until the window time is over, the cloud is triggered to give an alarm to prompt that the article out of the warehouse is suspected to be lost.
In the using stage, because of the large warehouse range, the network computing power used on the embedded system is limited, and in order to ensure the recognition accuracy, the whole system also operates a PTQ-QAT combined quantization method based on an edge model.
In the use stage, the intelligent security officer on the cloud server works based on the security measure knowledge graph generated in the preparation stage, and basic security emergency measures include, but are not limited to, fire extinguishing and evacuation schemes for fire, poisoning emergency treatment for toxic substances, leakage treatment and personnel evacuation schemes. According to the stored object table, inflammables are associated with fire schemes, toxic substances are associated with poisoning schemes, and when an alarm occurs at a certain place, a related emergency scheme can be provided according to the association of the stored objects.
The invention discloses a warehouse information recording method, which comprises the following steps:
Step 1, arranging hardware and binding information in a preparation stage, namely determining the direction of a camera, and confirming the proportional conversion relation between the object size and the picture pixels, and binding the warehouse position information corresponding to the camera. The camera lens parameters suitable for use in different warehouses are different, here referred to as cameras. Judging whether the camera belongs to horizontal shooting or vertical shooting, measuring a space angle formed by the whole platform and a horizontal plane by using an inclination sensor carried in the embedded platform so as to judge shooting conditions, and storing the data in a hardware platform to wait for networking information synchronization; besides the automatic measurement, the information of the shooting angle of the camera can be changed in the form of manual input data when the camera information is bound in the follow-up binding of the camera information. Assuming that the distance between the monitoring screen and the camera is Z, the actual size of one stored article is x×y, the focal length of the camera is f, and the center offset values of the projection point and the pixel range of the physical optical center of the camera on the imaging plane are c x and c y, the sizes u and v of the articles in the picture satisfy the following relationship:
Of these parameters, f, c x, and c y can be obtained from camera parameters, Z is the data measured when the camera is installed, uv can be obtained by measuring the camera image, and then the actual spatial position of the object in the camera can be solved. The position information of the warehouse corresponding to the hardware for storing the goods needs to be encoded according to the actual application scene, firstly, the encoding needs to inherit the encoding of the storage area, namely, the encoding is carried out according to the numbers which are sequentially and uniformly organized according to the storage specific positions of the warehouse, the goods yard, the goods shed, the goods stack, the goods shelf and the goods, and secondly, the encoding is carried out according to the shooting direction of the camera from left to right and from top to bottom according to the arrangement mode. After confirming the code, the device needs to be bound, then the cloud is connected, and the code is uploaded to become the device in the system.
And 2, acquiring and preprocessing image data of the main stored goods in the warehouse, wherein a large number of pictures containing the stored goods in the warehouse are required to be acquired, and labels comprising pixels of all goods targets are framed on each picture. The specific acquisition operation is that a camera on the embedded hardware is started to record a video in the time of warehouse access and use and then the video is uploaded to a cloud server, the server can call an API to split the video into pictures, then a YOLOv system which is preset by the cloud and used for training and identifying common storage products such as express cartons and containers is used for carrying out quick tag identification, or a manual marking tool which is built in the cloud is used for checking and modifying automatically generated tags.
The structure foundation of the target detection method in the step 3 is a network structure of YOLOv model, and certain improvement is made on the basis of the network structure, firstly, the original plate CSPDARKNET is replaced by FASTERNET adopting partial convolution on a main network, the diversity of the characteristics is reserved, lower delay is realized, and the training and reasoning speed is accelerated. The target is identified by a target detection method, which can be identified by the existing image identification method, and a result state x is generated.
The implementation process of the path tracking method in step 4 needs to be discussed in terms of two cases, namely, unregistered goods and registered goods.
For goods whose position record is not completed on the cloud server, the path tracking method is as follows, and the result state of each tracked object is modeled as follows after the target detection algorithm is completed.
Where u, v represent the two-dimensional pixel position of the center of the object, s represent the proportion of the bounding box of the object, r is the aspect ratio, and the other quantities represent the corresponding rates of change; r is the aspect ratio,Representing the motion speed of the two-dimensional pixel position of the center of the object; /(I)Representing the rate of change of aspect ratio. This result is then fed into a Kalman filter for predicting the position of its future bounding box for the observed value y k. Let K be the Kalman gain parameter,/>For outputting an estimated value, P is a covariance matrix of an input value, F is a state transition matrix, B is a control matrix, u is a control input, C is a relationship matrix for uniformly estimating input and actual observation, and R and Q are super parameters; the right subscript indicates the moment of output, e.g. "K" indicates the final output at the current moment and "K-1" indicates the last output; the upper right "-" indicates that the value is an a priori estimate, the upper right "T" indicates the matrix transpose, and the method of prediction is as follows.
In the parameters, F can be obtained by converting the input medium speed into a matrix form, B and u can be almost ignored in the problem, and the super parameters R and Q are obtained by experimental repeated adjustment. It is worth emphasizing that the x-axis of the coordinates of u and v is determined as the lower edge of the acquired picture, the origin is the leftmost endpoint, and the direction is from left to right; the y-axis of the coordinate is the left edge of the acquired picture, the origin is the lowest endpoint, and the direction is from bottom to top. The value range of u and v is [0,1], when calculating, u is obtained by dividing the value x of the pixel value at the center of the object by the corresponding picture length, and v is obtained by dividing the value x of the pixel value at the center of the object by the corresponding picture width.
The predicted position in the Kalman filter is then matched to the actual detected result in the next frame of image using the Hungarian algorithm. The algorithm completes the problem of assigning n targets to another n targets by operating the cost matrix. In the invention, n Kalman filters are needed to generate an object position prediction frame (hereinafter referred to as a prediction frame) predicted at a certain moment and are distributed to n object detection network detection result actual frames (hereinafter referred to as actual frames) at the moment, the object overlapping degree IoU of each input prediction frame and the actual frame is calculated to serve as a cost matrix, meanwhile, a threshold value of 0.3 is given to IoU to define incorrect matching, if the value of IoU is smaller than 0.3, two objects are considered to be not matched, otherwise, two objects are considered to be matched. The calculation formula of the overlapping degree IoU is as follows, wherein S o is the intersection area of two frame areas, and S u is the union area of two frame areas.
In order to solve the problem of continuous and correct tracking of targets during movement, each target has a corresponding number of life frames A. Once the actual frame of the target is correctly matched with the predicted frame, the target is considered to be kept in existence, the center coordinates (u, v) and (u ', v') of the actual frames of the real images of the two frames before and after can be connected as a path for moving the target between the two frames, and the life frame number of the target is returned to 0. If the predicted frame of the old target does not match with the actual frame, the life frame number corresponding to the target is increased, the matching is tried again when a new frame of image arrives, if the matching is carried out for a plurality of times until the life frame number of the old target exceeds the life threshold A', the scene is determined to leave for tracking, if the life threshold is set to be 5, the life frame number of the old target is increased by 1 every time the old target disappears, and when the life frame number is 5, the tracking of the old target is stopped. If the object detection network recognizes that a new object appears, the new object is considered as an old object that reappears after occlusion when the actual box of the object can be matched with the old object prediction box within the life threshold by the Hungarian algorithm, and its path is updated. If the new actual frame still cannot match any old target prediction frame after the number of frames experiencing the life threshold, then it is considered to be a new target and a new path is added to it.
In order to determine what condition the articles are actually stored in the warehouse and can record the position information, each warehouse target also has a static frame number B, the moving length L of the track is calculated according to the result obtained by the target detection when the track is updated for each frame of unregistered target, namely, the center coordinates (u, v) and (u ', v') of the target, when L is not 0, the target articles are considered to be moving, the static frame number of the target is normalized to 0, otherwise, the value is increased by 1, and the detection is continued. When the static frame number of a target does not move until the static threshold B' is finished, converting the u and v information corresponding to the target into the actual size X X Y by using the conversion formula mentioned in the step 1, and uploading the cloud completion position record by using a WiFi local area network after the calculation of the auxiliary binding information is completed, and then storing the coordinate frame information of the target.
For the purpose of completing the position registration, there are two possible changes that may occur: move or be blocked. When the target moves, the target which does not finish the position record is processed by the same process to track the target continuously. When the number of life frames of the article reaches a life threshold A 'because the article is blocked, recording a blocking frame number C, returning the value of C to 0 if the article is correctly matched in the next frame, otherwise increasing the value of C by 1, and when the value of C is increased to the blocking threshold C', checking whether a new target exists at the position, if the new target exists, keeping the position information of the article unchanged, uploading the coordinate frame of the article to the cloud and adding information of 'blocked', and registering the position of the new article, otherwise, triggering an alarm of 'article position recording abnormality 1' according to the disappearance of the article in the visual field area, and requesting the administrator to confirm.
The state of the object going out and going into storage can be determined by tracking the u, v change condition of the object through a path tracking method, when a new object appears, the u, v of the new object falls into the margin range marked in advance by an algorithm, namely the margin (0-a, 0+a), (x-a, x+a), (0-a, 0+a) and (y-a, y+a) in the x-axis direction for a horizontal camera, the margin (x-a, ±x+a) in the x-axis direction for a vertical camera, the margin a is the margin of error, the x, y are the margin coordinate values of the stock respectively, and then the rest position information of the new object is uploaded to the cloud, and then the object is considered as the object going into storage. For some new targets which suddenly appear on the non-edge, the new targets are matched with the blocked targets by a Hungarian method, if the new targets are matched with the blocked targets, the targets which are not blocked and recorded in the completion position are treated as the same, and tracking of the track is continued; otherwise, triggering an alarm of 'article position record abnormal 2', and requesting an administrator to confirm. For other targets with track records, when u and v fall into the edge range marked in advance by an algorithm again, and the Hungarian method matching is lost in the near future, if the target is a target which is not subjected to cloud position registration, the target is considered to be a passing target, and the system does not perform other processing; if the target is the target with the cloud position registered, the target is considered to be out of the warehouse at the moment, and the information of the target is synchronously sent to the cloud.
The PTQ-QAT combined quantization method based on the edge model in the step 5 mainly comprises three stages. The PTQ-QAT combined quantization method based on the edge model has the advantages that the model can be corrected regularly through the cloud platform, so that higher precision can be kept on the embedded chip, and the problem that the time and the calculation force are required to be large for the quantization perception training with large difficulty in identifying the number of classes in other schemes can be solved.
In the first stage, since the YOLOv program developed based on PyTorch mechanical learning library used in the target detection method cannot be directly used in the embedded device, after model training in the cloud server is completed, a built-in function "torch.onnx.export" provided by PyTorch is required to complete conversion from the trained YOLOv model to ONNX format, then the ONNX model is converted to TFLite format by corresponding format conversion functions in other mechanical learning related libraries onnx and tflite libraries in the Python, pruning of a convolutional layer of the YOLOv model and PTQ quantization of the type of weight to int16 of the type of FP32 are completed by functions in TFLite library, and finally the number of layers of the quantized deep learning network model is changed from complete N layers to N layers. The quantized result is transmitted to all hardware devices through the local area network, so that the hardware devices start to identify tasks. After quantization is completed, the server host also generates a pseudo quantization operator model containing QAT, and initial values of quantization parameters S and Z required in QAT are determined by the result of PTQ.
The second stage is edge feedback stage, when the edge terminal normally performs the goods carton identification task, the edge terminal sends a frame of picture currently shot to the host computer at intervals of fixed time intervals T, and the operation result of the edge terminal reasoning prediction(Including prediction category cls, target bounding box bbox, target box height h, target box width w, prediction anchor box anchor, etc.). Aiming at different light-weight target detection networks, setting the final result of an N-layer model quantized by a model with the total layer number of N before quantization as/>(The method is generally obtained through direct back propagation calculation of a server and is used for replacing a loss function to perform operation for improving accuracy. ). Then/>Can also be described as
The third stage is a model updating stage, firstly, the current frame picture returned in the second stage generates a label result e through a full-precision model, and then the label result e is subjected to counter propagation through a loss function of the QAT, and gradient is reduced. And (5) overlaying the original model on the edge terminal sent by the model, and returning to the second stage to perform cyclic reciprocation. Let us say that according to the loss function L of the different networks,For the set of parameters in the network currently,/>For the collection of parameters in network update, the loss value calculated by edge operation is/>Η is a superparameter, the process may be described as
And 6, constructing a safety measure emergency scheme library, wherein the technical form of the library is a knowledge graph, and the construction steps mainly comprise data collection, knowledge extraction and graph construction. The method for collecting the data is to upload a safety technical specification of related goods, or use a web crawler to perform data crawling on a network by taking all warehouse targets obtained in the step 2 and the step 3 as keywords, then perform data cleaning, and finally obtain reasonable corpus. The corpus obtained at present is unstructured data, so that the corpus can be used by knowledge extraction, and the specific steps are word segmentation, entity extraction, relation extraction and attribute extraction. And then, constructing a central node taking a warehouse target as a map through contents such as attributes, relations and the like. In this way, the required safety database can be finally obtained, and when the designated object gives an alarm, the database can be searched to complete emergency treatment.
Claims (8)
1. A path tracking method comprising tracking of unregistered goods and registered goods, characterized by:
The path tracking method of the unregistered goods comprises the following steps: first, modeling the state of each tracked object after completing the target detection algorithm:
Where u, v represents the two-dimensional pixel position of the center of the object, s represents the scale of the bounding box of the object, r is the aspect ratio, Representing the motion speed of the two-dimensional pixel position of the center of the object; /(I)Rate of change of aspect ratio;
Sending the modeling result as the observed value y k to the predicted boundary box position in the Kalman filter; let K be the kalman gain parameter, For outputting an estimated value, P is a covariance matrix of an input value, F is a state transition matrix, B is a control matrix, u is a control input, C is a relationship matrix for uniformly estimating input and actual observation, and R and Q are super parameters; the right subscript indicates the time of output, "K" indicates the final output at the current time, and "K-1" indicates the last output; the upper right "-" indicates that the value is an a priori estimate, the upper right "T" indicates the matrix transpose, and the prediction method is:
the predicted position in the Kalman filter is matched with the actual detection result in the next frame of image by using a Hungarian algorithm, and the specific process is as follows: n prediction frames are distributed to n actual frames, the object overlapping degree IoU of each input prediction frame and the actual frame is calculated to serve as a cost matrix, meanwhile, a threshold value of 0.3 is given to the overlapping degree IoU to define error matching, if the value of the overlapping degree IoU is smaller than 0.3, two targets are considered to be not matched, and otherwise, two targets are considered to be matched;
Setting a life frame number A for each target; the detection frame of the target is correctly matched with the prediction frame, the target is considered to be kept, the center coordinates (u, v) and (u ', v') of the target detection frame connecting the front frame and the rear frame of real images are used as the moving path of the target between the two frames, and the life frame number of the target is returned to 0; if the predicted frame of the old target is not matched with the predicted frame, the life frame number corresponding to the target is increased, the matching is tried again when a new frame of image arrives, and if the matching is repeated until the life frame number of the old target exceeds a life threshold A', the old target is determined to leave the scene to stop tracking; if a new detection frame exists, when the new detection frame is matched with an old target prediction frame within a life threshold through a Hungarian algorithm, the new target is regarded as an old target which reappears after occlusion, and the path of the new target is updated; if the new detection frame still cannot be matched with any old target prediction frame after the number of frames with the life threshold is passed, the new detection frame is considered as a new target, and a path is newly added;
Setting a static frame number B for each target, calculating the moving length L of the track according to the central coordinates (u, v) and (u ', v') of the target detection frame while updating the track for each unregistered target, and considering that the target object is moving when L is not 0, returning the static frame number of the target to 0, otherwise, increasing the value by 1, and continuing to detect; when the static frame number of a target does not move until the static threshold B' is finished, converting the u and v information corresponding to the target into actual size;
The path tracking method of the registered goods comprises the following steps: when the object moves, tracking is continued, after the life frame number reaches a life threshold A 'because of being blocked, recording the blocking frame number C is started, when the object is correctly matched in the next frame, the C value is reset to 0, otherwise, the C value is increased by 1, when the C is increased to the blocking threshold C', whether a new object exists in the position is checked, if the new object exists, the position information of the object is kept unchanged, otherwise, the object is seen to disappear in the visual field area.
2. A path tracing method according to claim 1, wherein: the x axis of the u and v coordinates is the lower edge of the acquired picture, the origin is the leftmost endpoint, and the direction is from left to right; the coordinate y-axis is the left edge of the acquired picture, the origin is the lowest endpoint, and the direction is from bottom to top; the value range of u and v is [0,1], when calculating, u is obtained by dividing the value x of the pixel value at the center of the object by the corresponding picture length, and v is obtained by dividing the value x of the pixel value at the center of the object by the corresponding picture width.
3. A path tracing method according to claim 1, wherein: the calculation formula of the overlap IoU is as follows,
Where S o is the intersection area of the two frame areas and S u is the union area of the two frame areas.
4. A method for recording warehouse information, which is characterized in that: the method comprises the following steps:
step 1, determining the direction of a camera, confirming the proportional conversion relation between the object size and the picture pixels and binding warehouse position information corresponding to the camera,
Step 2, image data acquisition and preprocessing of goods stored in a warehouse, namely acquiring a picture containing the goods stored in the warehouse and a label of pixels of all goods targets framed on each picture;
Step 3, identifying a target through a target detection method;
Step 4, tracking a target path; target path tracking employs a path tracking method as claimed in any one of claims 1 to 3.
5. A method of warehouse information logging as defined in claim 4, wherein: warehouse position information corresponding to the camera is: assuming that the distance between the monitoring screen and the camera is Z, the actual size of the stored goods is x×y, the focal length of the camera is f, and the center offset values of the projection point of the physical optical center of the camera on the imaging plane and the pixel range are c x and c y, the sizes u and v of the goods in the picture satisfy the following relationship:
The position information of the warehouse corresponding to the hardware for storing the goods needs to be encoded according to the actual application scene.
6. A method of warehouse information logging as defined in claim 4, wherein: the structure of the target detection method in the step 3 is a network structure of YOLOv model.
7. A method of warehouse information logging as defined in claim 4, wherein: the method also comprises a step 5, wherein the step 5 comprises the PTQ-QAT joint quantization step based on an edge model, and specifically comprises the following steps:
The first stage is an initialization stage, the trained YOLOv model is converted into ONNX format, the ONNX model is converted into TFLite format through corresponding format conversion functions in a machine learning related library onnx library and a tflite library in Python, pruning of a convolution layer of the YOLOv model and quantification of a weight of an FP32 type to a PTQ of an int16 type are completed through functions in the TFLite library, and the number of layers of the quantized deep learning network model is changed from a complete N layer to an N layer;
The second stage is an edge feedback stage, and when the goods carton identification task is normally carried out, a frame of picture shot currently and an operation result of edge terminal reasoning prediction are sent to a host computer at fixed time intervals T; setting the final result of the N-layer model after the model quantification with the total layer number of N before quantification according to different light-weight target detection networks, and calculating the result Is that
The third stage is a model updating stage, the current frame picture returned by the second stage generates a label result e by the full-precision model, according to the loss function L of different networks,For the set of parameters in the network currently,/>For the collection of parameters in network update, the loss value calculated by edge operation is/>Eta is a superparameter and the process is described as
8. A method of warehouse information logging as defined in claim 4, wherein: the method also comprises a step 6, wherein the step 6 is the construction of the intelligent map, and comprises data collection, knowledge extraction and map construction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410178872.0A CN117975368A (en) | 2024-02-17 | 2024-02-17 | Path tracking method and warehouse information recording method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410178872.0A CN117975368A (en) | 2024-02-17 | 2024-02-17 | Path tracking method and warehouse information recording method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117975368A true CN117975368A (en) | 2024-05-03 |
Family
ID=90859264
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410178872.0A Pending CN117975368A (en) | 2024-02-17 | 2024-02-17 | Path tracking method and warehouse information recording method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117975368A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118194608A (en) * | 2024-05-16 | 2024-06-14 | 深圳市辉熙智能科技有限公司 | Material monitoring and 3D simulation method and system based on digital twinning |
-
2024
- 2024-02-17 CN CN202410178872.0A patent/CN117975368A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118194608A (en) * | 2024-05-16 | 2024-06-14 | 深圳市辉熙智能科技有限公司 | Material monitoring and 3D simulation method and system based on digital twinning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109685066B (en) | Mine target detection and identification method based on deep convolutional neural network | |
| CN110163904B (en) | Object labeling method, movement control method, device, equipment and storage medium | |
| Medioni et al. | Event detection and analysis from video streams | |
| CN112734852B (en) | Robot mapping method and device and computing equipment | |
| CN111563442A (en) | Slam method and system for fusing point cloud and camera image data based on laser radar | |
| Teizer et al. | Personnel tracking on construction sites using video cameras | |
| US8447069B2 (en) | Apparatus and method for moving object detection | |
| US9361702B2 (en) | Image detection method and device | |
| EP2131328A2 (en) | Method for automatic detection and tracking of multiple objects | |
| CN113674328A (en) | Multi-target vehicle tracking method | |
| US20220391796A1 (en) | System and Method for Mapping Risks in a Warehouse Environment | |
| CN117975368A (en) | Path tracking method and warehouse information recording method thereof | |
| GB2528669A (en) | Image Analysis Method | |
| Ji et al. | RGB-D SLAM using vanishing point and door plate information in corridor environment | |
| CN110046677B (en) | Data preprocessing method, map construction method, loop detection method and system | |
| CN112541403B (en) | Indoor personnel falling detection method by utilizing infrared camera | |
| Arsic et al. | Applying multi layer homography for multi camera person tracking | |
| Konstantinidis et al. | AROWA: An autonomous robot framework for Warehouse 4.0 health and safety inspection operations | |
| GB2605948A (en) | Warehouse monitoring system | |
| Börcs et al. | Dynamic 3D environment perception and reconstruction using a mobile rotating multi-beam Lidar scanner | |
| CN113158816B (en) | Construction method of visual odometer quadric road sign for outdoor scene object | |
| Panahi et al. | Automated Progress Monitoring in Modular Construction Factories Using Computer Vision and Building Information Modeling | |
| KR20220083347A (en) | Method, apparatus, and computer program for measuring volume of objects by using image | |
| CN115457494A (en) | Object identification method and system based on infrared image and depth information fusion | |
| Niblock et al. | Fast model-based feature matching technique applied to airport lighting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |