CN112598660B - Automatic detection method for pulp cargo quantity in wharf loading and unloading process - Google Patents

Abstract

The invention discloses an automatic detection method for the quantity of paper pulp cargos in the loading and unloading process of a wharf, which comprises the following steps: s1: extracting a video stream of a transport vehicle loaded with pulp goods in real time at a certain frequency; s2: processing the extracted video image to segment a characteristic diagram of the paper pulp cargo part; s3: processing the characteristic diagram to obtain candidate connection points; s4: extracting a series of candidate line segment samples according to the candidate connecting points; s5: filtering the candidate line segment samples; s6: and counting the number of the candidate line segment samples to obtain the number of the pulp cargos. The invention can automatically detect the quantity of paper pulp goods, reduce the task amount and improve the detection efficiency, has no manual participation, has high intelligent degree and improves the detection accuracy.

Description

A method for automatic detection of pulp cargo quantity in terminal loading and unloading process

technical field

The invention relates to the technical field of computer vision and deep learning, in particular to an automatic detection method for the quantity of pulp cargo in the loading and unloading process of a wharf.

Background technique

At present, the terminal business is booming, and pulp cargo needs to be transported by means of transport (such as flatbed trucks) during the loading and unloading process at the terminal. Pulp cargo has the characteristics of one bundle and one package. During the loading and unloading process of pulp cargo, it is necessary to manually count the pulp cargo. After the manual inventory is carried out, the paper records the counted quantity, and then the truck can be released.

Such manual inventory work not only has a large workload, increases labor costs, but also has low efficiency. At the same time, there is a problem that the inventory quantity is wrong due to manual fatigue or negligence.

Therefore, there is a need for an intelligent and automated method for detecting the quantity of pulp cargoes, which can reduce the amount of tasks and improve the efficiency and accuracy of inventory.

SUMMARY OF THE INVENTION

The embodiment of the present invention provides an automatic detection method for the quantity of pulp cargo in the loading and unloading process of the terminal. First, the position of the pulp cargo part in the video image is determined, and then the quantity of the detected pulp cargo part is detected, so as to automatically detect the quantity of the pulp cargo and reduce the The workload is high and the detection efficiency is improved, no manual participation is required, the degree of intelligence is high, and the detection accuracy is improved.

In order to realize the above-mentioned purpose of the invention, the present invention adopts the following technical scheme to realize:

The present application relates to an automatic detection method for the quantity of pulp cargo in the loading and unloading process of a terminal, which is characterized in that it includes the following steps:

S1: Real-time extraction of the video stream of the transport vehicle loaded with pulp cargo at a certain frequency;

S2: Process the extracted video image, and segment the feature map of the pulp cargo part;

S3: Process the feature map to obtain candidate connection points;

S4: extracting a series of candidate line segment samples according to the candidate connection points;

S5: filter the candidate line segment samples;

S6: Count the number of the candidate line segment samples to obtain the number of pulp cargoes.

In this application, step S2 includes the following:

S21: Input the extracted video image to a residual network to obtain a feature image;

S22: Process the feature image to extract a rectangular candidate frame that distinguishes the foreground and the background;

S23: Map the extracted rectangular candidate frame to the feature image, and unify the window size of the rectangular candidate frame by using regional feature aggregation technology;

S24: Segment the feature image into a feature map of the pulp cargo portion according to the coordinate information of the rectangular candidate frame.

In the present application, the S2 also includes the following steps after S23 and before S24: S23': perform boundary frame regression on the rectangular candidate frame after the unified window in S23, so as to perform the coordinate information of the rectangular candidate frame. Correction.

In this application, step S3 includes the following steps:

S31: Perform grid division on the feature map to form M grid units W x ×H x ;

S32: Input the feature map to the convolution layer and the classification layer in turn for processing to calculate the confidence of each grid unit, and convert the feature map processed by the convolution layer into a connection point offset feature Figure O( x ),

；

;

表示网格单元x的中心位置；where V represents the point set, li represents the position of a connection point i in the point set V in the grid unit x ,

Represents the center position of the grid cell x ;

S33: Thresholding the calculated confidence level of each grid unit, obtaining a probability feature map P( x ) and classifying whether there is a connection point in each grid unit,

；

;

S34: Use the connection point offset feature map O( x ) to predict the relative position of the connection point in the corresponding grid unit;

S35: Use linear regression to optimize the relative positions of the connection points in the corresponding grid cells.

In the present application, step S3 further includes the following step: acquiring precise relative position information of the connection points in the grid unit by using a non-maximum value suppression technique.

In this application, step S4 includes:

S41: outputting the endpoint coordinate information of a series of line segment samples by adopting a mixed sampling mechanism of positive and negative samples;

S42: According to the endpoint coordinate information of each line segment sample, perform a fixed-length vectorization process on each line segment sample, and obtain a feature vector of each line segment sample, so as to extract a series of line segment samples.

In the present application, step S5 is specifically: filtering each line segment sample by using the intersection ratio between the areas of each rectangular frame formed by each line segment sample as an intersecting line.

In the present application, step S5 is specifically: filtering each line segment sample by using the Euclidean distance between each line segment sample.

The automatic detection method for the quantity of pulp cargo involved in this application has the following advantages and beneficial effects:

After the pulp cargo arrives at the port, the pulp cargo is loaded on the means of transportation, and the video stream of the means of transport loaded with the pulp cargo is monitored in real time on the monitoring screen, and the video image is extracted from the real-time video stream, and the pulp cargo is detected according to the video image. Part, combined with the characteristics of one bundle of pulp cargo, and then perform line segment detection according to the detected pulp cargo part, so as to realize the detection of the quantity of pulp cargo. The whole process does not require manual participation, reduces the amount of manual tasks, and realizes intelligence and automation. And the computer is used for calculation, the detection speed is fast, and the detection efficiency is improved.

Other features and advantages of the present invention will become more apparent after reading the detailed description of the present invention in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

Fig. 1 is the flow chart of one embodiment of the method for automatic detection of pulp cargo quantity in terminal loading and unloading process proposed by the present invention;

Fig. 2 is the original drawing of part of the pulp cargo involved in the embodiment of the method for automatic detection of pulp cargo quantity in the terminal loading and unloading process proposed by the present invention;

3 is a diagram of candidate connection points obtained in the embodiment of the method for automatic detection of the quantity of pulp cargo in the terminal loading and unloading process proposed by the present invention;

4 is a diagram of a real candidate line segment sample obtained in the embodiment of the method for automatic detection of pulp cargo quantity in the terminal loading and unloading process proposed by the present invention;

5 is a final detection diagram of mapping the true candidate line segment samples to the original image of the pulp cargo part in the embodiment of the method for automatic detection of pulp cargo quantity in the terminal loading and unloading process proposed by the present invention;

Fig. 6 is the video image that is extracted;

FIG. 7 is an effect diagram of the on-site detection of pulp cargo on the video image in FIG. 6 by using the method for automatic detection of pulp cargo quantity in the terminal loading and unloading process proposed by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments.

In order to avoid manual counting of the quantity of pulp cargo bales during loading and unloading at the wharf, the present application relates to an automatic detection method for the quantity of pulp cargoes. The main task of the method for automatic detection of pulp cargo quantity is to detect pulp cargo packages and calculate the contained pulp cargo quantity for all vehicles passing by in the surveillance video.

In this task, in order to realize the automatic detection of the quantity of pulp cargo on the transportation vehicle, the target detection and the line segment detection are comprehensively used.

Object detection mainly has two tasks: object classification and localization, which needs to separate foreground objects from the background, and determine the category and location information of the foreground.

Current target detection algorithms can be divided into candidate region-based and end-to-end based methods. In terms of detection accuracy and positioning accuracy, candidate region-based methods are more dominant. At the same time, due to the irregular placement of pulp cargoes, it is difficult to obtain auxiliary information such as volume and midpoint, which increases the difficulty for further determining the quantity of pulp cargoes.

Pulp cargo has the characteristic of being bundled in a bundle. Combined with the current line segment detection technology, the line segment detection method can effectively extract the quantity information.

Referring to Fig. 1 to Fig. 7, the realization of the method for automatic detection of pulp cargo quantity will be described in detail as follows.

S1: Real-time extraction of video streams of vehicles loaded with pulp cargo at a certain frequency.

After the pulp cargo arrives at the port, it is loaded onto a transport vehicle, such as a transport truck, which is monitored through the monitoring screen. In the monitored video stream, a certain frequency of frames is extracted according to the number of frames per second, and the video image is captured. Refer to Figure 2.

It should be noted that each package of pulp cargo loaded on the means of transport is basically a cuboid, and the height and cross-sectional area are basically the same, and are bundled into a package by straps (such as steel wires).

S2: Process the extracted video image to segment the feature map of the pulp cargo part.

The task of step S2 is to realize target detection, which is divided into the following parts for execution.

S21: The video image is input to the residual network to obtain the feature image.

In order to extract the semantic features of the video images and improve the ability to obtain the detailed features of the images, the video images are input into the residual network to obtain the feature map.

The residual network is an asymmetric coding-decoding structure fused with dilated convolutions.

In order to facilitate subsequent processing, the input video images of different sizes are first unified into a 512*512 square, and then sequentially input into multiple coding and decoding modules.

In the coding and decoding module, the coding part includes 3 convolution operations with a stride of 2, and there are 6 residual connection blocks after the convolution of each size, in which the convolution kernel used in the residual connection block performs The expansion rate of 2 is processed in order to obtain a larger receptive field. The decoding part restores the image to the input size.

The asymmetric coding and decoding structure is more sufficient for the extraction of details, and at the same time integrates a larger receptive field, so the residual module can provide rich details for subsequent processing.

S22: Process the feature image to extract a rectangular candidate frame that distinguishes the foreground and the background.

First, generate 9 rectangular boxes containing 3 shapes (aspect ratio ∈ {1:1, 1:2, 2:1}), iterate over each point in the feature image, and match these 9 for each point Rectangular frame, the rectangular frame that belongs to the foreground is judged by the Softmax classifier, which is a two-class problem; at the same time, the coordinate information of the rectangular frame is corrected by using the border boundary regression to form a more accurate rectangular candidate frame.

It should be noted that the foreground refers to the pulp cargo, and the background refers to the part other than the pulp cargo part.

S23: Map the extracted rectangular candidate frame to the feature image, and use the regional feature aggregation technology to unify the window size of the rectangular candidate frame.

The region feature aggregation technique is proposed when used in Mask RCNN to generate a fixed-size feature map from the generated candidate frame region proposal mapping, which is a technique commonly used in existing instance segmentation architectures.

S24: Segment the feature image according to the coordinate information of the rectangular candidate frame to obtain the feature map of the pulp cargo part.

According to the coordinate information of each rectangular candidate frame, the rectangular candidate frame belonging to the foreground is found, and the feature map of the pulp cargo part is segmented.

In order to ensure the accuracy of the rectangular candidate frame and improve the automatic detection accuracy of the pulp cargo quantity, before dividing the feature map of the pulp cargo part, the rectangular candidate frame obtained in S23 is subjected to boundary frame regression in sequence, and the rectangular candidate frame obtained in S23 The coordinate information is corrected to achieve accurate coordinate information.

According to the rectangular candidate frame with accurate coordinate information, the feature image of the pulp cargo part is segmented according to the coordinate information of the rectangular candidate frame.

The feature map of the pulp cargo part is segmented, and the target detection is completed.

After that, line segment detection needs to be performed on the feature map of the pulp cargo part to detect the quantity of pulp cargo in the video image.

The specific line segment detection part is described below with reference to FIGS. 1 to 7 .

S3: Process the feature map to obtain candidate connection points.

The feature map here refers to the feature map of the pulp cargo portion segmented in S24.

S31 : Perform grid division on the feature map to form M grid units Wx×Hx .

The feature map is gridded, and the W×H feature map is divided into M grid units, and the grid area of the grid unit is Wx×Hx , where V represents the point set.

In a certain grid unit x , it is necessary to predict whether there is a candidate connection point, and if there is a connection point, predict the relative position of the connection point in the grid unit x .

S32: Input the feature map to the convolutional layer and the classification layer in turn for processing to calculate the confidence of each grid unit, and convert the feature map processed by the convolutional layer into a connection point offset feature map O(x).

Specifically, a network including a 1*1 convolutional layer and a classification layer is used to process the feature map, and in the classification layer, the softmax classification function is used to calculate the confidence of whether there is a connection point in each grid unit.

And a network containing 1*1 convolutional layers is used to convert the feature map into a connection point offset feature map O( x ), as follows:

（1）。

(1).

表示网格单元x的中心位置。Among them, li represents the position of a connection point i in the point set V in the grid unit x ,

Represents the center position of grid cell x .

S33: Threshold the calculated confidence level of each grid unit, obtain a probability feature map P( x ), and classify whether there is a connection point in each grid unit.

The probability feature map P( x ) is as follows:

（2）。

(2).

That is, whether there are connection points in grid cells is a binary classification problem.

The calculated confidence of each grid unit is limited by a threshold p. If the confidence of grid unit x is greater than the threshold, then according to formula (3), P( x )=1, it is considered that the grid unit x exists in connection point, otherwise P( x )=0, it is considered that there is no connection point in the grid unit x .

If it is predicted that there is a connection point in the grid cell, continue to predict the relative position of the connection point in the grid cell (this part is described in detail below).

S34: Predict the relative position of the connection point in the corresponding grid cell using the connection point offset feature map O( x ).

O( x ) is set as the relative coordinate difference between the center point of the grid cell and the connection point i , which is used to predict the relative position of the connection point i in the grid cell x .

S35: Use linear regression to optimize the relative positions of the connection points in the corresponding grid cells.

If the grid cell x contains the connection point i, choose to use L2 linear regression to optimize the relative position of the connection point. The objective function of the L2 linear regression is:

（3）。

(3).

where Nv represents the number of connection points.

In addition, non-maximum suppression techniques are employed to further exclude "non-connected points" in each network element, ie to obtain more precise relative position information of connected points in a network element.

Specifically, it can be implemented through the maximum pooling operation in the program, which is used to obtain more accurate relative position information of the connection points in the network unit.

，参考图3。After the processes of S31 to S35 as described above, the relative positional relationship of the K candidate connection points with the highest confidence is finally output

, refer to Figure 3.

It should be noted that before step S3 is actually performed, the entire model in S3 needs to be trained by using the cross-entropy loss function. The trained model can be used directly in actual use, and when the feature map is used as input, the output is obtained. Candidate connection points as described above.

S4: Extract a series of candidate line segment samples according to the candidate connection points obtained in S3.

，获取T个候选线段样本

，其中

和

表示第z个候选线段样本的端点坐标。The purpose of this step is based on the K candidate connection points obtained in S3

, get T candidate line segment samples

,in

and

Represents the endpoint coordinates of the zth candidate line segment sample.

S41: Obtain the endpoint coordinate information of the T candidate line segment samples by adopting a mixed sampling mechanism of positive and negative samples.

It should be noted that the mixed sampling of positive and negative samples is the preparation work for model training. During the training process, the number of positive and negative samples of the K candidate connection points is greatly different, and the number of positive and negative samples needs to be balanced. The way of sample mixed training, in which positive samples are from real line segment samples marked, and negative samples are non-real line segment samples randomly generated through heuristic learning.

When there are almost no accurate positive samples in the extracted K candidate connection points or the training is saturated, adding quantitative positive/negative samples helps start the training. Moreover, the added positive samples help the prediction points to adjust their positions and improve the prediction performance.

S41: Perform vectorization processing on each line segment sample with a fixed length according to the endpoint coordinate information of each line segment sample, and obtain a feature vector of each line segment sample, so as to extract a series of candidate line segment samples.

和

对线段样本进行固定长度的向量化处理，即通过两个端点坐标计算出N _l 个均匀分布点，并在步骤S2中输出的特征图上通过双线性插值得到中间点坐标：According to a certain candidate line segment sample, such as the coordinates of the two endpoints of the zth candidate line segment sample

and

The fixed-length vectorization process is performed on the line segment samples, that is, N _l uniformly distributed points are calculated through the coordinates of the two endpoints, and the coordinates of the intermediate points are obtained by bilinear interpolation on the feature map output in step S2:

（5）。

(5).

In this way, the feature vector q of the line segment sample is extracted, which is C × N _l , where C is the number of channels of the feature map output in step S2.

At this point, the candidate line segment samples are extracted.

It should be noted that, before obtaining a series of candidate line segment samples through the endpoint coordinate information of the line segment samples, model training is required.

In the training process, the feature map output in step S2 and the candidate line segment samples output in step S4 are required as input.

The training process is briefly described as follows:

和

对线段样本进行固定长度的向量化处理，即通过两个端点坐标计算出N _l 个均匀分布点，并在步骤S2中输出的特征图上通过双线性插值得到中间点坐标：First, according to a candidate line segment sample, for example, the coordinates of the two endpoints of the zth candidate line segment sample

and

The fixed-length vectorization process is performed on the line segment samples, that is, N _l uniformly distributed points are calculated through the coordinates of the two endpoints, and the coordinates of the intermediate points are obtained by bilinear interpolation on the feature map output in step S2:

。

.

Then, the feature vector q is dimensionally reduced by a one-dimensional max-pooling operation with stride s to become C × N _l /s, and expanded into a one-dimensional feature vector.

。The one-dimensional feature vector is input into the fully connected layer for convolution processing to obtain a logical value. Specifically, the one-dimensional feature vector is subjected to two fully connected convolutions, and the log value is taken and returned.

.

与真实值y一起进行sigmoid损失计算并模型优化，以提高预测准确率，其中损失函数如下：

Perform sigmoid loss calculation and model optimization together with the true value y to improve the prediction accuracy, where the loss function is as follows:

（6）。

(6).

The true value y is the value returned by log after the eigenvalue of the true label of the line segment sample is convolved in the feature map output in S2.

）和该线段样本的真实标签对应的log值（即y）之间的误差，用于模型训练及优化。The loss is the log value of the computed prediction (i.e.

) and the log value (ie y) corresponding to the true label of the line segment sample, which is used for model training and optimization.

Since the phenomenon of repeated detection will inevitably occur in the detection, that is, two line segment samples overlap, therefore, in order to improve the detection accuracy of the line segment samples, it is necessary to filter the line segment samples output in S4.

The above filtering method is implemented in step S5, that is, using the intersection-over-union (IoU) ratio between the areas of each rectangular frame formed by each line segment sample as an intersecting line to filter each line segment sample, or using each line segment sample. The Euclidean distance between each line segment samples.

The filtering method is not limited here, as long as the purpose of filtering can be achieved.

Take the IoU filtering line segment sample as an example to illustrate.

Suppose the coordinates of two coincident line segment samples are L ₁ [( x ₁₁ , y ₁₁ ), ( x ₁₂ , y ₁₂ )] and L ₂ [( x ₂₁ , y ₂₁ ), ( x ₂₂ , y ₂₂ )], respectively.

The length H ₁ , the width W ₁ and its area A ₁ of the rectangular frame R1 formed by L ₁ , and the length H ₂ , the width W ₂ and its area A ₂ of the rectangular frame R2 formed by L ₂ are expressed as:

The length H, width H and area A of the intersection of the two rectangular frames R1 and R2 are respectively expressed as:

或

，IoU=0。否则，使用如下公式计算IoU：if

or

, IoU=0. Otherwise, calculate the IoU using the following formula:

Limit the IoU threshold, adjust the IoU threshold for filtering, and finally output the filtered line segment samples, see Figure 4.

S6: Count the number of filtered line segment samples to obtain the number of pulp cargoes.

The number of filtered line segment samples is the number of pulp obtained.

Figure 5 is the final detection map of the line segment candidate sample in Figure 4 mapped to the original image of the pulp cargo. It can be seen from the detection effect that the automatic detection of the quantity of pulp cargo is accurate.

Referring to Fig. 6 and Fig. 7, Fig. 6 is a captured original video image; Fig. 7 is an effect diagram of the on-site inspection of pulp cargoes applied to the video image in Fig. 6 by using the method for automatic detection of pulp cargo quantity proposed by the present application, and its output The number of line segment samples, that is, the number of pulp cargoes.

It can be seen from the detection effect diagrams in FIGS. 5 and 7 that the automatic detection method for the quantity of pulp cargo proposed in the present application has high detection accuracy.

Moreover, the method for automatic detection of pulp cargo quantity of the present application directly extracts video images from the video stream for detection, which is completely automated without manual participation, reduces the amount of manual tasks, and has intelligent computer calculation, fast detection speed, and improved detection efficiency.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art can still The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions claimed in the present invention.

Claims (7)

1. a method for automatic detection of pulp cargo quantity in a dock loading and unloading process, is characterized in that, comprises the steps: S1: Real-time extraction of the video stream of the transport vehicle loaded with pulp cargo at a certain frequency; S2: Process the extracted video image, and segment the feature map of the pulp cargo part; S3: Process the feature map to obtain K candidate connection points, where K>1; S4: extracting a series of candidate line segment samples according to the candidate connection points; S5: filter the candidate line segment samples; S6: Count the number of the candidate line segment samples to obtain the number of pulp cargoes; Wherein step S3 includes the following: S31: Perform grid division on the feature map to form M grid units W x ×H x ; S32: Input the feature map to the convolution layer and the classification layer in turn for processing to calculate the confidence of each grid unit, and convert the feature map processed by the convolution layer into a connection point offset feature Figure O( x ),

; where V represents the point set, li represents the position of a connection point i in the point set V in the grid unit x ,

Represents the center position of the grid cell x ; S33: Thresholding the calculated confidence level of each grid unit, obtaining a probability feature map P( x ) and classifying whether there is a connection point in each grid unit,

; S34: Use the connection point offset feature map O( x ) to predict the relative position of the connection point in the corresponding grid unit; S35: Use linear regression to optimize the relative positions of the connection points in the corresponding grid cells. 2. The method for automatically detecting the quantity of pulp goods according to claim 1, wherein step S2 comprises the following steps: S21: Input the extracted video image to a residual network to obtain a feature image; S22: Process the feature image to extract a rectangular candidate frame that distinguishes the foreground and the background; S23: Map the extracted rectangular candidate frame to the feature image, and unify the window size of the rectangular candidate frame by using regional feature aggregation technology; S24: Segment the feature image into a feature map of the pulp cargo portion according to the coordinate information of the rectangular candidate frame. 3. The method for automatic detection of pulp cargo quantity according to claim 2, wherein the S2 further comprises the following steps after S23 and before S24: S23': Perform boundary frame regression on the rectangular candidate frame after the unified window in S23, so as to correct the coordinate information of the rectangular candidate frame. 4. The method for automatically detecting the quantity of pulp goods according to claim 1, wherein step S3 further comprises the following steps: Accurate relative position information of connection points in grid cells is obtained by non-maximum suppression technique. 5. The method for automatically detecting the quantity of pulp goods according to claim 1, wherein step S4 comprises: S41: outputting the endpoint coordinate information of a series of line segment samples by means of mixed sampling of positive and negative samples; S42: According to the endpoint coordinate information of each line segment sample, perform a fixed-length vectorization process on each line segment sample, and obtain the feature vector of each line segment sample, so as to extract a series of line segment samples; Among them, the mixed sampling of positive and negative samples is the preparation work for model training. During the training process, the number of positive and negative samples of the K candidate connection points is greatly different, and the number of positive samples and negative samples needs to be balanced, and positive and negative samples are used for mixed training. method, where the positive samples are from the labeled real line segment samples, and the negative samples are non-real line segment samples randomly generated through heuristic learning. 6. The method for automatically detecting the quantity of pulp goods according to claim 1, wherein step S5 is specifically: Each line segment sample is filtered using the intersection ratio between the areas of each rectangular frame formed by each line segment sample as an intersecting line. 7. The method for automatically detecting the quantity of pulp goods according to claim 1, wherein step S5 is specifically: Filter each line segment sample using the Euclidean distance between each line segment sample.

Images