CN112465706A

CN112465706A - Automatic gate container residual inspection method

Info

Publication number: CN112465706A
Application number: CN202011513861.1A
Authority: CN
Inventors: 汤春明; 陈朋
Original assignee: Tianjin Polytechnic University
Current assignee: Tianjin Polytechnic University
Priority date: 2020-12-21
Filing date: 2020-12-21
Publication date: 2021-03-09

Abstract

The method relates to an automatic gate container residue checking method. Firstly, a container residue inspection frame taking multifunctional edge nodes as a core is provided based on an edge computing mode, and then the container residue inspection frame is decomposed into three modules which are mutually associated according to a gateway related workflow: the system comprises a container detection module, a container splicing module and a container residue checking module. And according to the tasks to be completed by the three modules, the research on the aspects of function analysis, algorithm realization, experimental test and the like is carried out one by one. In the aspect of algorithm, firstly, a container detection starting program based on a frame difference method is designed to judge whether the program starts to run or not. Secondly, a complete picture of the container is obtained by adopting an image splicing mode based on a phase correlation method. Finally, in the aspect of container residue inspection, researches are carried out on detection methods of deformation of an upper cross beam of the container, depression of the surface of the container and corrosion of the surface of the container, and on the basis of analyzing image characteristics of three defects, a machine vision related technology is used for respectively providing corresponding detection algorithms.

Description

Automatic gate container residual inspection method

Technical Field

The method belongs to the field of artificial intelligence and machine vision, relates to a method for automatically testing the disabled of a container passing through a gate, and can automatically test the disabled of the container passing through the gate.

Background

In recent years, the container throughput of each large port is on the rise, and the realization of the intelligence of port container business is an urgent task. The container damage inspection work is one of the indispensable research directions as an important component part of the container business, but is rarely researched by people. At present, the container detection mode of each large port gate mainly adopts manual detection; in few gaps, the staff detects through using auxiliary assembly, and its essence still is manual detection. Manual detection has many problems that are difficult to solve, such as high requirements for manual experience, danger of working environment, and the like.

In the aspect of the research of container gate inspection technology in China, two aspects of residue inspection and box number identification are mainly discussed. In the aspect of container residue inspection, a semi-automatic technology is mostly adopted, namely, auxiliary equipment such as a monitoring camera or a sensor is deployed on a gate field to assist workers in a monitoring room to judge, for example, Jeng and the like propose a method for detecting a container card based on a sensor, photographing each surface of the container by combining a CCD camera, and displaying the container card on a monitor for manual inspection; a container residue checking device with a similar principle is provided by a mansion door outer wheel tally limited company, and is mainly upgraded on the aspect of equipment, and comprises a PLC (programmable logic controller) main control module, a high-definition ball unit and the like, wherein the high-definition ball unit is arranged on each side and used for shooting. The exploration in the aspect of automatic residual inspection of the container is realized, the Wujun provides an optimized container residual inspection mode, and on the basis of PLC module control and monitoring snapshot, an OCR box number recognition technology is introduced for automatically recognizing each side of the container and then arranging each side image in one image for inspection by a detector, so that the workload is reduced. In terms of the special aspect of the detection method for various damage on the surface of the container, no relevant research method is searched at present. In the aspect of box number identification, a processing mode based on combination of machine learning and machine vision is researched more at present. Zhangieqi et al proposed a method based on PCA and Bayesian threshold estimation, which obtains the location of the container number and identifies the container number by analyzing the gray scale value of the image, but can only identify a single type of container number and require a long preprocessing time. The method adopts a morphological processing mode in a box number positioning module, ignores the diversity of containers, is easy to cause misjudgment, needs to manually establish a large number of character template libraries, consumes resources relatively and has relatively general recognition accuracy. Although the quality of box number identification in some aspects is improved to a certain extent by the research, the overall situation still has limitations, foreign port gates are rapidly developed in recent years, and the state of development and capital investment are in leading positions. smartPORT Logistics (SPL) App has been introduced in 2018 by hamburger port, germany, and has enabled the efficiency of truck transport at hamburger ports to be improved. SPL1.0APP is proposed at present, aiming at solving the technical development bottleneck faced by the truck transportation business. This APP can provide very effective information to the logistics company, and the truck drivers can download on the smart phone, through which they can see when they can enter the port, and also receive various instructions sent to them. The waiting time of the truck at the gate can be reduced to below 30 minutes. As the largest trade port in europe, port hartdan, which is the world information port for further creation, is in the international leading position in the direction of exploring and realizing port intellectualization. The Marsday project which is put into operation in 2015 4 months integrates advanced technologies such as intelligent control, Internet of things and big data calculation, and automatic integrated control is realized for working processes such as relevant transportation operation at the front edge of a wharf and relevant business or road access in a yard. And after the construction is completed in 2035 years, the operation rate of the wharf is improved by more than 50%. And advanced information technology means such as a local area network, cloud computing, a mobile terminal, an internet of things, a GIS (geographic information system) and a video monitoring system are utilized to carry out the informationized construction of the port. The American Virginia international container wharf adopts the operation process of 'single-trolley quay crane + straddle carrier + automatic rail crane', and is a semi-automatic wharf; the Global Container Terminal (Global Container Terminal bayone) is the closest Container Terminal to the entrance of new jersey, new york, the port adopts a streamline gate process, each gate is equipped with optical character recognition software (OCR), radio frequency identification technology (RFID) and remote communication technology, the passing efficiency of the truck is improved to the maximum extent, the truck is in a semi-automatic state, but all operation affairs are remotely guided and monitored by a background; the long beach container terminal adopts the operation flow of 'bridge crane + automatic guided vehicle + automatic track crane', and realizes horizontal transmission by lifting the automatic guided vehicle through a battery driving belt. And in the united states, due to the effects of the wharf's concerts, the exotic wharf has evolved toward a fully automated wharf, while the indigenous wharf has evolved toward a semi-automated wharf. The current development situation and trend in China: the Pacific international container terminal adopts a container number distributed OCR acquisition system to realize the identification of container numbers and ISO codes, and the container number distributed acquisition system shoots the positions of four container numbers of a container body in the running process of a container vehicle to finish the accurate identification of the container bodies and the container numbers of single containers and double containers; the container body residue inspection system completes image plane display of five side faces of the container body through a dynamic image splicing technology, and then completes container body residue inspection manually. The driver needs to swipe the card according to the prompt of the channel self-service processing terminal, the system is operated on site through key data stored in the RFID card, and the vehicle inspection system automatically lifts the rod after swiping the card. From 2018, the high-column national code intelligent electronic gate system under the Zhuhai harbor reunion flag has 8 lanes of 4-in and 4-out, has the functions of intelligent weighing, intelligent rod lifting and releasing and the like, and is provided with license plate snapshot and recognition equipment, container number automatic recognition equipment, IC automatic recognition equipment, electronic wagon balance, electronic license plate reader-writer equipment, electronic lock recognition equipment, channel control box and other supervision equipment. The Qingdao harbor was introduced into autonomous research and development by QQCTU in 2017, and became the first real intelligent gate in China. Compared with the traditional intelligent mode, the system has less camera requirements, and can reduce the construction and maintenance cost. The functions of automatically identifying the numbers of different types of containers, collecting card license plates, identifying the lead seal two-dimensional code, photographing the front box door and the rear box door, and checking the five sides of the box body for the residual photograph, and the like. The system is high in identification speed and accuracy, manpower input is greatly reduced, and high-efficiency operation is realized. At the key period of re-work and re-production of port and wharf, in order to realize the unmanned passing of documents and passing, a set of container paperless platform is developed and put into use in Chongqing orchard wharf. The i-Gate intelligent Gate system used in a matched mode is designed for intelligentization of container truck channel management, and mainly works on automatic weight collection, box number shooting and recognition, box body residue checking and shooting, license plate recognition, two-dimensional code recognition, IC card recognition, man-machine interaction and the like of a passing container truck, so that rapid Gate opening of an unattended and a current-collecting truck is achieved on a Gate. The I-Gate intelligent Gate system has leading identification rate of the industry, and the identification rate is more than 98%. Each party can also adopt WEB, WeChat and APP modes for self-service reservation, so that the working efficiency is improved. Although a certain amount of research has been carried out at home and abroad in terms of the overall architecture, no relevant method has been proposed so far about the specific implementation scheme and algorithm of the port gate container inspection system.

The method firstly provides a container residue checking frame taking multifunctional edge nodes as a core based on an edge computing mode. It is then broken down into three modules associated with each other: the container detection, container splicing and container residue checking module. And according to the tasks to be completed by the three modules, the research on the aspects of function analysis, algorithm realization, experimental test and the like is carried out one by one. In the aspect of algorithm, firstly, the method designs a container detection starting program based on a frame difference method to judge whether the program starts to run. Secondly, a complete picture of the container is obtained by adopting an image splicing mode based on a phase correlation method. Finally, researches are carried out on the detection methods of deformation of the upper cross beam of the container, depression of the surface of the container and corrosion of the surface of the container, and on the basis of analyzing the image characteristics of the three defects, a machine vision related technology is used for respectively providing corresponding detection algorithms.

Disclosure of Invention

The method firstly provides a container residue checking frame taking multifunctional edge nodes as a core based on an edge computing mode. It is then broken down into three modules associated with each other: the system comprises a container detection module, a container splicing module and a container residue checking module. And according to the tasks to be completed by the three modules, the research on the aspects of function analysis, algorithm realization, experimental test and the like is carried out one by one. In the aspect of algorithm, firstly, a container detection starting program based on a frame difference method is designed to judge whether the program starts to run or not. Secondly, a complete picture of the container is obtained by adopting an image splicing mode based on a phase correlation method. Finally, researches are carried out on the detection methods of deformation of the upper cross beam of the container, depression of the surface of the container and corrosion of the surface of the container, and on the basis of analyzing the image characteristics of the three defects, a machine vision related technology is used for respectively providing corresponding detection algorithms. The technical scheme for realizing the method comprises the following three steps:

step 1: monitoring the lanes through a start-up procedure based on container detection to decide whether to start up the entire system;

step 2: splicing the containers to obtain a container positive playing image;

and step 3: three kinds of damage to the surface of the container: the deformation of the upper beam of the container and the depression and corrosion on the surface of the container are respectively detected.

Compared with the prior art, the method has the outstanding characteristics that:

1. related work can be carried out through remote control, so that the workload of field workers is reduced, and the expenditure of the workers is reduced;

2. the severe working environment of a gate container carrying operation area is overcome, and the possibility of causing the life safety of field workers is reduced;

3. the speed of the container entering and exiting the gate can be increased, so that the efficiency of related work is improved;

4. can continuously run for a long time, and eliminates the high work experience requirement, the inconvenience brought by visual fatigue and the like.

Drawings

FIG. 1: flow chart of the method.

FIG. 2: the implementation process of the starting program of the container detection is shown. 2-1 is a monitoring visual field and an ROI lane area, 2-2 is a lane state diagram without vehicle, and 2-3 is a lane state diagram with vehicle.

FIG. 3: and (5) image splicing result graph. Fig. 3 shows the splicing result of the method.

FIG. 4: and 4-1 is a normal beam fitting graph, and 4-2 is a deformation beam fitting graph.

FIG. 5: the detection diagram of the surface depression of the vehicle body is shown, wherein 5-1 is an input image, and 5-2 is an output image.

FIG. 6: the vehicle body surface corrosion detection diagram is shown in the specification, wherein 6-1 is an input image, 6-2 is age map output, and 6-3 is a boundary segmentation output image.

Detailed Description

The lane monitoring video is exported, the video is processed into an image frame as input, and a corresponding processing result can be obtained after the processing of the method.

7 paths are called from a monitoring video of a certain gate of Tianjin harbor for verification, the frame rate is 25 frames/s, the average time of each path of video is 48 minutes, the detection success number Tp, the false detection number Fp, the missed detection number Fn, the detection accuracy Acc, the recall rate R and the recall rate f₁The performance of each algorithm in the index, disability test system is shown in table 1.

TABLE 1 results of the respective algorithms

In order to detect the real-time performance of the algorithm, the average time from entering the monitoring visual field to stopping of the container-carrying truck in each video is counted, and the average time required by the method for detecting the container is compared, as shown in table 2.

TABLE 2 statistical table for system detection

The data in the table show that the related algorithms of the method have good performance and can meet the requirement of real-time performance.

Claims

1. An automatic gate container residue checking method comprises three parts:

firstly, judging whether the program starts to run or not by detecting and starting the program through a container; secondly, acquiring a complete image of the container by an image splicing mode of a phase correlation method; finally, detecting three defects of deformation of the upper cross beam of the container, depression on the surface of the container and corrosion on the surface of the container respectively; on the basis of analyzing the image characteristics of the three defects, a machine vision related technology is used, and corresponding detection algorithms are respectively provided, and the method comprises the following steps:

A. container detection start-up procedure, the algorithm is based on the fact that: before detecting the container, judging whether a truck passes through a lane, and when a truck passes through a gate, the speed is relatively reduced so as to reserve time for a camera to shoot and a worker to check, and only needing to know whether the truck passes through without obtaining a full picture of the truck, based on the point, in order to save computing resources and accelerate the processing speed, firstly selecting a lane area as a processed ROI area, and selecting a frame-separated processing mode to perform frame difference processing in the ROI area: firstly, performing graying processing and low-pass filtering processing on read-in adjacent frame images respectively; secondly, applying a frame difference method to two adjacent frames of images to obtain a frame difference image; finally, performing binarization processing on the frame difference image, and calculating information contained in the frame difference image to judge whether the head of the truck enters the visual field of the camera; when the target (the container truck and the container carried by the container) is not in the monitoring visual field, the picture in the camera lens has no obvious change, the information in the frame difference image is 0, and when the vehicle head enters the monitoring visual field, the frame difference image contains the information of the phase difference of the two images; after the detection of the truck head, whether the truck carries the container is detected, and a frame difference method is also selected for judgment; firstly, the height h of each row of highest point white pixel points in the difference image_iThe total column number col of white pixel points 1, calculate the highest point mean of the image:

then, a first part of the current image is takenDividing the image into hx50 area, hx100 area of the second part and hx50 area of the third part, and calculating val2 of the three areas according to the steps of formula (1)₁、val2₂、val2₃Taking images of the front 1/2 area and the rear 1/2 area of the ROI area, and calculating val3 of the two areas₁、val3₂(ii) a Finally, if val2₁Is > val1 and val2₂Is > val1 and val2₃If the value is more than val1, the container is loaded in the collecting card, and the container can be tested for residue; if val3₁> 0 and val3₂When the value is 0, the truck is started;

B. image splicing: setting the container image at time t in the monitoring view as f₁(x, y), container image at time t + s is f₂(x, y), the displacement of the container in s time is Deltax, Deltay, then f₁(x, y) and f₂The change in position of the same part between (x, y) can be seen as a result of a displacement by Δ x, i.e., f₂(x, y) and f₁The relationship between (x, y) can be expressed as:

f₁(x，y)＝f₂(x-Δx，y-Δy) (2)

the Fourier transform can be used to convert the time domain variation into the frequency domain, and then the magnitude of the phase shift is used to obtain the expression displacement, and the Fourier expression of the formula (2) can be expressed in the form of the formula (3);

F₁(u，v)＝F₂(u，v)*e^{-j*2π*(u*Δx+v*Δy)} (3)

wherein F is₁(u, v) is f₁Representation in the frequency domain, F₂(u, v) is f₂In the representation of the frequency domain, the power density function is the signal power in the unit frequency band, and shows the change of the signal power along with the change of the frequency, and the F is obtained by₁And F₂Respectively obtain the conjugate F₁ ^*And F₂ ^*Then, f is obtained₂(x, y) and f₁The cross power spectrum of (x, y) is obtained by the following formula (4):

next, for cross power spectrum H_cps(u, v) taking an inverse fourier transform:

IFFT(H_cps(u，v))＝δ(u-Δx，v-Δy) (5)

obtaining a pulse function through a formula (5), finding the peak value coordinate of the pulse function to obtain an offset, splicing the images according to the offset, and obtaining a container panoramic image after splicing is finished after the tail of the vehicle enters the visual field;

C. and (3) residual inspection of the container: the method mainly aims at detecting damage on two side surfaces and the top surface of the container, and respectively detects three kinds of damage, namely deformation of a container beam, depression on the surface of the container and corrosion on the surface of the container; aiming at the detection of the deformation of the upper beam of the container, a detection algorithm based on the combination of edge information and a nonlinear least square method is provided; in order to save computing resources, an upper cross beam of a container is automatically acquired, then the extraction effect of an upper edge required by an experiment is superior to that of other two edge extraction operators through a Sobel operator, in order to more conveniently position the upper edge of the container, after non-maximum suppression is carried out on an edge image to eliminate a stray effect, a proper double-threshold algorithm is selected for detection and connection to acquire the upper edge of the upper cross beam, and therefore edge point information can be conveniently counted; performing linear fitting by adopting a least square method, and fitting a given scatter point set by searching an optimal linear line in a linear class y-kx + b, keeping the true value in the scatter point set as y, wherein m points are shared, the fitting value is y', and the error calculation formula is shown as a formula (6); the optimal straight line requires that k and b be found to minimize the error calculation of equation (7) by calculating the error E between the fitted value and the true value_rDetermining whether the upper cross beam deforms or not;

E_r＝|y-y′| (6)

for depression of the surface of a containerCombining the characteristics of the concave image information, a concave detection method based on transverse gradient information is provided, namely, a concave area is divided by making the transverse gradient information of the image more obvious and eliminating the surrounding background; selecting Sobel operator with strong extraction direction edge to extract transverse edge of the image with the dent defect, namely performing gradient detection on the image in the vertical direction, firstly graying the image, and then performing vertical gradient detection on the grayed image through the Sobel operator, and assuming F_img(x, y) denotes an actual image, M_img(x, y) represents a desired target image, N_img(x, y) represents an image including noise and other extraneous parts such as a background, and the relationship therebetween can be represented by equation (8):

F_img(x，y)＝M_img(x，y)+N_img(x，y) (8)

to segment defect aspects, we define G_mThe recessed area is distinguished from the other areas by the formula (9) for maximum gray value:

we filter the template K by the improved 5x5 mean_mTo the target image I_imgCarrying out filtering processing, wherein the processing formula is as follows:

wherein the improved template K_mThe expression is as follows:

introducing a parameter P on an original mean value filtering template, smoothing the image by changing the value of P, binarizing the obtained result, finally determining that the processing effect is better when the value of P is 10 through comparison, and then segmenting through a formula (9) to obtain a depressed area;

for the corrosion of the surface of the container, a container surface corrosion detection algorithm based on an age map and a gray gradient co-occurrence matrix is provided: assuming that the length and the width of the image are L, W respectively, dividing the image into hxh (20 x20 in the method) slices as source slices, wherein the size of each source slice is (L/h) x (W/h), and then determining the age map of each source slice by calculating the similarity between adjacent slices with the same size and the source slices, wherein the distances between the adjacent slices are W/2h and L/2h from the source slices; the luminance and gradient characteristics may represent an image, and thus the age of a source slice may be represented by simultaneously calculating the luminance and gradient similarities between the source slice and neighboring slices, as in equation (12):

wherein, I_sliceAs intensity map of slice image, G_sliceCalculating cosine distance between pixels to express similarity between the pixels for the gradient image; setting a certain source slice O_TThe gradient image of (1) is G_TLuminance image is L_TAdjacent to the slice O_NThe gradient image of (1) is G_NLuminance image is L_NAccording to the cosine similarity theorem, G_TAnd G_NThe similarity between them is calculated by equation (13):

in the same way, L can be obtained_TAnd L_NSimilarity between S_LWhile considering both the gradient feature and the brightness feature of the image, then O_TThe calculation mode of the agemap is shown as the formula (14):

wherein, O_TGIs O_TThe gray level images of (1) sequentially obtain the age maps of all source slices, finally obtain the age map of the whole image, and then graying the image to obtain G₁(ii) a By the pair G₁Observing, finding that the internal brightness of the rusty area is high and is uniformly distributed, and the gradient value of the edge of the rusty part and the normal part is high; according to the characteristics, the gray-gradient co-occurrence matrix is used for obtaining the edge of the rust and the background; setting the maximum gray value of the normalized image as P_fMaximum gradient value of P_gThen the expression form of the gray-gradient co-occurrence matrix is P_f x P_gA matrix of size, the matrix being divided into T₁～T₄Four regions, wherein: t is₁And T₃The region belongs to a rusty region or a normal region, T₂And T₄The area belongs to the edge of the rusted part and the normal part; four regions can be divided by only one dividing point (m, n), namely, a defect region can be found by finding (m, n); and if the probability that the gray value in the gray gradient co-occurrence matrix is x and the gradient value is y pixels is p (x, y), calculating T₁The pixel probability sum of the region definitions can be calculated by equation (15):

by the same token we can obtain T₂、T₃And T₄A sum of pixel probabilities for the region definition; only a normal box part, a rusted part, a background edge and a rusted edge exist in the image, so that the quadrant division points (m, n) are unique; firstly to T₂Normalizing the pixel probability of the region to obtain a normalized pixel probability P_T2(x, y); in the same way, T can be obtained₄Pixel probability after area normalization; t is₂The gradient value of the region is large, the gray value is small, the high probability is the background edge, and from the entropy point of view, P is_T2(x, y) is the calculation formula of the edge entropy of the background region:

T₄the region with large gradient value and large gray value is most likely to be rusty edge, P_T4(x, y) is the entropy H of the tarnish edge_entropy(edge | rust), can be calculated by equation (17):

according to the formula (16) and the formula (17), the entropy calculation formula of the image segmentation corrosion is the formula (18):

according to the maximum entropy method, let H_entropy(m, n) is the maximum point (m, n) is the result; and when the pixel meets the condition that the gray value is greater than m and the gradient value is greater than n, the pixel is an edge pixel, and finally the rusty segmentation image can be obtained.