CN112661013B - Automatic wharf bridge crane legacy lock pad detection method and system - Google Patents
Automatic wharf bridge crane legacy lock pad detection method and system Download PDFInfo
- Publication number
- CN112661013B CN112661013B CN202011502610.3A CN202011502610A CN112661013B CN 112661013 B CN112661013 B CN 112661013B CN 202011502610 A CN202011502610 A CN 202011502610A CN 112661013 B CN112661013 B CN 112661013B
- Authority
- CN
- China
- Prior art keywords
- lock pad
- lock
- image
- sample data
- bridge crane
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
- B65G67/00—Loading or unloading vehicles
- B65G67/60—Loading or unloading ships
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/16—Applications of indicating, registering, or weighing devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/18—Control systems or devices
- B66C13/48—Automatic control of crane drives for producing a single or repeated working cycle; Programme control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C15/00—Safety gear
- B66C15/06—Arrangements or use of warning devices
Landscapes
- Engineering & Computer Science (AREA)
- Mechanical Engineering (AREA)
- Ocean & Marine Engineering (AREA)
- Aviation & Aerospace Engineering (AREA)
- Automation & Control Theory (AREA)
- Control And Safety Of Cranes (AREA)
Abstract
The invention provides a method and a system for detecting a lock pad left behind an automatic wharf bridge crane, which belong to the technical field of target detection. In the process of automatic dock ship unloading operation, acquiring a box bottom angle real-time video image of a container on a bridge crane transfer platform pedestal through a box bottom angle video image real-time acquisition device, generating a lock pad detection result signal according to the box bottom angle real-time video image by adopting a lock pad legacy information real-time processing system, and transmitting the lock pad detection result signal to a bridge crane land side trolley electric control system; when the lock pad is left, the land side of the bridge crane is controlled to stop lifting from the trolley, and the lock pad is warned to be removed timely by field operators.
Description
Technical Field
The invention belongs to the technical field of target detection, and particularly relates to a method and a system for detecting a lock pad left by an automatic wharf bridge crane.
Background
At present, the ship unloading operation flow of the automatic container terminal is as follows: the main trolley at the sea side of the bridge crane picks up the container from the container ship at the shore, and the main trolley at the sea side of the bridge crane moves and places the container on a pedestal of a bridge crane transfer platform; manually removing lock pads at bottom corners of the container by field operators; then, the bridge crane land side trolley picks up and moves the container from the pedestal of the transferring platform, the container is placed on the ground AGV automatic navigation truck, the container is transported to a designated yard by the AGV automatic navigation truck, and then the container is picked up by the yard sea side rail crane, and the container is placed at the designated yard position.
The field operators can sometimes miss the lock pad of the bottom corner of the container on the bridge crane transferring platform, especially the lock pad with self-locking. If the unopened lock pad is not found, the lower container can be damaged when the container is placed in a storage yard, and more serious, the container with the self-locking left-behind lock pad can be fixedly connected with the lower container, and then when the container is lifted by the sea side track crane, the lower container can be lifted together, so that serious case breaking accidents are caused.
The control of the lock pad of the container at the sea side in the yard is an important concern in the port industry, the lock pad is dismantled and confirmed at present by field operators, and if the field operators neglect to remove the lock pad at the bottom corner of the container on the bridge crane transferring platform, the potential safety hazard is brought to the operation in the yard. Therefore, an automatic dock bridge crane legacy lock pad detection system is needed to detect and pre-warn the missed picking lock pad, realize the zero approach goal of the lock pad, and avoid accidents in the yard.
Disclosure of Invention
The invention aims to provide an automatic detection method and system for a left lock pad of a bridge crane of a wharf, and aims to solve the safety problem of missing lock pads entering a storage yard.
In order to achieve the above purpose, the invention adopts the following technical scheme:
an automated dock bridge crane legacy lock pad detection method, comprising:
acquiring a box bottom angle image in advance, and establishing a box bottom angle image library;
generating lock pad target training sample data and background training sample data according to the box bottom corner image library;
generating a legacy lock pad detection network model according to the lock pad target training sample data and the background training sample data;
acquiring a real-time video image of a box bottom angle in operation;
inputting the real-time video image of the box bottom angle in the operation to the legacy lock pad detection network model for detection, and generating a lock pad detection result signal; the lock pad detection result signal includes: with and without legacy lock pad signals;
when the lock pad detection result signal is a signal with a left lock pad, controlling the land side of the bridge crane to stop lifting from the trolley;
and when the lock pad detection result signal is a signal without a left lock pad, controlling the land side of the bridge crane to continuously lift from the trolley.
Preferably, the creating a box bottom angle image library includes:
placing a container on a bridge crane transfer platform pedestal; the containers comprise 45-ruler containers, 40-ruler containers, double 20-ruler containers and single 20-ruler containers;
hanging lock pads on the 4 bottom corners of the container respectively; the lock pad comprises a large lock pad, a middle lock pad and a small lock pad;
and collecting a plurality of container bottom images to establish the bottom corner image library.
Preferably, the generating the lock pad target training sample data and the background training sample data according to the box bottom corner image library includes:
intercepting and marking a lock pad area in any lock pad-carrying box bottom corner image in the box bottom corner image library to generate lock pad target training sample data; the lock pad target training sample data comprises a large lock pad sample, a middle lock pad sample and a small lock pad sample;
and randomly intercepting a non-locking pad area in any one image in the box bottom corner image library to generate the background training sample data.
Preferably, the generating a legacy lock pad detection network model according to the lock pad target training sample data and the background training sample data includes:
training the SSD target detection classification network by taking the lock pad target training sample data as a positive sample and the background training sample data as a negative sample;
defining training loss functions
Wherein L is loc (x, L, g) is a position error, L conf (x, c) is confidence error, N is positive sample number of a priori frame, c is category confidence predictive value, g is position parameter of positive sample, l is boundary frame predictive value, ++>
Is an indication parameter, which indicates that the ith prior frame is matched with the jth positive sample, the class of the positive sample is p, and alpha is a weight coefficient;
and optimizing the training loss function through the lock pad target training sample data and the background training sample data to obtain the legacy lock pad detection network model.
Preferably, inputting the real-time video image of the bottom corner in the operation to the legacy lock pad detection network model for detection, and generating a lock pad detection result signal includes:
inputting the real-time video image of the box bottom angle in the operation into a legacy lock pad detection network model to obtain feature maps with different sizes;
extracting feature graphs of Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2 layers, respectively constructing 6 bounding boxes with different scales at each point on the feature graphs, respectively classifying the bounding boxes, and generating a plurality of prediction boxes with category confidence;
and generating a lock pad detection result signal according to the prediction frame and the confidence coefficient.
Preferably, the generating a lock pad detection result signal according to the prediction frame and the confidence includes:
setting a confidence threshold;
filtering out a prediction frame with the category confidence coefficient lower than the confidence coefficient threshold according to the category confidence coefficient and the confidence coefficient threshold, and generating a prediction frame to be processed after background information is filtered;
decoding the predicted frames to be processed, removing overlapped or incorrect predicted frames by adopting a non-maximum suppression method, and determining the types of lock pads; the types of the lock pads comprise a large lock pad, a middle lock pad and a small lock pad;
and generating a lock pad detection result signal.
An automated dock bridge crane legacy lock pad detection system, comprising:
the real-time acquisition device of the box bottom angle video image is used for acquiring the real-time video image of the box bottom angle of the container on the bridge crane transfer platform pedestal;
the lock pad legacy information real-time processing system is connected with the box bottom angle video image real-time acquisition device and is used for generating a lock pad detection result signal according to the box bottom angle real-time video image and sending the lock pad detection result signal to a bridge crane land side trolley electric control system; the lock pad detection result signal includes: with and without legacy lock pad signals;
when the detection result signal of the detection lock pad is a signal with a left lock pad, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to stop lifting;
when the detection result signal of the detection lock pad is that no signal of the left lock pad exists, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to continuously lift.
Preferably, the lock pad legacy information real-time processing system includes:
the communication interface module is connected with the electric control system of the bridge crane land side slave trolley and is used for transmitting the operation information of the bridge crane land side slave trolley; the job information includes: a landing signal and an operation box;
the information processing module is connected with the communication interface module and is used for receiving the operation information, detecting the real-time video image of the box bottom angle according to the operation information, generating a lock pad detection result signal, and sending the lock pad detection result signal to a bridge crane land side trolley electric control system.
The method and the system for detecting the left lock pad of the automatic wharf bridge crane have the beneficial effects that: compared with the prior art, in the process of automatic dock ship unloading operation, the real-time box bottom angle video image of the container on the bridge crane transfer platform pedestal is acquired through the real-time box bottom angle video image acquisition device, the lock pad legacy information real-time processing system is adopted to generate a lock pad detection result signal according to the box bottom angle video image, and the lock pad detection result signal is sent to the bridge crane land side trolley electric control system; when the lock pad is left, the bridge crane land side slave trolley electric control system can control the bridge crane land side slave trolley to stop lifting, so as to warn site operators to remove the lock pad in time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for detecting a legacy lock pad of an automated dock bridge crane according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a classification network for providing SSD destination detection according to an embodiment of the present invention;
FIG. 3 is a schematic view of a camera mounting position according to an embodiment of the present invention;
fig. 4 is a flowchart of an automated dock bridge legacy lock pad detection system according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The method for detecting the left lock pad of the bridge crane of the automatic wharf provided by the invention is described, and referring to fig. 1, the method comprises the following steps:
step 1: acquiring a box bottom angle image in advance, and establishing a box bottom angle image library;
the step 1 specifically comprises the following steps: placing a container on a bridge crane transfer platform pedestal; the containers comprise 45-ruler containers, 40-ruler containers, double 20-ruler containers and single 20-ruler containers;
hanging lock pads on the 4 bottom corners of the container respectively; the lock pad comprises a large lock pad, a middle lock pad and a small lock pad;
and collecting a plurality of container bottom images to establish the bottom corner image library.
In practical application, 45-ruler containers, 40-ruler containers, double 20-ruler containers and single 20-ruler containers are respectively placed on a bridge crane transferring platform pedestal, large lock pads, middle lock pads and small lock pads are respectively installed on bottom corners of containers with different sizes, bottom corner images with the lock pads are acquired in daytime and at night, and a bottom corner image library with the number of not less than 1 ten thousand is constructed.
Step 2: generating lock pad target training sample data and background training sample data according to the box bottom corner image library;
the step 2 specifically comprises the following steps:
intercepting and marking a lock pad area in any image in the box bottom corner image library to generate lock pad target training sample data; the lock pad target training sample data comprises a large lock pad sample, a middle lock pad sample and a small lock pad sample;
randomly intercepting a non-lock pad area in any image in a box bottom corner image library to generate background training sample data;
in practical application, any box bottom corner image with a lock pad in a box bottom corner image library is intercepted and marked in an image, a lock pad area is marked as a large lock pad, a middle lock pad and a small lock pad, a background negative sample is intercepted randomly from a non-lock pad target area in the image, lock pad target and background training sample data are obtained, 8000 large lock pad target positive samples, 7000 middle lock pad target positive samples, 4000 small lock pad target positive samples and 5 ten thousand background negative samples can be obtained.
Step 3: generating a legacy lock pad detection network model according to the lock pad target training sample data and the background training sample data;
the step 3 specifically comprises the following steps:
training the SSD target detection classification network by taking the lock pad target training sample data as a positive sample and the background training sample data as a negative sample;
defining training loss functions
Wherein L is loc (x, L, g) is a position error, L conf (x, c) is confidence error, N is positive sample number of a priori frame, c is category confidence predictive value, g is position parameter of positive sample, l is boundary frame predictive value, ++>
The index parameter is used for indicating that the ith priori frame is matched with the jth positive sample, the class of the positive sample is p, and alpha is a weight coefficient;
and optimizing a training loss function through the lock pad target training sample data and the background training sample data to obtain a legacy lock pad detection network model.
In practical applications, please refer to fig. 2, wherein Image represents an Image, conv represents a convolution layer, FC represents a full connection layer, classifer represents a classifier, classes is a category, extra Feature Layers represents an additional functional layer, and detections represent classification results. Training an SSD (solid State disk) target detection classification network according to the lock pad target training sample data and the background training sample data; the base network portion of the SSD employs VGG16 to, during training,firstly, matching the training data of the positive sample with a priori frame, wherein a boundary frame corresponding to the priori frame matched with the positive sample is used for predicting the positive sample; after the bounding box corresponding to the training sample is determined, a training loss function L (x, c, L, g) is defined as a position error L loc (x, L, g) and confidence error L conf Weighted sum of (x, c)
Where N is the number of positive samples of the prior frame, c is the class confidence predictor, g is the position parameter of the positive sample, l is the bounding box predictor, x represents +.>
Is an indication parameter, which indicates that the ith priori frame is matched with the jth lock pad target positive sample, and the type of the lock pad target positive sample is p; the legacy lock pad detection network model can be pre-trained by training samples to optimize the training loss function L (x, c, L, g).
Step 4: acquiring a real-time video image of a box bottom angle in operation; according to the invention, the real-time video image of the bottom corner of the container on the pedestal of the bridge crane transfer platform is acquired by adopting the real-time video image acquisition device of the bottom corner.
Step 5: inputting the real-time video image of the bottom corner of the box into a legacy lock pad detection network model for detection, and generating a lock pad detection result signal; the lock pad detection result signal includes: with and without legacy lock pad signals;
the step 5 specifically comprises the following steps:
step 501: inputting real-time video images of the bottom corners of the box into a legacy lock pad detection network model to obtain feature maps with different sizes;
in practical application, for a real-time video image of a bottom corner acquired in real time, taking a marked container bottom corner at a position of a rough pixel in the video image of the bottom corner as a central part, intercepting a 300×300 image in the video image of the bottom corner, and inputting the 300×300 image into a pre-trained target detection classification network (a legacy lock pad detection network model) to obtain feature maps with different sizes.
Step 502: extracting the feature graphs of Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2 layers, constructing 6 bounding boxes with different scales at each point on the feature graphs, classifying the bounding boxes, and generating a plurality of prediction boxes with category confidence.
Step 503: and generating a lock pad detection result signal according to the prediction frame and the confidence coefficient.
Step 503 specifically includes:
setting a confidence threshold;
filtering out a prediction frame with the category confidence coefficient lower than the confidence coefficient threshold according to the category confidence coefficient and the confidence coefficient threshold, and generating a prediction frame to be processed after background information is filtered;
decoding the predicted frames to be processed, removing overlapped or incorrect predicted frames by adopting a non-maximum suppression method, and determining the types of lock pads; the types of the lock pads comprise a large lock pad, a middle lock pad and a small lock pad;
and generating a lock pad detection result signal.
In practical application, a confidence coefficient threshold value of 0.7 is set, and for each prediction frame, the prediction frame with lower confidence coefficient is filtered according to the confidence coefficient threshold value of 0.7; decoding the rest predicted frames, arranging the rest predicted frames in a descending order according to the confidence level, and reserving 200 predicted frames; and removing overlapped or incorrect prediction frames by a non-maximum suppression method (NMS), wherein the rest prediction frames are final lock pad detection results, and further judging whether a lock pad target exists in the real-time video image of the bottom corner of the box.
Step 6: when the lock pad detection result signal is a signal with a left lock pad, controlling the land side of the bridge crane to stop lifting from the trolley;
step 7: and when the lock pad detection result signal is a signal without a left lock pad, controlling the land side of the bridge crane to continuously lift from the trolley.
The automatic wharf bridge crane legacy lock pad detection method disclosed by the invention is constructed based on an SSD target detection algorithm, has the advantages of high recognition speed and high detection precision, adopts an END-TO-END training mode, and has accurate classification results even when pictures with smaller resolution are processed. According to test data statistics, the embodiment of the system and the method for detecting the legacy locking pads of the automatic wharf bridge crane can realize real-time processing of video images with 1280 multiplied by 1080 resolution and 25 frames/second, and the detection and identification accuracy of the legacy locking pads is more than 99%.
The invention provides an automatic wharf bridge crane legacy lock pad detection system. An automated dock bridge crane legacy lock pad detection system, comprising:
the box bottom angle video image real-time acquisition device and the lock pad legacy information real-time processing system; the real-time acquisition device of the box bottom angle video image is used for acquiring the box bottom angle video image of the container on the bridge crane transfer platform pedestal in real time; the real-time video image of the box bottom angle can observe and image all the box bottom angles of containers such as 45 ruler, 40 ruler, double 20 ruler, single 20 ruler and the like placed on a bridge crane transit platform base; the lock pad legacy information real-time processing system is connected with the box bottom angle video image real-time acquisition device and is used for receiving the box bottom angle real-time video image, generating a lock pad detection result signal according to the box bottom angle video image and sending the lock pad detection result signal to the bridge land side trolley electric control system; the lock pad detection result signal includes: with and without legacy lock pad signals;
when the detection result signal of the detection lock pad is a signal with a left lock pad, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to stop lifting;
when the detection result signal of the detection lock pad is that no signal of the left lock pad exists, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to continuously lift.
In practical application, the device for acquiring the video image of the box bottom corner in real time comprises a plurality of groups of image acquisition sub-devices, and the image acquisition sub-devices are preferably cameras. The camera of the invention is preferably a Kawav visual small hemispherical network camera. On a bridge crane transfer platform pedestal, cameras are arranged on a transfer platform vertical support which is closest to the bottom corners of each box type box, the installation height of each camera is approximately equal to the bottom of a container placed on the transfer platform pedestal, and each camera is used for observing the bottom corners of the box type boxes such as a 45-scale container, a 40-scale container, a double 20-scale container, a single 20-scale container and the like placed on the bridge crane transfer platform pedestal in the operation process; the camera on the vertical support of the transfer platform can observe the container bottom angle nearest to the camera by adjusting the visual angle of the camera, and mark the approximate pixel position of the container bottom angle in the observed image; the size of the camera does not exceed the width of the vertical support and does not exceed the outer contour of the vertical support, and the camera has a light supplementing lamp function; the fixed support of the camera on the vertical support of the transfer platform is connected with the vertical support of the transfer platform in a welding mode; the rain cover is arranged above the camera on the vertical support of the transfer platform and does not cross the outer contour of the vertical support.
As another embodiment of the present application, a lock pad legacy information real-time processing system includes: a communication interface module and an information processing module; the communication interface module is connected with the electric control system of the bridge crane land side slave trolley and is used for transmitting the operation information of the bridge crane land side slave trolley; the job information includes: a landing signal and an operation box; the information processing module is connected with the communication interface module and used for receiving the operation information, generating a lock pad detection result signal according to the operation information detection box bottom angle video image, and sending the lock pad detection result signal to the bridge crane land side slave trolley electric control system through the communication interface module to control the stop and lifting of the bridge crane land side slave trolley.
As another embodiment of the application, the information processing module reads the landing signal of the bridge land side slave trolley PLC and the operation information of the operation box type and the like through the communication interface module, after receiving the landing signal of the bridge land side slave trolley PLC, the information processing module starts to process the 4 box bottom angle real-time video images corresponding to the operation box bottom angle, detects whether the lock pad exists in the 4 box bottom angle real-time video images through the automatic dock bridge legacy lock pad detection method, and sends the lock pad detection result signal to the bridge land side slave trolley electronic control system through the communication interface module, so as to control the stop or lifting of the bridge land side slave trolley.
When the information processing module detects a lock pad target in any one of the 4 video images, a lock pad left-behind alarm signal is output to the bridge crane land side slave trolley electric control system through the communication interface module, the bridge crane land side slave trolley electric control system stops lifting of the bridge crane land side slave trolley, and the on-site operation personnel are warned to manually detach the left-behind lock pad; when the information processing module does not detect the lock pad target in all 4 video images, the land side of the bridge crane is lifted from the trolley all the time. And a bypass switch is further arranged at the transfer platform, and after the left lock pad is detached, the on-site operator resumes lifting operation of the bridge crane from the trolley on the land side by operating the bypass switch.
The communication interface module is also used for transmitting the timing heartbeat signal of the normal working state of the information processing module to the electric control system of the slave trolley at the land side of the bridge crane. If the bridge land side trolley electric control system does not receive the normal working state timing heartbeat signal of the information processing module within 2 continuous heartbeat signal periods, the bridge land side trolley electric control system locks the lifting of the bridge land side slave trolley until the bridge land side trolley electric control system can receive the normal working state timing heartbeat signal of the information processing module.
In practical application, the mounting positions of the real-time box bottom angle video image acquisition device provided by the invention are respectively provided with 1 camera on 10 vertical supports at two sides of each pedestal on the bridge crane transfer platform, and the total of 10 cameras (5 cameras are arranged at each side), referring to fig. 3, the mounting height of the cameras is approximately equal to the bottom of a container placed on the pedestal of the transfer platform, the cameras adopt small hemispherical cameras with the function of light supplementing lamps, the size of the cameras does not exceed the width of the vertical supports and does not exceed the outer contour of the vertical supports; the fixed support of the camera on the vertical support of the transfer platform is connected with the vertical support of the transfer platform in a welding mode; the viewing angle of the camera is adjusted to observe the container bottom angle nearest to the camera, and the approximate pixel position of the container bottom angle in the observed image is marked. The mounting mode enables the real-time acquisition device of the video image of the bottom corner to observe the bottom corners of the bridge crane transfer platform, which are placed on the pedestal, of all the cases, such as a 45-ruler container, a 40-ruler container, a double 20-ruler container, a single 20-ruler container and the like.
Referring to fig. 4, the workflow of the present invention is: the information processing module of the lock pad legacy information real-time processing system reads operation information such as landing signals of a bridge land side slave trolley PLC and operation box types of containers through the communication interface module, after receiving the landing signals of the bridge land side slave trolley PLC, the information processing system starts to process 4 box bottom angle video images corresponding to bottom angles of the containers, the 4 box bottom angle real-time video images are detected through an automatic dock bridge legacy lock pad detection method, a large lock pad target or a middle lock pad target or a small lock pad target is detected in any one of the 4 video images, then a lock pad legacy alarm signal is output to a bridge land side slave trolley electric control system through the communication interface module, the bridge land side slave trolley electric control system stops lifting of the bridge land side slave trolley, and site operators are warned of manually detached lock pads; after the left-over large lock pad is detached, the on-site operator resumes the lifting operation of the bridge crane land side from the trolley by operating the bypass switch.
The method and the system for detecting the left lock pad of the automatic wharf bridge crane have the beneficial effects that: compared with the prior art, in the process of automatic dock ship unloading operation, the real-time box bottom angle video image of the container on the bridge crane transfer platform pedestal is acquired through the real-time box bottom angle video image acquisition device, the lock pad legacy information real-time processing system is adopted to generate a lock pad detection result signal according to the box bottom angle video image, and the lock pad detection result signal is sent to the bridge crane land side trolley electric control system; when the lock pad is left, the bridge crane land side slave trolley electric control system can control the bridge crane land side slave trolley to stop lifting so as to remind on-site operators to remove the lock pad in time.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (3)
1. An automated dock bridge crane legacy lock pad detection method is characterized by comprising the following steps:
acquiring a box bottom angle sample image;
wherein, before the acquiring the bottom corner sample image, further comprises:
placing a container on a bridge crane transfer platform pedestal; the containers comprise 45-ruler containers, 40-ruler containers, double 20-ruler containers and single 20-ruler containers;
collecting a plurality of container bottom images to generate a bottom corner sample image;
establishing a lock pad image library according to the box bottom corner sample image;
generating lock pad target sample data and background training sample data according to the lock pad image library;
the generating the lock pad target sample data and the background training sample data according to the lock pad image library comprises:
intercepting and marking a lock pad area in any one image in the lock pad image library to generate lock pad target sample data; the lock pad target sample data comprises a large lock pad sample, a middle lock pad sample and a small lock pad sample;
the large lock pad sample is generated by marking lock pad images of the 45-ruler container; the middle lock pad sample is generated by marking lock pad images of the 40-ruler container; the small lock pad sample is generated by marking lock pad images of the double 20-ruler container and the single 20-ruler container;
randomly intercepting a non-lock pad area in any one image in the lock pad image library to generate the background training sample data;
generating a legacy lock pad detection network model according to the lock pad target sample data and the background training sample data;
the generating a legacy lockpad detection network model from the lockpad target sample data and the background training sample data includes:
training the SSD target detection classification network by taking the lock pad target sample data as a positive sample and the background training sample data as a negative sample;
defining training loss functions
Wherein L is loc (x, L, g) is a position error, L conf (x, c) is confidence error, N is positive sample number of a priori frame, c is category confidence predictive value, g is position parameter of positive sample, l is boundary frame predictive value, ++>
Is an indication parameter, which indicates that the ith prior frame is matched with the jth positive sample, the class of the positive sample is p, and alpha is a weight coefficient;
optimizing the training loss function through the lock pad target sample data and the background training sample data to obtain the legacy lock pad detection network model;
acquiring a box bottom corner video image;
inputting the box bottom corner video image into the legacy lock pad detection network model for detection, and generating a lock pad detection result signal; the lock pad detection result signal includes: with and without legacy lock pad signals;
when the lock pad detection result signal is a signal with a left lock pad, controlling the land side of the bridge crane to stop from the trolley;
when the lock pad detection result signal is a signal without a left lock pad, controlling the land side of the bridge crane to continuously lift from the trolley;
inputting the box bottom corner video image to the legacy lock pad detection network model for detection, and generating a lock pad detection result signal comprises the following steps:
intercepting an image from the box bottom corner video image to generate a box bottom image;
inputting the box bottom image into a legacy lock pad detection network model to obtain feature maps with different sizes;
extracting feature graphs of Conv4_3, conv7, conv8_2, conv9_2, conv10_2 and Conv11_2 layers, respectively constructing 6 bounding boxes with different scales at each point on the feature graphs, respectively classifying the bounding boxes, and generating a plurality of prediction boxes with category confidence;
generating a lock pad detection result signal according to the prediction frame;
the generating a lock pad detection result signal according to the prediction frame includes:
determining the category of the lock pad according to the category confidence, and filtering out a prediction frame belonging to the background at the same time to generate a prediction frame after background information is filtered; the types of the lock pads comprise a large lock pad, a middle lock pad and a small lock pad;
acquiring a confidence threshold;
filtering the prediction frames with the category confidence coefficient lower than the confidence coefficient threshold according to the confidence coefficient threshold, and generating the prediction frames to be processed after filtering background information and lower than the confidence coefficient threshold;
and decoding the predicted frame to be processed, removing overlapped or incorrect predicted frames by adopting a non-maximum value inhibition method, and generating a lock pad detection result signal.
2. An automated dock bridge crane legacy lock pad detection system, comprising:
the real-time box bottom angle image acquisition device is used for acquiring a box bottom angle video image of a container on a bridge crane transfer platform pedestal;
the lock pad legacy information real-time processing system is connected with the box bottom angle image real-time acquisition device and is used for generating a lock pad detection result signal according to the box bottom angle video image and sending the lock pad detection result signal to a bridge crane land side trolley electric control system; the lock pad detection result signal includes: with and without legacy lock pad signals;
the step of generating the lock pad detection result signal comprises the following steps:
acquiring a box bottom angle sample image;
wherein, before the acquiring the bottom corner sample image, further comprises:
placing a container on a bridge crane transfer platform pedestal; the containers comprise 45-ruler containers, 40-ruler containers, double 20-ruler containers and single 20-ruler containers;
collecting a plurality of container bottom images to generate a bottom corner sample image;
establishing a lock pad image library according to the box bottom corner sample image;
generating lock pad target sample data and background training sample data according to the lock pad image library;
the generating the lock pad target sample data and the background training sample data according to the lock pad image library comprises:
intercepting and marking a lock pad area in any one image in the lock pad image library to generate lock pad target sample data; the lock pad target sample data comprises a large lock pad sample, a middle lock pad sample and a small lock pad sample;
the large lock pad sample is generated by marking lock pad images of the 45-ruler container; the middle lock pad sample is generated by marking lock pad images of the 40-ruler container; the small lock pad sample is generated by marking lock pad images of the double 20-ruler container and the single 20-ruler container;
randomly intercepting a non-lock pad area in any one image in the lock pad image library to generate the background training sample data;
generating a legacy lock pad detection network model according to the lock pad target sample data and the background training sample data;
the generating a legacy lockpad detection network model from the lockpad target sample data and the background training sample data includes:
training the SSD target detection classification network by taking the lock pad target sample data as a positive sample and the background training sample data as a negative sample;
defining training loss functions
Wherein L is loc (x, L, g) is a position error, L conf (x, c) is confidence error, N is the number of positive samples of the prior box, cIs a class confidence predictor, g is the position parameter of the positive sample, l is a bounding box predictor, +.>
Is an indication parameter, which indicates that the ith prior frame is matched with the jth positive sample, the class of the positive sample is p, and alpha is a weight coefficient;
optimizing the training loss function through the lock pad target sample data and the background training sample data to obtain the legacy lock pad detection network model;
acquiring a box bottom corner video image;
inputting the box bottom corner video image into the legacy lock pad detection network model for detection, and generating a lock pad detection result signal;
when the detection result signal of the detection lock pad is a signal with a left lock pad, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to stop by the electric control system of the bridge crane land side slave trolley;
when the detection result signal of the detection lock pad is that no signal of the left lock pad exists, the bridge crane land side slave trolley is controlled by the bridge crane land side slave trolley to continuously lift.
3. The automated dock bridge-hung legacy lock pad detection system of claim 2, wherein the lock pad legacy information real-time processing system comprises:
the communication interface module is connected with the bridge crane land side slave trolley and is used for transmitting operation information of the bridge crane land side slave trolley; the job information includes: a landing signal and an operation box;
the information processing module is connected with the communication interface module and is used for receiving the operation information, detecting the box bottom angle video image according to the operation information, generating a lock pad detection result signal, and sending the lock pad detection result signal to a bridge crane land side trolley electric control system.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011502610.3A CN112661013B (en) | 2020-12-17 | 2020-12-17 | Automatic wharf bridge crane legacy lock pad detection method and system |
| PCT/CN2021/123337 WO2022127311A1 (en) | 2020-12-17 | 2021-10-12 | Left lock pad detection method and system of automated wharf bridge crane |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011502610.3A CN112661013B (en) | 2020-12-17 | 2020-12-17 | Automatic wharf bridge crane legacy lock pad detection method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112661013A CN112661013A (en) | 2021-04-16 |
| CN112661013B true CN112661013B (en) | 2023-06-30 |
Family
ID=75406325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011502610.3A Active CN112661013B (en) | 2020-12-17 | 2020-12-17 | Automatic wharf bridge crane legacy lock pad detection method and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112661013B (en) |
| WO (1) | WO2022127311A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112661013B (en) * | 2020-12-17 | 2023-06-30 | 北京航天自动控制研究所 | Automatic wharf bridge crane legacy lock pad detection method and system |
| CN115293443B (en) * | 2022-08-17 | 2024-03-22 | 上海国际港务（集团）股份有限公司 | Bridge crane and container ship loading and unloading operation time prediction method, system and medium |
| CN116074616B (en) * | 2023-04-06 | 2023-06-20 | 上海知率智能科技有限公司 | Container image acquisition system |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207879B (en) * | 2012-01-17 | 2016-03-30 | 阿里巴巴集团控股有限公司 | The generation method and apparatus of image index |
| JP2014174021A (en) * | 2013-03-08 | 2014-09-22 | Mitsui Eng & Shipbuild Co Ltd | Center of gravity measuring system and center of gravity measuring method for a truck loading container |
| CN106203239B (en) * | 2015-05-04 | 2020-08-04 | 杭州海康威视数字技术股份有限公司 | Information processing method, device and system for container tallying |
| CN106600581B (en) * | 2016-12-02 | 2019-09-24 | 北京航空航天大学 | A kind of train operation automatic fault detection system and method based on binocular stereo vision |
| CN107680092B (en) * | 2017-10-12 | 2020-10-27 | 中科视拓（北京）科技有限公司 | Container lock catch detection and early warning method based on deep learning |
| CN108491851B (en) * | 2018-01-29 | 2020-07-31 | 江苏大学 | Container lockhole rapid identification and hanger deviation correction method based on machine vision |
| CN108846440B (en) * | 2018-06-20 | 2023-06-02 | 腾讯科技（深圳）有限公司 | Image processing method and device, computer readable medium and electronic equipment |
| CN109335964B (en) * | 2018-09-21 | 2020-05-12 | 北京航天自动控制研究所 | Container twist lock detection system and detection method |
| CN110033026B (en) * | 2019-03-15 | 2021-04-02 | 深圳先进技术研究院 | Target detection method, device and equipment for continuous small sample images |
| CN211644372U (en) * | 2019-12-20 | 2020-10-09 | 青岛港国际股份有限公司 | Container lock pad frame hoisting device for quayside container crane |
| CN111275682A (en) * | 2020-01-19 | 2020-06-12 | 上海箱云物流科技有限公司 | Container detection method, device and computer readable storage medium |
| CN112661013B (en) * | 2020-12-17 | 2023-06-30 | 北京航天自动控制研究所 | Automatic wharf bridge crane legacy lock pad detection method and system |
-
2020
- 2020-12-17 CN CN202011502610.3A patent/CN112661013B/en active Active
-
2021
- 2021-10-12 WO PCT/CN2021/123337 patent/WO2022127311A1/en active Application Filing
Non-Patent Citations (1)
| Title |
|---|
| 刘祥龙等编著.《飞桨PaddlePaddle深度学习实战》.北京：机械工业出版社,2020,228-232. * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022127311A1 (en) | 2022-06-23 |
| CN112661013A (en) | 2021-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112661013B (en) | Automatic wharf bridge crane legacy lock pad detection method and system | |
| US7508956B2 (en) | Systems and methods for monitoring and tracking movement and location of shipping containers and vehicles using a vision based system | |
| CN106067023B (en) | Container number and truck number identification system and method based on image processing | |
| CN112528721B (en) | Bridge crane integrated card safety positioning method and system | |
| CN109335964B (en) | Container twist lock detection system and detection method | |
| CN111832415B (en) | Truck safety intelligent protection system for container hoisting operation | |
| WO2015022001A1 (en) | Method and system for automatically landing containers on a landing target using a container crane | |
| CN109534177A (en) | A kind of anti-hoisting device of truck based on machine vision and truck are prevented slinging method | |
| CN116208745A (en) | Dock ship berthing monitoring and recording system and method | |
| WO2009052854A1 (en) | Device, method and system for recording inspection data about a freight container | |
| CN107200274A (en) | A kind of anti-container truck based on machine vision is lifted method | |
| CN114998824A (en) | Vehicle loading and unloading task monitoring method, device and system | |
| CN111163290A (en) | Device and method for detecting and tracking night navigation ship | |
| CN111539344B (en) | Video stream and artificial intelligence based integrated card anti-lifting control system and method | |
| CN115588121A (en) | Tower crane lifting object type detection method and system based on sensing data and image sequence | |
| CN113435278A (en) | Crane safety detection method and system based on YOLO | |
| CN208802612U (en) | Ship loader operating system and ship loader | |
| CN115849189B (en) | Point cloud-based lifting appliance secondary anchoring method, system, equipment and storage medium | |
| CN113415727A (en) | Container loading and unloading identification safety guiding device and method | |
| WO2004058623A1 (en) | A system and a method for container identification | |
| CN211733642U (en) | Automatic identification system for container shipping berth | |
| CN114782359A (en) | Industrial conveyor belt surface defect detection method based on data heterogeneous fusion | |
| CN110516551B (en) | Vision-based line patrol position deviation identification system and method and unmanned aerial vehicle | |
| CN114332824B (en) | Automatic winding and unwinding devices of trouble warning of commodity circulation vehicle | |
| CN220300222U (en) | Device for detecting non-separated state of lifting appliance and container based on vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |