WO2023070954A1

WO2023070954A1 - Container truck guidance and single/double-container identification method and apparatus based on machine vision

Info

Publication number: WO2023070954A1
Application number: PCT/CN2022/072004
Authority: WO
Inventors: 郑智辉; 闫威; 唐波; 郭宸瑞; 王硕; 董昊天; 闫涛; 李钊
Original assignee: 北京航天自动控制研究所
Priority date: 2021-10-29
Filing date: 2022-01-14
Publication date: 2023-05-04
Also published as: CN114119741A

Abstract

A container truck guidance and single/double-container identification method and apparatus based on machine vision, which method and apparatus belong to the technical field of operation assistance for quayside container cranes at harbors, and solve the problem of existing methods of the efficiency of container grabbing/placing operations of quayside container cranes being low due to the container truck positioning precision being low and the single/double container determination accuracy being insufficient. The method comprises: calibrating a target parking position of a container truck and the height estimation of the container truck loaded with a container; calculating an initial positioning deviation distance on the basis of a target rectangular area and the target parking position; performing image cropping on a vehicle body image, so as to generate a vehicle body sub-image; identifying container hole coordinates and characters or frame guide plate coordinates by using a second target detection model; performing single/double container determination on the basis of whether the target rectangular area is a fusion area, and the container hole coordinates and the characters; estimating the height of the container; calculating a movement deviation distance, so as to guide the container truck to the target parking position; and adjusting the shape of a spreader according to determined single/double containers, so as to perform a container grabbing or placing operation. The container truck positioning precision and the single/double container determination accuracy are high, thereby improving the efficiency of container grabbing/placing operations of a quayside container crane.

Description

A machine vision-based collection truck guidance and single and double box identification method and device

technical field

The present application relates to the technical field of port quay crane operation assistance, in particular to a machine vision-based quay crane control method and device.

Background technique

Containers are an important carrier in the process of modern logistics and transportation, and the efficiency of loading and unloading of containers at ports directly affects the efficiency of the entire logistics and transportation. As the name implies, the quay crane refers to the bridge-type equipment used on the shore. It is an important working tool for unloading containers from the ship to the wharf or loading them on the ship from the wharf. The driver controls the trolley and the spreader through the handle, and carries out the operation of grabbing and releasing boxes from the collection truck in the port. Since the truck driver does not have the correct target guidance, the truck driver needs to manually observe when the spreader reaches the frame or container of the inner truck every time the inner truck grabs and releases the container. Or the guidance of external personnel, in order to carry out accurate grasping and putting operations. On the other hand, the quay crane driver also has to concentrate on observing the single and double box conditions in the current operating vehicle, and then control the different forms of the spreader. This not only greatly reduces the operating efficiency of the container, but also increases the workload of the drivers of the trucks and quayside cranes. Collection truck is the abbreviation of container truck, which is divided into inner collection truck and outer collection truck. The inner collection truck refers to the truck running in the container port, and the outer collection truck refers to the truck from the outside to the container port.

Traditionally, the lidar scanning method is generally used to identify the container or frame of the truck, but because the lidar is expensive, the function is single, and the accuracy cannot be effectively guaranteed.

The existing safety positioning method for bridge cranes and trucks uses the camera to calibrate the parking point of the vehicle in advance, and then when the vehicle enters the recognition area, a certain range is added or subtracted according to the calibrated parking point of the vehicle, the image is cropped, and the Mask-RCNN algorithm is used to identify and segment out The area of the truck container or frame, and obtain the center point, and calculate its Euclidean distance with the pre-marked parking point, so as to guide the vehicle to the accurate position.

In the existing method, there are the following problems:

1. The existing method assumes that the vehicle needs to drive to the vicinity of the exact target parking position first, so it cannot effectively segment the collection trucks when the deviation distance is large, that is, beyond a certain range, which leads to inaccurate guidance.

2. When the height changes due to changes in container specifications, this method cannot effectively estimate the height of the container on the current truck, resulting in a guidance error.

3. Due to the complexity of the shape of the inner truck frame, the method is difficult to effectively and accurately segment the truck frame, which leads to guidance errors.

4. Since this method needs to divide and binarize the truck containers or frames in various working conditions and weathers, it needs to label a large number of container or frame contour mask data, which requires high manual research and development costs. The operation of the segmentation 2D algorithm is also time-consuming, which may cause processing delay or increase hardware costs.

5. This method does not clearly provide a determination method for the direction of the vehicle, that is, the vehicle should move forward or backward.

The machine vision part of the existing double box detection method is to use a camera to shoot the middle of the container, and then obtain the image of the middle of the container, use the box hole recognition model to identify the box hole, and then determine the single and double boxes.

In the existing method, there are the following problems:

1) This method only relies on the box hole recognition in the middle image of the container to determine whether there are double boxes, and it does not deal with the missing or false detection of the box holes, so there is a hidden danger of double box misjudgment, which reduces the correct rate of discrimination.

2) This method cannot guarantee the accuracy of data collection, that is, it cannot guarantee that the collected image is exactly the middle area of the container, which leads to misjudgment.

Contents of the invention

In view of the above analysis, the embodiment of the present application aims to provide a machine vision-based quay crane control method and device to solve the problem of low positioning accuracy of the collection truck and insufficient accuracy of single and double tank judgments in the existing method, which cause the quay crane The problem of low efficiency in picking and placing boxes.

On the one hand, the embodiment of the present application provides a machine vision-based quay crane control method, including: calibrating the target parking position of the collection truck and the estimated height of the collection truck loaded with containers; acquiring and using the first target detection model to identify the vehicle body image The target rectangular area of the container or frame in the vehicle body image, and calculate the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area includes the minimum circumscribed rectangle; for the container in the vehicle body image or the target rectangular area of the vehicle frame to crop the image to generate the sub-image of the vehicle body; obtain and use the second target detection model to identify the box hole coordinates and text on the container in the sub-image of the vehicle body or the frame guide plate coordinates on the vehicle frame ; Based on whether the target rectangular area of the container in the body sub-image is a fusion area, the box hole coordinates and the text of the middle sub-area in the body sub-image, carry out single and double box judgment; based on the box hole coordinates or the The frame guide plate coordinates are used to obtain the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image to estimate the height of the container; based on the height of the container or the frame and the box hole or The target straight line generated by the frame guide plate calculates the movement deviation distance, guides the truck to the target parking position according to the movement deviation distance; and when the spreader of the quay bridge reaches the container or the vehicle frame When it is above, adjust the form of the spreader of the quay crane according to the judged single and double boxes, and carry out the operation of grabbing or putting boxes.

The beneficial effects of the above-mentioned technical solution are as follows: the initial positioning of the collection truck increases the distance range within which the collection truck can be guided and positioned in the quayside crane, and at the same time, the positioning accuracy of the internal collection truck is improved through the detection and positioning of small targets such as box holes or guide plates. The precise guidance and positioning of the inner collection truck is realized, and the positioning accuracy of the collection truck is improved. Using the comprehensive container quantity, box hole and the text information on the container surface, the single and double box judgment method is carried out. Through this joint judgment method, the misjudgment caused by relying on a single information judgment can be avoided, thereby effectively improving the accuracy of single and double box discrimination. Therefore, the operating efficiency of the quay crane grabbing and releasing boxes is improved.

Based on a further improvement of the above method, the calibration of the target parking position of the collection truck and the estimated height of the container loaded container further includes: pre-collecting the body image of the target parking position of the collection truck and calculating the body image of the target parking position Carry out identification to obtain the image coordinates of the container and the corresponding box hole or the frame and the corresponding frame guide plate image coordinates of the truck at the target parking position; using the box hole image coordinates or the frame guide plate image coordinates Generate the target line and estimate the height-dependent pixel distance factor.

Based on the further improvement of the above method, acquiring and using the first target detection model to identify the target rectangular area of the container or frame in the vehicle body image further includes: acquiring multiple historical vehicle body images from the database and analyzing the multiple historical vehicle body images Mark the container or vehicle frame in; establish the first neural network Yolov5 and use the multiple pieces of historical vehicle body images marked to train the first neural network Yolov5 to obtain the first target detection model; and collect the current vehicle body image in real time, and Use the first target detection model to identify the target rectangular area of the container or frame in the current vehicle body image, so as to judge whether to grab or put the box, wherein the determination is made according to the size of the target rectangular area of the container. Whether the container is 20 feet or not, and the target rectangular areas of two 20 feet containers on the same container are merged to form the fusion area.

Based on a further improvement of the above method, performing image clipping on the target rectangular area of the container or vehicle frame in the vehicle body image to generate a vehicle body sub-image further includes: clipping the historical vehicle body image or the current vehicle body image into a historical vehicle body sub-image image or the current body sub-image, wherein, when the target rectangular area is a container area, the historical body sub-image or the current body sub-image includes a first upper sub-area, a middle sub-area and a first lower sub-area , wherein box hole and text detection is performed on the first middle sub-region, and box hole detection is performed on the first upper sub-region image and the first lower sub-region image; and when the target rectangular region is frame area, the historical body sub-image or the current body sub-image includes a second upper sub-area and a second lower sub-area, wherein the second upper sub-area and the second lower sub-area are Frame guide inspection.

Based on the further improvement of the above method, acquiring and using the second target detection model to identify the box hole coordinates on the container or the frame guide plate coordinates on the frame in the sub-image of the vehicle body further includes: Mark the box hole or the frame guide plate; establish the second neural network Yolov5 and utilize the historical body image of the label to train the second neural network Yolov5 to obtain the second target detection model; and use the second target detection The model identifies box holes or frame guides in the current body subimage, and obtains box hole coordinates and text and frame guide coordinates.

Based on the further improvement of the above method, the first camera, the second camera, the third camera, the fourth camera, the fifth camera and the sixth camera are installed at the isolation strips on the opposite sides of the lane on the quay bridge beam, wherein, using the The first camera to the fourth camera capture the head image of the truck to confirm the identity of the vehicle and the moving direction of the vehicle; and use the fifth camera and the sixth camera to capture the body image of the truck, To calculate the initial positioning deviation distance and the movement deviation distance of the collection truck.

Based on the further improvement of the above method, after calibrating the target parking position of the collection truck and the estimated height of the collection truck loaded with containers, it further includes: acquiring a plurality of historical vehicle front images from the database and analyzing the vehicle front and two in the historical vehicle front images Dimensional codes are marked; the third neural network Yolov5 is established, and the third neural network Yolov5 is trained to obtain the third target detection model by using the marked historical front image; the current front image is collected in real time, and the third target detection model is used Recognizing the collection truck head in the current front image and the two-dimensional code pasted on the collection truck head and confirming the collection card identity code; and connecting the data receiving unit in the cab of the corresponding collection truck through the network according to the collection card identity code.

Based on a further improvement of the above method, calculating the initial positioning deviation distance based on the target rectangular area and the target parking position further includes: based on the identified target rectangular area of the container or frame in the current vehicle body image and the pre-acquired Target parking position, calculate the initial positioning deviation distance of the collection truck; and transmit the initial positioning deviation distance of the collection truck to the data receiving unit through the network, and display it through the LED display screen to guide the collection truck driver to adjust the collection truck position, wherein the initial positioning deviation distance of the collection truck is calculated by the following formula:

offset＝D(yy ₀ )

Among them, y represents the ordinate of the current vehicle area image, y ₀ represents the ordinate of the target parking position collected in advance, and D represents the actual pixel distance factor related to the height.

Based on a further improvement of the above method, the moving deviation distance is calculated based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, and the collection truck is guided to the target parking according to the moving deviation distance The position further includes: calculating the movement deviation distance of the collection card by the following formula further includes:

Among them, x ₀ , y ₀ are the midpoints of the two currently detected box holes, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height; through the The network sends the moving deviation distance to the data receiving unit and displays the moving deviation distance on the LED display to continue to guide the driver to adjust the truck collection position until the truck collection is completed when the moving deviation distance is less than a threshold Positioning guidance.

On the other hand, the embodiment of the present application provides a machine vision-based quay crane control device, including: a video processor and a control module, the video processor includes a calibration module, an initial positioning module, an image cropping module, an identification module, The single and double box judgment module, the height estimation module and the movement deviation distance calculation module, wherein the calibration module is used to calibrate the target parking position of the collection truck and the height estimation of the collection truck loaded with containers; the initial positioning module uses Obtaining and using the first target detection model to identify the target rectangular area of the container or frame in the body image, and calculating the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area Including the minimum circumscribed rectangle; the image cropping module is used to crop the target rectangular area of the container or frame detected in the vehicle body image to generate a sub-image of the vehicle body; the identification module is used to obtain and use the first The second target detection model recognizes the coordinates of the box hole on the container in the sub-image of the body and the coordinates of the frame guide plate on the text or the frame; Whether the target rectangular area is a fusion area, the box hole coordinates and text of the middle sub-area in the body sub-image, and perform single and double box judgment; the height estimation module is used to base on the box hole coordinates or the car body frame guide plate coordinates to obtain the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image to estimate the height of the container; the movement deviation distance calculation module is used for The height of the frame and the target straight line generated by the box hole or the frame guide plate calculate the movement deviation distance, so as to guide the collection truck to the target parking position according to the movement deviation distance; and the control module is used for When the spreader of the quay crane arrives directly above the container or the frame, adjust the shape of the spreader of the quay crane according to the judged single or double container, and carry out the operation of grabbing or releasing the container.

Based on the further improvement of the above-mentioned device, it includes a camera, which is used to pre-collect the vehicle body image of the target parking position of the truck, and intermittently collect the vehicle body image; the calibration module also includes a target position calibration sub-module and a height estimation calibration The submodule, wherein, the target position calibration submodule of the camera is used to identify the body image of the target parking position, and obtain the container and the corresponding box hole image of the truck at the target parking position coordinates or frame and corresponding frame guide plate image coordinates to generate a target straight line; and the height estimation and calibration sub-module uses the box hole image coordinates or the frame guide plate image coordinates to estimate a height-related pixel distance factor.

Based on a further improvement of the above device, the initial positioning module includes a labeling submodule, a first target detection model, and a target rectangle generation submodule, wherein the first labeling submodule is used to obtain multiple historical vehicle body images from the database And label the container or vehicle frame in the multiple historical vehicle body images; the first target detection model is used to establish the first neural network Yolov5 and utilize the multiple historical vehicle body images of the label to analyze the first neural network Yolov5 is trained to obtain the first target detection model; and the target rectangle generation submodule is used to collect the current vehicle body image in real time, and utilize the first target detection model to identify the container or frame in the current vehicle body image The target rectangular area of the container is used to determine whether the container is 20 feet or not according to the size of the target rectangular area of the container, and the two 20-foot containers on the same collection card are The target rectangular areas of the containers are fused to form the fused area.

Based on the further improvement of the above device, the image cropping module is used to crop the historical vehicle body image or the current vehicle body image into a historical vehicle body sub-image or a current vehicle body sub-image, wherein when the target rectangular area is a container area , the historical body sub-image or the current body sub-image includes a first upper sub-area, a middle sub-area and a first lower sub-area, wherein the box hole and text detection is performed on the first middle sub-area, and the The first upper sub-region image and the first lower sub-region image are subjected to box hole detection; and when the target rectangular region is a frame region, the historical body sub-image or the current body sub-image includes The second upper sub-area and the second lower sub-area, wherein the detection of the frame guide plate is performed on the second upper sub-area and the second lower sub-area.

Based on the further improvement of the above device, the recognition module includes a second labeling submodule, a second target detection model, and a box hole and frame guide plate recognition submodule, wherein the second labeling submodule is used to identify the historical The box hole or the frame guide plate in the car body sub-image are marked; the second target detection model is used to set up a second neural network Yolov5 and utilize the marked historical car body sub-image to train the second neural network Yolov5 to obtain The second target detection model; and the box hole and frame guide plate identification submodule, used to use the second target detection model to identify the box hole and text or the frame guide plate in the current body sub-image, and obtain the box hole coordinates or frame guide coordinates.

Based on the further improvement of the above-mentioned device, the cameras include a first camera, a second camera, a third camera, a fourth camera, a fifth camera and a sixth camera, which are installed at the isolation strips on opposite sides of the lane on the cross beam of the quay bridge , wherein, the first camera to the fourth camera are used to capture the head image of the truck to confirm the identity of the vehicle and the moving direction of the vehicle; and the fifth camera and the sixth camera are used to Taking a body image of the collection truck to calculate the initial positioning deviation distance and the movement deviation distance of the collection truck.

Based on the further improvement of the above-mentioned device, the quay bridge control device based on machine vision also includes a data receiving unit, and the identification module also includes a third labeling submodule, a third target detection model and a vehicle head and identity confirmation submodule, wherein the The third labeling sub-module is used to obtain multiple historical car head images from the database and to mark the car head and the two-dimensional code in the historical car head images; the third target detection model is used to establish a third neural network Yolov5, and use the marked historical head image to train the third neural network Yolov5 to obtain the third target detection model; the head and identity confirmation sub-module is used to collect the current head image in real time, and use the third target detection model to identify Find the collection truck head in the current head image and the two-dimensional code pasted on the collection truck head and confirm the collection card identity code; and the data receiving unit is used to be located in the collection truck cab and according to the collection card identity code Connect with the video processor through the network.

Based on the further improvement of the above-mentioned device, the quay crane control device based on machine vision also includes an LED display screen, and the initial positioning module is used for pre-acquired Calculate the initial positioning deviation distance of the collection truck, wherein, the initial positioning deviation distance of the collection truck is calculated by the following formula:

offset＝D(yy ₀ )

Wherein, y represents the ordinate of the current vehicle area image, y ₀ represents the ordinate of the target parking position collected in advance, and D represents the actual pixel distance factor related to the height; the data receiving unit transfers the set The initial positioning deviation distance of the card is transmitted to the data receiving unit; and the LED display screen is located in the cab of the collection truck and communicated with the data receiving unit for displaying the initial positioning deviation distance to guide the collection truck driver to adjust Card location.

Based on the further improvement of the above device, the movement deviation distance calculation module is used to calculate the movement deviation distance of the truck through the following formula and further includes:

Wherein, x ₀ , y ₀ are the midpoints of the two box holes currently detected respectively, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height; The data receiving unit receives the moving deviation distance through the network; the LED display is used to display the moving deviation distance, so as to continue to guide the driver to adjust the truck collection position until the truck collection is completed when the moving deviation distance is less than a threshold Positioning guidance.

Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

1. By installing a camera on the crossbeam of the quay bridge, this application proposes an initial positioning of the truck based on the detection of the container or frame area of the truck, and accurate positioning through the common features of the container or frame (box holes and guide plates). method. Through the initial positioning of the collection truck, the distance range for the guidance and positioning of the internal collection truck of the quayside crane is increased. At the same time, the positioning accuracy of the internal collection truck is improved through the detection and positioning of small targets such as box holes or guide plates, and the precise guidance of the internal collection truck is realized. position.

2. This application adopts a height estimation method based on the distance of the container hole. This method effectively distinguishes the guidance error caused by the different heights of the containers, and by estimating the height of the collection truck in the loaded container, it effectively adapts to the operating conditions of various container heights and improves the adaptability of the system.

3. This application adopts the comprehensive container quantity, container hole and text information on the upper surface of the container to determine single and double containers. Through the joint judgment method, the misjudgment caused by relying on a single information judgment is avoided, and the accuracy of single and double box discrimination is effectively improved.

In the present application, the above technical solutions can also be combined with each other to realize more preferred combination solutions. Additional features and advantages of the application will be set forth in the description which follows, and some of the advantages will be apparent from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the matter particularly pointed out in the written description and accompanying drawings.

Description of drawings

The drawings are for the purpose of illustrating specific embodiments only and are not to be considered limiting of the application, and like reference numerals refer to like parts throughout the drawings.

Fig. 1 is a flowchart of a machine vision-based control method for quay cranes according to an embodiment of the present application.

Fig. 2 is a schematic diagram of installation arrangement of a camera device according to an embodiment of the present application.

Fig. 3 is a schematic diagram of collection cards collected by cameras, their recognition results and deviations according to an embodiment of the present application.

Fig. 4 is a specific flow chart of a machine vision-based quay crane control method according to an embodiment of the present application.

Fig. 5 is a schematic diagram of height estimation according to an embodiment of the present application.

Fig. 6 is a schematic diagram of vehicle head and two-dimensional code recognition according to an embodiment of the present application.

Fig. 7 is a schematic diagram of clipping and fusion of loaded container regions according to an embodiment of the present application.

Fig. 8 is a schematic diagram of clipping and fusion of frame regions according to an embodiment of the present application.

Fig. 9 is a schematic diagram of the working principle of the container truck in the quay crane according to the embodiment of the present application.

Fig. 10 is a block diagram of a machine vision-based quay crane control device according to an embodiment of the present application.

Detailed ways

Preferred embodiments of the application are described in detail below in conjunction with the accompanying drawings, wherein the drawings constitute a part of the application and together with the embodiments of the application are used to explain the principles of the application and are not intended to limit the scope of the application.

A specific embodiment of the present application discloses a machine vision-based control method for quay cranes. As shown in Figure 1, the machine vision-based quay crane control method includes: in step S102, calibrate the target parking position of the collection truck and the estimated height of the collection truck loaded with containers; in step S104, obtain and use the first target The detection model identifies the target rectangular area of the container or frame in the body image, and calculates the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area includes the minimum circumscribed rectangle; in step S106, the body image The target rectangular area of the container or frame in the image is cropped to generate the body sub-image; in step S108, obtain and use the second target detection model to identify the box hole coordinates and text or frame on the container in the body sub-image frame guide plate coordinates on the upper frame; in step S110, whether the target rectangular area of the container in the sub-image of the vehicle body is a fusion area, the box hole coordinates and the text of the middle sub-region in the sub-image of the vehicle body, carry out single and double container judgment; In step S112, the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image are obtained based on the box hole coordinates or the frame guide plate coordinates to estimate the height of the container; in step S114, based on the container or vehicle frame height and the target straight line generated by the box hole or the frame guide plate to calculate the movement deviation distance, and guide the truck to the target parking position according to the movement deviation distance; When it is above, adjust the spreader shape of the quay crane according to the judged single and double boxes, and carry out the operation of grabbing or putting boxes.

Compared with the prior art, in the machine vision-based quay crane control method provided by this embodiment, the distance range within which the quay truck can be guided and positioned by the collection truck in the quay crane is increased through the initial positioning of the collection truck, and at the same time, small targets such as box holes or guide plates The detection and positioning of the inner collection truck improve the positioning accuracy of the inner collection truck, and then realize the precise guidance and positioning of the inner collection truck, and improve the positioning accuracy of the collection truck. Using the comprehensive container quantity, box hole and the text information on the container surface, the single and double box judgment method is carried out. Through this joint judgment method, the misjudgment caused by relying on a single information judgment can be avoided, thereby effectively improving the accuracy of single and double box discrimination. Therefore, the operating efficiency of the quay crane grabbing and releasing boxes is improved.

Hereinafter, steps S102 to S116 in the machine vision-based quay crane control method will be described in detail with reference to FIG. 1 .

First, the first camera, the second camera, the third camera, the fourth camera, the fifth camera and the sixth camera are installed at the isolation strips on the opposite sides of the lane on the cross beam of the quay bridge, wherein, using the first camera to the fourth camera The camera captures the front image of the truck to confirm the identity of the vehicle and the moving direction of the vehicle; and uses the fifth camera and the sixth camera to capture the body image of the truck to calculate the initial positioning deviation distance and the movement deviation distance of the truck. Store the captured body image and head image in the database.

In step S102, the target parking position of the truck and the estimated height of the truck loaded with containers are calibrated. Specifically, the calibration of the target parking position of the collection truck and the estimated height of the collection truck loaded with containers further includes: pre-collecting the body image of the target parking position of the collection truck and identifying the body image of the target parking position, so as to obtain the vehicle body image of the collection truck at the target parking position. The image coordinates of the container and the corresponding box hole or the frame and the corresponding frame guide plate image coordinates at the target parking position; use the box hole image coordinates or the frame guide plate image coordinates to generate the target line L and estimate the height-related pixel distance factors D and E .

After calibrating the target parking position of the truck and the estimated height of the truck loaded with containers, it further includes: obtaining multiple historical front images from the database and marking the front and QR codes in the historical front images; establishing a third nerve Network Yolov5, and use the marked historical head image to train the third neural network Yolov5 to obtain the third target detection model; collect the current head image in real time, and use the third target detection model to identify the set truck head and The QR code pasted on the front of the truck and confirm the identity code of the truck; and connect the data receiving unit in the cab of the corresponding truck through the network according to the ID code of the truck.

In step S104, acquire and use the first object detection model to identify the target rectangular area of the container or frame in the body image, and calculate the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area includes the minimum The bounding rectangle. Specifically, acquiring and using the first target detection model to identify the target rectangular area of the container or the frame in the body image further includes: acquiring multiple historical body images from the database and detecting the container or frame in the multiple historical body images Carry out labeling; Establish the first neural network Yolov5 and use the multiple pieces of historical body images of the label to train the first neural network Yolov5 to obtain the first target detection model; and collect the current vehicle body image in real time, and use the first target detection model to identify The target rectangular area of the container or vehicle frame in the current body image is used to determine whether to grab or put the container, wherein, according to the size of the target rectangular area of the container, it is determined whether the container is 20 feet or not, and the The target rectangular areas of the two 20-foot containers merge to form a fusion area. Calculating the initial positioning deviation distance based on the target rectangular area and the target parking position further includes: calculating the initial positioning deviation distance of the truck based on the recognized target rectangular area of the container or frame in the current body image and the pre-acquired target parking position; And the initial positioning deviation distance of the collection truck is transmitted to the data receiving unit through the network, and displayed on the LED display screen to guide the collection truck driver to adjust the position of the collection truck. The initial positioning deviation distance of the collection truck is calculated by the following formula:

offset＝D(yy ₀ )

Among them, y represents the ordinate of the current vehicle area image, y ₀ represents the ordinate of the target parking position collected in advance, and D represents the actual pixel distance factor related to height in mm/pixel, in other words, at a single height level When it is 2.4m, 2.6m or 2.9m, the actual pixel distance factor D is also different.

In step S106 , image cropping is performed on the target rectangular area of the container or frame in the vehicle body image to generate a body sub-image. Specifically, performing image cropping on the target rectangular area of the container or frame in the vehicle body image to generate the vehicle body sub-image further includes: clipping the historical vehicle body image or the current vehicle body image into the historical vehicle body sub-image or the current vehicle body sub-image, wherein, when When the target rectangular area is a container area, the historical body sub-image or the current body sub-image includes the first upper sub-area, the middle sub-area and the first lower sub-area, wherein the first middle sub-area is subjected to box hole and text detection, Carrying out box hole detection on the first upper sub-region image and the first lower sub-region image; and when the target rectangular region is the frame region, the historical body sub-image or the current body sub-image includes the second upper sub-region and the second lower A sub-area, wherein the frame guide plate detection is performed on the second upper sub-area and the second lower sub-area. Specifically, when the Y coordinate of the center point of the detected rectangular area is less than 1/3 of the image height, the cropping ratio is 0.15, 0.4, 0.45; when the Y coordinate is greater than 1/3 and less than 2/3, it is 0.25, 0.4, 0.35; When the coordinate is greater than 2/3, the scale is 0.35, 0.4, 0.25. Since the image in the middle sub-region has a fixed image ratio and is shifted up and down at the center of the entire image as needed, this image cropping method can guarantee the detection of box holes and text.

In step S108 , acquire and use the second target detection model to identify the box hole coordinates and text on the container in the vehicle body sub-image or the coordinates of the frame guide plate on the frame. Specifically, acquiring and using the second target detection model to identify the box hole coordinates on the container or the frame guide plate coordinates on the frame in the body sub-image further includes: Annotate; set up the second neural network Yolov5 and utilize the historical body image of labeling to train the second neural network Yolov5 to obtain the second target detection model; and utilize the second target detection model to identify the box holes and text or frame guide and get box hole coordinates or frame guide coordinates.

In step S110, based on whether the target rectangular area of the container in the body sub-image is a fusion area, that is, whether it is two 20-foot containers, the box hole coordinates and the text of the middle sub-area in the body sub-image, perform single and double container judge. Calculate the total score according to whether the container area in the container image is two 20-foot containers, the box hole and the text in the middle sub-area. The total score is calculated by the following formula:

Among them, weight ₀ indicates the weight of the number of containers, whether weight ₁ has text weight and weight ₂ has box hole weight, R ₀ respectively indicates whether there are two containers, whether R ₁ has text, and whether R ₂ has box holes, and if there is, it is 1 , nothing is 0. When the total score calculated by the above formula is greater than the threshold, it is judged that the vehicle-mounted container is a double box; and when the total score calculated by the above formula is less than or equal to the threshold, it is judged that the vehicle-mounted container is a single box.

In step S112, the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image is acquired based on the box hole coordinates or the frame guide plate coordinates to estimate the height of the container. Specifically, the height estimation is performed for the container-laden container trucks, taking into account the individual height classes 2.4, 2.6 and 2.9 of the container. It is fixed according to the position of the camera, and the distance between the camera and the camera is different according to the height of the container, which in turn causes the distance of the box hole in the image to be different. For example, when the height level is 2.4, the distance between the box hole in the image is the smallest, and when At height class 2.9, the boxhole distance in the image is at its maximum. Therefore, it is necessary to estimate the container height of the container truck. First, calculate the closest position line between the detected center points of the two container holes, and then compare the distance d between the two container holes in the current image with the target stop when the height is calibrated. The size of the distance d _k between the two box holes in the body image of the position, dd _k > T ₀ is considered to be 2.9 level, if dd _k < T ₁ is considered to be 2.4 level, if it is within the range of T ₀ ~ T ₁ , Then it is regarded as a grade of 2.6, where T ₀ =3 and T ₁ =1. Also, the height of the frame is fixed, so there is no need to estimate the height of the frame.

In step S114, the movement deviation distance is calculated based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, and the collection truck is guided to the target parking position according to the movement deviation distance. Specifically, calculating the movement deviation distance based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, guiding the collection truck to the target parking position according to the movement deviation distance further includes: calculating the movement of the collection truck by the following formula Bias distances further include:

Among them, x ₀ , y ₀ are the midpoints of the two currently detected box holes, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height; through the network Send the movement deviation distance to the data receiving unit and display the movement deviation distance on the LED display to continue to guide the driver to adjust the position of the collection truck until the movement deviation distance is less than the threshold to complete the collection truck positioning guidance.

In step S116, when the spreader of the quay crane reaches directly above the container or the vehicle frame, adjust the form of the spreader of the quay crane according to the judged single or double container, and carry out the operation of grabbing or releasing the container.

Another specific embodiment of the present application discloses a machine vision-based quay crane control device. Referring to Fig. 10, the quay crane control device based on machine vision includes: a video processor 1002 and a control module 1018, and the video processor 1002 includes a calibration module 1004, an initial positioning module 1006, an image cropping module 1008, an identification module 1010, and single and double box judgment Module 1012, height estimation module 1014, movement deviation distance calculation module 1016, data receiving unit and LED display screen.

The camera is used for pre-collecting the vehicle body image of the target parking position of the collecting card, and intermittently collecting the vehicle body image. Referring to Fig. 2, the camera comprises a first camera 201, a second camera 202, a third camera 203, a fourth camera 204, a fifth camera 205 and a sixth camera 206. place. The first camera 201 to the fourth camera 204 are used to capture the front image of the collection truck to confirm the identity of the vehicle and the direction of movement of the vehicle; and the fifth camera 205 and the sixth camera 206 are used to capture the body image of the collection truck to calculate The initial positioning deviation distance and moving deviation distance of the truck.

The calibration module 1004 is configured to calibrate the target parking position of the truck and the estimated height of the truck loaded with containers. The calibration module also includes: a target position calibration sub-module and a height estimation calibration sub-module. The target position calibration sub-module is used to identify the body image of the target parking position, and obtain the image coordinates of the container and the corresponding box hole or the image coordinates of the vehicle frame and the corresponding frame guide plate at the target parking position to generate the target straight line L. The height estimation and calibration sub-module uses the box hole image coordinates or the frame guide image coordinates to estimate the height-related pixel distance factor.

The initial positioning module 1006 is configured to obtain and use the first target detection model to identify the target rectangular area of the container or frame in the body image, and calculate the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area Include the smallest bounding rectangle. The initial positioning module 1006 includes a labeling submodule, a first target detection model, and a target rectangle generation submodule. The first labeling submodule is used to acquire multiple historical vehicle body images from the database and label the containers or vehicle frames in the multiple historical vehicle body images. The first object detection model is used to establish the first neural network Yolov5 and train the first neural network Yolov5 by using multiple marked historical body images to obtain the first object detection model. The target rectangle generation sub-module is used to collect the current body image in real time, and use the first target detection model to identify the target rectangle area of the container or frame in the current body image. The initial positioning module 1006 is used to calculate the initial positioning deviation distance of the collection truck based on the recognized target rectangular area of the container or frame in the current body image and the pre-acquired target parking position, wherein the initial positioning deviation distance of the collection truck is calculated by the following formula Positioning deviation distance:

offset＝D(yy ₀ )

The image cropping module 1008 is configured to perform image cropping on the target rectangular area of the container or vehicle frame detected in the vehicle body image to generate a body sub-image. The image cropping module 1008 is used for cutting the historical body image or the current body image into a historical body sub-image or a current body sub-image, wherein, when the target rectangular area is a container area, the historical body sub-image or the current body sub-image includes the first upper part sub-areas, middle sub-areas and first lower sub-areas, wherein box hole and text detection is performed on the first middle sub-area, and box hole detection is performed on the first upper sub-area image and the first lower sub-area image; and When the target rectangular area is the frame area, the historical body sub-image or the current body sub-image includes a second upper sub-area and a second lower sub-area, wherein the frame guide plate is performed on the second upper sub-area and the second lower sub-area detection.

The recognition module 1010 is configured to obtain and use the second target detection model to recognize the box hole coordinates and text on the container in the body image or the frame guide plate coordinates on the frame. The recognition module 1010 includes a third labeling submodule, a third target detection model, and a vehicle front and identity confirmation submodule. The third labeling sub-module is used to acquire multiple historical vehicle front images from the database and label the vehicle front and the two-dimensional code in the historical vehicle front images. The third object detection model is used to establish the third neural network Yolov5, and use the marked historical head images to train the third neural network Yolov5 to obtain the third object detection model. The vehicle front and identity confirmation sub-module is used to collect the current vehicle front image in real time, and use the third target detection model to identify the collection truck head in the current vehicle front image and the QR code pasted on the collection truck head and confirm the identity code of the collection truck. In addition, the recognition module 1010 also includes a second labeling submodule, a second target detection model, and a box hole and frame guide plate recognition submodule. The second labeling sub-module is used to mark the box hole or the frame guide plate in the historical car body sub-image; the second target detection model is used to establish the second neural network Yolov5 and use the marked historical car body sub-image to the second neural network The network Yolov5 is trained to obtain the second target detection model; and the box hole and frame guide plate identification submodule is used to use the second target detection model to identify the box hole or frame guide plate in the current body sub-image, and obtain the box hole coordinates and frame guide coordinates.

The single and double box judging module 1012 is used to determine single and double boxes based on whether the target rectangular area of the container in the body sub-image is a fusion area, the box hole coordinates and the text of the middle sub-area in the body sub-image.

The height estimation module 1014 is configured to obtain the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image based on the box hole coordinates or the frame guide plate coordinates to estimate the height of the container.

The movement deviation distance calculation module 1016 is used to calculate the movement deviation distance based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, so as to guide the collection truck to the target parking position according to the movement deviation distance.

The data receiving unit is used to be located in the truck cab and connect with the video processor through the network according to the identity code of the truck. The data receiving unit receives the initial positioning deviation distance of the collection card through the network and transmits it to the data receiving unit, and receives the moving deviation distance through the network. The LED display is located in the cab of the truck and communicated with the data receiving unit, and is used to display the initial positioning deviation distance to guide the truck driver to adjust the truck position; and display the moving deviation distance to continue to guide the driver to adjust the truck position until When the movement deviation distance is less than the threshold, the positioning guidance of the collection card is completed.

The control module 1018 is used to adjust the shape of the spreader of the quay crane according to the judged single or double container when the spreader of the quay crane arrives directly above the container or the frame, and carry out the operation of grabbing or releasing the container. Specifically, the movement deviation distance calculation module is used to calculate the movement deviation distance of the collection card through the following formula and further includes:

Among them, x ₀ and y ₀ are the midpoints of the two currently detected box holes, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height.

Hereinafter, with reference to FIG. 1 to FIG. 9 , the machine vision-based control method of the quay crane will be described in detail by way of specific examples.

The technical problem to be solved in this application is to solve the long-distance guided positioning of the internal collection truck through the initial positioning of the internal collection truck, that is, to increase the positioning distance range, and at the same time to perform effective height estimation of the collection truck container and the common characteristics of the frame and the container (guide plate or box hole) detection to complete the precise positioning of the collection truck, and at the same time determine the driving direction of the collection truck through the judgment of the front of the truck, so as to better guide the collection truck driver. On the other hand, through the image acquisition after the initial positioning of the collection truck, the middle image area of the container can be effectively obtained, and the container detection results, the text detection results on the container, and the number and distance information of the container holes can be used to jointly determine the single and double containers, thereby effectively improving the single and double containers. Box discrimination accuracy.

This application provides a machine vision-based quay crane internal collection truck positioning system and single and double container discrimination method, which is characterized in that the intelligent video analysis technology is used to effectively identify the container area, box hole, text, and frame and container on the internal collection truck. The frame guide plate and other information are analyzed and judged to realize the positioning of the container truck in the quayside crane and the judgment of single and double boxes. On the other hand, through the video data, the special logo of the two-dimensional code on the head of the truck is identified, and the vehicle identity is confirmed and the direction of the vehicle is determined. With reference to Figure 4, the specific steps of the method are as follows:

1. Referring to Figure 2, install six cameras on the beam of the quay bridge, four of them (number: 201, 202, 203, 204) to shoot the front part of the inner truck, and the other two (number: 205, 206) to shoot the inner For the truck body part, the camera function of the front part is to confirm the identity of the vehicle and the moving direction of the vehicle, and the body camera is mainly responsible for the calculation of the positioning deviation distance of the inner truck and the judgment of single and double boxes.

An LED display and a data receiving unit are installed in the cab of the inner truck to display the current deviation distance and direction of the inner truck.

2. By pre-collecting the image data of the target parking position with different container types or empty frames at different positions, that is, the inner container truck to be installed, the data is analyzed and identified, and the box body, box hole or frame of the vehicle at the target position is obtained , the image coordinates of the frame guide plate, and the actual distances A mm/pixel and E mm/pixel represented by each pixel of the corresponding height.

3. By pre-intermittently collecting the image data of the inner container truck slowly driving through the field of view of the

cameras

205 and 206, and then calculating the Euclidean distance of the image coordinates of the two container holes near the camera side at different intervals of movement, according to Carry out the subsequent height estimation of the inner container of the loaded container.

4. When the vehicle enters the field of view of the 205 and 206 cameras, firstly, the target recognition algorithm YoloV5 is used to identify the container or empty frame area on the inner set card, and the container area is recognized to determine the size, distinguishing between 20 feet and non-20 feet , and then according to certain conditions, the 20-foot boxes on the same collection card are merged, and at the same time, according to the identified category (container or frame), it is judged that it is a box grabbing or boxing operation. And use the SORT (Simple Online and Realtime Tracking) target tracking method to track the inner collection card area.

5. According to the tracking results, the iterative judgment of the two frames of data before and after, when the Euclidean distance between the geometric centers of the recognition areas of the two consecutive frames is less than a certain threshold, and the number of statistical frames is greater than a certain number, the vehicle parking judgment is performed and the current vehicle is determined For working vehicles and work lanes.

6. Use the front recognition camera to identify the front of the internal truck and the QR code pasted on the front, confirm the ID number of the internal truck, and connect to the data receiving unit of the cab of the internal truck through the network.

7. By identifying the tracked box area center and the pre-acquired target position coordinates in step 2, compare the y coordinates to make a difference, and calculate the initial positioning deviation distance of the collection truck as follows:

offset＝D(yy ₀ )

Among them, y represents the ordinate of the current vehicle area image, y ₀ represents the pre-collected target position, and D represents the actual pixel distance factor.

The calculation result is sent to the data receiving unit of the corresponding driver through the network, and displayed on the LED display screen to guide the driver to adjust the position of the internal collection card.

8. After the initial positioning is completed, image cropping is performed on the detected container area or frame area of the internal collection truck. For the collection truck loaded with containers, the cropping area is divided into three sub-area images, and the middle cropping area is used for box holes and text Detection, the other two areas only detect the box hole, and for the frame area, the image is cropped into two areas, and both areas detect the frame guide.

9. According to whether the container truck area currently loaded is a fusion area, that is, whether it is two 20 feet, combined with the box hole and text detection information in the middle sub-area, determine single and double containers, as follows:

Among them, weight ₀ indicates the weight of container quantity (0.4), weight ₁ indicates whether there is text weight (0.2), and weight ₂ indicates whether there is box hole weight (0.4), R ₀ indicates whether there are two containers, R ₁ indicates whether there is text and R ₂ indicates whether there is a box hole, 1 if it is present, and 0 if it is not. When the total score is greater than a certain threshold (0.6), it is determined to be a double box, otherwise it is a single box.

10. Estimate the total height of the on-board container by using the distance and position of the lower sub-region box holes. Assuming the frames are then the same height, no height estimation is required.

11. Combining the height information and the target straight line L generated by the target box hole of the corresponding lane, the accurate calculation of the movement deviation distance of the inner truck is performed.

Among them, x ₀ , y ₀ are the midpoints of the two currently detected boxholes, A, B, and C represent the pre-detected two boxholes. The parameter E of the straight line equation represents the height-related pixel distance factor.

12. Continuously adjust and iteratively calculate in step 11. When the deviation distance offset is less than a certain threshold, the positioning guidance of the inner set card is completed.

The specific embodiment of the application can be divided into three parts:

1. Calibration of the target parking position of the container truck and the estimated total height of the container container.

(1) Park the inner collection trucks loaded with four different box types (single 20 feet, double 20 feet, 40 feet, 45 feet) and three different box heights (2.4m, 2.6m, 2.9m) at the The target position of each lane, that is, the position where the spreader can accurately grab the container, please refer to Figure 3, and then save the image coordinates of the center of the container rectangular area and the corresponding box hole for different box types, different box heights, and different lanes. data. On the other hand, the empty frame is parked at the target position of each lane on the inner collection trucks of different box types (front 20 feet, rear 20 feet, middle 20 feet, 40 feet, 45 feet), that is, the spreader can Accurately place the box, and then save the center of the rectangular area of the frame and the image coordinate data of the corresponding frame guide for different box locations, box types, lane information, etc., and use the box hole size to estimate the pixel distance A mm/pixel, The same operation is performed on the empty frame to generate the target straight line L and the estimated pixel distance factor E mm/pixel.

(2) Slowly drive a container loaded with 40 feet and a height of 2.6m through each lane, and then collect image data through the gap, and save the Euclidean distance between the two front box holes when the truck is at different positions in the same lane, Refer to Figure 5 as the basis for subsequent determination of the height of the container truck.

2. Inner truck guidance and identification of single and double boxes: Confirm the current ID number of the inner truck through real-time video data, and calculate the deviation of the inner truck at the same time, and send the truck guidance information to the cab of the inner truck through the network, and carry out LED display, and at the same time detect and identify single and double boxes on the inner collection card, and send the identification results to the quay crane control system.

(1) First, the pre-trained target detection algorithm Yolov5-0 (the output of the algorithm model is two categories, namely the container on the truck and the frame) and the target detection algorithm Yolov5-1 (the output of the algorithm model is three The two categories are box holes, frame guides, and text), and then use Yolov5-0 for target recognition, and judge whether to grab or put a box according to the recognition category, and then use the size of the rectangle to determine whether it is 20 feet or not, and then according to With certain constraints, the 20-foot box is fused, that is, it is judged whether the width and height of the circumscribed rectangular areas of any two rectangular areas in the image are within a certain range. If the conditions are met, the two rectangular areas are merged into one area, and then Use the SORT tracking algorithm for target tracking. When the tracking distance of the current and subsequent frames of data is less than a certain threshold, and the number of statistical frames less than the threshold is greater than a certain number, it is determined that the vehicle stops and is determined as the current working vehicle. According to the center point of the area, Determine the current working lane.

(2) After calibration, refer to Figure 6, use the pre-trained Yolov5-2 detection model of the front of the car and the two-dimensional code positioning and recognition algorithm Apriltag to confirm the identity code and driving direction of the collection card, and connect the collection card through the network accordingly Data receiving unit in the cab.

(3) Calculate the initial positioning deviation according to the center coordinate y of the target rectangular area detected and identified by Yolov5-0 and the pre-acquired target position coordinate y ₀ saved in step 1, and send it to the data receiving unit of the truck cab, and pass The receiving unit displays the deviation data and the schematic diagram, and when the initial positioning deviation offset is less than a certain threshold, the initial positioning of the collection truck is completed.

(4) Crop the image of the container or the rectangular area of the frame, please refer to Figure 7 and Figure 8, the cropped image will be recognized using the YoloV5-1 target detection algorithm, which will increase the resolution of the target recognition sample, and then Better recognition of small objects such as box holes, text or guides. Among them, Yolo V5 is a deep learning neural network target recognition algorithm, which mainly learns offline through samples, trains the model, and then recognizes the specified target after collecting images in real time.

(5) If the current operation is to grab boxes, perform single and double box detection, the steps are as follows:

First, divide the detected rectangular area into three sub-areas, and use Yolov5-1 for batch detection and recognition processing to obtain the detection results.

Secondly, the detection results of the middle sub-region are analyzed, box hole analysis: if the number of detected box holes is less than 2, it is considered that no box hole has been detected, otherwise the detected box holes will be sorted according to the X coordinate, if the sorted minimum value is the same as If the minimum value in the X direction is within a certain threshold range, the box hole is considered to be detected. Text analysis: If text is detected in the middle sub-area, and the height and width of its rectangle meet a certain range, it is considered that the text on the upper surface of the container has been detected.

Finally, the integrated fusion information, box hole information and text information are used to determine single and double boxes, as follows:

In this application, the weight parameters of weight ₀ , weight ₁ , and weight ₂ are 0.4, 0.2, and 0.4 respectively. If the total score is greater than 0.6, it is considered a double box, otherwise it is considered a single box.

If the current operation is to place boxes, cut the rectangular area into two sub-areas, use Yolov5-1 to detect the guide plate, and then obtain the detected coordinates of the guide plate.

(6) Carry out misdetection to the detection result of the box hole or the guide plate of the upper and lower sub-regions of the container or vehicle frame, and its method is to calculate the angle between the modulus sum of the vector formed by any two points in the detection result and the horizontal direction, Whether to remove false detection points within a certain range.

(7) Estimate the height of the container-loaded inner container, taking into account the individual height classes 2.4, 2.6 and 2.9 of the container. Therefore, it is necessary to estimate the height of the container truck. First, calculate the closest position line between the center points of the detected two container holes, and then compare the current distance d between the two container holes with the distance d between the two container holes when the height is calibrated. The size of the distance d _k , if dd _k >T ₀ , it is considered as 2.9 level, if dd _k <T ₁ , it is considered as 2.4 level, if it is within the range of T ₀ ~ T ₁ , it is considered as 2.6 level, T ₀ =3, T ₁ =1.

(8) Carry out the fine positioning of the inner collection card guidance through the box hole or the frame guide plate. Please refer to Figure 3. After obtaining the midpoint P of the two detected box holes or guide plates, then use the information of grabbing and placing boxes, lane information, The height information and box shape information obtain the target straight-line distance L, and then calculate the straight-line distance from point P to the target straight line L, and then multiply it by E, then the actual deviation distance of the inner set card can be obtained. And send the distance deviation to the cab of the internal truck to display the deviation distance.

Through the continuous iteration of step 8, the guidance and positioning of the inner set card is completed.

3. The internal collection truck guides and fixes the internal collection truck according to the actual deviation distance, and the quay crane control system adjusts according to the judged single and double container when the spreader of the quay crane reaches the container or the frame. The form of the spreader of the quay crane is used for grabbing or putting boxes.

Referring to Figure 9, the industrial control computer stores the AI algorithm, and its main function is to collect the video image information of the 1-6 cameras, and run the inner set card guidance and positioning and single and double box discrimination algorithm software. What PLC mainly receives from the video processor is the discrimination result of single and double boxes, and judges whether it is single box or double box. The main purpose of the video processor to receive the operation information of the quay crane is to more accurately judge whether there is an internal set card in the quay crane, whether it is necessary to run the guidance operation and to distinguish single and double tanks.

The term "video processor" includes various devices, devices, and machines for processing data, for example, a video processor includes a programmable processor, a computer, multiple processors, or multiple computers, and the like. In addition to hardware, the apparatus may include code for creating an execution environment for the computer program in question, for example, constituting processor firmware, a protocol stack, a database management system, an operating system, or a runtime environment, or one or more thereof. combined code.

The methods and logic flows described in this specification can be performed by one or more programmable processors, wherein the programmable processors execute one or more computer programs to perform operations on surveillance video and generate object detection results these functions.

Typically, the computer also includes or is operably coupled to one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing historical video data and data sets, to receive data from the mass storage device or to transmit data to the mass storage device, or both. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices including, for example: semiconductor memory devices such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically erasable programmable read-only memory) and flash memory devices; magnetic disks, such as internal or removable hard disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

Those skilled in the art can understand that all or part of the processes of the methods in the above embodiments can be implemented by instructing related hardware through computer programs, and the programs can be stored in a computer-readable storage medium. Wherein, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, and the like.

The above is only a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in this application Replacement should be covered within the protection scope of this application.

Claims

A machine vision-based quay crane control method is characterized in that it includes:

Calibrate the target parking position of the truck and estimate the height of the truck loaded with containers;

Acquire and use the first target detection model to identify the target rectangular area of the container or frame in the body image, and calculate the initial positioning deviation distance based on the target rectangular area and the target parking position, wherein the target rectangular area includes The smallest circumscribing rectangle;

Carry out image clipping on the target rectangular area of the container or vehicle frame in the vehicle body image to generate the body sub-image;

Acquiring and using the second target detection model to identify the box hole coordinates and text on the container in the body sub-image or the frame guide plate coordinates on the frame;

Based on whether the target rectangular area of the container in the body sub-image is a fusion area, the box hole coordinates and the text of the middle sub-area in the body sub-image, single and double boxes are judged;

Obtaining the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image based on the box hole coordinates or the frame guide plate coordinates to estimate the height of the container;

calculating a moving deviation distance based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, and guiding the collection truck to the target parking position according to the moving deviation distance; and

When the spreader of the quay crane arrives directly above the container or the vehicle frame, the form of the spreader of the quay crane is adjusted according to the judged single or double container, and the operation of grabbing or putting the container is carried out.
The quay crane control method based on machine vision according to claim 1, wherein the calibration of the target parking position of the collection truck and the estimated height of the collection truck loaded with containers further comprises:

Pre-acquisition of the body image of the target parking position of the collection truck and identifying the body image of the target parking position, so as to obtain the image coordinates or frame of the container and the corresponding box hole of the collection truck at the target parking position and the image coordinates of the corresponding frame guide plate;

Using the box hole image coordinates or the frame guide plate image coordinates to generate a target straight line and estimate a height-related pixel distance factor.
The quay crane control method based on machine vision according to claim 1, wherein obtaining and utilizing the first target detection model to identify the target rectangular area of the container or vehicle frame in the body image further comprises:

Acquiring multiple historical body images from the database and labeling the containers or frames in the multiple historical body images;

Establishing a first neural network Yolov5 and using a plurality of marked historical body images to train the first neural network Yolov5 to obtain a first target detection model; and

Collecting the current body image in real time, and using the first target detection model to identify the target rectangular area of the container or frame in the current body image, so as to judge whether to grab or put the box, wherein, according to the target of the container The size of the rectangular area determines whether the container is 20 feet or not, and the target rectangular areas of two 20-foot containers on the same container are fused to form the fusion area.
The quay crane control method based on machine vision according to claim 3, wherein, performing image clipping on the target rectangular area of the container or vehicle frame in the vehicle body image to generate the vehicle body sub-image further comprises:

Clipping the historical vehicle body image or the current vehicle body image into a historical vehicle body sub-image or a current vehicle body sub-image, wherein,

When the target rectangular area is a container area, the historical body sub-image or the current body sub-image includes a first upper sub-area, a middle sub-area and a first lower sub-area, wherein, for the first middle performing box hole and text detection in the sub-region, performing box hole detection on the first upper sub-region image and the first lower sub-region image; and

When the target rectangular area is a frame area, the historical body sub-image or the current body sub-image includes a second upper sub-area and a second lower sub-area, wherein, for the second upper sub-area and the second upper sub-area The second lower sub-area is used to detect the frame guide plate.
The quay crane control method based on machine vision according to claim 4, characterized in that, acquiring and utilizing the second target detection model to identify the box hole coordinates on the container or the vehicle frame on the vehicle frame in the sub-image of the vehicle body Guide plate coordinates further include:

Marking the box hole or the frame guide plate in the sub-image of the historical vehicle body;

Establishing a second neural network Yolov5 and utilizing the marked historical car body images to train the second neural network Yolov5 to obtain a second target detection model; and

Using the second target detection model to identify box holes and characters or frame guide plates in the current body sub-image, and obtain box hole coordinates or frame guide plate coordinates.
The quay bridge control method based on machine vision according to claim 1, wherein a first camera, a second camera, a third camera, and a fourth camera are installed at the isolation strips on opposite sides of the lane on the quay bridge crossbeam , the fifth camera and the sixth camera, wherein,

Using the first camera to the fourth camera to capture the head image of the truck to confirm the identity of the vehicle and the moving direction of the vehicle; and

Using the fifth camera and the sixth camera to shoot the body image of the truck to calculate the initial positioning deviation distance and the movement deviation distance of the truck.
The quay crane control method based on machine vision according to claim 6, further comprising:

Obtain a plurality of historical vehicle front images from the database and mark the vehicle front and the two-dimensional code in the historical vehicle front images;

Set up the 3rd neural network Yolov5, and utilize the historical car head image of mark to train described 3rd neural network Yolov5 to obtain the 3rd target detection model;

Collecting the current head image in real time, and using the third target detection model to identify the truck head in the current head image and the QR code pasted on the head of the truck, and confirm the identity code and driving direction of the truck; and

The data receiving unit in the cab of the corresponding collection truck is connected through the network according to the collection card identity code.
The machine vision-based shore crane control method according to claim 7, wherein calculating the initial positioning deviation distance based on the target rectangular area and the target parking position further comprises:

Based on the recognized target rectangular area of the container or vehicle frame in the current body image and the pre-acquired target parking position, calculate the initial positioning deviation distance of the truck; and

The initial positioning deviation distance of the collection truck is transmitted to the data receiving unit through the network, and displayed on the LED display to guide the collection truck driver to adjust the location of the collection truck, wherein the initial positioning of the collection truck is calculated by the following formula Positioning deviation distance:

offset＝D(yy 0 )

Among them, y represents the ordinate of the current vehicle area image, y 0 represents the ordinate of the target parking position collected in advance, and D represents the actual pixel distance factor related to the height.
The machine vision-based quay crane control method according to claim 8, characterized in that, based on the height of the container or vehicle frame and the target straight line generated by the box hole or vehicle frame guide plate, the movement deviation distance is calculated, and according to the movement The deviation distance guiding the truck to the target parking position further includes:

Calculating the moving deviation distance of the collection card by the following formula further includes:

Among them, x 0 , y 0 are the midpoints of the two currently detected box holes, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height;

Send the movement deviation distance to the data receiving unit through the network and display the movement deviation distance on the LED display to continue to guide the driver to adjust the position of the truck until the movement deviation distance is less than the threshold and complete Set card positioning guide.
A quay crane control device based on machine vision, characterized in that it includes: a video processor and a control module, the video processor includes a calibration module, an initial positioning module, an image cropping module, an identification module, a single and double box judging module, A height estimation module and a movement deviation distance calculation module, wherein,

The calibration module is used to calibrate the target parking position of the collection truck and the estimated height of the collection truck loaded with containers;

The initial positioning module is configured to obtain and use the first target detection model to identify the target rectangular area of the container or frame in the body image, and calculate the initial positioning deviation distance based on the target rectangular area and the target parking position, Wherein, the target rectangular area includes a minimum circumscribed rectangle;

The image cropping module is used to crop the target rectangular area of the container or vehicle frame detected in the vehicle body image to generate a body sub-image;

The recognition module is used to obtain and use the second target detection model to recognize the box hole coordinates and text on the container in the body sub-image or the frame guide plate coordinates on the frame;

The single and double container judging module is used to determine single and double containers based on whether the target rectangular area of the container in the body sub-image is a fusion area, the box hole coordinates and the text of the middle sub-region in the body sub-image ;

The height estimation module is used to obtain the distance and position of the box hole or the frame guide plate of the lower sub-region image in the body sub-image based on the box hole coordinates or the frame guide plate coordinates to estimate the height of the container high

The movement deviation distance calculation module is used to calculate the movement deviation distance based on the height of the container or the vehicle frame and the target straight line generated by the box hole or the frame guide plate, so as to guide the collection truck to the said target parking location; and

The control module is used to adjust the shape of the spreader of the quay crane according to the judged single or double container when the spreader of the quay crane arrives directly above the container or the frame, so as to grab or put the container Operation.
The quay crane control device based on machine vision according to claim 10, characterized in that it includes a camera for pre-collecting the vehicle body image of the target parking position of the truck, and intermittently collecting the vehicle body image;

The calibration module also includes: a target position calibration sub-module and a height estimation calibration sub-module, wherein,

The target position calibration sub-module is used to identify the body image of the target parking position, and obtain the image coordinates of the container and the corresponding box hole or the frame and the corresponding frame of the truck at the target parking position Guide plate image coordinates to generate the target line; and

The height estimation and calibration sub-module estimates a height-related pixel distance factor by using the box hole image coordinates or the frame guide plate image coordinates.
The machine vision-based shore crane control device according to claim 10, wherein the initial positioning module includes a labeling submodule, a first target detection model, and a target rectangle generation submodule, wherein,

The first labeling submodule is used to acquire multiple historical vehicle body images from the database and label the containers or frames in the multiple historical vehicle body images;

The first target detection model is used to establish a first neural network Yolov5 and use a plurality of marked historical body images to train the first neural network Yolov5 to obtain a first target detection model; and

The target rectangle generation sub-module is used to collect the current vehicle body image in real time, and use the first target detection model to identify the target rectangular area of the container or frame in the current vehicle body image, so as to grab or put the box Judging, wherein, according to the size of the target rectangular area of the container, it is determined whether the container is 20 feet or not, and the target rectangular areas of two 20-foot containers on the same collection card are fused to form the fusion area .
The machine vision-based quay crane control device according to claim 12, wherein the image cropping module is used to clip the historical vehicle body image or the current vehicle body image into a historical vehicle body sub-image or a current vehicle body sub-image ,in,

When the target rectangular area is a container area, the historical body sub-image or the current body sub-image includes a first upper sub-area, a middle sub-area and a first lower sub-area, wherein, for the first middle performing box hole and text detection in the sub-region, performing box hole detection on the first upper sub-region image and the first lower sub-region image; and

When the target rectangular area is a frame area, the historical body sub-image or the current body sub-image includes a second upper sub-area and a second lower sub-area, wherein, for the second upper sub-area and the second upper sub-area The second lower sub-area is used to detect the frame guide plate.
The shore bridge control device based on machine vision according to claim 13, wherein the recognition module includes a second labeling submodule, a second target detection model and a box hole and frame guide plate recognition submodule, wherein,

The second labeling submodule is used to label the box hole or the frame guide plate in the historical vehicle body image;

The second target detection model is used to establish a second neural network Yolov5 and use the marked historical car body images to train the second neural network Yolov5 to obtain a second target detection model; and

The box hole and frame guide identification sub-module is used to use the second target detection model to identify box holes or frame guides in the current body sub-image, and obtain box hole coordinates and frame guide coordinates.
The machine vision-based quay bridge control device according to claim 10, wherein the cameras include a first camera, a second camera, a third camera, a fourth camera, a fifth camera and a sixth camera, installed on At the isolation strips on opposite sides of the lane on the cross beam of the quay bridge, where,

The first camera to the fourth camera are used to capture the head image of the truck to confirm the identity of the vehicle and the moving direction of the vehicle; and

The fifth camera and the sixth camera are used to take images of the body of the collection truck to calculate the initial positioning deviation distance and the movement deviation distance of the collection truck.
The shore crane control device based on machine vision according to claim 15, further comprising a data receiving unit, and the identification module further comprising a third labeling sub-module, a third target detection model and a vehicle head and an identification sub-module module, where

The third labeling submodule is used to acquire multiple historical vehicle front images from the database and label the vehicle front and the two-dimensional code in the historical vehicle front images;

The third target detection model is used to set up a third neural network Yolov5, and utilizes the marked history front image to train the third neural network Yolov5 to obtain a third target detection model;

The vehicle head and identity confirmation sub-module is used to collect the current vehicle head image in real time, and use the third target detection model to identify the collection truck head in the current vehicle head image and the QR code pasted on the collection truck head and confirm the identity code of the collection truck and direction of travel; and

The data receiving unit is configured to be located in the truck cab and connect to the video processor through a network according to the ID code of the truck.
The machine vision-based quay crane control device according to claim 16, further comprising an LED display screen,

The initial positioning module is used to calculate the initial positioning deviation distance of the truck based on the recognized target rectangular area of the container or frame in the current body image and the pre-acquired target parking position, wherein, the following formula is used to calculate the The initial positioning deviation distance of the set card:

offset＝D(yy 0 )

Among them, y represents the ordinate of the current vehicle area image, y 0 represents the ordinate of the target parking position collected in advance, and D represents the actual pixel distance factor related to height;

The data receiving unit transmits the initial positioning deviation distance of the collection card to the data receiving unit through the network; and

The LED display screen is located in the cab of the collection truck and communicated with the data receiving unit, and is used to display the deviation distance of the initial positioning, so as to guide the truck driver to adjust the position of the collection truck.
The quay crane control device based on machine vision according to claim 17, wherein the movement deviation distance calculation module is used to calculate the movement deviation distance of the truck by the following formula further comprising:

Among them, x 0 , y 0 are the midpoints of the two currently detected box holes, A, B, and C represent the linear equation parameters of the two box holes detected in advance, and E represents the actual pixel distance factor related to the height;

The data receiving unit receives the movement deviation distance through the network;

The LED display is used to display the movement deviation distance to continue to guide the driver to adjust the position of the truck until the movement deviation distance is less than a threshold to complete the positioning guidance of the collection truck.