CN116563803B - Visual-based shipping cement loading and unloading monitoring method and system - Google Patents
Visual-based shipping cement loading and unloading monitoring method and system Download PDFInfo
- Publication number
- CN116563803B CN116563803B CN202310833246.6A CN202310833246A CN116563803B CN 116563803 B CN116563803 B CN 116563803B CN 202310833246 A CN202310833246 A CN 202310833246A CN 116563803 B CN116563803 B CN 116563803B
- Authority
- CN
- China
- Prior art keywords
- unloading
- loading
- cement loading
- cement
- event information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000004568 cement Substances 0.000 title claims abstract description 325
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012544 monitoring process Methods 0.000 title claims abstract description 65
- 230000000007 visual effect Effects 0.000 title claims description 16
- 238000001514 detection method Methods 0.000 claims abstract description 108
- 238000002372 labelling Methods 0.000 claims abstract description 35
- 238000012549 training Methods 0.000 claims abstract description 23
- 238000009432 framing Methods 0.000 claims abstract description 17
- 230000003321 amplification Effects 0.000 claims abstract description 13
- 238000003199 nucleic acid amplification method Methods 0.000 claims abstract description 13
- 238000004364 calculation method Methods 0.000 claims description 16
- 230000004927 fusion Effects 0.000 claims description 16
- 238000012216 screening Methods 0.000 claims description 14
- 238000012806 monitoring device Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 6
- 230000001131 transforming effect Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 30
- 230000006399 behavior Effects 0.000 abstract description 9
- 238000012545 processing Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000004566 building material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009440 infrastructure construction Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/54—Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
- G06Q10/0832—Special goods or special handling procedures, e.g. handling of hazardous or fragile goods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
- G06Q10/0833—Tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- Economics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Development Economics (AREA)
- Health & Medical Sciences (AREA)
- General Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Computational Linguistics (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the field of cement transportation, and discloses a vision-based shipping cement loading and unloading monitoring method and system, which are used for monitoring and improving loading and unloading efficiency and quality in real time in the transportation process of ships and preventing the occurrence of stealing and unloading behaviors in the transportation process. The method comprises the following steps: collecting information of cement loading and unloading pipelines; extracting video data, carrying out image framing and information labeling to obtain a labeling data set; carrying out data amplification to obtain an extended data set and carrying out model training on a YOLOV5 target detection model to obtain a ship cement loading and unloading object detection model; inputting target image data to be detected into a ship cement loading and unloading detection model to carry out cement loading and unloading detection to obtain first cement loading and unloading event information; and carrying out secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and acquiring the ship ID input by a user to inquire the second cement loading and unloading event information.
Description
Technical Field
The invention relates to the field of cement transportation, in particular to a vision-based shipping cement loading and unloading monitoring method and system.
Background
Cement is a common building material and plays a vital role in construction engineering. With the rapid development of infrastructure construction, the scale of the cement industry is also expanding, which puts higher demands on the development of the cement transportation industry. Because cement has the advantages of easy dispersibility and easy volatility, ensuring the quality and the safety of the cement in the transportation and loading and unloading processes is an important task of cement enterprises. Among them, shipping cement plays a critical role in cement logistics transportation. As it allows for rapid transport of cement between large volumes of material. The traditional cement loading and unloading mode is mostly dependent on manual monitoring, so that the efficiency is low, and the problems of potential safety hazard, goods stealing and changing and the like exist, so that the transportation cost is increased. Therefore, the advanced monitoring technology is adopted to monitor and manage the loading and unloading processes of the shipping cement, and the cement loading and unloading device is a necessary measure for improving the transportation efficiency and the safety, reducing the operation cost and bringing more benefits to cement enterprises.
At present, a pair of slide tube monitoring cameras are arranged at the upper ends of two sides of a slide tube to monitor whether a feed opening is positioned at the middle position of a cabin of a ship, and two upright post ship monitoring cameras are arranged at the edge of a code head to monitor the position and the state of the ship, so that the monitoring and automatic control of the cement material loading and unloading of the ship are realized, the loading and unloading efficiency can be provided, and the safety risk of manual operation is reduced. However, this method is effective only for ships that enter the port to the shore, and once the ship is detached from the port after loading and unloading is completed, it cannot be effectively monitored during transportation. The other method is that a camera is installed on a specific site position to acquire an event image sequence, and the acquired event image sequence is respectively used for target detection and optical flow processing to obtain a corresponding target position image and an optical flow information image; then, fusing the obtained target position image and the optical flow information image to obtain a fused image; and finally, carrying out loading and unloading event detection on the fused image. The method has fixed installation site and can not detect in real time at any place and any time.
Disclosure of Invention
The invention provides a ship cement loading and unloading monitoring method and system based on vision, which are used for monitoring and improving loading and unloading efficiency and quality in real time in the transportation process of ships and preventing the occurrence of stealing and unloading behaviors in the transportation process.
The first aspect of the invention provides a vision-based shipping cement loading and unloading monitoring method, which comprises the following steps:
respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, collecting cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
acquiring cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and carrying out image framing and information labeling on the video data to obtain a labeling data set;
performing data amplification on the marked data set to obtain an extended data set, and performing model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
acquiring target image data to be detected, inputting the target image data into the ship cement loading and unloading detection model to carry out cement loading and unloading detection, and obtaining first cement loading and unloading event information;
Performing secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform;
and acquiring the ship ID input by the user, inquiring the second cement loading and unloading event information from the server platform according to the ship ID, and carrying out visual management on the second cement loading and unloading event information.
With reference to the first aspect, in a first implementation manner of the first aspect of the present invention, the obtaining cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and performing image framing and information labeling on the video data to obtain a labeled data set includes:
acquiring cement loading and unloading time, and extracting video data from the terminal hard disk according to the cement loading and unloading time;
carrying out image framing on the video data through a preset ffmpeg tool to obtain a plurality of video frames;
performing image classification and image screening on the plurality of video frames to obtain loading event images and unloading event images;
and labeling the loading event image and the unloading event image by a preset labelme image labeling tool to obtain a labeling data set.
With reference to the first aspect, in a second implementation manner of the first aspect of the present invention, the performing data amplification on the labeling data set to obtain an extended data set, and performing model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model, where the method includes:
turning, rotating, color transforming and scaling the marked data set to obtain an expanded data set;
and (3) a preset YOLOV5 target detection model is paired with the extended data set, wherein the YOLOV5 target detection model comprises: backbone, neck and head networks;
predicting the extended data set through the YOLOV5 target detection model to obtain a prediction result;
and performing parameter tuning on the YOLOV5 target detection model based on the prediction result to obtain a ship cement loading and unloading detection model.
With reference to the first aspect, in a third implementation manner of the first aspect of the present invention, the acquiring target image data to be detected, and inputting the target image data into the ship cement loading and unloading detection model to perform cement loading and unloading detection, to obtain first cement loading and unloading event information, includes:
Acquiring target image data to be detected, and inputting the target image data into the ship cement loading and unloading object detection model;
performing feature extraction through a backbone network in the ship cement loading and unloading object detection model to obtain feature image data;
inputting the characteristic image data into a neck network in the ship cement loading and unloading object detection model to perform characteristic fusion to obtain characteristic fusion data;
and inputting the characteristic fusion data into a head network in the ship cement loading and unloading detection model to detect cement loading and unloading and output first cement loading and unloading event information.
With reference to the first aspect, in a fourth implementation manner of the first aspect of the present invention, the performing a secondary determination on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform, includes:
when the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
when the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
And uploading the second cement loading and unloading event information to a preset server platform.
With reference to the first aspect, in a fifth implementation manner of the first aspect of the present invention, when the first cement loading and unloading event information is a loading event, performing a secondary determination on the first cement loading and unloading event information based on a preset loading event determination rule to obtain second cement loading and unloading event information, where the method includes:
when the first cement loading and unloading event information is a loading event, four coordinate point positions in the feeding pipeline image are obtained；
Acquiring a first probability value of a feeding pipeline in an image and according to the pre-predictionA first probability threshold value is setPerforming preliminary screening on the first probability value;
by the position of the coordinate pointCoordinate point of furthest charging port->And the nearest fill port coordinate point->Performing distance calculation to obtain a target distance value;
the calculation formula of the target distance value is as follows:
；
；
；
wherein ,representing +.>And furthest fill port coordinate point->Vertical distance between (I)>Representing +.>And most recent chargesMouth coordinate pointVertical distance between (I)>Representing a target distance value;
For the target distance valueDistance from the predetermined distance threshold->Comparing when->≤/>When a loading event is determined to exist, and the loading event is output as second cement loading and unloading event information.
With reference to the first aspect, in a sixth implementation manner of the first aspect of the present invention, when the first cement loading and unloading event information is a loading and unloading event, performing a secondary determination on the first cement loading and unloading event information based on a preset loading and unloading event determination rule, to obtain second cement loading and unloading event information, where the method includes:
when the first cement loading and unloading event information is an unloading event, four coordinate point positions in an unloading pipeline image are obtained；
Acquiring a second probability value of the discharge pipeline in the image, and according to a preset second probability threshold valuePerforming preliminary screening on the second probability value;
setting a rectangular frame and four rectangular frame coordinate points in the unloading pipeline imageCalculating the intersection ratio of the rectangular frames according to the coordinate points of the four rectangular frames to obtain an IOU value;
the calculation formula of the IOU value is as follows:
；
；
；
；
；
；
；
；
；
wherein ,representing the width of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Indicating unloading The height of the rectangle formed by the material pipeline at four coordinate points of the image, +.>Representing the width of a rectangle formed by setting four rectangular frame coordinate points, < >>Representing the height of a rectangle formed by setting four rectangular frame coordinate points, +.>Minimum abscissa value representing rectangle formed by intersection of two rectangular frames, ">Representing the smallest ordinate value of the rectangle formed by the intersection of two rectangular frames, < >>Maximum abscissa value representing rectangle formed by intersection of two rectangular frames, +.>Representing the maximum ordinate value of a rectangle formed by the intersection of two rectangular frames,/for>Represents the area of a rectangle formed by the intersection of two rectangular frames, < > and>representing the area of the two rectangular boxes minus the area of the intersecting rectangular box;
for the IOU value and a preset IOU threshold valueComparing, when IOU is greater than or equal to->And when the cement loading and unloading event information is determined to exist, and the loading and unloading event information is output as second cement loading and unloading event information.
A second aspect of the present invention provides a vision-based shipping cement loading and unloading monitoring system comprising:
the acquisition module is used for respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, acquiring cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
The marking module is used for acquiring cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and carrying out image framing and information marking on the video data to obtain a marked data set;
the training module is used for carrying out data amplification on the labeling data set to obtain an extended data set, and carrying out model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
the detection module is used for acquiring target image data to be detected, inputting the target image data into the ship cement loading and unloading detection model for cement loading and unloading detection, and obtaining first cement loading and unloading event information;
the judging module is used for carrying out secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform;
and the query module is used for acquiring the ship ID input by the user, querying the second cement loading and unloading event information from the server platform according to the ship ID, and carrying out visual management on the second cement loading and unloading event information.
A third aspect of the present invention provides a vision-based shipping cement loading and unloading monitoring device comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the vision-based shipping cement loading and unloading monitoring device to perform the vision-based shipping cement loading and unloading monitoring method described above.
A fourth aspect of the invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the vision-based shipping cement loading and unloading monitoring method described above.
In the technical scheme provided by the invention, the information of the cement loading and unloading pipelines is collected; extracting video data, carrying out image framing and information labeling to obtain a labeling data set; carrying out data amplification to obtain an extended data set and carrying out model training on a YOLOV5 target detection model to obtain a ship cement loading and unloading object detection model; inputting target image data to be detected into a ship cement loading and unloading detection model to carry out cement loading and unloading detection to obtain first cement loading and unloading event information; the method comprises the steps of carrying out secondary judgment on first cement loading and unloading event information to obtain second cement loading and unloading event information, and obtaining the ship ID input by a user to inquire the second cement loading and unloading event information. Meanwhile, the loading and unloading efficiency and quality can be improved, and the occurrence of stealing and unloading behaviors in the transportation process can be prevented. The real-time monitoring of the cement loading and unloading process is realized by utilizing a visual sensor and a computer visual technology. The loading and unloading efficiency and the quality are improved, and meanwhile, the stealing and unloading behavior in the transportation process can be prevented. The invention can detect in real time at any time and any place, is not limited by sites, and can effectively monitor ships entering and exiting ports.
Drawings
FIG. 1 is a schematic view of one embodiment of a vision-based shipping cement loading and unloading monitoring method in accordance with an embodiment of the present invention;
FIG. 2 is a flow chart of model training in an embodiment of the invention;
FIG. 3 is a flow chart of cement loading and unloading detection in an embodiment of the invention;
FIG. 4 is a flow chart of a secondary decision in an embodiment of the invention;
FIG. 5 is a schematic view of one embodiment of a vision-based shipping cement loading and unloading monitoring system in accordance with an embodiment of the invention;
FIG. 6 is a schematic view of one embodiment of a vision-based shipping cement loading and unloading monitoring device in accordance with an embodiment of the present invention;
FIG. 7 is a schematic view of a camera mounting a monitoring charging port in an embodiment of the invention;
FIG. 8 is a schematic view of a camera mounting for monitoring a discharge port in an embodiment of the present invention;
FIG. 9 is a block diagram of a training architecture for a YOLOV5 object detection model in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a vision-based shipping cement loading and unloading monitoring method and system, which are used for monitoring and improving the loading and unloading efficiency and quality in real time in the transportation process of ships and preventing the occurrence of stealing and unloading behaviors in the transportation process. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present invention will be described below with reference to fig. 1, and one embodiment of a vision-based shipping cement loading and unloading monitoring method according to the embodiment of the present invention includes:
s101, respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, collecting cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
it will be appreciated that the subject of the present invention may be a vision-based shipping cement loading and unloading monitoring system, or may be a terminal or server, and is not limited in this regard. The embodiment of the invention is described by taking a server as an execution main body as an example.
It should be noted that, this embodiment adopts technologies such as high definition surveillance camera head, GPS, wireless communication module, realizes monitoring the loading and unloading process of shipping cement, realizes data acquisition, transmission, processing and show through the internet of things, has realized the real-time supervision of shipping cement loading and unloading. The embodiment consists of a monitoring module, a data processing module and a data display module. The monitoring module comprises two high-definition monitoring cameras, a GPS and a wireless communication module and is used for collecting images, position and state information of the shipping cement loading and unloading process; the data processing module comprises a terminal host, a YOLOV5 target detection model, a secondary judgment rule and a network transmission module, and is used for processing and analyzing the acquired data and transmitting the result to the data display module through a network; the data display module comprises a service platform and a user interface and is used for displaying real-time data, historical data, ship information and the like.
Specifically, the server performs equipment installation, installs a terminal host in the cockpit and is connected with a power supply in the cockpit; two high-definition monitoring cameras are arranged on the shell of the ship's bow cockpit, and are respectively aligned with the charging port and the discharging port of the ship cement and connected with a terminal host; the GPS wire and the signal receiving wire are arranged on the surface of the cockpit shell and are connected with the terminal host; the SIM card is inserted into the terminal host and is connected with the service platform to bind related information, so that information transmission between the SIM card and the service platform is facilitated through the network.
S102, acquiring cement loading and unloading time, extracting video data from a terminal hard disk according to the cement loading and unloading time, and carrying out image framing and information labeling on the video data to obtain a labeled data set;
specifically, the server acquires cement loading and unloading time, and video data are extracted from a terminal hard disk according to the cement loading and unloading time; the server carries out image framing on the video data through a preset ffmpeg tool to obtain a plurality of video frames; the server performs image classification and image screening on a plurality of video frames to obtain loading event images and unloading event images; the server marks the loading event image and the unloading event image through a preset labelme image marking tool to obtain a marked data set.
S103, carrying out data amplification on the marked data set to obtain an extended data set, and carrying out model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
it should be noted that, the server performs overturn, rotation, color transformation and scaling on the labeling data set to obtain an extended data set; the server pairs the expanded data set with a preset YOLOV5 target detection model, wherein the YOLOV5 target detection model comprises: backbone, neck and head networks; the server predicts the extended data set through a YOLOV5 target detection model to obtain a prediction result; and the server performs parameter tuning on the YOLOV5 target detection model based on the prediction result to obtain a ship cement loading and unloading object detection model.
S104, acquiring target image data to be detected, and inputting the target image data into a ship cement loading and unloading detection model to carry out cement loading and unloading detection to obtain first cement loading and unloading event information;
specifically, the server acquires target image data to be detected, and inputs the target image data into a ship cement loading and unloading object detection model; the server performs characteristic extraction through a backbone network in the ship cement loading and unloading object detection model to obtain characteristic image data; the server inputs the characteristic image data into a neck network in the ship cement loading and unloading object detection model to perform characteristic fusion, so as to obtain characteristic fusion data; the server inputs the characteristic fusion data into a head network in a ship cement loading and unloading detection model to carry out cement loading and unloading detection, and outputs first cement loading and unloading event information.
S105, carrying out secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform;
specifically, when the first cement loading and unloading event information is a loading event, the server carries out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information; when the first cement loading and unloading event information is a loading event, the server carries out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information; and uploading the second cement loading and unloading event information to a preset server platform by the server.
S106, acquiring the ship ID input by the user, inquiring the second cement loading and unloading event information from the server platform according to the ship ID, and performing visual management on the second cement loading and unloading event information.
Specifically, the user queries. The user can input the ship ID to be inquired through the website or the application program, and the service platform can inquire corresponding ship information from the database according to the input ID information, wherein the ship information comprises a loading and unloading picture, a live real-time picture, a ship navigation track, a history picture and the like of the ship. The query result is presented to the user in the form of images or characters, and the user can view the relevant information of the ship at any time and any place. Meanwhile, the service platform also provides functions of ship state monitoring, alarming and the like, and can inform a user in time and process when abnormal conditions occur to the ship. In order to ensure the safety of the user information, the service platform encrypts the user information and adopts a secure transmission protocol to ensure that the privacy of the user is not revealed.
According to the embodiment, two high-definition monitoring cameras are arranged on the shell of the ship's bow cockpit, and are respectively aligned with the loading port and the unloading port of shipping cement and used for collecting image data in the cement loading and unloading process; and storing the cement loading and unloading pipeline information acquired by the camera in a terminal hard disk, extracting the hard disk video data, and converting the extracted hard disk video data into an image format. The images are screened, cleaned and analyzed, leaving images with loading and unloading events. Calibrating the image by using a professional image marking tool, framing a pipeline containing cement loading and unloading materials, and obtaining calibration data; inputting calibration data into a YOLOV5 model, and training the model YOLOV 5; the trained model is deployed into camera host terminal equipment, the terminal equipment acquires camera head end data in real time, and then the camera head end data are input into the model to detect cement loading and unloading events and upload event result information to a server platform after secondary judgment rule calculation is carried out; and the user opens a platform website, searches for the ship ID, and displays the cement loading and unloading events of the ship and corresponding images and other information according to the ship ID. The cement loading and unloading events, loading and unloading time, ship navigation tracks, real-time pictures of the ship, detailed information of the ship and the like of the ship can be checked by a user at any time and any place, the ship can be better monitored, the transportation efficiency and the safety of the cement are improved, and the occurrence of a goods stealing and changing event in the transportation process is prevented.
In the embodiment of the invention, the information of the cement loading and unloading pipelines is collected; extracting video data, carrying out image framing and information labeling to obtain a labeling data set; carrying out data amplification to obtain an extended data set and carrying out model training on a YOLOV5 target detection model to obtain a ship cement loading and unloading object detection model; inputting target image data to be detected into a ship cement loading and unloading detection model to carry out cement loading and unloading detection to obtain first cement loading and unloading event information; the method comprises the steps of carrying out secondary judgment on first cement loading and unloading event information to obtain second cement loading and unloading event information, and obtaining the ship ID input by a user to inquire the second cement loading and unloading event information. Meanwhile, the loading and unloading efficiency and quality can be improved, and the occurrence of stealing and unloading behaviors in the transportation process can be prevented. The real-time monitoring of the cement loading and unloading process is realized by utilizing a visual sensor and a computer visual technology. The loading and unloading efficiency and the quality are improved, and meanwhile, the stealing and unloading behavior in the transportation process can be prevented. The invention can detect in real time at any time and any place, is not limited by sites, and can effectively monitor ships entering and exiting ports.
In a specific embodiment, the process of executing step S102 may specifically include the following steps:
(1) Acquiring cement loading and unloading time, and extracting video data from a terminal hard disk according to the cement loading and unloading time;
(2) Carrying out image framing on the video data through a preset ffmpeg tool to obtain a plurality of video frames;
(3) Performing image classification and image screening on a plurality of video frames to obtain loading event images and unloading event images;
(4) Labeling the loading event image and the unloading event image by a preset labelme image labeling tool to obtain a labeling data set.
Specifically, the server collects cement loading and unloading labeling data, and video data collected by the monitoring camera is extracted from a hard disk of the terminal host according to cement loading and unloading time. Firstly, extracting video data by using a ffmpeg tool, and decomposing the video data into image data frame by frame; then, processing the obtained image data, wherein the processed image data comprises an event-free image, a loading event image and a unloading event image; and finally, manually labeling the image by using a labelme image labeling tool, and labeling the position of the event target and the category of the event target by using a rectangular frame containing the loading and unloading events in the image. The invention aims to monitor loading and unloading of shipping cement, and therefore, a cement loading pipeline and a cement unloading pipeline need to be marked, as shown in fig. 7 and 8, wherein fig. 7 is a schematic view of a camera of a monitoring loading port, and fig. 8 is a schematic view of a camera of a monitoring unloading port.
In a specific embodiment, as shown in fig. 2, the process of performing step S103 may specifically include the following steps:
s201, turning over, rotating, performing color transformation and scaling on the marked data set to obtain an expanded data set;
s202, a data set is expanded to a preset YOLOV5 target detection model, wherein the YOLOV5 target detection model comprises: backbone, neck and head networks;
s203, predicting the extended data set through a YOLOV5 target detection model to obtain a prediction result;
and S204, performing parameter tuning on the YOLOV5 target detection model based on the prediction result to obtain a ship cement loading and unloading object detection model.
Specifically, the server performs model training based on the annotation data. In this embodiment, the images need to be processed using a machine learning method in order to achieve detection of the shipping cement loading and unloading. Before model training, the image data is required to be subjected to data amplification so as to solve the problems of small data quantity and strong repeatability, thereby improving the robustness and accuracy of the model. The methods used include flipping, rotation, color transformation, scaling, etc. After image data expansion, training was performed using the YOLOV5 object detection model. YOLOv5 is a target detection algorithm, and the core idea is to consider a target detection task as a regression problem, and finally obtain information such as the position, the category and the like of a target by extracting multi-channel characteristics of each pixel point in an image and mapping the multi-channel characteristics onto a target detection result. The YOLOV5 target detection model comprises three parts, namely a backbone network (for feature extraction), a neck network (for feature fusion) and a head network (for detection result output), and particularly as shown in fig. 9, fig. 9 is a training structure block diagram of the YOLOV5 target detection model. According to the embodiment, before model training, data expansion processing is performed on the calibration image data, and on the premise that target events are not influenced, the diversity of the image data is increased, so that the characteristics of input image data are enriched, and the robustness and generalization capability of the model are better improved. Therefore, the actual requirements can be better met, and the accuracy and the reliability of cement loading and unloading event detection are improved.
In a specific embodiment, as shown in fig. 3, the process of executing step S104 may specifically include the following steps:
s301, acquiring target image data to be detected, and inputting the target image data into a ship cement loading and unloading object detection model;
s302, carrying out feature extraction through a backbone network in a ship cement loading and unloading object detection model to obtain feature image data;
s303, inputting the characteristic image data into a neck network in a ship cement loading and unloading object detection model to perform characteristic fusion to obtain characteristic fusion data;
s304, inputting the characteristic fusion data into a head network in a ship cement loading and unloading detection model to detect cement loading and unloading, and outputting first cement loading and unloading event information.
Specifically, the server acquires target image data to be detected, and inputs the target image data into a ship cement loading and unloading object detection model; the server performs characteristic extraction through a backbone network in the ship cement loading and unloading object detection model to obtain characteristic image data; the server inputs the characteristic image data into a neck network in the ship cement loading and unloading object detection model to perform characteristic fusion, so as to obtain characteristic fusion data; the server inputs the characteristic fusion data into a head network in a ship cement loading and unloading detection model to carry out cement loading and unloading detection, and outputs first cement loading and unloading event information.
In a specific embodiment, as shown in fig. 4, the process of performing step S105 may specifically include the following steps:
s401, when the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
s402, when the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
s403, uploading the second cement loading and unloading event information to a preset server platform.
Specifically, the server ships the cement loading and unloading event detection. First, a trained YOLOV5 model is deployed into an end host. And secondly, the terminal host acquires image data of the camera end, inputs the image data into a deployed model, and calculates to obtain preliminary first cement loading and unloading event information, including coordinate position and category information of the target. And secondly, carrying out secondary judgment according to the coordinate position, the category and the probability of the target, wherein a secondary judgment rule is obtained by adopting a method of combining priori knowledge and a rule of thumb through multiple experimental debugging, and the misjudgment rate of the model can be effectively reduced.
In a specific embodiment, the process of executing step S401 may specifically include the following steps:
(1) When the first cement loading and unloading event information is a loading event, four coordinate point positions in the feeding pipeline image are acquired；
(2) Acquiring a first probability value of a feeding pipeline in an image, and according to a preset first probability threshold valuePerforming primary screening on the first probability value;
(3) By position of coordinate pointsCoordinate point of furthest charging port->And the nearest fill port coordinate point->Performing distance calculation to obtain a target distance value;
the calculation formula of the target distance value is as follows:
；
；
；
wherein ,representing +.>And furthest fill port coordinate point->Vertical distance between (I)>Representing +.>And the nearest fill port coordinate pointVertical distance between (I)>Representing a target distance value;
(4) For the target distance valueDistance from the predetermined distance threshold->Comparing when->≤/>When the cement loading event is determined to exist, the loading event is output as second cement loading event information.
Specifically, when the event detected initially is a shipment event, the event willObtaining four coordinate point positions of the feeding pipeline in the image from left to right and from top to bottom The corresponding probabilities; setting a first probability threshold +.>Preliminary screening out below the first probability threshold based on the probability of the event +.>Event of (2); then, the lowest two coordinate points of the event in the image are +.>Coordinate point of farthest charging port of ship in imageAnd the nearest fill port coordinate point->And (3) performing distance calculation, wherein an average distance calculation formula is as follows:
；
；
；
wherein ,representing +.>And furthest fill port coordinate point->Vertical distance between (I)>Representing +.>And the nearest fill port coordinate pointVertical distance between (I)>Representing a target distance value;
finally, setting a distance thresholdWhen->≤/>And when the cement loading event information is stored, indicating that the loading event exists, and outputting the loading event as second cement loading event information.
In a specific embodiment, the process of executing step S402 may specifically include the following steps:
(1) When the first cement loading and unloading event information is an unloading event, four coordinate point positions in the unloading pipeline image are obtained；
(2) Acquiring a second probability value of the discharge pipeline in the image, and according to a preset second probability threshold valuePerforming preliminary screening on the second probability value;
(3) Setting a rectangular frame and four rectangular frame coordinate points in the unloading pipeline image Calculating the intersection ratio of the rectangular frames according to the coordinate points of the four rectangular frames to obtain an IOU value; />
The calculation formula of the IOU value is as follows:
；
；
；
；
；
；
；
；
；
wherein ,representing the width of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Representing the formation of discharge pipes at four coordinate points of an imageRectangular height,/->Representing the width of a rectangle formed by setting four rectangular frame coordinate points, < >>Representing the height of a rectangle formed by setting four rectangular frame coordinate points, +.>Minimum abscissa value representing rectangle formed by intersection of two rectangular frames, ">Representing the smallest ordinate value of the rectangle formed by the intersection of two rectangular frames, < >>Maximum abscissa value representing rectangle formed by intersection of two rectangular frames, +.>Representing the maximum ordinate value of a rectangle formed by the intersection of two rectangular frames,/for>Represents the area of a rectangle formed by the intersection of two rectangular frames, < > and>representing the area of the two rectangular boxes minus the area of the intersecting rectangular box;
(4) For IOU value and preset IOU threshold valueComparing, when IOU is greater than or equal to->And when the cement loading and unloading event information is determined to exist, and the loading and unloading event information is output as second cement loading and unloading event information.
In particular, when the event detected initially is a discharge event, the method is as follows The positions of four coordinate points are also obtained from left to right and from top to bottomThe corresponding probabilities; setting a second probability threshold +.>Preliminary screening out events below a second probability threshold value according to the probability of the event>Event of (2); then, a rectangular frame is defined in the camera image picture of the discharge opening, and four coordinate points are set>And calculating the intersection ratio between the preliminarily detected rectangular frame formed by the four coordinate points and the set rectangular frame, namely IOU (IntersetionOver Union) value, wherein the calculation formula is as follows:
；
；
；
；
；
；/>
；
；
；
wherein ,representing the width of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Representing the height of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Representing the width of a rectangle formed by setting four rectangular frame coordinate points, < >>Representing the height of a rectangle formed by setting four rectangular frame coordinate points, +.>Minimum abscissa value representing rectangle formed by intersection of two rectangular frames, ">Representing the smallest ordinate value of the rectangle formed by the intersection of two rectangular frames, < >>Maximum abscissa value representing rectangle formed by intersection of two rectangular frames, +.>Representing the maximum ordinate value of a rectangle formed by the intersection of two rectangular frames,/for >Represents the area of a rectangle formed by the intersection of two rectangular frames, < > and>representing the area of the two rectangular boxes minus the area of the intersecting rectangular box;
finally, setting the IOU threshold valueWhen IOU is greater than or equal to%>When a discharge event is indicated.
After the cement loading and unloading event information is acquired, the terminal host machine can send the related information to the server platform through a network for storage and display.
The method for monitoring the shipment cement based on vision in the embodiment of the present invention is described above, and the system for monitoring the shipment cement based on vision in the embodiment of the present invention is described below, referring to fig. 5, one embodiment of the system for monitoring the shipment cement based on vision in the embodiment of the present invention includes:
the acquisition module 501 is used for respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, acquiring cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
the marking module 502 is configured to obtain cement loading and unloading time, extract video data from the terminal hard disk according to the cement loading and unloading time, and perform image framing and information marking on the video data to obtain a marked data set;
The training module 503 is configured to perform data amplification on the labeling data set to obtain an extended data set, and perform model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
the detection module 504 is configured to obtain target image data to be detected, and input the target image data into the ship cement loading and unloading detection model to perform cement loading and unloading detection, so as to obtain first cement loading and unloading event information;
the judging module 505 is configured to perform a secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and upload the second cement loading and unloading event information to a preset server platform;
and the query module 506 is configured to obtain a ship ID input by a user, query the server platform for the second cement loading and unloading event information according to the ship ID, and perform visual management on the second cement loading and unloading event information.
Collecting the information of the cement loading and unloading pipelines through the cooperative cooperation of the components; extracting video data, carrying out image framing and information labeling to obtain a labeling data set; carrying out data amplification to obtain an extended data set and carrying out model training on a YOLOV5 target detection model to obtain a ship cement loading and unloading object detection model; inputting target image data to be detected into a ship cement loading and unloading detection model to carry out cement loading and unloading detection to obtain first cement loading and unloading event information; the method comprises the steps of carrying out secondary judgment on first cement loading and unloading event information to obtain second cement loading and unloading event information, and obtaining the ship ID input by a user to inquire the second cement loading and unloading event information. Meanwhile, the loading and unloading efficiency and quality can be improved, and the occurrence of stealing and unloading behaviors in the transportation process can be prevented. The real-time monitoring of the cement loading and unloading process is realized by utilizing a visual sensor and a computer visual technology. The loading and unloading efficiency and the quality are improved, and meanwhile, the stealing and unloading behavior in the transportation process can be prevented. The invention can detect in real time at any time and any place, is not limited by sites, and can effectively monitor ships entering and exiting ports.
The vision-based shipping cement loading and unloading monitoring system in the embodiment of the present invention is described in detail above in terms of the modularized functional entity, and the vision-based shipping cement loading and unloading monitoring apparatus in the embodiment of the present invention is described in detail below in terms of hardware processing.
Fig. 6 is a schematic structural diagram of a vision-based shipping cement shipment monitoring apparatus 600 according to an embodiment of the present invention, which may vary considerably in configuration or performance, may include one or more processors (central processing units, CPU) 610 (e.g., one or more processors) and a memory 620, and one or more storage mediums 630 (e.g., one or more mass storage devices) storing applications 633 or data 632. Wherein the memory 620 and the storage medium 630 may be transitory or persistent storage. The program stored on the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations on the vision-based shipping cement loading and unloading monitoring device 600. Still further, the processor 610 may be configured to communicate with the storage medium 630 to execute a series of instructional operations in the storage medium 630 on the vision-based shipping cement loading and unloading monitoring device 600.
The vision-based shipping cement shipment monitoring device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input/output interfaces 660, and/or one or more operating systems 631, such as Windows Serve, macOS X, unix, linux, freeBSD, and the like. It will be appreciated by those skilled in the art that the vision-based shipping cement shipment monitoring apparatus configuration shown in fig. 6 is not limiting and that more or fewer components than shown may be included or certain components may be combined or a different arrangement of components may be included.
The invention also provides a visual-based shipping cement loading and unloading monitoring device, which comprises a memory and a processor, wherein the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the visual-based shipping cement loading and unloading monitoring method in the above embodiments.
The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, and which may also be a volatile computer readable storage medium, the computer readable storage medium having instructions stored therein which, when executed on a computer, cause the computer to perform the steps of the vision-based shipping cement loading and unloading monitoring method.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (randomacceS memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. The vision-based shipping cement loading and unloading monitoring method is characterized by comprising the following steps of:
respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, collecting cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
acquiring cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and carrying out image framing and information labeling on the video data to obtain a labeling data set;
Performing data amplification on the marked data set to obtain an extended data set, and performing model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
acquiring target image data to be detected, inputting the target image data into the ship cement loading and unloading detection model to carry out cement loading and unloading detection, and obtaining first cement loading and unloading event information;
performing secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform, wherein the method specifically comprises the following steps of: when the first cement loading and unloading event information is a loading event, four coordinate point positions in the feeding pipeline image are obtained；
Acquiring a first probability value of a feeding pipeline in an image, and according to a preset first probability threshold valuePerforming preliminary screening on the first probability value;
by the position of the coordinate pointCoordinate point of furthest charging port->And the nearest fill port coordinate point->Performing distance calculation to obtain a target distance value;
the calculation formula of the target distance value is as follows:
；
；
；
wherein ,Representing +.>And furthest fill port coordinate point->Vertical distance between (I)>Representing +.>And the nearest fill port coordinate point->Vertical distance between (I)>Representing a target distance value;
for the target distance valueDistance from the predetermined distance threshold->Comparing when->≤/>When the cement loading and unloading event information is stored, determining that a loading event exists, and outputting the loading event as the second cement loading and unloading event information; when the first cement loading and unloading event information is an unloading event, four coordinate point positions +_in the unloading pipeline image are acquired>；
Acquiring a second probability value of the discharge pipeline in the image, and according to a preset second probability threshold valuePerforming preliminary screening on the second probability value;
setting a rectangular frame and four rectangular frame coordinate points in the unloading pipeline imageCalculating the intersection ratio of the rectangular frames according to the coordinate points of the four rectangular frames to obtain an IOU value;
the calculation formula of the IOU value is as follows:
；
；
；
；
；
；
；
；
；
wherein ,representing the width of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Representing the height of the rectangle formed by the discharge tube at four coordinate points of the image, +.>Representing the width of a rectangle formed by setting four rectangular frame coordinate points, < > >Representing the height of a rectangle formed by setting four rectangular frame coordinate points, +.>Minimum abscissa value representing rectangle formed by intersection of two rectangular frames, ">Representing the smallest ordinate value of the rectangle formed by the intersection of two rectangular frames, < >>Maximum abscissa value representing rectangle formed by intersection of two rectangular frames, +.>Representing the maximum ordinate value of a rectangle formed by the intersection of two rectangular frames,/for>Representing the area of a rectangle formed by the intersection of two rectangular boxes,representing the area of the two rectangular boxes minus the area of the intersecting rectangular box;
for the IOU value and a preset IOU threshold valueComparing, when IOU is greater than or equal to->When the cement loading and unloading event information is stored, determining that a loading and unloading event exists, and outputting the loading and unloading event information as second cement loading and unloading event information;
and acquiring the ship ID input by the user, inquiring the second cement loading and unloading event information from the server platform according to the ship ID, and carrying out visual management on the second cement loading and unloading event information.
2. The vision-based shipping cement loading and unloading monitoring method according to claim 1, wherein the obtaining cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and performing image framing and information labeling on the video data to obtain a labeling data set, comprises:
Acquiring cement loading and unloading time, and extracting video data from the terminal hard disk according to the cement loading and unloading time;
carrying out image framing on the video data through a preset ffmpeg tool to obtain a plurality of video frames;
performing image classification and image screening on the plurality of video frames to obtain loading event images and unloading event images;
and labeling the loading event image and the unloading event image by a preset labelme image labeling tool to obtain a labeling data set.
3. The vision-based shipping cement loading and unloading monitoring method according to claim 1, wherein the performing data amplification on the labeling data set to obtain an extended data set, and performing model training on a preset YOLOV5 target detection model based on the extended data set to obtain a shipping cement loading and unloading detection model comprises:
turning, rotating, color transforming and scaling the marked data set to obtain an expanded data set;
and (3) a preset YOLOV5 target detection model is paired with the extended data set, wherein the YOLOV5 target detection model comprises: backbone, neck and head networks;
predicting the extended data set through the YOLOV5 target detection model to obtain a prediction result;
And performing parameter tuning on the YOLOV5 target detection model based on the prediction result to obtain a ship cement loading and unloading detection model.
4. The vision-based shipping cement loading and unloading monitoring method according to claim 1, wherein the acquiring the target image data to be detected and inputting the target image data into the ship cement loading and unloading detection model to perform cement loading and unloading detection, and obtaining the first cement loading and unloading event information, comprises:
acquiring target image data to be detected, and inputting the target image data into the ship cement loading and unloading object detection model;
performing feature extraction through a backbone network in the ship cement loading and unloading object detection model to obtain feature image data;
inputting the characteristic image data into a neck network in the ship cement loading and unloading object detection model to perform characteristic fusion to obtain characteristic fusion data;
and inputting the characteristic fusion data into a head network in the ship cement loading and unloading detection model to detect cement loading and unloading and output first cement loading and unloading event information.
5. The vision-based shipping cement loading and unloading monitoring method according to claim 1, wherein the performing the secondary determination on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform, comprises:
When the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
when the first cement loading and unloading event information is a loading event, carrying out secondary judgment on the first cement loading and unloading event information based on a preset loading and unloading event judgment rule to obtain second cement loading and unloading event information;
and uploading the second cement loading and unloading event information to a preset server platform.
6. The utility model provides a shipment cement loading and unloading monitored control system based on vision which characterized in that, shipment cement loading and unloading monitored control system based on vision includes:
the acquisition module is used for respectively installing high-definition monitoring cameras at a cement loading port and a cement unloading port of a target ship, acquiring cement loading and unloading pipeline information through the high-definition monitoring cameras, and storing the cement loading and unloading pipeline information to a preset terminal hard disk;
the marking module is used for acquiring cement loading and unloading time, extracting video data from the terminal hard disk according to the cement loading and unloading time, and carrying out image framing and information marking on the video data to obtain a marked data set;
The training module is used for carrying out data amplification on the labeling data set to obtain an extended data set, and carrying out model training on a preset YOLOV5 target detection model based on the extended data set to obtain a ship cement loading and unloading object detection model;
the detection module is used for acquiring target image data to be detected, inputting the target image data into the ship cement loading and unloading detection model for cement loading and unloading detection, and obtaining first cement loading and unloading event information;
the judging module is used for carrying out secondary judgment on the first cement loading and unloading event information to obtain second cement loading and unloading event information, and uploading the second cement loading and unloading event information to a preset server platform, and specifically comprises the following steps: when the first cement loading and unloading event information is a loading event, four coordinate point positions in the feeding pipeline image are obtained；
Acquiring a first probability value of a feeding pipeline in an image, and according to a preset first probability threshold valuePerforming preliminary screening on the first probability value;
by the position of the coordinate pointCoordinate point of furthest charging port->And the nearest fill port coordinate point->Performing distance calculation to obtain a target distance value;
The calculation formula of the target distance value is as follows:
；
；
；
wherein ,representing +.>And furthest fill port coordinate point->Vertical distance between (I)>Representing +.>And the nearest fill port coordinate point->Vertical distance between (I)>Representing a target distance value;
for the target distance valueDistance from the predetermined distance threshold->Comparing when->≤/>When the cement loading and unloading event information is stored, determining that a loading event exists, and outputting the loading event as the second cement loading and unloading event information; when the first cement loading and unloading event information is an unloading event, four coordinate point positions +_in the unloading pipeline image are acquired>；
Acquiring a second probability value of the discharge pipeline in the image, and according to a preset second probability threshold valuePerforming preliminary screening on the second probability value;
setting a rectangular frame and four rectangular frame coordinate points in the unloading pipeline imageCalculating the intersection ratio of the rectangular frames according to the coordinate points of the four rectangular frames to obtain an IOU value;
the calculation formula of the IOU value is as follows:
；
；
；
；
；
；
；
；
；
wherein ,indicating discharge pipeWidth of rectangle formed by four coordinate points of image, +.>Representing the height of the rectangle formed by the discharge tube at four coordinate points of the image, +. >Representing the width of a rectangle formed by setting four rectangular frame coordinate points, < >>Representing the height of a rectangle formed by setting four rectangular frame coordinate points, +.>Minimum abscissa value representing rectangle formed by intersection of two rectangular frames, ">Representing the smallest ordinate value of the rectangle formed by the intersection of two rectangular frames, < >>Maximum abscissa value representing rectangle formed by intersection of two rectangular frames, +.>Representing the maximum ordinate value of a rectangle formed by the intersection of two rectangular frames,/for>Representing the area of a rectangle formed by the intersection of two rectangular boxes,representing the area of the two rectangular boxes minus the area of the intersecting rectangular box;
for the IOU value and a preset IOU threshold valueComparing, when IOU is greater than or equal to->When the cement loading and unloading event information is stored, determining that a loading and unloading event exists, and outputting the loading and unloading event information as second cement loading and unloading event information;
and the query module is used for acquiring the ship ID input by the user, querying the second cement loading and unloading event information from the server platform according to the ship ID, and carrying out visual management on the second cement loading and unloading event information.
7. The utility model provides a shipment cement loading and unloading goods supervisory equipment based on vision, its characterized in that, shipment cement loading and unloading goods supervisory equipment based on vision includes: a memory and at least one processor, the memory having instructions stored therein;
The at least one processor invoking the instructions in the memory to cause the vision-based shipping cement loading and unloading monitoring device to perform the vision-based shipping cement loading and unloading monitoring method of any one of claims 1-5.
8. A computer readable storage medium having instructions stored thereon, which when executed by a processor, implement the vision-based shipping cement loading and unloading monitoring method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310833246.6A CN116563803B (en) | 2023-07-10 | 2023-07-10 | Visual-based shipping cement loading and unloading monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310833246.6A CN116563803B (en) | 2023-07-10 | 2023-07-10 | Visual-based shipping cement loading and unloading monitoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116563803A CN116563803A (en) | 2023-08-08 |
| CN116563803B true CN116563803B (en) | 2023-09-15 |
Family
ID=87496873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310833246.6A Active CN116563803B (en) | 2023-07-10 | 2023-07-10 | Visual-based shipping cement loading and unloading monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116563803B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8170272B1 (en) * | 2010-02-23 | 2012-05-01 | The United States Of America As Represented By The Secretary Of The Navy | Method for classifying vessels using features extracted from overhead imagery |
| KR20120119557A (en) * | 2011-04-22 | 2012-10-31 | 대양전기공업 주식회사 | Ship condition monitoring service system |
| CN112016730A (en) * | 2019-05-31 | 2020-12-01 | 中电科海洋信息技术研究院有限公司 | Port berth loading and unloading efficiency excavating method, device, equipment and storage medium |
| CN113743377A (en) * | 2021-11-02 | 2021-12-03 | 深圳市信润富联数字科技有限公司 | Cargo handling monitoring method, system, equipment and computer storage medium |
| CN114014037A (en) * | 2021-11-17 | 2022-02-08 | 安徽海螺信息技术工程有限责任公司 | Intelligent cement loading and unloading monitoring system for wharf |
| CN114187564A (en) * | 2021-12-17 | 2022-03-15 | 江苏智库智能科技有限公司 | Cross-equipment linkage method and vision-aided linkage system in intelligent warehouse |
| CN114241355A (en) * | 2021-11-19 | 2022-03-25 | 上海浦东发展银行股份有限公司 | Warehouse management method and device, computer equipment and storage medium |
| CN114596239A (en) * | 2020-11-19 | 2022-06-07 | 顺丰科技有限公司 | Loading and unloading event detection method and device, computer equipment and storage medium |
| CN115131704A (en) * | 2022-06-29 | 2022-09-30 | 上海伟视清数字技术有限公司 | Cabin positioning method and system combining quadrilateral detection and deep learning |
| CN115303828A (en) * | 2022-08-10 | 2022-11-08 | 福州大学 | Port ship unloader parking positioning system based on vision |
-
2023
- 2023-07-10 CN CN202310833246.6A patent/CN116563803B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8170272B1 (en) * | 2010-02-23 | 2012-05-01 | The United States Of America As Represented By The Secretary Of The Navy | Method for classifying vessels using features extracted from overhead imagery |
| KR20120119557A (en) * | 2011-04-22 | 2012-10-31 | 대양전기공업 주식회사 | Ship condition monitoring service system |
| CN112016730A (en) * | 2019-05-31 | 2020-12-01 | 中电科海洋信息技术研究院有限公司 | Port berth loading and unloading efficiency excavating method, device, equipment and storage medium |
| CN114596239A (en) * | 2020-11-19 | 2022-06-07 | 顺丰科技有限公司 | Loading and unloading event detection method and device, computer equipment and storage medium |
| CN113743377A (en) * | 2021-11-02 | 2021-12-03 | 深圳市信润富联数字科技有限公司 | Cargo handling monitoring method, system, equipment and computer storage medium |
| CN114014037A (en) * | 2021-11-17 | 2022-02-08 | 安徽海螺信息技术工程有限责任公司 | Intelligent cement loading and unloading monitoring system for wharf |
| CN114241355A (en) * | 2021-11-19 | 2022-03-25 | 上海浦东发展银行股份有限公司 | Warehouse management method and device, computer equipment and storage medium |
| CN114187564A (en) * | 2021-12-17 | 2022-03-15 | 江苏智库智能科技有限公司 | Cross-equipment linkage method and vision-aided linkage system in intelligent warehouse |
| CN115131704A (en) * | 2022-06-29 | 2022-09-30 | 上海伟视清数字技术有限公司 | Cabin positioning method and system combining quadrilateral detection and deep learning |
| CN115303828A (en) * | 2022-08-10 | 2022-11-08 | 福州大学 | Port ship unloader parking positioning system based on vision |
Non-Patent Citations (1)
| Title |
|---|
| 自动化散货装船机物位检测技术;李强;宓超;赵宁;沈阳;;上海海事大学学报(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116563803A (en) | 2023-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111310645B (en) | Method, device, equipment and storage medium for warning overflow bin of goods accumulation | |
| US20210183050A1 (en) | Pipe feature identification using pipe inspection data analysis | |
| CN110084165B (en) | Intelligent identification and early warning method for abnormal events in open scene of power field based on edge calculation | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| CN112446363A (en) | Image splicing and de-duplication method and device based on video frame extraction | |
| CN113822385B (en) | Coal conveying abnormity monitoring method, device and equipment based on image and storage medium | |
| CN111814739B (en) | Method, device, equipment and storage medium for detecting express package volume | |
| CN113255590A (en) | Defect detection model training method, defect detection method, device and system | |
| CN111126196A (en) | Equipment oil leakage detection method | |
| CN115797811B (en) | Agricultural product detection method and system based on vision | |
| CN112784821A (en) | Building site behavior safety detection and identification method and system based on YOLOv5 | |
| CN111310826A (en) | Method and device for detecting labeling abnormity of sample set and electronic equipment | |
| CN111753612A (en) | Method and device for detecting sprinkled object and storage medium | |
| CN112580600A (en) | Dust concentration detection method and device, computer equipment and storage medium | |
| CN114140684A (en) | Method, device and equipment for detecting coal blockage and coal leakage and storage medium | |
| CN107657682A (en) | A kind of power transformation method for inspecting based on augmented reality | |
| CN112794274B (en) | Safety monitoring method and system for oil filling port at bottom of oil tank truck | |
| CN116563803B (en) | Visual-based shipping cement loading and unloading monitoring method and system | |
| CN108629310B (en) | Engineering management supervision method and device | |
| CN113408630A (en) | Transformer substation indicator lamp state identification method | |
| CN117132872A (en) | Intelligent collision recognition system for material transport vehicle on production line | |
| CN114550129B (en) | Machine learning model processing method and system based on data set | |
| Wang et al. | Insulator defect detection based on improved YOLOv5 algorithm | |
| CN110363109B (en) | Computer vision-based coverage type oil leakage detection system and method | |
| CN114596239A (en) | Loading and unloading event detection method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |