CN115328983A - Port data comprehensive application method and system based on big data platform - Google Patents

Port data comprehensive application method and system based on big data platform Download PDF

Info

Publication number
CN115328983A
CN115328983A CN202210902709.5A CN202210902709A CN115328983A CN 115328983 A CN115328983 A CN 115328983A CN 202210902709 A CN202210902709 A CN 202210902709A CN 115328983 A CN115328983 A CN 115328983A
Authority
CN
China
Prior art keywords
data
port
ship
berth
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210902709.5A
Other languages
Chinese (zh)
Inventor
邰伟鹏
金明秀
李伟
刘富豪
王小林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Anhui University Of Technology Science Park Co ltd
Original Assignee
Anhui University Of Technology Science Park Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anhui University Of Technology Science Park Co ltd filed Critical Anhui University Of Technology Science Park Co ltd
Priority to CN202210902709.5A priority Critical patent/CN115328983A/en
Publication of CN115328983A publication Critical patent/CN115328983A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29—Geographical information databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/083—Shipping

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Human Resources & Organizations (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Economics (AREA)
  • General Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Game Theory and Decision Science (AREA)
  • Remote Sensing (AREA)
  • Educational Administration (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention provides a port data comprehensive application method and a port data comprehensive application system based on a big data platform, which relate to the field of big data application; the method comprises the following steps: building a big data platform, and constructing a structure table of a data warehouse according to original data of a port production system and an equipment material system and extracting data; performing ETL processing on the extracted data; performing secondary data cleaning on the ETL-processed data; carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data; and calculating the comprehensive index efficiency of the port according to the scheduling data. The invention carries out ETL processing and standardization on port data by constructing a big data platform, analyzes the scheduling problem of a ship in the process of going from port, loading and unloading the ship and then leaving the port, optimizes the scheduling, further calculates the comprehensive index efficiency of the port, fully solves the problem of data island of the port data and perfects the operation management of the port.

Description

Port data comprehensive application method and system based on big data platform
Technical Field
The invention relates to the technical field of big data statistics, in particular to a port data comprehensive application method and system based on a big data platform.
Background
With the development of national economy and the increase of international trade in China, the status and the function of ports are greatly changed in the development of social economy. Ports are the economic catalyst of the cities they serve, promoting the integration of the economic industry and the convergence of services, thus creating social and economic benefits. In recent years, the construction period of ports is prolonged, the investment cost is increased, the types of goods are diversified, the unit price factors of the goods are influenced, and the data resources are rapidly increased. The port data analysis method can be developed continuously only by continuously improving the service quality of the port, analyzing the port data, reducing data redundancy and maximally utilizing the data to analyze the comprehensive efficiency of the port, so that the comprehensive efficiency becomes an important reference for a manager to make a decision. The comprehensive efficiency of the port is an important index of port competitiveness, plays a vital role in shortening the berthing time of ships in the port, accelerating the turnover of the ships and reducing the transportation cost, and directly influences the selection of a ship company for hanging and berthing on an air route.
Some solutions for calculating the working efficiency of the port by using port data are also disclosed in the prior art; for example, patent application CN112016730a discloses a digging method, device, equipment and storage medium for loading and unloading efficiency of port berth, the method includes: acquiring at least one piece of ship basic information, wherein each piece of ship basic information comprises historical ship draft data, a ship model, and maximum draft data and maximum load data corresponding to the ship model; inputting at least one historical ship draft data into a pre-constructed cargo estimation model to determine a maximum cargo handling capacity of a ship berthing at a port and berth, the cargo handling capacity being a capacity of a ship berthing at the port for loading or unloading; acquiring the minimum ship berthing time corresponding to the maximum cargo loading and unloading amount; the loading and unloading efficiency of the port berth is calculated based on the maximum cargo loading and unloading amount and the minimum ship berthing time. However, only relevant information of ships is considered in the application, and the problem of calculation of the loading and unloading efficiency at large-scale port berths is not considered.
As well as the port planning big data processing method and system based on ship dynamic information disclosed in patent CN112396216B, the method divides all ships berthing at all ports into a plurality of groups according to port berths, and each group of ships is allocated to a waiting area corresponding to the matched port berths; when ships stop at the distributed port berths, the ships are divided into two rows according to the sequence of the port-entering stopping time points and are arranged in the waiting area, and the ships in the waiting area enter the port berths according to the sequence; guiding ships which do not arrive at the later time according to the berthing time point to a general regulation area to berth and distributing the ships to port berths corresponding to the general regulation area; the ship is regulated and controlled to be transferred to the port berth of the ship without waiting for berthing so as to improve the use efficiency of the port berth. The scheme only considers the scheduling problem of the berths of the ships and the ports, does not consider the problems of the ships and the storage yards and the goods and the storage yards, and does not consider the matching degree and the scheduling problem of the integral operation of the ports.
The technical scheme disclosed above only utilizes partial port data, and the problem of data island of the port data is not completely solved; meanwhile, the comprehensive efficiency of the port cannot be integrally counted, so that the comprehensive scheduling problem of the port is not greatly solved.
Disclosure of Invention
The invention aims to provide a port data comprehensive application method and a port data comprehensive application system based on a big data platform.
In order to achieve the above purpose, the invention provides the following technical scheme: a port data comprehensive application method based on a big data platform comprises the following steps: a port data comprehensive application method based on a big data platform comprises the following steps:
building a big data platform, and constructing a structure table of a data warehouse according to original data of a port production system and an equipment material system and extracting data; performing ETL processing on the extracted data;
carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data;
carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data;
and calculating the comprehensive index efficiency of the port according to the scheduling data.
Further, the ETL processed data are sequentially stored in an ODS layer, a DWD layer and a DWS layer;
the ODS layer stores port operation ticket information, ship information and port original data of wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculation according to the DWD layer data shipunload And berthing time T of ship shipberth And device age T machine 。
Further, the data cleansing rule includes:
static data cleaning rules and dynamic data cleaning rules;
the dynamic data cleaning rule comprises a missing value processing rule, a repeated value processing rule and an abnormal value processing rule; the missing value processing rule is a similar mean interpolation method filling missing values, and the repeated value processing rule is comparison repeated data deduplication;
the abnormal value processing rule is according to the ship loading and unloading time T shipunloa And berthing time T of ship shipberth The exception handling is carried out according to the size of the ratio k, and the exception handling comprises the following steps:
when k is more than 1, correcting the berthing time T of the ship shipberth ；
When k is>>1, correcting T by adopting a repeated value processing rule shipunload Or to correct the ship departure time.
Further, the obtaining process of the scheduling data is as follows:
acquiring ship information and berth information of a port coming within a planning date, and carrying out berth scheduling;
determining a storage yard for storing goods according to the result of the berth scheduling and the storage yard information;
the berthing scheduling process is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure BDA0003771447280000031
wherein,
Figure BDA0003771447280000032
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure BDA0003771447280000033
Figure BDA0003771447280000034
wherein,
Figure BDA0003771447280000035
indicating the distance, L, of the vessel to the kth berth min[ship,bert] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure BDA0003771447280000036
wherein,
Figure BDA0003771447280000037
representing the berthing time of the ith ship;
when a plurality of berths of the port are occupied and have no empty berths, scheduling the ship with short berthing time to the closest berth operation of the port according to the shortest time and shortest distance priority scheduling principle;
Figure BDA0003771447280000038
Figure BDA0003771447280000039
the yard is determined as follows:
after determining the berthing position of the incoming port ship, calculating the distance from the berthing position to any yard, and determining the yard with the shortest berthing position distance as the berthing yard of the incoming port ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure BDA0003771447280000041
Figure BDA0003771447280000042
q＝1,…m，
i＝0,1,…r,
Figure BDA0003771447280000043
wherein L is min[p,q] Represents the shortest distance, L, from the berth p to the yard q [p,q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the i-th inflection point on the path from the berth p to the yard q.
Further, the calculation process of the comprehensive index efficiency of the port is as follows:
according to ship loading and unloading time T shipunload And the berthing time T of the ship shipbert h calculating the handling efficiency E of the port unload ：
E unload ＝T shipunloa /(T shipberth -T invalid -T shiipbreak )
Wherein, T invalid Due to the dead time, T, caused by natural factors shiipbreak Due to maintenance time resulting from vessel failure;
according to the time T of use of the equipment machi Computing device efficiency E machine ：
E machine ＝T machineactual /(T machine -T machinebreak )
Wherein, T machineactual Is the actual operating time of the apparatus, T machinebrea Maintenance time due to equipment failure;
calculating the efficiency E of the loading and unloading personnel according to the information of the personnel of the loading and unloading working group in one loading and unloading work of the port ship oneworker ：
E worker ＝W cargo /Work nums *(T workend -T workstart -T rest )
Wherein, W cargo Weight indicating loading or unloading of goods in one-time operation ticket, work nums Indicating the number of workers involved in the unloading work, T, in a job ticket workend Indicating the end time of loading and unloading, T, by the loader workstart Indicating the loading and unloading start time, T, of the loader rest Representing the rest time of the handler;
calculating ship loading and unloading personnel efficiency E according to ship loading and unloading work ticket information shipworker ：
Figure BDA0003771447280000051
Wherein n is the loading and unloading times or the number of operation tickets for completing the loading and unloading work of one ship.
The invention also provides a port data comprehensive application system based on a big data platform, which comprises:
the building module is used for building a big data platform, building a structure table of a data warehouse according to original data of a port production system and an equipment material system, and extracting data;
the data processing module is used for carrying out ETL processing on the extracted data;
the data cleaning module is used for carrying out secondary data cleaning on the data after the ETL processing according to a preset cleaning rule to obtain cleaned port data;
the first calculation module is used for carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data;
and the second calculation module is used for calculating the comprehensive index efficiency of the port according to the scheduling data.
Further, the data processed by the data processing module are sequentially stored in an ODS layer, a DWD layer and a DWS layer;
the ODS layer stores port operation ticket information, ship information and port original data of wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculation according to the DWD layer data shipunloa And berthing time T of ship shipberth And device age T machiine 。
Further, the cleaning rule for the data cleaning module to clean the data includes:
static data cleaning rules and dynamic data cleaning rules;
the dynamic data cleaning rule comprises a missing value processing rule, a repeated value processing rule and an abnormal value processing rule; the missing value processing rule is a similar mean interpolation method filling missing values, and the repeated value processing rule is comparison repeated data deduplication;
the abnormal value processing rule is according to the ship loading and unloading time T shipunload And berthing time T of ship shipberth The exception handling is carried out according to the size of the ratio k, and the exception handling comprises the following steps:
when k is more than 1, correcting the berthing time T of the ship shipberth ；
When k is>>1, correcting T by adopting a repeated value processing rule shipunloa Or to correct the departure time of the ship.
Further, the execution unit of the first calculation module for calculating and obtaining the scheduling data includes:
the acquisition unit is used for acquiring the ship information and berth information of the incoming port in the planning date;
the berth scheduling unit is used for scheduling the berth according to the information acquired by the acquisition unit;
the determining unit is used for determining a storage yard for storing goods according to the berth scheduling result and the storage yard information;
the execution process of the berthage scheduling unit is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure BDA0003771447280000061
wherein,
Figure BDA0003771447280000062
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure BDA0003771447280000063
Figure BDA0003771447280000064
wherein,
Figure BDA0003771447280000065
indicating the distance, L, of the vessel to the kth berth min[ship,berth] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure BDA0003771447280000066
wherein,
Figure BDA0003771447280000067
representing the berthing time of the ith ship;
when a plurality of berths of the port are occupied and have no empty berths, scheduling the ship with short berthing time to the closest berth operation of the port according to the shortest time and shortest distance priority scheduling principle;
Figure BDA0003771447280000068
Figure BDA0003771447280000069
the yard is determined as follows:
after determining the berthing position of the incoming port ship, calculating the distance from the berthing position to any yard, and determining the yard with the shortest berthing position distance as the berthing yard of the incoming port ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure BDA00037714472800000610
Figure BDA0003771447280000071
q＝1,…m,
i＝0,1,…r,
Figure BDA0003771447280000072
wherein L is min[p,q] Represents the shortest distance, L, from the berth p to the yard q [p,q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the i-th inflection point on the path from the berth p to the yard q.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein when the processor runs the computer program, the port data comprehensive application method based on the big data platform is realized.
According to the technical scheme, the technical scheme of the invention provides the following beneficial effects:
the invention discloses a port data comprehensive application method and a system based on a big data platform, wherein the method comprises the following steps: building a big data platform, building a structure table of a data warehouse according to original data, and extracting data; performing ETL processing on the extracted data; carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data; carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data; and calculating the comprehensive index efficiency of the port according to the scheduling data. The invention carries out ETL processing and standardization on port data by constructing a big data platform, analyzes the scheduling problem of a ship in the process of going from port, loading and unloading the ship and then leaving the port, optimizes the scheduling, further calculates the comprehensive index efficiency of the port, fully solves the problem of data island of the port data and perfects the operation management of the port. The invention has the following specific advantages:
1) The invention effectively solves the problems of integration, storage and calculation of data of a port production system and a device system by using Hadoop and other big data technologies, and avoids insufficient memory during calculation;
2) The invention effectively processes the data of the production system and the equipment system by using the data warehouse technology, and when ETL processing and calculation are carried out on the data, the use efficiency of temporary data is improved and the calculation time of temporary intermediate results is saved due to the use of the layering mode of the data warehouse;
3) The invention uses the dispatching algorithm and the calculation of the comprehensive efficiency, provides multi-angle analysis for the port, benefits the important reference of the decision of the manager and promotes the development of the port.
It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail below can be considered as part of the inventive subject matter of this disclosure unless such concepts are mutually inconsistent.
The foregoing and other aspects, embodiments and features of the present teachings will be more fully understood from the following description taken in conjunction with the accompanying drawings. Additional aspects of the present invention, such as features and/or advantages of exemplary embodiments, will be apparent from the description which follows, or may be learned by practice of specific embodiments in accordance with the teachings of the present invention.
Drawings
The figures are not intended to be drawn to scale with true references. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. Embodiments of various aspects of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a comprehensive port data application method based on a big data platform according to the invention;
FIG. 2 is a schematic view of a berth-yard process of the present invention;
FIG. 3 is a flow chart of the outlier cleaning rule application of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention. Unless defined otherwise, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs.
The use of "first," "second," and similar terms in the description and in the claims of the present application does not denote any order, quantity, or importance, but rather the terms are used to distinguish one element from another. Similarly, the singular forms "a," "an," or "the" do not denote a limitation of quantity, but rather denote the presence of at least one, unless the context clearly dictates otherwise. The terms "comprises," "comprising," or the like, mean that the elements or items listed before "comprises" or "comprising" encompass the features, integers, steps, operations, elements, and/or components listed after "comprising" or "comprising," and do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. "upper", "lower", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute position of the object to be described is changed, the relative positional relationships may also be changed accordingly.
Based on the gradually enhanced role of ports in national economy and international trade development, the service quality of the ports needs to be improved to improve the competitiveness of the ports in the industry; however, the current port only preliminarily completes production, equipment, materials, office and other informatization systems, port data has the characteristics of large data volume and overflowing memory, and the port cannot be integrally applied to information data acquired by each department and is integrally represented as a data island phenomenon. The invention provides a port data comprehensive application method and a port data comprehensive application system based on a big data platform, which aim at the problems by building data of each gate of a big data platform comprehensive application port, calculating dispatching data among port berths, ships, goods and storage yards and calculating the comprehensive index efficiency of the port according to the dispatching data, fully solves the problems of large port data volume and memory overflow, conveniently obtains the comprehensive index efficiency of the port, perfects the operation management of the port and improves the market competitiveness.
The port data comprehensive application method and system based on the big data platform disclosed by the invention are further specifically described below with reference to the attached drawings.
With reference to fig. 1, the embodiment of the disclosed port data comprehensive application method based on the big data platform includes the following steps:
step S102, a big data platform is built, a structure table of a data warehouse is built according to original data of a port production system and an equipment material system, and data extraction is carried out;
the production system of the port comprises a plurality of subsystems such as a business department management system, a production department management system, a port first company, a port third company and the like, the equipment material system comprises a plurality of subsystems such as an equipment static management subsystem, an equipment dynamic management subsystem and a material management subsystem, and original data come from each component system and are stored in a database. The acquired original data is mainly related data which is arranged according to business requirements and takes a ship as a main line, comprises berth information, stock yard information, equipment information, ship information, operation ticket information and ship voyage information, and relates to the fact that the related data comes from a ship forecasting module of a business department management system; a ship data query module, a production day and night plan module, a tally management module, a ship scheduling management module and an operation ticket module of the production department management system; a port affair third company job ticket filling condition query module and a material receiving module of the equipment material system.
The big data platform is constructed by adopting Hadoop, hive, spark and other frames, six data structure tables are newly built in an oracle database, and are respectively a berth data structure table, a yard data structure table, an equipment data structure table, a ship data structure table, an operation ticket data structure table and a ship voyage data structure table which are respectively used for storing final data; specifically, the berth data structure table stores berth information, which comprises basic berth information, berth codes, berth names and basic belonged wharf information; the storage yard data structure table stores storage yard information, which comprises basic information of storage yard number, name, area and capacity; the equipment data structure table stores equipment information which comprises equipment numbers, names, types and basic information of affiliated departments; the ship data structure table stores ship information, including ship code, name, identification number, registration number, type, total length and total tonnage basic information; the operation ticket data structure table stores operation ticket information, records basic information of each operation of workers, comprises working time, starting time, a belonging department, a ship name, a ship voyage number, a cargo name, cargo weight, a cargo code, a docking dock and basic information of working equipment, and is associated with the equipment information table, the berth information table and the ship information table; the ship voyage data structure table stores ship voyage information comprising ship voyage number, ship code, ship name, arrival time, departure time, operation duration and actual cargo weight basic information.
After the data table is constructed, data are directly extracted into Hive through Sqoop; the data extracted into the Hive are extracted in different stages, a full-quantity extraction mode is adopted for data extraction for the first time, all the data are extracted and stored into the HDFS in one time, the occupied resources are more, the consumed time is longer, an incremental extraction mode is adopted for the data extraction mode, and the data are extracted from the current data by judging the maximum timestamp of the original data.
Step S104, ETL processing is carried out on the extracted data;
in the embodiment, the ETL processed data is stored in an ODS layer, a DWD layer and a DWS layer in sequence; the ODS layer stores port original data of port operation ticket information, ship information and wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculating according to DWD layer data shipunload And berthing time T of ship shipberth And device age T machiine 。
In practice, the ship loading and unloading time T shipunload The calculation is as follows:
Figure BDA0003771447280000101
wherein, T unloadstart Indicating the loading and unloading start time, T unloadend Indicating the loading and unloading end time;
berthing time T of ship shipberth The calculation is as follows:
T shipberth ＝T shiipdeparture -T shiipcome
wherein, T shiipcome Indicating the time of arrival of the vessel, T shiipdeparture Representing the ship departure time;
device usage time T machine The calculation is as follows:
Figure BDA0003771447280000102
wherein, T machinestart Indicating the time at which the device is starting to operate,T machineend indicating the end of the device on-time period.
Step S106, carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data;
the method is characterized in that the source of original data is mainly divided into two parts based on port use, data are collected by manual input and RFID (radio frequency identification) detection technologies respectively, due to abnormal reasons such as manual input errors, detection equipment faults, communication system faults and environments, the collected data have the phenomena of redundancy, omission, errors and inaccuracy, data cleaning is a process of simplifying and standardizing a database to remove repeated records and convert the residual data into data values according with actual conditions, the data are cleaned through a series of steps, and the cleaned data are output in an expected format. Data cleansing deals with problems of data such as missing values, out-of-bounds values, inconsistent codes, repeated data and the like in terms of accuracy, integrity, consistency, validity, uniqueness and the like of the data. The accuracy of the data is to describe whether the data is consistent with the characteristics of its corresponding objective entity. The integrity of data is to describe whether the data has missing records or missing fields. The consistency of data is to describe whether the values of the same attribute of the uniform entity are consistent in different systems. The validity of the data is to describe whether the data meets a user-defined condition or is within a certain threshold. The uniqueness of the data is to describe whether the data has duplicate records. The data is cleaned mainly through two aspects, on one hand, the static data is cleaned, the error rate of the static data is low, and the probability that most static data is mistaken again after being cleaned is low; on the other hand, aiming at the cleaning of dynamic data, the frequency of updating the working time of ships from harbor, goods loading and unloading to harbor and personnel is high, the timeliness of data transmission and storage is also high, and the requirements on the fineness of the dynamic data in the aspects of system operation analysis, auxiliary decision making and the like are also high.
In the embodiment, the following cleaning rules are adopted to clean the port data, including: static data cleaning rules and dynamic data cleaning rules, wherein the dynamic data cleaning rules comprise missing value processing rules, repeated value processing rules and abnormal value processing rules; the specific content of each cleaning rule is as follows:
for the static data cleaning rule, the method mainly comprises the steps of timely acquiring related equipment information through long-term cooperation with related departments, establishing an information tracking mechanism and a data file, and timely updating related data to ensure the integrity and the effectiveness of the data; when the data is missing or abnormal, the data is timely linked with related departments, and the data is checked and corrected from the data source head.
For the missing value processing rule, a method of filling missing values by means of mean interpolation of the same kind is mainly adopted, for example, when data of the loading and unloading efficiency of a certain cargo is missing, the data is firstly classified from white class, night class and the loaded and unloaded cargo, and the average working time of each person of the same type of cargo is calculated to perform interpolation filling of the missing values.
For the repeated value processing rule, the repeated data is compared to remove the repeated data; the repeated record is generated in the merging process of multiple data sources, and repeated data is compared, and if the data are repeated, repeated processing is carried out.
For abnormal value processing rules, ship loading and unloading time T is adopted shipunload And berthing time T of ship shipberth The method for exception handling according to the magnitude of the ratio k, as shown in fig. 3, includes:
when k is more than 1, the departure time of the ship is smaller than the finish time of the last loading and unloading of the worker, which indicates T shipberth Data abnormality and correction of ship berthing time T shipberth ；
When k is>>The method includes two cases, one is that data is repeated, the time of loading and unloading operation is repeated, for example, the loading and unloading operation time of the ith time includes the loading and unloading operation time of the jth time, and the description is T shipunload Data abnormity, load and unload time length needing to be removed by repeated calculation, and T is corrected by adopting repeated value processing rule shipunload (ii) a Secondly, the manual input information is wrong, the docking time of the ship approaches to the departure time of the ship, namely T shipberth Data abnormality requiring repairPositive ship departure time.
Step S108, carrying out scheduling calculation on the process of the ship going in and out of the port according to the port data to obtain scheduling data;
known port operations are mainly performed around ships and cargos, and according to the characteristics of the ships and the cargos, machinery, manpower and infrastructure are effectively combined and reasonably distributed, so that the ships can stop and the cargos can be loaded, unloaded and transported; therefore, the dispatching of the port ship according to the port data is mainly planned from four aspects of berthing, ships, storage yards and goods:
1) The method comprises the steps that the planning date and basic information of arrival of ships at ports and basic information of loaded cargos are obtained in advance and provided to port related departments;
2) The berth is in an idle state and meets the hard constraint condition of the berth of the ship, and the ship can enter the berth to start operation;
3) A single berth only receives one ship to berth once, and a single ship can only berth at one berth once;
4) Each yard is allowed to be stacked with only one cargo, and if the weight of the cargo is higher than the maximum capacity of the yard, the cargo needs to be split according to the capacity of the yard.
5) The port yard total amount, port operating personnel and acting mechanical equipment meet the requirements.
Then, through the information of the ship coming to the port within the known planning date, the ship berthing scheduling is arranged, which specifically comprises the following steps: acquiring ship information and berth information of a port coming within a planning date, and carrying out berth scheduling;
determining a storage yard for storing goods according to the result of the berth scheduling and the storage yard information;
in an embodiment, the berthage scheduling process is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure BDA0003771447280000121
wherein,
Figure BDA0003771447280000122
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure BDA0003771447280000123
Figure BDA0003771447280000131
wherein,
Figure BDA0003771447280000132
indicating the distance, L, of the vessel to the kth berth min[ship,bert] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure BDA0003771447280000133
wherein,
Figure BDA0003771447280000134
representing the berthing time of the ith ship;
when a plurality of berths are occupied without empty berths in the port, scheduling the ship with short berthing time to berth at the closest wharf berth according to the shortest time and shortest distance priority scheduling principle;
Figure BDA0003771447280000135
Figure BDA0003771447280000136
the yard is determined as follows:
after determining the berth of the incoming port ship, calculating the distance from the berth to any storage yard, wherein the shorter the distance is, the lower the operation cost is, wherein the situation that the nonlinear distance between the berth and the storage yard contains a plurality of inflection points and different routes is shown in fig. 2.
Determining the yard with the shortest berthing distance as the berthing yard of the ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure BDA0003771447280000137
Figure BDA0003771447280000138
q＝1,…m，
i＝0,1,…r,
Figure BDA0003771447280000139
wherein L is min[p,q] Represents the shortest distance, L, from the berth p to the yard q [p,q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the i-th inflection point on the path from the berth p to the yard q.
And step S110, calculating the comprehensive index efficiency of the port according to the scheduling data.
The calculation content is as follows: according to ship loading and unloading time T shipunload And the berthing time T of the ship shipber h calculating the handling efficiency E of the port unload ：
E unload ＝T shipunload /(T shipbert h-T invalid -T shiipbreak )
Wherein, T invalid Due to the dead time, T, caused by natural factors shiipbreak Due to maintenance time resulting from vessel failure;
according to the time T of use of the equipment machine Computing device efficiency E machine ：
E machi ＝T machineactual /(T machine -T machinebreak )
Wherein, T machineactual Is the actual operating time of the apparatus, T machinebreak Maintenance time due to equipment failure;
calculating the efficiency E of the loading and unloading personnel according to the information of the personnel of the loading and unloading working group in one loading and unloading work of the port ship oneworker ：
E worker ＝W cargo /Work nums *(T workend -T workstart -T rest )
Wherein, W cargo Weight indicating loading or unloading of goods in one-time operation ticket, work nums Indicating the number of workers involved in the unloading work, T, in a job ticket workend Indicating the end time of loading and unloading, T, by the loader workstart Indicating the loading and unloading start time, T, of the loader rest Representing the rest time of the handler;
calculating ship loading and unloading personnel efficiency E according to ship loading and unloading work ticket information shipworker ：
Figure BDA0003771447280000141
Wherein n is the loading and unloading times or the number of operation tickets for completing the loading and unloading work of one ship.
The calculation process of the service time, the scheduling data and the port comprehensive index efficiency of the ship at the port is illustrated below.
For example, calculate the ship loading and unloading time T of 344739 and WANZHOUNGFA1598 shipunload 。
T shipunload ＝(2021-11-17 18:30:00-2021-11-17 12:00:00)+(2021-11-18 02:10:00-2021-11-17 23:30:00)+(2021-11-18 04:30:00-2021-11-18 03:00:00)+(2021-11-18 12:15:00-2021-11-18 08:30:00)+(2021-11-18 16:30:00-2021-11-18 13:20:00)+(2021-11-19 01:40:00-2021-11-18 20:00:00)＝23.251
And calculating the ship berthing time of the ship with the number of 344739 and the name of WANZHOUNGFA 1598.
T shipberth ＝2021-11-19 02:22:28-2021-11-17 06:00:00＝44.374
The number of the ship is 344739, and the name of the ship is WANZHOUNGFA 1598.
Figure BDA0003771447280000155
For another example, the shortest docking time and shortest distance from the #7 berth are calculated for the ship named WANZHOUNGFA 1598.
Figure BDA0003771447280000151
Figure BDA0003771447280000152
The ship name is new seat 1678 at berth #7 to the minimum distance of yard D405.
Figure BDA0003771447280000153
f(2)＝L min *y nm ＝2km*1＝2km
And calculating the loading and unloading efficiency of the ship with the ship number of 344739 and the ship name of WANZHOUNGFA 1598.
E unload ＝T shipunload /(T shipberth -T invalid -T shiipbreak )＝23.251/44.374-2-1＝56.2％
Calculate the equipment efficiency using a #8 gate machine with a ship number of 344739, and a ship name of wanzhonggfa 1598.
E machiine ＝T machiineactual /T machine -T machinebreak ＝10/16.751-2＝67.8％
Calculating the efficiency of the person loading and unloading the first-time operation ticket with the ship number of 344739 and the ship name of WANZHHOUNGFA 1598, wherein the unit is as follows: ton/(man hour).
E worker ＝W cargo /Work nums *(T workend -T workstart -T rest )
＝2080/6*(2021-11-17 18:30:00-2021-11-17 12:00:00-0.5)＝57.7
Calculating the efficiency E of the person handling the ship with the ship number of 344739 and the ship name of WANZHOUNGFA1598 shiipworker The unit is: ton/(man hour).
Figure BDA0003771447280000154
After calculation, all calculation results are written into a business database, such as an Oracle database.
According to the application method, the raw data generated by a large data platform regular port production and equipment system is built, the data is subjected to layering and standardized processing, and the processed data is used for scheduling planning of port ships, so that the problems of data redundancy, memory overflow, incapability of integrally planning data islands and the like caused by large data volume are solved, the comprehensive efficiency of analyzing ports by utilizing the data can be maximized, and the market competitiveness of the ports is improved.
In an embodiment of the present invention, an electronic device is further provided, where the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the port data comprehensive application method based on the big data platform in the above embodiment is implemented.
The programs described above may be run on a processor or may also be stored in memory, i.e., a computer readable medium, which may include non-transitory and non-transitory, removable and non-removable media, which may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media such as modulated data signals and carrier waves.
These computer programs may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks and corresponding method steps may be implemented by different modules.
In this embodiment, there is provided an apparatus or system, which may be referred to as a big data platform based port data integrated application system, the system comprising: the building module is used for building a big data platform, building a structure table of a data warehouse according to original data of a port production system and an equipment material system, and extracting data; the data processing module is used for carrying out ETL processing on the extracted data; the data cleaning module is used for carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data; the first calculation module is used for carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data; and the second calculation module is used for calculating the comprehensive index efficiency of the port according to the scheduling data.
The steps of the system for implementing the port data comprehensive application method based on the big data platform disclosed in the above embodiments have already been described, and are not described herein again.
For example, the data processed by the data processing module are sequentially stored in an ODS layer, a DWD layer and a DWS layer; the ODS layer stores port original data of port operation ticket information, ship information and wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculating according to DWD layer data shipunloa And berthing time T of ship shipberth And device age T machiine 。
For another example, the cleansing rule for the data cleansing module to cleanse the data includes: static data cleaning rules and dynamic data cleaning rules; the dynamic data cleaning rule comprises a missing value processing rule, a repeated value processing rule and an abnormal value processing rule; the missing value processing rule is a similar mean interpolation method filling missing values, and the repeated value processing rule is comparison repeated data deduplication; the abnormal value processing rule is according to the ship loading and unloading time T shipunload And berthing time T of ship shipberth The exception handling is carried out according to the size of the ratio k, and the exception handling comprises the following steps: when k is more than 1, correcting the berthing time T of the ship shipberth (ii) a When k is>>1, correcting T by adopting a repeated value processing rule shipunloa Or to correct the ship departure time. .
For another example, the execution unit of the first calculation module to calculate and obtain the scheduling data includes: the system comprises an acquisition unit, a storage unit and a control unit, wherein the acquisition unit is used for acquiring ship information and berth information of a port coming within a planned date; the berth scheduling unit is used for scheduling the berth according to the information acquired by the acquisition unit; the determining unit is used for determining a storage yard for storing goods according to the berth scheduling result and the storage yard information; the execution process of the berthage scheduling unit is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure BDA0003771447280000171
wherein,
Figure BDA0003771447280000172
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure BDA0003771447280000173
Figure BDA0003771447280000174
wherein,
Figure BDA0003771447280000175
indicating the distance, L, of the vessel to the kth berth min[ship,bert] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure BDA0003771447280000181
wherein,
Figure BDA0003771447280000182
representing the berthing time of the ith ship;
when a plurality of berths of the port are occupied and have no empty berths, scheduling the ship with short berthing time to the closest berth operation of the port according to the shortest time and shortest distance priority scheduling principle;
Figure BDA0003771447280000183
Figure BDA0003771447280000184
the yard is determined as follows:
after determining the berthing position of the incoming port ship, calculating the distance from the berthing position to any yard, and determining the yard with the shortest berthing position distance as the berthing yard of the incoming port ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure BDA0003771447280000185
Figure BDA0003771447280000186
q＝1,…m,
i＝0,1,…r,
Figure BDA0003771447280000187
wherein L is min[p,q] Represents the shortest distance, L, from the berth p to the yard q [p,q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the i-th inflection point on the path from the berth p to the yard q.
According to the method and the system, a big data platform is constructed by using Hadoop, hive, spark and other big data technologies, the storage and calculation problems of data generated by a port production system and a device system are effectively normalized, and the problem of large data volume caused by data redundancy is solved; meanwhile, the ETL processing is carried out on the data by adopting a data warehouse technology, the use efficiency of temporary data is effectively improved, and the calculation time of temporary intermediate results is saved. The invention further provides a new scheduling algorithm according to the big data platform, and calculates the comprehensive index efficiency of the port according to the scheduling data, thereby providing multi-angle analysis for the port and promoting the development of the port.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims (10)

1. A port data comprehensive application method based on a big data platform is characterized by comprising the following steps:
building a big data platform, and constructing a structure table of a data warehouse according to original data of a port production system and an equipment material system and extracting data;
performing ETL processing on the extracted data;
carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data;
carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data;
and calculating the comprehensive index efficiency of the port according to the scheduling data.
2. The port data comprehensive application method based on the big data platform as claimed in claim 1, wherein the ETL processed data are stored in an ODS layer, a DWD layer and a DWS layer in sequence;
the ODS layer stores port operation ticket information, ship information and port original data of wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculation according to the DWD layer data shipunloa Ship berthing time T shipberth And device age T machiine 。
3. The big data platform based port data comprehensive application method as claimed in claim 2, wherein the data cleaning rule comprises:
static data cleaning rules and dynamic data cleaning rules;
the dynamic data cleaning rule comprises a missing value processing rule, a repeated value processing rule and an abnormal value processing rule; the missing value processing rule is a similar mean interpolation method filling missing values, and the repeated value processing rule is comparison repeated data deduplication;
the abnormal value processing rule is according to the ship loading and unloading time T shipunload And berthing time T of ship shipberth The exception handling is carried out according to the size of the ratio k, and the exception handling comprises the following steps:
when k is more than 1, correcting the berthing time T of the ship shipberth ；
When k is>>1, correcting T by adopting a repeated value processing rule shipunload Or to correct the departure time of the ship.
4. The port data comprehensive application method based on the big data platform as claimed in claim 2, wherein the process of acquiring the scheduling data is as follows:
acquiring ship information and berth information of a port coming within a planning date, and carrying out berth scheduling;
determining a storage yard for storing goods according to the result of the berth scheduling and the storage yard information;
the berthing scheduling process is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure FDA0003771447270000021
wherein,
Figure FDA0003771447270000022
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure FDA0003771447270000023
Figure FDA0003771447270000024
wherein,
Figure FDA0003771447270000025
indicating the distance, L, of the vessel to the kth berth min[ship，berth] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure FDA0003771447270000026
wherein,
Figure FDA0003771447270000027
representing the berthing time of the ith ship;
when a plurality of berths of the port are occupied and have no empty berths, scheduling the ship with short berthing time to the closest berth operation of the port according to the shortest time and shortest distance priority scheduling principle;
Figure FDA0003771447270000028
Figure FDA0003771447270000029
the yard is determined as follows:
when the berthing position of the incoming port ship is determined, calculating the distance from the berthing position to any one yard, and determining the yard with the shortest berthing distance as the berthing yard of the incoming port ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure FDA00037714472700000210
Figure FDA00037714472700000211
Figure FDA00037714472700000212
wherein L is min[p，q] Represents the shortest distance, L, from the berth p to the yard q [p，q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the i-th inflection point on the path from the berth p to the yard q.
5. The port data comprehensive application method based on the big data platform as claimed in claim 2, wherein the port comprehensive index efficiency is calculated as follows:
according to ship loading and unloading time T shipunload And the berthing time T of the ship shipberth Calculating the handling efficiency E of a port unload ：
E unload ＝T shipunload /(T shipberth -T invalid -T shipbreak )
Wherein, T invalid Due to the dead time, T, caused by natural factors shipbreak Due to maintenance time resulting from vessel failure;
according to the time T of use of the equipment machine Computing device efficiency E machi ：
E machine ＝T machineactual //(T machine -T machinebrea )
Wherein, T machineactua Is the actual working time, T, of the equipment machinebrea Maintenance time due to equipment failure;
calculating the efficiency E of the loading and unloading personnel according to the information of the personnel of the loading and unloading working group in one loading and unloading work of the port ship oneworker ：
E worker ＝W cargo /Work nums *(T workend -T workstart -T rest )
Wherein, W cargo Weight indicating loading or unloading of goods in one-time operation ticket, work nums Indicating the number of workers involved in the unloading work, T, in a job ticket workend Means for indicating the loading or unloading of workersEnd time of discharge, T workstart Indicating the loading and unloading start time, T, of the loader rest Representing the rest time of the handler;
calculating ship loading and unloading personnel efficiency E according to ship loading and unloading work ticket information shipworker ：
Figure FDA0003771447270000031
Wherein n is the loading and unloading times or the number of operation tickets for completing the loading and unloading work of one ship.
6. A port data comprehensive application system based on a big data platform is characterized by comprising:
the building module is used for building a big data platform, building a structure table of a data warehouse according to original data of a port production system and an equipment material system, and extracting data;
the data processing module is used for carrying out ETL processing on the extracted data;
the data cleaning module is used for carrying out secondary data cleaning on the ETL-processed data according to a preset cleaning rule to obtain cleaned port data;
the first calculation module is used for carrying out scheduling calculation on the process of the ship entering and exiting the port according to the port data to obtain scheduling data;
and the second calculation module is used for calculating the comprehensive index efficiency of the port according to the scheduling data.
7. The port data comprehensive application system based on the big data platform as claimed in claim 6, wherein the data processed by the data processing module are stored in an ODS layer, a DWD layer and a DWS layer in sequence;
the ODS layer stores port operation ticket information, ship information and port original data of wharf information, the DWD layer stores formatted structured data and semi-structured data of the ODS layer, and the DWS layer stores ship loading and unloading time T obtained by calculation according to the DWD layer data shipunload And berthing time T of ship shipberth And device age T machine 。
8. The big data platform based port data comprehensive application system as claimed in claim 7, wherein the cleaning rule of the data cleaning module for data cleaning comprises:
static data cleaning rules and dynamic data cleaning rules;
the dynamic data cleaning rule comprises a missing value processing rule, a repeated value processing rule and an abnormal value processing rule; the missing value processing rule is a similar mean interpolation method filling missing values, and the repeated value processing rule is comparison repeated data deduplication;
the abnormal value processing rule is according to the ship loading and unloading time T shipunloa And berthing time T of ship shipberth The exception handling is carried out according to the size of the ratio k, and the exception handling comprises the following steps:
when k is more than 1, correcting the berthing time T of the ship shipberth ；
When k is more than 1, the repeated value processing rule is adopted to correct T shipunload Or to correct the ship departure time.
9. The big data platform-based port data comprehensive application system as claimed in claim 7, wherein the execution unit of the first calculation module for calculating the obtained scheduling data comprises:
the system comprises an acquisition unit, a storage unit and a control unit, wherein the acquisition unit is used for acquiring ship information and berth information of a port coming within a planned date;
the berth scheduling unit is used for scheduling the berth according to the information acquired by the acquisition unit;
the determining unit is used for determining a storage yard for storing goods according to the berth scheduling result and the storage yard information;
the execution process of the berthage scheduling unit is as follows:
when the port has an empty berth and no waiting ship exists in front of the port when the ship arrives, dispatching the ship coming to the port to directly enter the empty berth;
when the port has an empty berth and a waiting ship is in front when the ship arrives, scheduling according to the arrival time priority principle, namely:
Figure FDA0003771447270000051
wherein,
Figure FDA0003771447270000052
representing the arrival time of the i-th ship;
when the port has a plurality of empty berths, calculating the distance from the ship to each empty berth, and dispatching the ship from the first port to the nearest berth according to the port time priority principle, namely:
Figure FDA0003771447270000053
Figure FDA0003771447270000054
wherein,
Figure FDA0003771447270000055
indicating the distance, L, of the vessel to the kth berth min[ship，berth] Representing the distance from the ship to the shortest path berth;
when only one berth is occupied and no empty berth exists in the port and the ship waits to berth in front when the ship arrives at the port, scheduling the ship with short berthing time to berth preferentially according to the shortest time priority service principle;
Figure FDA0003771447270000056
wherein,
Figure FDA0003771447270000057
representing the berthing time of the ith ship;
when a plurality of berths of the port are occupied and have no empty berths, scheduling the ship with short berthing time to the closest berth operation of the port according to the shortest time and shortest distance priority scheduling principle;
Figure FDA0003771447270000058
Figure FDA0003771447270000059
the yard is determined as follows:
after determining the berthing position of the incoming port ship, calculating the distance from the berthing position to any yard, and determining the yard with the shortest berthing position distance as the berthing yard of the incoming port ship; shortest distance L min The calculation is as follows:
f(2)＝L min *y yard
Figure FDA00037714472700000510
Figure FDA00037714472700000511
Figure FDA00037714472700000512
wherein L is min[p，q] Represents the shortest distance, L, from the berth p to the yard q [p，q] Representing any path from the berth p to the storage yard q, N representing the number of paths from the berth p to the storage yard q, i representing the total number of possible inflection points of a nonlinear path from the berth p to the storage yard q, and k i The representation is the ith from the berth p to the path of the yard qAnd (6) inflection points.
10. An electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the big data platform-based port data comprehensive application method according to any one of claims 1 to 5.
CN202210902709.5A 2022-07-29 2022-07-29 Port data comprehensive application method and system based on big data platform Pending CN115328983A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210902709.5A CN115328983A (en) 2022-07-29 2022-07-29 Port data comprehensive application method and system based on big data platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210902709.5A CN115328983A (en) 2022-07-29 2022-07-29 Port data comprehensive application method and system based on big data platform

Publications (1)

Publication Number Publication Date
CN115328983A true CN115328983A (en) 2022-11-11

Family

ID=83919680

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210902709.5A Pending CN115328983A (en) 2022-07-29 2022-07-29 Port data comprehensive application method and system based on big data platform

Country Status (1)

Country Link
CN (1) CN115328983A (en)

Similar Documents

Publication Publication Date Title
Dotoli et al. A decision support system for optimizing operations at intermodal railroad terminals
US20080140597A1 (en) System and method suitable for optimizing linehaul operations
CN113077103B (en) Transportation network planning method and device
US20100262450A1 (en) Method and system for mobile asset management
CN101493911A (en) Virtual inventory management system for harbor equipment manufacturing enterprise
Bititci et al. Driving continuous improvement
Alcalde et al. Optimal space for storage yard considering yard inventory forecasts and terminal performance
CN116569118A (en) Information management method and device
CN115936588A (en) Data simulation method for cross-border e-commerce supply chain
CN106548257B (en) Method for making quota of spare parts based on decision tree model
CN115049325A (en) Intelligent scheduling method, system, equipment and storage medium for container port transshipment
Gaete et al. A dwell time-based container positioning decision support system at a port terminal
US20120323641A1 (en) Terminal performance analytical computer system
CN115328983A (en) Port data comprehensive application method and system based on big data platform
Mazouz et al. Container–terminal–gate–system optimization
CN114524004B (en) Locomotive operation statistical method, device, equipment and storage medium
Dotolia et al. A train load planning optimization model for intermodal freight transport terminals: A case study
CN106203914A (en) A kind of boats and ships storage management method and system
CN115293443A (en) Method, system and medium for predicting loading and unloading operation time of bridge crane and container ship
JP7402986B2 (en) Container storage planning device, container storage planning system, and container storage planning method
CN114897471A (en) Logistics transportation management method, electronic equipment and storage medium
CN115169658A (en) Inventory consumption prediction method, system and storage medium based on NPL and knowledge graph
Razouk et al. Adapted Bin-Packing algorithm for the yard optimization problem
CN114066055A (en) Method, device and server for predicting late-stage approach of vehicle in logistics transportation
Marzuki et al. Factor Influence Of Container Loading And Unloading As Productivity Support On Mirah Terminal

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination