CN117544620A - Automatic processing method for container terminal message based on distributed task scheduling platform - Google Patents
Automatic processing method for container terminal message based on distributed task scheduling platform Download PDFInfo
- Publication number
- CN117544620A CN117544620A CN202310918978.5A CN202310918978A CN117544620A CN 117544620 A CN117544620 A CN 117544620A CN 202310918978 A CN202310918978 A CN 202310918978A CN 117544620 A CN117544620 A CN 117544620A
- Authority
- CN
- China
- Prior art keywords
- message
- task
- configuring
- container terminal
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 30
- 230000003993 interaction Effects 0.000 claims description 5
- 230000001419 dependent effect Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 7
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000011084 recovery Methods 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 2
- 101100509468 Arabidopsis thaliana JASON gene Proteins 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0876—Aspects of the degree of configuration automation
- H04L41/0886—Fully automatic configuration
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Automation & Control Theory (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses an automatic processing method of a container terminal message based on a distributed task scheduling platform, which is based on a distributed and multithreaded large-scale data batch processing technology, and is characterized in that the processing of a message task execution period, failure warning, recovery and the like is flexibly configured through a self-defined message format and automatic generation of the message task, so that the self-defined configuration and automatic parallel execution of various messages are realized, the rapid receiving and processing of messages of different types and formats, which are interacted by a container terminal, a shipcompany, a customs, a client and a logistics station, are supported, the problem of how high the number of the messages of the terminal is concurrent is solved, and the early warning and the rapid tracking of the problems can be timely carried out in the message processing process, thereby improving the processing efficiency and the accuracy of the container terminal message.
Description
Technical Field
The invention belongs to the technical field of container terminals, and particularly relates to an automatic processing method of a container terminal message based on a distributed task scheduling platform.
Background
With the rapid development of global trade and the continuous expansion of international logistics industry, container terminals are an important component of logistics industry, and have become one of important supporting facilities of global logistics system.
In the container terminal business, the container message is an important information communication mode, and throughout the logistics management process of the whole terminal, the message types comprise the message types such as ship planning, cargo dynamic state, ship loading state, release supervision state, loading and unloading ship in the terminal, entrance and exit, expense bill and the like, and the interaction and message transceiving processing which are required to meet the different time requirements of shipcompanies, customs, clients, logistics stations and the like are required.
At present, a single-task mechanism single line Cheng Chuanhang processing mode based on the traditional is generally adopted for processing the container terminal message, and once the size of the mechanism message reaches a certain degree, the problems of low task processing efficiency, poor expandability and the like are encountered. In addition, if the message is sent and received in a timing polling mode, when different types of messages and different message sending and receiving times are faced, the problem of low processing efficiency is easy to occur, and the conditions of long message processing time, message congestion and the like can occur. Currently, many terminal enterprises adopt commercialized software to solve the problem of automatic message processing, the commercialized software is usually based on a traditional single-task processing mechanism, and needs to perform a great deal of configuration and programming work, and in addition, if new message types need to be added or message transceiving time needs to be adjusted, the commercialized software also needs to perform corresponding configuration and programming adjustment, so that the information interaction work of the container terminal becomes more complicated and difficult.
Disclosure of Invention
The invention aims to provide an automatic processing method of a container terminal message based on a distributed task scheduling platform, which solves the technical problems of low processing efficiency and accuracy of the container terminal message caused by the problems of multiple message types, different message format standards, real-time dynamic message content, need of receiving and transmitting according to a certain time rule and the like on the container code head based on a distributed automatic processing means of the distributed message which is executed in parallel by a cluster scheduling center and an executor.
The invention is realized by adopting the following technical scheme:
the invention provides an automatic processing method of a container terminal message based on a distributed task scheduling platform, wherein the distributed task scheduling platform comprises a scheduler and an executor; the method comprises the following steps:
s1, configuring a message according to a message interaction object, a receiving and transmitting attribute and a service category to which the message belongs;
s2, carrying out custom configuration on the message;
s3, carrying out association configuration on the message configuration and the message task, wherein the method comprises the following steps: generating a message task by configuring a plurality of messages of the same type with the same execution period, respectively generating the message task by configuring the messages of the same type with different execution periods, and generating different message tasks by configuring the messages of different types; the message tasks comprise timing tasks, dependent tasks and event triggering tasks;
s4, configuring an executor for the message task;
s5, the dispatcher dispatches the message task based on asynchronous dispatching, asynchronous execution and asynchronous callback, generates a dispatching instruction and sends the dispatching instruction to an executor of the configured message task; during the execution of the scheduling instruction by the executor, the scheduler performs execution management and log tracking on the message task;
s6, the executor executes the related message task of the scheduling instruction in a multithreading mode, and returns information to the scheduler after the execution is finished.
In some embodiments of the present invention, the configuration message includes: configuring basic message information including message name, message storage path, message type, message receiver, receiving mode and last execution time; configuring message row information comprising a message head, a message body, a circulation head, a circulation body, a circulation tail and a message tail; configuring the message line type and separator of each message line; the fields contained in each message row are configured, including field names, field types, field lengths, field precision and field sequences.
In some embodiments of the present invention, the method further includes a step of customizing the message, including: defining basic message information including message name, message type and message grouping; defining message line information according to the message type, wherein the message line information comprises a message head, a message body, a circulation head, a circulation body, a circulation tail and a message tail; defining the content of the message rows, wherein the content comprises fields, data and separators contained in each message row; the data configuration comprises a table from which the data is fetched, conditions and parameters for fetching the data, a field name, a field length, a field precision, a custom value and a field sequence.
In some embodiments of the present invention, the method further includes the steps of adding, deleting, modifying and querying the message type.
In some embodiments of the present invention, the method further comprises registering the executives with the dispatch center for unified management by the scheduler, so that the dispatch center can perceive all the executives online.
In some embodiments of the present invention, the method further includes a step of configuring a number of failed retries of the message task.
In some embodiments of the present invention, the method further includes the step of configuring the frequency, time and data range of execution of the message task.
Compared with the prior art, the invention has the advantages and positive effects that: the automatic processing method of the container terminal message based on the distributed task scheduling platform is based on a distributed and multithreaded large-scale data batch processing technology, and processes such as self-defining message format, automatically generating message tasks, flexibly configuring message task execution period, failure alarm and recovery are realized, the self-defining configuration and automatic parallel execution of various messages are realized, the rapid receiving and processing of the messages of different types and formats, which are interacted by a container terminal, a shipcompany, a customs, a client and a logistics station, are supported, the problem of concurrency of the quantity of the messages of the terminal is solved, and the timely early warning and the rapid tracking of the problems can be realized in the message processing process, thereby improving the processing efficiency and the accuracy of the container terminal message.
Other features and advantages of the present invention will become more apparent from the following detailed description of embodiments of the present invention, which is to be read in connection with the accompanying drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention, without limitation to the invention. It is evident that the figures in the following description are only examples, from which other figures can be obtained, without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of processing steps of an automatic processing method for a container terminal message based on a distributed task scheduling platform.
It should be noted that these drawings and the written description are not intended to limit the scope of the inventive concept in any way, but to illustrate the inventive concept to those skilled in the art by referring to the specific embodiments.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions in the embodiments will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention, and the following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
The invention provides a container terminal message automatic processing method based on a distributed task scheduling platform, which is applied to the distributed task scheduling platform, wherein the distributed task scheduling platform is established based on large-scale data batch processing technologies such as a distributed technology, a multithreading technology, a cache technology and the like and comprises a scheduler and an executor, wherein the scheduler is used for uniformly distributing message tasks, generating scheduling instructions aiming at the message tasks and issuing the scheduling instructions to the executor; and the executor executes specific task operation after receiving the scheduling instruction, and returns information to the scheduler after the execution is finished. In task execution, the scheduler also manages tasks in real time and logs.
In the invention, the scheduler adopts asynchronous design, including asynchronous scheduling, asynchronous operation, asynchronous callback and other processing modes to execute task scheduling, and effectively carries out flow peak clipping on dense scheduling, thereby realizing large-scale batch execution of messages, avoiding the problems of low efficiency caused by that each type of message corresponds to one application program and only one message can be processed at the same time in the traditional single line Cheng Chuanhang processing mode, avoiding the problems of program occurrence influencing the execution of other messages and the like.
Based on the above, as shown in fig. 1, the method for automatically processing the container terminal message based on the distributed task scheduling platform provided by the invention comprises the following steps:
s1: and configuring the message according to the message interaction object, the receiving and transmitting attribute and the service category to which the message belongs.
In the embodiment of the invention, service personnel flexibly configure fixed format messages and dynamic SQL format messages through a message configuration page provided by a distributed task scheduling platform, including but not limited to:
a) The basic information of the configuration message comprises a message name, a message access path, a message type, a message receiver, a receiving mode, the last execution time and the like.
b) Configuring message line information, including message line information such as message header, message body, message tail, circulation header, circulation body, circulation tail and the like, and configuring message line types and separators; and when the message is analyzed later, the message can be automatically processed according to the configuration of the message row.
c) Configuring fields in a message row, wherein the fields comprise field names, field types, field lengths, field precision, field sequences and the like; when the message is generated and analyzed, the message is analyzed or the message content is generated according to the message row and the configuration information of the fields in the message row.
S2: and carrying out custom configuration on the message.
In the embodiment of the invention, service personnel customize the message through a message custom page provided by a distributed task scheduling platform, so as to achieve the purpose of generating the message without adjusting a program, including but not limited to:
a) Defining basic message information including message name, message type and message grouping; the message packets here are generated for example according to different vessels into different message files or message packets.
b) Defining message line information according to the message type, wherein the message line information comprises a message head, a message body, a circulation head, a circulation body, a circulation tail and a message tail;
c) Defining the content of the message rows, wherein the content comprises fields, data and separators contained in each message row; the data configuration comprises a table from which the data is fetched, conditions and parameters for fetching the data, a field name, a field length, a field precision, a custom value and a field sequence. The data may be taken from a plurality of data sources.
In the embodiment of the invention, the distributed task scheduling platform provides a management interface to manage the message types, and processes such as adding, deleting, modifying and inquiring the message types are implemented.
S3: and implementing message task association on the message configuration.
In the embodiment of the invention, the distributed task scheduling platform correlates the message configuration implemented in the S1 and the S2 with the message task, and comprises the following steps:
a) And configuring a plurality of messages of the same type with the execution period to generate a message task.
b) And respectively generating message tasks by configuring the same type of messages with different execution periods.
c) Different message tasks are generated by configuring different types of messages.
The above execution cycle refers to the number of times of execution within a specified period of time or execution within a certain period of time.
In the embodiment of the invention, the message task comprises, but is not limited to, a timing task, a dependent task and an event triggering task. The dependent tasks are subtasks defined by pointers to the parent tasks, and the subtasks are started depending on the parent tasks, but have own configuration.
S4: and configuring an executor for the message task.
In the embodiment of the invention, a task configuration page is provided, and parameters for configuring the execution of the message task are provided, including but not limited to which executor the message task is allocated to, the frequency of executing the message task, the time for executing the message task, the data range and the like, so that the scheduler performs unified scheduling on the message task based on the configuration information.
The invention enables the dispatching and the task to be decoupled in a mode of receiving the dispatching instruction and executing the dispatching instruction through the unified dispatching of the dispatcher, and the dispatching performance is not limited by the execution of the message task, thereby improving the usability and the stability of the message processing system.
In the embodiment of the invention, cluster information (namely, scheduling instructions corresponding to all message tasks) of the message task execution request is maintained, the executors need to be registered in a scheduling center to be uniformly managed by a scheduler, and the scheduling center can sense all the executors on line; the executor receives the dispatching instruction of the dispatcher, executes the message task, terminates the message task, requests the log and the like based on the dispatching instruction.
S5: the dispatcher dispatches the message task based on asynchronous dispatching, asynchronous execution and asynchronous callback, generates a dispatching instruction and sends the dispatching instruction to an executor where the message task is configured; and during the execution of the scheduling instruction by the executor, the scheduler performs execution management and log tracking on the message task.
In the embodiment of the invention, a scheduler implements static scheduling and/or dynamic scheduling; in the static scheduling, scheduling is carried out according to the attribute of the message task; in the dynamic scheduling, selecting available nodes in real time according to the current node condition and the task condition to perform task scheduling; and monitoring, quickly starting and stopping the task state.
S6: the executor executes the relevant message task of the dispatching instruction in a multithreading mode and returns information to the dispatcher after the execution is finished.
The executor dynamically generates XLS, JASON, TXT, XML formats with message header and message content according to message configuration, supports various messages of default separator segmentation and fixed length splicing, further analyzes the message and creates multiple threads to execute message tasks in a parallel mode.
In the embodiment of the invention, the executor supports large-scale concurrent task processing in a distributed and multithreaded mode, and the analysis result of the message is written into a database or a production receipt message.
S7: and monitoring and early warning are carried out on the message task.
And during the execution of the message task by the executor, monitoring the message task, including but not limited to task flow, task state, error log and the like, and carrying out corresponding alarm and processing, wherein the task failure retry times can be configured, and when the task fails, the retry is actively carried out according to the preset failure retry times.
Compared with the traditional mode that each message needs to be read by a separate writing program or generated, the method for automatically processing the messages of the container terminal based on the distributed task scheduling platform has the advantages of strong message customization, flexible message configuration and remarkable reduction of the message maintenance workload; the invention utilizes the distributed scheduling platform, can automatically process various messages in batch and concurrently, realizes the load balance of message tasks by different executors, has traceable and controllable process, overcomes the problems of low efficiency and easy error of traditional manual processing of the messages, and obviously improves the message processing efficiency and accuracy; in view of the configurable and extensible mechanism provided by the invention, the invention can be suitable for different wharf business requirements and has good compatibility and adaptability.
It should be noted that the above description is not intended to limit the invention, but rather the invention is not limited to the above examples, and that variations, modifications, additions or substitutions within the spirit and scope of the invention will be within the scope of the invention.
Claims (7)
1. The method for automatically processing the container terminal message based on the distributed task scheduling platform comprises a scheduler and an executor;
characterized in that the method comprises the steps of:
s1, configuring a message according to a message interaction object, a receiving and transmitting attribute and a service category to which the message belongs;
s2, carrying out custom configuration on the message;
s3, carrying out association configuration on the message configuration and the message task, wherein the method comprises the following steps: generating a message task by configuring a plurality of messages of the same type with the same execution period, respectively generating the message task by configuring the messages of the same type with different execution periods, and generating different message tasks by configuring the messages of different types; the message tasks comprise timing tasks, dependent tasks and event triggering tasks;
s4, configuring an executor for the message task;
s5, the dispatcher dispatches the message task based on asynchronous dispatching, asynchronous execution and asynchronous callback, generates a dispatching instruction and sends the dispatching instruction to an executor of the configured message task; during the execution of the scheduling instruction by the executor, the scheduler performs execution management and log tracking on the message task;
s6, the executor executes the related message task of the scheduling instruction in a multithreading mode, and returns information to the scheduler after the execution is finished.
2. The automatic processing method of container terminal messages based on a distributed task scheduling platform according to claim 1, wherein configuring the messages comprises:
configuring basic message information including message name, message storage path, message type, message receiver, receiving mode and last execution time;
configuring message row information comprising a message head, a message body, a circulation head, a circulation body, a circulation tail and a message tail;
configuring the message line type and separator of each message line;
the fields contained in each message row are configured, including field names, field types, field lengths, field precision and field sequences.
3. The method for automatically processing the container terminal message based on the distributed task scheduling platform according to claim 1, wherein the method further comprises the step of customizing the message, and the method comprises the following steps:
defining basic message information including message name, message type and message grouping;
defining message line information according to the message type, wherein the message line information comprises a message head, a message body, a circulation head, a circulation body, a circulation tail and a message tail;
defining the content of the message rows, wherein the content comprises fields, data and separators contained in each message row; the data configuration comprises a table from which the data is fetched, conditions and parameters for fetching the data, a field name, a field length, a field precision, a custom value and a field sequence.
4. The method for automatically processing a container terminal message based on a distributed task scheduling platform according to claim 1, wherein the method further comprises the steps of adding, deleting, modifying and inquiring the message type.
5. The method for automatically processing the container terminal message based on the distributed task scheduling platform according to claim 1, wherein the method further comprises registering the executors to the scheduling center to be uniformly managed by the scheduler so that the scheduling center can perceive all the executors on line.
6. The method for automatically processing a container terminal message based on a distributed task scheduling platform according to claim 1, wherein the method further comprises the step of configuring the number of failed retries of the message task.
7. The method for automatically processing a container terminal message based on a distributed task scheduling platform according to claim 1, wherein the method further comprises the step of configuring the frequency, time and data range of the execution of the message task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310918978.5A CN117544620A (en) | 2023-07-25 | 2023-07-25 | Automatic processing method for container terminal message based on distributed task scheduling platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310918978.5A CN117544620A (en) | 2023-07-25 | 2023-07-25 | Automatic processing method for container terminal message based on distributed task scheduling platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117544620A true CN117544620A (en) | 2024-02-09 |
Family
ID=89781309
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310918978.5A Pending CN117544620A (en) | 2023-07-25 | 2023-07-25 | Automatic processing method for container terminal message based on distributed task scheduling platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117544620A (en) |
-
2023
- 2023-07-25 CN CN202310918978.5A patent/CN117544620A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111737329A (en) | Unified data acquisition platform for rail transit | |
| EP0422310A1 (en) | Distributed mechanism for the fast scheduling of shared objects | |
| CN110611707B (en) | Task scheduling method and device | |
| CN102467412B (en) | Method, device and business system for processing operation request | |
| US7478130B2 (en) | Message processing apparatus, method and program | |
| CN114253748B (en) | Message processing system and message processing method | |
| CN110795254A (en) | Method for processing high-concurrency IO based on PHP | |
| CN114490112B (en) | Message processing method, device and system | |
| CN111405130B (en) | Voice interaction system and method | |
| CN108008950A (en) | The implementation method and device of a kind of user interface updating | |
| CN114116259A (en) | Message queue-based real-time message pushing method, system, device and medium | |
| CN113658351A (en) | Product production method and device, electronic equipment and storage medium | |
| CN117544620A (en) | Automatic processing method for container terminal message based on distributed task scheduling platform | |
| CN108509257B (en) | Message processing method and device based on multithreading | |
| CN112559202A (en) | Aircraft application software communication method based on embedded real-time operating system | |
| CN116362213A (en) | Method, device, processor and storage medium for generating report file | |
| CN114328722A (en) | Data synchronization method and device supporting multiple data sources and computer equipment | |
| CN115617480A (en) | Task scheduling method, device and system and storage medium | |
| CN115150244A (en) | Adaptation system and method for managing external data interaction with a funding system | |
| CN111176799A (en) | Service scheduling method, equipment and storage medium | |
| CN114840320A (en) | Method for automatically producing and consuming timed tasks based on RabbitMQ | |
| CN114090210A (en) | Service calling implementation method and device | |
| CN115379012B (en) | Industrial interconnection platform message queue deployment method and device based on identification analysis | |
| CN113032110A (en) | High-availability task scheduling method based on distributed peer-to-peer architecture design | |
| CN117311936A (en) | LMAX-based production consumption model multithreading batch processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |