CN114020535B - Backtracking analysis system based on shipping data snapshot - Google Patents
Backtracking analysis system based on shipping data snapshot Download PDFInfo
- Publication number
- CN114020535B CN114020535B CN202111303459.5A CN202111303459A CN114020535B CN 114020535 B CN114020535 B CN 114020535B CN 202111303459 A CN202111303459 A CN 202111303459A CN 114020535 B CN114020535 B CN 114020535B
- Authority
- CN
- China
- Prior art keywords
- data
- information
- snapshot
- ais
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 101
- 230000004931 aggregating effect Effects 0.000 claims abstract description 3
- 230000000875 corresponding effect Effects 0.000 claims description 86
- 238000000034 method Methods 0.000 claims description 34
- 238000003860 storage Methods 0.000 claims description 28
- 238000004364 calculation method Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 17
- 238000012216 screening Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 8
- 230000003203 everyday effect Effects 0.000 claims description 4
- 238000004873 anchoring Methods 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 3
- 230000002596 correlated effect Effects 0.000 claims description 2
- 235000019580 granularity Nutrition 0.000 description 27
- 238000010586 diagram Methods 0.000 description 21
- 238000007726 management method Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000013500 data storage Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000013075 data extraction Methods 0.000 description 3
- 238000011068 loading method Methods 0.000 description 3
- 230000008929 regeneration Effects 0.000 description 3
- 238000011069 regeneration method Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 238000013501 data transformation Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- PMHURSZHKKJGBM-UHFFFAOYSA-N isoxaben Chemical compound O1N=C(C(C)(CC)CC)C=C1NC(=O)C1=C(OC)C=CC=C1OC PMHURSZHKKJGBM-UHFFFAOYSA-N 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003245 coal Substances 0.000 description 1
- 230000006854 communication Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Library & Information Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the field of shipping informatization, in particular to a backtracking analysis system based on shipping data snapshot, which comprises the following steps: the storage center is used for storing data such as data snapshot; the snapshot generation module is used for generating data snapshots according to snapshot time granularity aiming at carefully chosen snapshot samples; the snapshot searching module is used for directly correlating ship data snapshots and rapidly searching out the latest state data of the designated slicing time; and the business analysis module is used for forming trend data taking the time slices as time axes based on the data snapshots, defining the dynamically updated characteristic data snapshots into attribute groups according to categories, counting the characteristic data of each time slice, and aggregating the time slices, namely forming cross-period trend statistics, so as to perform next-step demand analysis. According to the invention, by combining with shipping data characteristics, snapshot samples are selected and snapshot time granularity is customized, and a more flexible and fine-granularity backtracking analysis system is constructed based on data snapshots generated by full historical data.
Description
Technical Field
The invention relates to the technical field of shipping informatization, in particular to a backtracking analysis system based on a shipping data snapshot.
Background
Along with the wide application and rapid development of technologies such as big data and cloud computing in various industries, the informatization of the shipping field is gradually developed and strengthened, and especially the gradual increase of shipping information data is particularly important for the storage and analysis utilization of shipping big data. The shipping history data snapshot is mainly used for backtracking the history state or analyzing the history trend, and is an extremely important technology.
The storage network industry association defines the snapshot as: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (copy start point in time), the snapshot may be a copy of the data it represents, or may be a replica of the data. The snapshot has the functions of mainly carrying out online data backup and recovery, carrying out rapid data recovery when the storage equipment has application faults or file damages, rolling back data to a state of a qualified available time point, providing another data access channel for a storage user, and carrying out online application processing on metadata, wherein the user can access snapshot data, can also carry out testing and other works by utilizing the snapshot, and all storage systems, regardless of high, medium and low ends, become an indispensable function as long as being applied to the online system.
Currently, snapshot data based on the existing backtracking analysis system or application system mainly depends on data obtained by a traditional data snapshot generation method, such as the representation of patent CN111552437a, and such snapshots are mainly used in the field of safe operation and maintenance, such as rapid restoration or repair of a system environment. Another part is the snapshot technology for data, represented by patent CN102096613B, which is also focused on the data operation and maintenance level to recover the data as a main purpose and function. The generation of such snapshots requires predefining, such as the data range, time granularity, execution plan, etc. of the snapshots, once the generation of the snapshots is missed, the snapshot generation is difficult to compensate, so that a backtracking analysis system based on the data snapshots is difficult to achieve more accurate and precise analysis and historical backtracking, especially in the situation facing the requirement of continuously updating iteration of a shipping data system.
In addition, in the field of shipping, the existing analysis systems are mainly focused on management, search and supervision of ships; for example, CN110135776a, a shipping information management method, system and computer storage medium make the updating of shipping information timely and convenient, and at the same time, it is convenient for the supervision departments such as port and navigation to supervise. Or CN205508317U, can effectively display port shipping information. Or, for example, CN104112172a, can better manage and schedule shipping without being affected by bad weather. Further, as in CN102103802a, a passenger ship shipping management system and a control method thereof using AIS (Automatic IdentificationSystem, automatic ship identification system) for bi-directional communication are disclosed. The analysis system in the prior art is mainly applied to analysis based on data which is reported by AIS periodically, and is mainly applied to management, positioning, navigation, supervision and the like of ships, and an accurate analysis backtracking system is lacking to carry out further analysis backtracking of big data of the ships so as to provide more analysis application functions for shipping informatization fields or other users.
In the face of the important characteristic that data applied to a shipping platform or a related analysis system is different from data in other existing fields, most of the data are intermediate calculation results, when calculation logic is changed, the data need to be recalculated, data snapshots need to be updated based on the latest data, and the analysis system of the data snapshots obtained based on the existing snapshot generation method or technology is difficult to meet the analysis requirement of the shipping data.
Therefore, how to meet the background requirement of the snapshot of the newly generated data based on the latest calculation result in the technical field, and how to quickly respond, accurately analyze and trace back the shipping snapshot data based on the change to help meet specific analysis and application requirements of shipping companies or related companies and users is also a problem to be solved in the technical field of shipping informatization.
Disclosure of Invention
The invention provides a backtracking analysis system based on the shipping data snapshot, which aims to meet the background requirement of the snapshot based on the latest calculation result regeneration data in the technical field, and the shipping snapshot data based on the change can be subjected to quick response, accurate analysis and backtracking to help meet specific analysis and application requirements of shipping companies or related companies and users.
The invention provides the following scheme:
the invention provides a backtracking analysis system based on shipping data snapshot, which is characterized by comprising the following steps:
the storage center is used for storing data snapshot formed based on snapshot sample data and storing data generated in the running process of the system;
the snapshot generation module is used for generating shipping data snapshots according to a certain snapshot time granularity aiming at different snapshot samples such as the latest AIS information, the AIS total historical table, the avigation section information total historical table and the like which are fixedly reported, and storing the shipping data snapshots in a storage table form in the storage center; the selection of the snapshot time granularity is determined according to the cost of shipping and the refined satisfaction of shipping services;
the snapshot searching module is used for providing a query searching function based on the latest AIS information in the time dimension; the method comprises the steps of acquiring a data snapshot of a ship, and acquiring the latest state data of the ship at a specified slicing time, wherein the data snapshot is used for switching the association relation between a latest navigation section information table and ship basic information into the association between the latest navigation section information snapshot table and the ship basic information based on the data snapshot, and directly associating the data snapshot of the ship when different ship states at different periods are searched for, so that the latest state data of the ship at the specified slicing time is quickly searched for;
The business analysis module is used for data analysis service of ships, based on the data snapshot, rapidly forming trend data taking corresponding time slices as time axes, defining the required feature data which are continuously and dynamically updated into attribute groups according to categories, counting the number of the ships of each corresponding time slice based on the corresponding time slices and the attribute groups by utilizing the classified data snapshot, aggregating the corresponding time slices, namely forming cross-period trend statistics, and assisting a user to perform next-step demand analysis based on the trend statistics.
Further, the snapshot generating module further includes a snapshot sample processing unit, including:
the snapshot sample data sub-module is used for obtaining latest AIS information of the fixed report, wherein the latest AIS information defines key information including AIS reporting time, ship position longitude and latitude, ground speed, ship heading, AIS reporting time and the like; all AIS information table history data are collected to form an AIS total history table;
the dynamic updating latest avigation information sub-module is used for updating latest avigation information based on the acquired latest AIS information of the ship, wherein the latest avigation information consists of the latest AIS information reported fixedly and avigation information which dynamically changes along with calculation logic; all the latest navigation segment information historical data are collected to form a navigation segment information total historical table;
The sub-module for acquiring other important classification information is used for acquiring historical data based on the navigation section information and analyzing and summarizing additional classification attributes such as trade, operation, commercial value and the like of the ship, and acquiring internal and external trade operation states and the like of the ship.
Further, the dead leg information following the dynamic change of the calculation logic means that in the actual calculation process, based on AIS historical data and a ship navigation dynamic judgment algorithm, ship dynamic information is calculated, wherein the ship dynamic information comprises navigation dynamic berthing, anchoring, origin port and destination port information and pre-supporting information; the ship navigation dynamic judgment algorithm is a dynamic algorithm which is continuously optimized, and with each round of optimization of the dynamic algorithm, the ship dynamic information is recalculated, and the latest navigation segment information is generated and updated based on the recalculated ship dynamic information.
Further, the snapshot generating module further includes a generating unit, including:
generating a latest AIS information data sub-module: generating latest AIS information data based on AIS data in the AIS data source extracted at fixed time, and copying the latest AIS information data into an AIS information table;
The direct generation snapshot submodule is used for configuring a timing task by utilizing an ETL tool, copying the set of the latest navigation segment information at fixed time every day, and directly generating the direct snapshot after the slicing time is associated;
the indirect snapshot generation sub-module is used for selecting snapshot time granularity based on the AIS total historical table, circularly extracting sample data near a designated time point according to a slice time sequence corresponding to the selected snapshot time granularity, and indirectly generating a data snapshot associated with latest AIS information; and selecting snapshot time granularity based on the total historical list of the leg information, circularly extracting sample data near a designated time point according to a slicing time sequence corresponding to the selected snapshot time granularity, and indirectly generating a data snapshot associated with the latest leg information.
Further, in the generating the latest AIS information data sub-module, the method further includes:
and acquiring an external data module: the AIS data processing method comprises the steps of regularly extracting AIS data of an external AIS data source in an interval period, wherein the AIS data is reported and updated by the AIS data source;
a data increment determining module: the device is used for cleaning and converting the incremental data in the interval period and storing the incremental data as a temporary table;
and a generation information module: and the method is used for constructing PRE AIS information for newly added AIS data, updating the latest AIS information containing the PRE AIS information into a storage table of the latest AIS information, namely, the latest AIS information set, and copying the latest AIS information set into the AIS full history table at the same time.
Further, the data snapshot in the storage center is stored in a storage table, each piece of data in the storage table contains two parts of data, the first part is slicing time, which means slicing time of a point on snapshot time granularity, and the slicing time main key is used for uniquely defining a slice; the second part is slicing service, which is the service data corresponding to the slicing, wherein each row of service data comprises a unique ship identifier, longitude, latitude, navigational speed, reporting time and the like.
Further, the snapshot searching module is configured to switch, based on the data snapshot, an association relationship between the latest leg information table and the ship foundation information into an association relationship between the latest leg information snapshot table and the ship foundation information; wherein the data snapshot comprises the latest generated segment information data snapshot, namely the directly generated snapshot.
Further, in the service analysis module, the method further includes:
the inquiry information acquisition sub-module is used for acquiring information input by a user inquiry, wherein the inquiry information comprises information of regional analysis dimensions such as basic tag information of a ship, port name information, geographical position information and the like, and time dimension information such as analysis time slices, reference process time and the like;
And the quick response and analysis sub-module is used for quickly responding and calling corresponding data snapshot according to the acquired query information, analyzing, returning the return value of the analysis result to the display interface, and generating a corresponding analysis trend chart for reference of a user.
Further, the indirect snapshot generation submodule further comprises a circulation module, wherein the circulation module is used for generating a slice time sequence corresponding to the snapshot based on the corresponding snapshot time range; judging whether the screened slicing time is in the corresponding slicing time sequence or not, and further judging whether the screened cycle is ended or not; if yes, directly exiting the circulation, and stopping screening; if not, generating a corresponding data snapshot for the corresponding slicing time through an AIS full-scale historical table or a avionics full-scale historical table; and finally, ending all the loops until all the judging processes are stopping screening, and ending the indirect snapshot generation.
Further, the indirect snapshot generating sub-module further includes:
the latest AIS information data snapshot generating module: the AIS data of the ship with all reporting time before the corresponding slicing time is selected based on the AIS total historical table, and the AIS data is defined as a temporary result corresponding to the AIS data; screening the latest reporting time of each ship identifier according to the grouping of the ship identifiers based on the corresponding temporary result, forming a reporting time data set closest to the corresponding slicing time, and defining a corresponding temporary data set; and correlating the corresponding temporary result with the corresponding temporary data set through Cartesian products, and screening sample data of (ship identification, latest reporting time) combined values in the corresponding temporary data set from the corresponding temporary result, namely indirectly generating a data snapshot of correlated latest AIS information;
The data snapshot generating module of the latest navigation segment information: the navigation section information data of the ship before the corresponding slicing time in all time periods is selected based on the navigation section information total historical table, and the temporary result corresponding to the navigation section information data is defined; and based on the corresponding temporary result, grouping according to the ship identification, acquiring corresponding latest leg information data when the slicing time falls in the corresponding leg period, and screening out (ship identification and leg information) combined values, namely indirectly generating a data snapshot associated with the latest leg information.
Compared with the prior art, the invention generates the snapshot according to different time granularities, generates the data snapshot based on the latest leg information data, and generates the latest state data snapshot based on massive historical data (AIS full history and leg information full history), thereby constructing a system which is convenient for backtracking and analyzing the history by utilizing the snapshot data. The backtracking analysis system combines the characteristics of shipping data, carefully selects a sample set of snapshots, and realizes more flexible and fine-granularity analysis backtracking system based on historical data by a snapshot generation method based on the latest data and the historical data through self-defining snapshot time granularity under the condition that source data are changed and based on the snapshot data of the whole history by one key, thereby changing the traditional snapshot characteristics, namely, the current data are required to be captured, namely, the analysis backtracking system based on the historical data is constructed, and more complex shipping data analysis is realized compared with the prior art.
The specific flexibility is embodied in the following aspects:
regional analysis latitude: based on the section information snapshot data, the port-to-port, port-to-country, country-to-port and country-to-region capacity trend conditions can be easily analyzed, and the analysis latitude supported in the section information can be flexibly supported.
Time analysis latitude: based on snapshot data in a unit of day, the running state of the ship can be recorded accurately, the data storage granularity is adjusted from a minute level to a day level, the storage cost is reduced by nearly thousand times, the retrieval efficiency response is improved by nearly hundred times, and the multi-table association and complex service scene query are supported more easily.
And the snapshot data regeneration technology is convenient for generating more accurate snapshots based on the latest calculation result after the algorithm is optimized, so as to obtain more accurate analysis results.
Drawings
FIG. 1 is a block diagram of a backtracking analysis system based on a shipping data snapshot according to the present invention.
FIG. 2 is a diagram of an exemplary storage of a data snapshot generated in a shipping data snapshot based backtracking analysis system.
FIG. 3 is a block diagram of a snapshot generation module in the system according to the present invention.
FIG. 4 is a block diagram of a snapshot sample processing unit in a snapshot generation module in the system.
Fig. 5 is a block diagram of a snapshot generating unit in the system according to the present invention.
Fig. 6 is a block diagram of a program for generating the latest AIS information data sub-module in the system according to the present invention.
FIG. 7 is a block diagram of an indirect snapshot generation sub-module in the system according to the present invention.
Fig. 8 is a program module diagram of a service analysis module in the system according to the present invention.
Fig. 9 is a graph of the result of trend analysis of the ship for both internal and external trade based on the backtracking analysis system of the shipping data snapshot.
FIG. 10 is a system deployment diagram of another shipping data snapshot based backtracking analysis system provided by the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
First, partial terms or terminology appearing in describing embodiments of the present application are applicable to the following explanation:
the automatic ship identification system (Automatic Identification System, AIS) is a navigation aid system originally applied to marine safety and communication between a ship and a shore, and plays an important role in the field of marine service such as reduction of ship collision accidents. The huge amount of data information accumulated by the AIS reflects the real-time dynamic state of the global marine ship, and the real-time dynamic state is a precious big data resource which can be used for analyzing the port and ship management problem. Besides being used for ship collision prevention, the huge amount of data can be applied to analysis and decision in the field of port and navigation management through technical methods such as data processing and mining, and has huge utilization potential, such as: port operation management, ship operation management, dynamic monitoring of a fleet, maritime management and the like.
The international maritime organization prescribes that ships with more than 300 total tons of international airlines, ships with more than 500 total tons of non-international airlines and all passenger ships need to be provided with AIS equipment; the AIS message comprises the current dynamic information (such as position and navigational speed) of the ship, static information (such as ship length and ship width) and navigation time related information (such as destination, expected arrival time); the broadcasting frequency of the AIS message is 2s-30s, and the AIS message is transmitted through a VHF channel; AIS incorporates standardized VHF transceivers, positioning systems (e.g., GPS receivers), and other electronic navigation sensors (e.g., gyrocompass, rotation speed indicators). Information data about the vessel itself will be broadcast transmitted to other receivers by means of a dedicated VHF transmitter; the transmission distance limits for inter-vessel and inter-shore AIS signals are about 20nmi and 40nmi, respectively.
Reporting of on-board AIS equipment data is generally achieved through satellites, coastal base stations, on-board signal equipment and the like, and AIS data providers integrate collected AIS data to form AIS data service and synchronize the AIS data service to business partners in batches according to a certain time frequency. In the present invention, as an AIS data consumer, the latest AIS data is purchased and periodically extracted from the AIS service provider.
ETL data extraction, transformation and loading techniques are abbreviations for English Extract-Transform-Load, and are used to describe the process of extracting (extracting), transforming (transforming) and loading (Load) data from a source to a destination. The main use of ETL in the invention includes, dispose the timing execution task, call the database storage course according to the plan; configuring a remote execution task, remotely executing a command according to a plan, and informing an execution result by mail; and configuring a data checking task, checking and notifying the result according to the execution plan data.
Kettle data Extraction and transformation, kettle was at the earliest an open-source ETL tool, and is known as KDE Extraction, transport, transformation and Loading Environment.
In addition, the structure of the total AIS history table and the latest AIS information table referred to in the present invention is the same, except that there is only one latest record for each ship in the latest AIS information table, but there is a difference in the instantaneous latitude between all the histories in the total history table.
In order to make the advantages of the technical scheme of the invention more clear, the invention is described in detail below with reference to the accompanying drawings and examples.
The invention provides a backtracking analysis system based on shipping data snapshot, as shown in fig. 1, the invention provides a program block diagram of the backtracking analysis system based on shipping data snapshot, comprising:
The storage center 101 is used for storing data snapshots formed based on snapshot sample data and data generated during the operation of the storage system. In this embodiment, based on snapshot data formed by the latest navigation segment information of the ship and the important classification information of the ship, the data volume is smaller than the whole volume history, the data volume can be stored in a single naming space under the same database, and can be directly associated with and accessed to service data, and a complete analysis and backtracking system is formed by combining the snapshot data and the latest data. The hardware devices of the storage center 101 are built using a virtual server environment of the cloud data center, and in practical applications, the hardware devices are not limited to the above-mentioned environment, but include local or remote, virtual or actual physical devices that can be implemented by those skilled in the art.
The snapshot generating module 102 is configured to generate a shipping data snapshot according to a certain snapshot time granularity for different snapshot samples such as the latest AIS information, the AIS full-scale history table, the avigation section information full-scale history table, and the like, which are fixedly reported, and store the shipping data snapshot in a storage table form in the storage center 101; the selection of the snapshot time granularity is determined based on the cost of shipping and the refined satisfaction of the shipping business. Based on different snapshot samples including the latest AIS information, AIS full-quantity history list, avionic section information full-quantity history list and the like which can be finely selected, a foundation is provided for providing more complex scenes and fine-granularity analysis backtracking for the system.
A snapshot search module 103, configured to provide a query search function based on the latest AIS information in a time dimension; and the method is used for switching the association relation between the latest navigation segment information table and the ship basic information into the association between the latest navigation segment information table and the ship basic information based on the data snapshot, and directly associating the data snapshot of the ship when different ship states in different periods are searched, so that the latest state data of the ship with the designated slicing time is quickly searched. That is, if the searching of a single piece (or a point) of data in the massive data is originally based, a larger granularity searching basis is provided, for example, the data of the latest navigation segment data slice is searched in the massive shipping data, the searching time is shortened, and the quick searching is realized, so that the basis is provided for the following analysis.
The service analysis module 104 is configured to quickly form trend data using corresponding time slices as time axes based on the data snapshot, define the required feature data that is continuously and dynamically updated into attribute groups according to categories, utilize the classified data snapshot, count the number of vessels in each corresponding time slice based on the corresponding time slices and the attribute groups, and aggregate the corresponding time slices, i.e. form cross-period trend statistics, and assist a user in performing next-step demand analysis based on the trend statistics. The business analysis module 104 of the invention has analysis functions, relies on the generation module and the search module to carry out further analysis backtracking of the big data of the ship (including periodically extracting the latest AIS data from AIS service providers and especially sorting the latest navigation segment data, history data, other classification data and the like which are generated and carefully selected), so as to provide more analysis application functions for the shipping informatization field or other users.
In the snapshot generating module 102, the selection of the snapshot time granularity is mainly selected according to the actual situation of industry, for example, in the field of shipping data, the selection of the snapshot time granularity is determined according to the cost of shipping and the refined satisfaction degree of shipping service; specifically, in the snapshot sample, the update frequency of the latest AIS information is in units of minutes, the sailing dynamic information is in units of hours or days, and the important classification information is in units of Zhou Shenzhi months.
Preferably, in view of factors such as the main stream commercial ship design speed, actual speed, voyage, comprehensive cost, efficiency, service acceptance degree and the like, the selection of the snapshot time granularity takes a day as a unit, namely, a snapshot is extracted every day aiming at the latest voyage information and other important classification information of the ship; further refining the snapshot time granularity, selecting a specific time point of the current day based on the local business actual operation time of the specific region time zone, and uniformly selecting UTC time 00:00, namely 08:00 under the GMT+8 time zone as a sampling time point.
In actual process, based on the snapshot sample, the update frequency of the latest AIS information is in units of minutes, the sailing dynamic information is in units of hours or days, and the important classification information is in units of Zhou Shenzhi months.
The AIS data is characterized by unequal density and larger magnitude, wherein the AIS data is reported once in an average of 10 minutes by a single ship, the AIS fault time is removed, the average reporting time of the AIS data in the whole year is 300, and the AIS data amount of one ship in one year is 43200. According to the data of the Rough class society, the number of vessels in the global volume is 300 ten thousand, partial small fishing vessels, short barge passenger vessels and small special operation vessels are removed, and according to the calculation of 200 ten thousand commercial vessels, the AIS data generated by the global commercial vessels in one year exceeds 800 hundred million. In the service system associated with this embodiment, the key tracking analysis is about 10 tens of thousands of vessels, and about 80 hundreds of millions of AIS history data in 2 years. When the business system performs history analysis, the business system is difficult to quickly respond and has poor user experience when directly searching the full AIS history data, so that data with larger granularity is required to be stored independently, and therefore, the selection of data snapshot storage with large granularity and the selection of proper slicing time granularity of the data snapshot are extremely important.
Considering that the current market design speed in the world is basically below 25 knots, namely 25 seas/hour, the actual speed is lower, taking Asia-Mexi airlines as an example, the airlines are usually above 10 days, the state change in one day is not great in more than 5000 seas, the comprehensive cost, efficiency, service acceptance degree and other factors, the snapshot time granularity of the shipping data samples is unified into one day, namely the latest leg information and other important classification information of the ship are extracted every day. The specific time point of the daily snapshot can be in principle any time point in the day, and for convenience of operation, the UTC time 00:00 point, namely the 08:00 point in the GMT+8 time zone, can be uniformly selected as the sampling time point.
In this embodiment, the data snapshot in the storage center 101 is stored in the form of a storage table, where each piece of data in the storage table includes two parts of data, and the first part is a slicing time, which refers to a slicing time of a point on the snapshot time granularity, including a slicing time primary key, which is used to uniquely define a slice; the second part is slicing service, which is the service data corresponding to the slicing, wherein each row of service data comprises a unique ship identifier, longitude, latitude, navigational speed, reporting time and the like. For the convenience of describing the implementation process of the present invention, corresponding symbols are defined for some nouns, such as unique ship identification MMSI, longitude LON, latitude LAT, navigational speed SOG and reporting time POSTITME; it is noted that the methods of the present invention are not limited to the definitions set forth herein, but include all similar term designations in the actual practice of the invention.
Preferably, as shown in fig. 2, in the exemplary diagram of storage of a data snapshot generated in the system provided by the present invention, in fig. 2, taking the latest AIS information as an example, it is assumed that the object name of the storage table of the data is ais_new, unlike the AIS history table that includes all AIS history records of vessels each day, in the ais_new table, each vessel only stores the latest AIS record, and each row of data includes at least a unique vessel identifier MMSI, longitude LON, latitude LAT, speed SOG, and reporting time posttime. The ais_new SNAPSHOT table and the ais_new table are stored separately, for example, named as snappshot_ais_new, each snappshot_ais_new needs to contain at least two parts of data, the first part is a SLICE TIME key slice_time used for uniquely defining a SLICE, the second part is business data corresponding to the SLICE, as shown in fig. 2, data in the same SLICE can be locked through the slice_time, and different SLICEs are distinguished through different slice_time.
FIG. 3 is a program module diagram of a snapshot generating module in the system provided by the present invention. As shown in fig. 3, the snapshot generating module further includes a snapshot sample processing unit 1021 and a generating unit 1022. The snapshot sample processing unit 1021 is mainly used for providing different snapshot samples; the generating unit 1022 is configured to generate a shipping data snapshot according to different snapshot samples according to a certain snapshot time granularity and store the shipping data snapshot in a storage table form in the storage center 101.
As shown in fig. 4, the present invention provides a program module diagram of the snapshot sample processing unit 1021 in the snapshot generating module 102 in the system.
Wherein, the snapshot sample processing unit 1021 includes:
the snapshot sample data sub-module 10211 for obtaining the latest AIS information of the fixed report, wherein the latest AIS information defines key information including AIS reporting time, ship position longitude and latitude, ground speed, ship heading, AIS reporting time and the like.
A dynamic update latest avigation information sub-module 10212, configured to update latest avigation information based on the acquired latest AIS information of the ship, where the latest avigation information is composed of the latest AIS information reported fixedly and avigation information dynamically changing along with calculation logic; all the latest navigation segment information historical data are collected to form a navigation segment information total historical table; in this embodiment, the dead leg information following the dynamic change of the calculation logic means that in the actual calculation process, based on AIS historical data and a ship navigation dynamic judgment algorithm, ship dynamic information is calculated, where the ship dynamic information includes navigation dynamic berthing, anchoring, origin port and destination port information, and pre-supporting information; the ship navigation dynamic judgment algorithm is a dynamic algorithm which is continuously optimized, and with each round of optimization of the dynamic algorithm, the ship dynamic information is recalculated, and the latest navigation segment information is generated and updated based on the recalculated ship dynamic information.
The acquire other important classification information sub-module 10213 is configured to acquire historical data based on the leg information and analyze and summarize data in combination with additional classification attributes such as trade, operation, and commercial value of the ship, and acquire internal and external trade operation states of the ship. In this embodiment, the other important classification information is data that is analyzed and summarized based on the historical data in the full-scale historical table of the leg information and additional classification attributes such as trade, operation, commercial value and the like of the ship, including internal and external trade operation status data and the like of the ship. The other important classification information, such as the information of the internal and external trade operation states of the ship, is based on the dynamic statistics and induction of the history of the ship; preferably, the data snapshot based on the operating state of the foreign trade and the foreign trade of the ship is that whether the port hanging record of the past period contains the overseas port or not is determined, and the slice data is formed to record the characteristics of the foreign trade and the foreign trade of the slice time node, that is, whether the port hanging record of the past period contains the overseas port or not is determined, and the slice data is formed to record the characteristics of the foreign trade and the foreign trade of the slice time node. The data of the internal and external trade operation states and the like can be used for providing services such as foreign trade analysis, consultation, pricing and cost reference based on the services for a user or a shipping enterprise, and preferably, as shown in fig. 7, the trend analysis result diagram for the internal and external trade facultative ship based on the backtracking analysis system of the shipping data snapshot is provided.
As shown in fig. 5, the present invention provides a program module diagram of the generating unit 1022 of the snapshot generating module 102 in the system.
Wherein the generating unit 1022 includes:
generating the latest AIS information data sub-module 10221: for generating up-to-date AIS information data based on AIS data in the time extracted AIS data source, while copying into the AIS full history table.
The direct snapshot generation sub-module 10222 is configured to perform data extraction and conversion by using an ETL tool, and in this embodiment, a keyle is used to configure a timing task, copy the generated set of the latest leg information at regular intervals each day, and directly generate the direct snapshot after correlating slicing time.
An indirect snapshot generation sub-module 10223, configured to circularly extract sample data near a specified time point according to a slice time sequence based on the AIS full-scale history table, and indirectly associate a data snapshot of the latest AIS information; and the data snapshot is used for selecting snapshot time granularity based on the full-scale historical table of the navigation segment information, circularly extracting sample data near a designated time point according to a slicing time sequence corresponding to the selected snapshot time granularity, and indirectly generating data snapshot associated with the latest navigation segment information.
Preferably, in the generating latest AIS information data sub-module 10221, as shown in fig. 6, a program module diagram of the generating latest AIS information data sub-module in the system provided by the invention includes:
acquiring external data module 1022101: the AIS data processing method comprises the steps of regularly extracting AIS data of an external AIS data source in an interval period, wherein the AIS data is reported and updated by the AIS data source;
the determine data delta module 1022102: the device is used for cleaning and converting the incremental data in the interval period and storing the incremental data as a temporary table;
the generation information module 1022103: and the method is used for constructing PRE AIS information for newly added AIS data, updating the latest AIS information containing the PRE AIS information into a storage table of the latest AIS information, namely, the latest AIS information set, and copying the latest AIS information set into the AIS full history table at the same time.
In the indirect snapshot generating sub-module 10223, as shown in fig. 7, a program module diagram of the indirect snapshot generating sub-module in the system provided by the present invention includes:
a loop module 1022301 for generating a slice time sequence corresponding to a snapshot based on a corresponding snapshot time range; judging whether the screened slicing time is in the corresponding slicing time sequence or not, and further judging whether the screened cycle is ended or not; if yes, directly exiting the circulation, and stopping screening; if not, generating a corresponding data snapshot for the corresponding slicing time through an AIS full-scale historical table or a avionics full-scale historical table; and finally, ending all the loops until all the judging processes are stopping screening, and ending the indirect snapshot generation.
The latest AIS information data snapshot generation module 1022302: the AIS data of the ship with all reporting time before the corresponding slicing time is selected based on the AIS total historical table, and the AIS data is defined as a temporary result corresponding to the AIS data; screening the latest reporting time of each ship identifier according to the grouping of the ship identifiers based on the corresponding temporary result, forming a reporting time data set closest to the corresponding slicing time, and defining a corresponding temporary data set; and correlating the corresponding temporary result with the corresponding temporary data set through Cartesian products, and screening sample data (of ship identification and latest reporting time) of a combined value in the corresponding temporary data set from the corresponding temporary result, namely indirectly generating a data snapshot of associated latest AIS information.
The data snapshot generation module 1022303 of the latest leg information: the navigation section information data of the ship before the corresponding slicing time in all time periods is selected based on the navigation section information total historical table, and the temporary result corresponding to the navigation section information data is defined; and based on the corresponding temporary result, grouping according to the ship identification, acquiring corresponding latest leg information data when the slicing time falls in the corresponding leg period, and screening out (ship identification and leg information) combined values, namely indirectly generating a data snapshot associated with the latest leg information.
In this embodiment, preferably, in the snapshot searching module 103, the association relationship between the latest leg information table and the ship base information is switched to the association relationship between the latest leg information snapshot table and the ship base information based on the data snapshot; wherein the data snapshot comprises the latest AIS information data snapshot generated, namely the snapshot generated directly. Therefore, through the snapshot data regeneration technology, after the algorithm is optimized, more accurate snapshots can be conveniently and quickly generated based on the latest calculation results, and further more accurate search query results can be obtained when searching the query.
In this embodiment, as shown in fig. 8, the program module diagram of the service analysis module 104 in the system provided by the present invention, where the service analysis module 104 further includes:
the query information obtaining submodule 1041 is configured to obtain information input by a user, where the query information includes information of analysis dimensions of areas such as basic tag information of a ship, port name information, geographical location information, and time dimension information such as analysis time slice and reference process time;
the quick response and analysis sub-module 1042 is used for quick response and calling corresponding data snapshot according to the obtained query information, analyzing, returning the return value of the analysis result to the display interface, and generating a corresponding analysis trend chart for reference of the user.
Wherein, regional analysis latitude: based on the section information snapshot data, the port-to-port, port-to-country, country-to-port and country-to-region capacity trend conditions can be easily analyzed, and the analysis latitude supported in the section information can be flexibly supported. Further, time analysis latitude: based on snapshot data in a unit of day, the running state of the ship can be recorded accurately, the data storage granularity is adjusted from a minute level to a day level, the storage cost is reduced by nearly thousand times, the retrieval efficiency response is improved by nearly hundred times, and the multi-table association and complex service scene query are supported more easily.
Preferably, the service analysis module 104 is capable of easily forming trend data using SLICE_TIME as a TIME axis based on snapshot data, and taking a ship classification status data snapshot as an example for illustration. As shown in FIG. 9, the backtracking analysis system based on the shipping data snapshot provided by the invention is aimed at a trend analysis result diagram of an internal and external trade facultative ship; and analyzing the internal and external trade characteristics of the ship based on the hanging history of the ship, and defining characteristic data into VESSEL_CATALOG according to categories, wherein the data are dynamic data and are updated once a day. Using the classified SNAPSHOT data SNAPSHOT_VESSEL_CATALOG, the number of internal and external trade ships per SLICE_TIME can be counted based on the SLICE_TIME and the internal and external trade attribute groups, and the aggregate SLICE_TIME can further form a cross-cycle trend statistic.
In order to further illustrate the invention, a system deployment diagram of another backtracking analysis system based on shipping data snapshots provided by the invention is shown, as shown in fig. 10, wherein the system deployment diagram mainly comprises deployment of storage of data, deployment of snapshot generation and deployment of backtracking analysis based on the data snapshots, wherein the generated data snapshots are stored in a data storage system, a snapshot sample data in the data storage system is called in a data snapshot generation process, and the backtracking analysis system calls the data storage system and stores backtracking analysis results with the data storage system. The deployment of data storage comprises the deployment of AIS data, air segment data and pre-support data; the deployment of snapshot generation mainly comprises the deployment of direct snapshot generation and the deployment of indirect snapshot generation; the deployment of the backtracking analysis comprises a ship history retrieval system, a cargo capacity analysis system, an internal and external trade trend analysis system and the like.
Preferably, if based on the data snapshot of the leg information, the freight conditions between designated ports in the past period of time are statistically analyzed, for example, the freight quantity analysis from black draand to dry bulk cargoes in Qingdao in China in 3 months is performed, the snapshot data in 3 months is selected according to the time slicing, the leg information in each day in the sliced data is statistically consistent with the ship load ton from black draand to dry bulk cargoes in Qingdao, the freight tendency is generated, and the freight capacity condition of coal or iron ore is tracked and analyzed.
Preferably, if based on the data snapshot of the navigation segment information, the condition of the internal and external trade attributes of different ships in the past period is statistically analyzed, namely, the departure destination port is the domestic port and the departure destination port is the foreign port, and the number of the internal trade and the foreign trade can be further analyzed through the snapshot data to form the internal and external trade transport capacity index.
Preferably, the accuracy of the service such as the inquiry snapshot analysis service is close to that of the direct search AIS total history, because the dynamic change period of the ship is longer, and the dynamic change is usually in days; the response time of the snapshot analysis service is one tenth of that of the direct search AIS full-quantity history, because AIS full-quantity history data of 10 ten thousand business ships for 3 years can reach 100 hundred million levels, the search time is very long, and more data tables are difficult to associate to realize complex query.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (7)
1. A backtracking analysis system based on a shipping data snapshot, comprising:
the storage center is used for storing data snapshot formed based on snapshot sample data and storing data generated in the running process of the system;
the snapshot generation module is used for generating shipping data snapshots according to a certain snapshot time granularity aiming at the snapshot samples with different latest AIS information, AIS total historical tables and avigation information total historical tables which are fixedly reported, and storing the shipping data snapshots in a storage table form in the storage center; the selection of the snapshot time granularity is determined according to the cost of shipping and the refined satisfaction of shipping services; the snapshot generating module comprises a snapshot sample processing unit and a generating unit;
the snapshot sample processing unit includes:
the snapshot sample data sub-module is used for obtaining latest AIS information of the fixed report, wherein the latest AIS information defines information including AIS reporting time, ship position longitude and latitude, ground speed, ship heading and AIS reporting time key; all AIS information table history data are collected to form an AIS total history table;
the dynamic updating latest avigation information sub-module is used for updating latest avigation information based on the acquired latest AIS information of the ship, wherein the latest avigation information consists of the latest AIS information reported fixedly and avigation information which dynamically changes along with calculation logic; all the latest navigation segment information historical data are collected to form a navigation segment information total historical table; the following calculation logic dynamic change of the navigation section information refers to calculation of the ship dynamic information based on AIS historical data and a ship navigation dynamic judgment algorithm in the actual calculation process, wherein the ship dynamic information comprises navigation dynamic berthing, anchoring, originating port and destination port information and pre-supporting information; the ship navigation dynamic judgment algorithm is a dynamic algorithm which is continuously optimized, and as each round of the dynamic algorithm is optimized, the ship dynamic information is recalculated, and the latest navigation segment information is generated and updated based on the recalculated ship dynamic information;
The sub-module is used for acquiring other important classification information, and is used for acquiring historical data based on the navigation section information and analyzing and summarizing data by combining trade, operation and commercial value added classification attributes of the ship to acquire internal and external trade operation states of the ship; the generation unit includes:
generating a latest AIS information data sub-module: generating latest AIS information data based on AIS data in the AIS data source extracted at fixed time, and copying the latest AIS information data into an AIS information table;
the direct generation snapshot submodule is used for configuring a timing task by utilizing an ETL tool, copying the set of the latest navigation segment information at fixed time every day, and directly generating the direct snapshot after the slicing time is associated;
the indirect snapshot generation sub-module is used for selecting snapshot time granularity based on the AIS total historical table, circularly extracting sample data near a designated time point according to a slice time sequence corresponding to the selected snapshot time granularity, and indirectly generating a data snapshot associated with latest AIS information; selecting snapshot time granularity based on the total historical list of the navigation segment information, circularly extracting sample data near a designated time point according to a slicing time sequence corresponding to the selected snapshot time granularity, and indirectly generating a data snapshot associated with the latest navigation segment information;
The snapshot searching module is used for providing a query searching function based on the latest AIS information in the time dimension; the method comprises the steps of acquiring a data snapshot of a ship, and acquiring the latest state data of the ship at a specified slicing time, wherein the data snapshot is used for switching the association relation between a latest navigation section information table and ship basic information into the association between the latest navigation section information snapshot table and the ship basic information based on the data snapshot, and directly associating the data snapshot of the ship when different ship states at different periods are searched for, so that the latest state data of the ship at the specified slicing time is quickly searched for;
the business analysis module is used for data analysis service of ships, based on the data snapshot, rapidly forming trend data taking corresponding time slices as time axes, defining the required feature data which are continuously and dynamically updated into attribute groups according to categories, counting the number of the ships of each corresponding time slice based on the corresponding time slices and the attribute groups by utilizing the classified data snapshot, and aggregating the corresponding time slices to form cross-period trend statistics, and assisting a user to perform next-step demand analysis based on the trend statistics.
2. The backtracking analysis system of claim 1, further characterized in that in the generate-up-to-date AIS information data sub-module, further comprising:
And acquiring an external data module: the AIS data processing method comprises the steps of regularly extracting AIS data of an external AIS data source in an interval period, wherein the AIS data is reported and updated by the AIS data source;
a data increment determining module: the device is used for cleaning and converting the incremental data in the interval period and storing the incremental data as a temporary table;
and a generation information module: and the method is used for constructing PRE AIS information for newly added AIS data, updating the latest AIS information containing the PRE AIS information into a storage table of the latest AIS information, namely, the latest AIS information set, and copying the latest AIS information set into the AIS full history table at the same time.
3. The backtracking analysis system of claim 1, further characterized in that the data snapshot in the storage center is stored in a form of a storage table, wherein each piece of data in the storage table contains two parts of data, and the first part is a slice time, which refers to a slice time of a point on a snapshot time granularity, and includes a slice time master key for uniquely defining a slice; the second part is slicing service, which is the service data corresponding to the slicing, wherein each row of service data comprises a unique ship identifier, longitude, latitude, navigational speed and reporting time.
4. The backtracking analysis system of claim 1, further characterized in that the snapshot search module is configured to switch, based on a data snapshot, an association relationship between a latest leg information table and ship base information to an association relationship between a latest leg information snapshot table and ship base information; wherein the data snapshot comprises the latest generated segment information data snapshot, namely the directly generated snapshot.
5. The backtracking analysis system of claim 1, further characterized in that in the traffic analysis module, further comprising:
the inquiry information acquisition sub-module is used for acquiring information input by a user inquiry, wherein the inquiry information comprises basic tag information of a ship, port name information, information of geographical position information area analysis dimension, analysis time slice and reference process time dimension information;
and the quick response and analysis sub-module is used for quickly responding and calling corresponding data snapshot according to the acquired query information, analyzing, returning the return value of the analysis result to the display interface, and generating a corresponding analysis trend chart for reference of a user.
6. The backtracking analysis system of claim 1, further characterized by,
the indirect snapshot generation submodule further comprises a circulation module, wherein the circulation module is used for generating a slice time sequence corresponding to the snapshot based on the corresponding snapshot time range; judging whether the screened slicing time is in the corresponding slicing time sequence or not, and further judging whether the screened cycle is ended or not; if yes, directly exiting the circulation, and stopping screening; if not, generating a corresponding data snapshot for the corresponding slicing time through an AIS full-scale historical table or a avionics full-scale historical table; and finally, ending all the loops until all the judging processes are stopping screening, and ending the indirect snapshot generation.
7. The backtracking analysis system of claim 1 or 6, further characterized in that the indirect snapshot generation sub-module further comprises:
the latest AIS information data snapshot generating module: the AIS data of the ship with all reporting time before the corresponding slicing time is selected based on the AIS total historical table, and the AIS data is defined as a temporary result corresponding to the AIS data; screening the latest reporting time of each ship identifier according to the grouping of the ship identifiers based on the corresponding temporary result, forming a reporting time data set closest to the corresponding slicing time, and defining a corresponding temporary data set; and correlating the corresponding temporary result with the corresponding temporary data set through Cartesian products, and screening sample data of (ship identification, latest reporting time) combined values in the corresponding temporary data set from the corresponding temporary result, namely indirectly generating a data snapshot of correlated latest AIS information;
the data snapshot generating module of the latest navigation segment information: the navigation section information data of the ship before the corresponding slicing time in all time periods is selected based on the navigation section information total historical table, and the temporary result corresponding to the navigation section information data is defined; and based on the corresponding temporary result, grouping according to the ship identification, acquiring corresponding latest leg information data when the slicing time falls in the corresponding leg period, and screening out (ship identification and leg information) combined values, namely indirectly generating a data snapshot associated with the latest leg information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111303459.5A CN114020535B (en) | 2021-11-05 | 2021-11-05 | Backtracking analysis system based on shipping data snapshot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111303459.5A CN114020535B (en) | 2021-11-05 | 2021-11-05 | Backtracking analysis system based on shipping data snapshot |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114020535A CN114020535A (en) | 2022-02-08 |
| CN114020535B true CN114020535B (en) | 2024-04-09 |
Family
ID=80061482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111303459.5A Active CN114020535B (en) | 2021-11-05 | 2021-11-05 | Backtracking analysis system based on shipping data snapshot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020535B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116319836B (en) * | 2023-05-22 | 2023-08-01 | 江苏物润船联网络股份有限公司 | Service flow data synchronous processing method and system, electronic equipment and storage medium |
| CN117520313B (en) * | 2024-01-02 | 2024-03-26 | 北京淇瑀信息科技有限公司 | Data backtracking method and device based on multidimensional associated data warehouse slice table |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862321A (en) * | 2021-02-08 | 2021-05-28 | 国家海洋信息中心 | Ocean transportation statistical system based on AIS big data and statistical method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8332862B2 (en) * | 2009-09-16 | 2012-12-11 | Microsoft Corporation | Scheduling ready tasks by generating network flow graph using information receive from root task having affinities between ready task and computers for execution |
-
2021
- 2021-11-05 CN CN202111303459.5A patent/CN114020535B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112862321A (en) * | 2021-02-08 | 2021-05-28 | 国家海洋信息中心 | Ocean transportation statistical system based on AIS big data and statistical method thereof |
Non-Patent Citations (2)
| Title |
|---|
| 下一代空管系统运行概念及其关键技术;严勇杰;曹罡;;指挥信息系统与技术;20180702(03);全文 * |
| 试述基于多传感信息融合的内河水运统计应用研究;何建春;;中国水运;20171015(10);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114020535A (en) | 2022-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114020535B (en) | Backtracking analysis system based on shipping data snapshot | |
| CN112862321B (en) | Ocean transportation statistical system based on AIS big data and statistical method thereof | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| Mitchell et al. | Waterway performance monitoring with automatic identification system data | |
| CN113179173B (en) | Operation and maintenance monitoring system for expressway system | |
| CN113256146B (en) | Ship supervision method and system | |
| US10621873B1 (en) | Systems and methods for generating geofences | |
| US7933693B2 (en) | System and method for harvesting business intelligence from maritime communications | |
| CN104050289A (en) | Detection method and system for abnormal events | |
| CN114037230B (en) | Method and system for rapidly and accurately identifying predicted empty ship information | |
| Štepec et al. | Machine learning based system for vessel turnaround time prediction | |
| CN116257565A (en) | Ship abnormal behavior detection method | |
| CN110555452A (en) | network problem processing method and device based on intelligent clustering | |
| CN117764282A (en) | Ship energy consumption tracking method based on multi-sensor data fusion | |
| Herodotou et al. | Big maritime data management | |
| Zhang et al. | How liner shipping heals schedule disruption: A data-driven framework to uncover the strategic behavior of port-skipping | |
| CN104767534A (en) | Error-controllable vehicle track compression storage method and error-controllable vehicle track compression storage system | |
| Patroumpas | Online mobility tracking against evolving maritime trajectories | |
| CN116126926B (en) | Intelligent management method and system for full life cycle dynamic data of ship | |
| CN111061793A (en) | Data processing system and method | |
| CN114020534B (en) | Shipping data snapshot generating method | |
| CN116596435A (en) | Global ship cargo quantity statistical analysis method and system | |
| CN110718022A (en) | Alarm method of intelligent electric meter, server and computer readable storage medium | |
| CN114387812A (en) | Garbage collection and transportation monitoring method, device, equipment and storage medium | |
| Doğan et al. | A Novel Approach for Knowledge Discovery from AIS Data: An Application for Transit Marine Traffic in the Sea of Marmara. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |