CN115985099A - Rail transit data unloading method and device - Google Patents
Rail transit data unloading method and device Download PDFInfo
- Publication number
- CN115985099A CN115985099A CN202211737940.XA CN202211737940A CN115985099A CN 115985099 A CN115985099 A CN 115985099A CN 202211737940 A CN202211737940 A CN 202211737940A CN 115985099 A CN115985099 A CN 115985099A
- Authority
- CN
- China
- Prior art keywords
- data
- storage space
- unloading
- target
- target storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for unloading rail transit data, wherein the method comprises the following steps: determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred; determining a transfer strategy corresponding to target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space; and transferring the target data to be transferred to the target storage space based on the transfer strategy. The method and the device for unloading the rail transit data can simply and efficiently unload the rail transit data under the condition that the types of the data source and the storage space are different, can synchronously unload the data to a plurality of storage spaces, and can improve the flexibility of unloading the rail transit data.
Description
Technical Field
The invention relates to the technical field of rail transit, in particular to a method and a device for unloading rail transit data.
Background
With the development of science and technology, rail transit brings great convenience to people's life, and has become an indispensable part of modern life.
The normal operation of the rail transit depends on the support of various rail transit systems. During the actual operation process of rail transit, the data generated by the system can be called as rail transit data, and the unloading requirement of the rail transit data exists between different rail transit systems.
However, it is difficult to implement the conventional method for unloading the track traffic data simply and efficiently. Therefore, how to more simply and efficiently perform the unloading of the rail transit data is a technical problem to be solved in the field.
Disclosure of Invention
The invention provides a method and a device for unloading rail transit data, which are used for solving the defect that the unloading of the rail transit data is difficult to realize simply and efficiently in the prior art and realizing the unloading of the rail transit data more simply and efficiently.
The invention provides a rail transit data unloading method, which comprises the following steps:
acquiring target data to be transferred;
determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred;
determining a transfer strategy corresponding to the target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space;
and unloading the target data to be unloaded to the target storage space based on the unloading strategy.
According to the rail transit data unloading method provided by the invention, the unloading strategy corresponding to the target data to be unloaded is determined based on the type of the target storage space, and the method comprises the following steps:
under the condition that the type of the target storage space is MySQL or Qracle, determining that the unloading strategy corresponding to the target data to be unloaded comprises a first unloading strategy;
the first unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a Mybatis frame and a Springboot frame.
According to the rail transit data unloading method provided by the invention, the unloading strategy corresponding to the target data to be unloaded is determined based on the type of the target storage space, and the method comprises the following steps:
under the condition that the type of the target storage space is a message queue, determining that a unloading strategy corresponding to the target data to be unloaded comprises a second unloading strategy;
and the second unloading strategy comprises the steps of acquiring a message queue actuator based on the configuration information of the target storage space, analyzing the target data to be unloaded by using JSONObject, and unloading the analyzed target data to be unloaded to the target storage space based on the message queue actuator.
According to the rail transit data unloading method provided by the invention, the unloading strategy corresponding to the target data to be unloaded is determined based on the protocol corresponding to the target storage space, and the method comprises the following steps:
under the condition that the protocol corresponding to the target storage space is an HTTP (hyper text transport protocol), a UDP (user datagram protocol) or a TCP (transmission control protocol), determining that the unloading strategy corresponding to the target data to be unloaded comprises a third unloading strategy;
and the third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a protocol corresponding to the target storage space.
According to the rail transit data unloading method provided by the invention, the unloading strategy corresponding to the target data to be unloaded is determined based on the type of the target storage space, and the method comprises the following steps:
under the condition that the type of the target storage space is ES, determining that the unloading strategy corresponding to the target data to be unloaded comprises a fourth unloading strategy;
and the fourth unloading strategy comprises the steps of acquiring an implementation class corresponding to the target storage space based on the operation type of the target storage space, and unloading the target data to be unloaded to the target storage space based on the implementation class.
According to the rail transit data unloading method provided by the invention, the acquiring of the target data to be unloaded comprises the following steps:
acquiring original data to be transferred;
carrying out data verification on the original data to be transferred;
under the condition that the original data to be transferred and stored pass the data verification, performing data processing on the original data to be transferred and stored;
determining the original data to be transferred and stored after data processing as the target data to be transferred and stored;
wherein the data check comprises a data format check; the data processing includes data filtering.
According to the rail transit data unloading method provided by the invention, the unloading of the target data to be unloaded to the target storage space based on the unloading strategy comprises the following steps:
and under the condition that the target storage space passes the verification, unloading the target data to be unloaded to the target storage space based on the unloading strategy.
The invention also provides a track traffic data unloading device, which comprises:
the unloading data acquisition module is used for acquiring target data to be unloaded;
the storage space determining module is used for determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred;
a unloading strategy determining module, configured to determine an unloading strategy corresponding to the target data to be unloaded based on the type of the target storage space or a protocol corresponding to the target storage space;
and the data unloading module is used for unloading the target data to be unloaded to the target storage space based on the unloading strategy.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the rail transit data unloading method.
The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the rail transit data unloading method as described in any of the above.
The invention also provides a computer program product, which comprises a computer program, and the computer program realizes the rail transit data unloading method when being executed by a processor.
According to the method and the device for unloading the rail transit data, after the target storage space corresponding to the target data to be unloaded is determined based on the configuration information of the target data to be unloaded, the unloading strategy corresponding to the target data to be unloaded is determined based on the type or the corresponding protocol of the target storage space, and then the target data to be unloaded is unloaded to the target storage space based on the unloading strategy.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a rail transit data unloading method provided by the present invention;
fig. 2 is a second schematic flow chart of the rail transit data unloading method provided by the present invention;
FIG. 3 is a schematic structural diagram of a track traffic data unloading device provided by the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the data transmission mode adopted by the rail transit system is usually specific, the data transmission methods adopted by different rail transit systems may not be the same, and the data transmission of the rail transit system has the characteristics of strong limitation, poor universality, high expansion cost and the like.
Due to the fact that data transmission modes adopted by different rail transit systems are difficult to unify, the transfer of rail transit data is difficult to achieve simply and efficiently under the condition that rail transit data transfer requirements exist among different rail transit systems, and obstacles are caused to cooperation of multiple rail transit systems.
Therefore, the invention provides a rail transit data unloading method based on an HTTP protocol. The rail transit data unloading method provided by the invention can realize data unloading among different types of databases more simply and efficiently, can synchronously unload the data to a plurality of databases, and can improve the flexibility of rail transit data unloading.
Fig. 1 is a schematic flow chart of a rail transit data unloading method provided by the present invention. The track traffic data unloading method of the present invention is described below with reference to fig. 1. As shown in fig. 1, the method includes: step 101, obtaining target data to be transferred.
The execution subject of the embodiment of the present invention is a track traffic data unloading device.
Specifically, the target data to be dumped in the embodiment of the invention is rail transit data. The target data to be transferred is a transfer object of the rail transit data transfer method provided by the invention. The target storage space is a transfer target of the target data to be transferred. The track traffic data unloading method provided by the invention can be used for unloading the target data to be unloaded to the target storage space.
It should be noted that, in the embodiment of the present invention, a data source of target data to be transferred may be referred to as a data source to be transferred. The type of the data source to be transferred and stored can be the same as or different from that of the target storage space; the protocol corresponding to the data source to be transferred may also be the same as or different from the protocol corresponding to the target storage space.
It should be noted that the number of the target storage spaces in the embodiment of the present invention may be one or more. For example: the target storage space can be a MySQL database; alternatively, the target storage space may include a MySQL database, an Oracle database, and a kafka message queue.
In the embodiment of the present invention, the target data to be transferred may be acquired in various ways, for example: the original data to be transferred and stored output by the data source to be transferred and stored can be obtained and used as the target data to be transferred and stored; or after the original data to be transferred and stored output by the data source to be transferred and stored is obtained, the original data to be transferred and stored is subjected to data processing, and the original data to be transferred and stored after the data processing is determined as the target data to be transferred and stored. The embodiment of the invention does not limit the specific way of obtaining the target data to be transferred.
As an optional embodiment, the obtaining target data to be transferred includes: and acquiring original data to be transferred.
It should be noted that the track traffic data unloading method provided by the present invention is based on the HTTP protocol, and the HTTP interface is exposed to the outside.
Correspondingly, in the embodiment of the present invention, the original data to be transferred output by the data source to be transferred may be acquired based on an HTTP interface.
Carrying out data verification on original data to be transferred; the data check includes a data format check.
Specifically, after the original data to be transferred is obtained, data verification may be performed on the original data to be transferred.
Optionally, in the embodiment of the present invention, the data verification on the original data to be transferred may include: and based on a preset format, carrying out format verification on the original data to be transferred, carrying out integrity verification and repeated submission prevention verification on the original data to be transferred, and the like.
Under the condition that the original data to be transferred and stored pass data verification, data processing is carried out on the original data to be transferred and stored;
wherein the data processing comprises data filtering.
Specifically, after the data verification is performed on the original data to be transferred, if the original data to be transferred passes the data verification, the data processing may be performed on the original data to be transferred.
It should be noted that, in the embodiment of the present invention, the data processing on the original data to be dumped may include, but is not limited to, data filtering, and the like.
And determining the original data to be transferred and stored after data processing as target data to be transferred and stored.
Specifically, after the data processing is performed on the original data to be transferred, the original data to be transferred after the data processing may be determined as the target data to be transferred.
According to the embodiment of the invention, the original data to be transferred is subjected to data verification, under the condition that the original data to be transferred passes the data verification, the original data to be transferred is further subjected to data processing, and the original data to be transferred after the data processing is determined as the target data to be transferred, so that the safety of the data to be transferred of the target can be improved, and the transfer failure caused by the problems of format error/repetition and the like can be avoided.
And 102, determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred.
Specifically, the configuration information of the target data to be dumped may include an interface request parameter topic _ id.
After the target data to be transferred and stored is obtained, a target storage space corresponding to the target data to be transferred and stored can be determined based on topic _ id in the target data to be transferred and stored.
It should be noted that, after determining the target storage space corresponding to the target data to be transferred, the configuration information of the target storage space may be further obtained. The configuration information may include a type of the target storage space and/or a protocol corresponding to the target storage space.
It should be noted that the types of the storage space in the embodiment of the present invention may include, but are not limited to, a MySQL database, a Qracle database, an ES database, a message queue, and the like; the storage space corresponding protocol may include, but is not limited to, HTTP protocol, UDP protocol, TCP protocol, and the like.
It should be noted that, based on the configuration information of the target storage space, the target table structure may also be queried, and the queried target table structure may be updated.
Specifically, after the target storage space corresponding to the target data to be transferred is determined, the type of the target storage space or the protocol corresponding to the target storage space may be determined based on the configuration information of the target storage space.
After the type of the target storage space or the protocol corresponding to the target storage space is determined, a unloading strategy corresponding to the target data to be unloaded may be determined based on a corresponding relationship between the type of the storage space and a preset unloading strategy, or based on a corresponding relationship between the protocol corresponding to the storage space and a preset unloading strategy.
It should be noted that the preset unloading strategy, the correspondence between the type of the storage space and the preset unloading strategy, and the correspondence between the protocol corresponding to the storage space and the preset unloading strategy may be predetermined based on prior knowledge and/or actual conditions. In the embodiment of the present invention, the preset unloading strategy, the corresponding relationship between the type of the storage space and the preset unloading strategy, and the corresponding relationship between the protocol corresponding to the storage space and the preset unloading strategy are not specifically limited.
And 104, transferring the target data to be transferred to the target storage space based on the transfer strategy.
Specifically, after determining a unloading strategy corresponding to the target data to be unloaded, unloading the target data to be unloaded based on the unloading strategy, and unloading the target data to be unloaded to the target storage space.
As an optional embodiment, the unloading the target data to be unloaded to the target storage space based on the unloading policy includes: and under the condition that the target storage space passes the verification, unloading the target data to be unloaded to the target storage space based on an unloading strategy.
Specifically, in the embodiment of the present invention, before executing the unloading policy corresponding to the target data to be unloaded, the target storage space may be verified, and whether the target storage space is empty is determined, so that an unloading failure of the target data to be unloaded due to the empty target storage space can be avoided.
When the target storage space passes the verification, it may be determined that the target storage space is not empty, and at this time, the target data to be transferred may be transferred to the target storage space by using a transfer policy corresponding to the target data to be transferred.
It should be noted that, after the target data to be transferred is transferred to the target storage space, the transfer result may be returned to the interface caller.
According to the embodiment of the invention, after the target storage space corresponding to the target data to be transferred is determined based on the configuration information of the target data to be transferred, the transfer strategy corresponding to the target data to be transferred is determined based on the type of the target storage space or the corresponding protocol, and then the target data to be transferred is transferred to the target storage space based on the transfer strategy, so that the transfer of the rail transit data can be realized more simply and more efficiently under the condition that the types of the data source and the storage space are different, the data can be synchronously transferred to a plurality of storage spaces, and the flexibility of the transfer of the rail transit data can be improved.
Based on the content of the foregoing embodiments, determining a load transfer policy corresponding to target data to be loaded for transfer based on the type of the target storage space includes: under the condition that the type of the target storage space is MySQL, determining that a transfer strategy corresponding to target data to be transferred comprises a first transfer strategy;
the first unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a Mybatis frame and a Springboot frame.
It should be noted that MySQL is a relational database management system, and the MySQL database can increase the data storage speed and improve the data storage flexibility by storing data in different tables.
It should be noted that, in the case that the number of the target storage space is one, if the type of the target storage space is MySQL, it may be determined that the target storage space is a MySQL database;
if the type of any target storage space is MySQL, the target storage space may be determined to be a MySQL database.
Specifically, under the condition that the type of the target storage space is MySQL, it may be determined that the unloading policy corresponding to the target data to be unloaded includes the first unloading policy based on the correspondence between the type of the storage space and the preset unloading policy, and then the target data to be unloaded may be unloaded to the MySQL database based on the first unloading policy. The first unloading strategy comprises unloading the target unloading data to the MySQL database based on a Mybatis framework and a Springboot framework.
And based on the first unloading strategy, unloading the target data to be unloaded to the MySQL database in an unloading data mode of the DBHandler.
The specific step of transferring the target data to be transferred to the MySQL database based on the first transfer strategy comprises the following steps: firstly, based on the Mybatis framework and the DataSourceID in the configuration information of the target storage space, the dump path of the target data to be dumped can be switched to the MySQL database.
Mybatis is a persistent layer framework that can support customized SQL, stored procedures, and advanced mapping, and avoids almost all JDBC codes and manual parameter setting and result set acquisition. Mybatis can also use simple XML or annotations to configure and map native information, mapping interfaces and Java pojs (Plain organisary Java Object) to records in the database.
Springboot is a framework that simplifies Spring development, providing a way to quickly use springs.
In the embodiment of the invention, the environment architecture for transferring the target transfer and storage data to the MySQL database is a Springboot framework.
It should be noted that, based on the Mybatis framework and the DataSourceID in the configuration information of the target storage space, the MySQL database may be obtained from the cache first. If the MySQL database is not acquired in the cache, the transfer path of the target data to be transferred may be switched to a default storage space, then all storage spaces are traversed, a storage space consistent with the DataSourceID is acquired, after the storage space consistent with the DataSourceID is acquired, the storage space may be set to a dynamic storage space manager, and the storage space is stored to the cache.
Then, according to the operation Type information (on Type) in the configuration information of the target data to be transferred, an SQL generation policy corresponding to the operation Type information may be generated, and an SQL statement corresponding to the target data to be transferred is generated.
And finally, based on the Mybatis framework, the SQL statement can be executed, so that the target data to be transferred and stored can be transferred and stored to the MySQL database in the target storage space.
In the embodiment of the invention, under the condition that the type of the target storage space is MySQL, the first unloading strategy corresponding to the target data to be unloaded is determined to comprise the first unloading strategy, and the first unloading strategy comprises unloading the target data to be unloaded into the MySQL database based on a Mybatis frame and a Springboot frame, so that the target data to be unloaded can be more simply and more efficiently unloaded into the MySQL database.
It should be noted that Oracle is a relational database management system, has good portability, convenient use and strong function, and is suitable for various large, medium and small microcomputer environments.
It should be noted that, under the condition that the number of the target storage space is one, if the type of the target storage space is Oracle, it may be determined that the target storage space is an Oracle database;
when the number of the target storage spaces is multiple, if the type of any target storage space is Oracle, it can be determined that the target storage space is an Oracle database.
Specifically, under the condition that the type of the target storage space includes Oracle, it may be determined that the unloading policy corresponding to the target data to be unloaded includes the first unloading policy based on the correspondence between the type of the storage space and the preset unloading policy, and then the target data to be unloaded may be unloaded to the Oracle database based on the first unloading policy.
And based on the first unloading strategy, unloading the target data to be unloaded to the Oracle database in an unloading data mode of the DBHandler.
The specific steps of transferring the target data to be transferred to the Oracle database based on the first transfer strategy comprise: firstly, based on the Mybatis framework and the DataSourceID in the configuration information of the target storage space, the unloading path of the target data to be unloaded can be switched to the Oracle database.
Then, according to the operation Type information (on Type) in the configuration information of the target data to be transferred, an SQL generation policy corresponding to the operation Type information may be generated, and an SQL statement corresponding to the target data to be transferred may be generated.
And finally, based on the Mybatis framework, the SQL statement can be executed, so that the target data to be transferred and stored can be transferred and stored to an Oracle database in the target storage space.
According to the embodiment of the invention, under the condition that the type of the target storage space is Oracle, the unloading strategy corresponding to the target data to be unloaded is determined to comprise the first unloading strategy, the first unloading strategy comprises unloading the target data to be unloaded into the Oracle database based on the Mybatis frame and the Spingboot frame, and the target data to be unloaded can be unloaded into the Oracle database more simply and more efficiently.
Based on the content of the foregoing embodiments, determining a load transfer policy corresponding to target data to be loaded based on the type of the target storage space includes: under the condition that the type of the target storage space is a message queue, determining that a transfer strategy corresponding to target data to be transferred comprises a second transfer strategy;
and the second unloading strategy comprises the steps of acquiring a message queue actuator based on the configuration information of the target storage space, analyzing the target data to be unloaded by using JSONObject, and unloading the analyzed target data to be unloaded to the target storage space based on the message queue actuator.
It should be noted that, a Message Queue (MQ) generally refers to Message middleware, and can implement forwarding of messages.
It should be noted that, under the condition that the number of the target storage space is one, if the type of the target storage space is a message queue, it may be determined that the target storage space is the message queue;
when the number of the target storage spaces is plural, if the type of any one of the target storage spaces is a message queue, it may be determined that the target storage space is a message queue.
It should be noted that the message queue in the embodiment of the present invention may include a Kafka message queue and/or an ActiveMq message queue.
Specifically, when the type of the target storage space is the message queue, it may be determined that the unloading policy corresponding to the target data to be unloaded includes the second unloading policy based on the correspondence between the type of the storage space and the preset unloading policy, and then the target data to be unloaded may be unloaded to the message queue based on the second unloading policy.
Based on the second unloading strategy, the specific step of unloading the target data to be unloaded to the message queue in the target storage space may include: firstly, generating a KEY value based on topicId and URL in the configuration information of a target storage space; acquiring a message queue executor (MqProducer) from a cache based on the KEY value;
in a case where the message queue executor is not obtained from the cache based on the KEY value, the type (MqType) of the target storage space may be obtained based on code attribute information in configuration information of the target storage space, and the message queue executor may be generated based on the MqType, topicId, and URL.
And secondly, analyzing the target data to be transferred and stored by using the JSONObject, and analyzing json data in the target data to be transferred and stored into a JSONObejct object.
And finally, based on the message queue actuator, transferring the analyzed target data to be transferred to a message queue.
In the embodiment of the invention, when the type of the target storage space is the message queue, the unloading strategy corresponding to the target data to be unloaded is determined to comprise a second unloading strategy, the second unloading strategy comprises configuration information based on the target storage space, a message queue actuator is obtained, the JSONObject is utilized to analyze the target data to be unloaded, and then the analyzed target data to be unloaded is unloaded to the message queue based on the message queue actuator, so that the target data to be unloaded can be more simply and more efficiently unloaded to the message queue.
Based on the content of the foregoing embodiments, determining a load transfer policy corresponding to target data to be loaded and transferred based on a protocol corresponding to a target storage space includes: under the condition that the protocol corresponding to the target storage space is an HTTP (hyper text transport protocol), a UDP (user datagram protocol) or a TCP (transmission control protocol), determining that the unloading strategy corresponding to the target data to be unloaded comprises a third unloading strategy;
and the third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on the protocol corresponding to the target storage space.
Specifically, under the condition that the protocol corresponding to the target storage space is the HTTP protocol, it may be determined that the unloading policy corresponding to the target data to be unloaded includes a third unloading policy, and the target unloading data may be unloaded to the target storage space based on the third unloading protocol. The third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on the HTTP.
And under the condition that the protocol corresponding to the target storage space is the HTTP protocol, the target data to be transferred can be transferred to the target storage space through the data storage method of the HTTP handle based on the third transfer strategy.
Under the condition that the protocol corresponding to the target storage space is the HTTP protocol, the specific step of transferring the target data to be transferred to the target storage space based on the third transfer strategy comprises the following steps: firstly, acquiring URL (uniform resource locator) and code attribute information in configuration information of a target storage space;
secondly, an httpclient technology is adopted to create objects such as HttpPost, requestConfig and the like, corresponding attributes are set, and an HTTP request is executed, so that target data to be transferred and stored are transferred and stored to a target storage space.
Under the condition that the protocol corresponding to the target storage space is the UDP protocol, it may be determined that the unloading policy corresponding to the target data to be unloaded includes a third unloading policy, and the target unloading data may be unloaded to the target storage space based on the third unloading protocol. And the third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a UDP protocol.
And under the condition that the protocol corresponding to the target storage space is the UDP protocol, the target data to be transferred and stored can be transferred and stored to the target storage space through a data storage method of the UDP handle based on the third transfer and storage strategy.
Under the condition that the protocol corresponding to the target storage space is a UDP protocol, the specific step of transferring the target data to be transferred to the target storage space based on the third transfer strategy comprises the following steps: firstly, acquiring a URL and a topicId in configuration information of a target storage space, and acquiring an ip and a port of the target storage space by analyzing the URL;
secondly, creating a datagrampocket object, and transferring the target data to be transferred to the target storage space through the DatagramSocket object, so that the datagrampocket object can be created based on a TCP protocol, and the target data to be transferred to the target storage space can be transferred.
Specifically, under the condition that the protocol corresponding to the target storage space is a TCP protocol, it may be determined that the unloading policy corresponding to the target data to be unloaded includes a third unloading policy, and the target unloading data may be unloaded to the target storage space based on the third unloading protocol. And the third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a TCP protocol.
And under the condition that the protocol corresponding to the target storage space is a TCP (transmission control protocol), the target data to be transferred can be transferred to the target storage space through a data storage method of TcpHandle based on the third transfer strategy.
Under the condition that the protocol corresponding to the target storage space is a TCP (transmission control protocol), the specific step of transferring the target data to be transferred to the target storage space based on the third transfer strategy comprises the following steps: firstly, acquiring a URL and a topicId in configuration information of a target storage space, acquiring an ip and a port of the target storage space by analyzing the URL, analyzing target data to be transferred and stored, and analyzing json type numbers in the target data to be transferred and stored into a byte data packet;
secondly, creating a Socket object and obtaining an output stream OutputStream, and unloading the target data to be unloaded to the target storage space through the output stream, so that the unloading of the target data to be unloaded to the target storage space can be realized by establishing a Socket (Socket) based on a TCP protocol.
Based on the content of the foregoing embodiments, determining a load transfer policy corresponding to target data to be loaded for transfer based on the type of the target storage space includes: under the condition that the type of the target storage space is ES, determining that the unloading strategy corresponding to the target data to be unloaded comprises a fourth unloading strategy;
and the fourth unloading strategy comprises the steps of obtaining an implementation class corresponding to the target storage space based on the operation type of the target storage space, and unloading the target data to be unloaded to the target storage space based on the implementation class.
It should be noted that the elastic search (ES for short) is a distributed, highly-extended, highly-real-time search and data analysis engine, and has the capability of enabling a large amount of data to be searched, analyzed and explored.
It should be noted that, under the condition that the number of the target storage space is one, if the type of the target storage space is ES, it may be determined that the target storage space is an ES database;
when the number of the target storage spaces is plural, if the type of any one of the target storage spaces is ES, it may be determined that the target storage space is an ES database.
Specifically, when the type of the target storage space is ES, it may be determined that the unloading policy corresponding to the target data to be unloaded includes a fourth unloading policy based on a corresponding relationship between the type of the storage space and a preset unloading policy, and the target data to be unloaded may be unloaded to the ES database based on the fourth unloading policy.
The specific step of transferring the target data to be transferred to the ES database based on the fourth transfer policy includes: first, an operation type (opType) of the ES database may be obtained in the ESContext, and an espandler implementation class corresponding to the ES database may be obtained according to the operation type, for example: ESInsertHandler, ESUpdateHandler, ESDeleteHandler, ESReplaceHandler; the ESHandler is a processor corresponding to the ES database;
secondly, acquiring a URL and a topicId in configuration information of the ES database, acquiring an ip and a port of the ES database by analyzing the URL, and further initializing a RestHighLevelClient based on the ip and the port of the ES database;
thirdly, creating an IndexRequest based on topiId in the configuration information of the ES database, and setting the type, the id and the data resource;
finally, indexRequest is transmitted through the RestHighLevelClient.
The embodiment of the invention determines that the unloading strategy corresponding to the target data to be unloaded comprises a fourth unloading strategy under the condition that the type of the target storage space is ES, wherein the first unloading strategy comprises the fourth unloading strategy and comprises an operation type based on the target storage space, the implementation class corresponding to the target storage space is obtained, the target data to be unloaded is unloaded to the target storage space based on the implementation class, and the target data to be unloaded can be unloaded to an ES database more simply and more efficiently.
In order to facilitate understanding of the rail transit data unloading method provided by the invention, the rail transit data unloading method provided by the invention is described below by an example.
Fig. 2 is a second schematic flow chart of the rail transit data unloading method provided by the present invention. As shown in fig. 2, after the interface is called to obtain the original data to be transferred, data verification may be performed on the original data to be transferred, whether the original data to be transferred has a format error and includes duplicated data is determined, and when the original data to be transferred passes the data verification, data processing is performed on the original data to be transferred, so as to obtain target data to be transferred.
And under the condition that the original data to be transferred does not pass the data verification, prompting can be carried out on a user interaction interface to prompt a user that the original data to be transferred has an error format, an incomplete format or a repeated format.
Based on the configuration information of the target data to be transferred, the target storage space corresponding to the target data to be transferred can be determined, and whether the target storage space is empty or not can be judged.
And under the condition that the target storage space is empty, prompting can be performed on a user interaction interface to prompt a user that the target storage space is empty.
Under the condition that the target storage space is not empty, a dump protocol can be determined based on the type of the target storage space or a protocol corresponding to the target storage space, and then the target data to be dumped can be dumped to the target storage space based on the dump protocol.
After the target data to be transferred is transferred to the target storage space, the transfer result can be returned to the interface caller.
Fig. 3 is a schematic structural diagram of a track traffic data unloading device provided by the invention. The track traffic data unloading device provided by the present invention is described below with reference to fig. 3, and the track traffic data unloading device described below and the track traffic data unloading method provided by the present invention described above may be referred to correspondingly. As shown in fig. 3, a dump data obtaining module 301, a storage space determining module 302, a dump policy determining module 303, and a data dump module 304.
A dump data obtaining module 301, configured to obtain target data to be dumped;
a storage space determining module 302, configured to determine, based on configuration information of the target data to be transferred and stored, a target storage space corresponding to the target data to be transferred and stored;
a unloading strategy determining module 303, configured to determine an unloading strategy corresponding to the target data to be unloaded, based on the type of the target storage space or a protocol corresponding to the target storage space;
and the data unloading module 304 is configured to unload the target data to be unloaded to the target storage space based on an unloading policy.
Specifically, the unloading data obtaining module 301, the storage space determining module 302, the unloading strategy determining module 303 and the data unloading module 304 are electrically connected.
The dump data obtaining module 301 may obtain the target data to be dumped through a plurality of ways, for example: the original data to be transferred and stored output by the data source to be transferred and stored can be obtained and used as the target data to be transferred and stored; or after the original data to be transferred and stored output by the data source to be transferred and stored is obtained, the original data to be transferred and stored is subjected to data processing, and the original data to be transferred and stored after the data processing is determined as the target data to be transferred and stored. The specific way of obtaining the target to-be-transferred data is not limited in the embodiment of the invention.
The storage space determining module 302 may determine, based on the topic _ id in the target data to be transferred, a target storage space corresponding to the target data to be transferred.
The unloading policy determining module 303 may determine the type of the target storage space or a protocol corresponding to the target storage space based on the configuration information of the target storage space.
After the type of the target storage space or the protocol corresponding to the target storage space is determined, a unloading strategy corresponding to the target data to be unloaded may be determined based on the corresponding relationship between the type of the storage space and the preset unloading strategy, or based on the corresponding relationship between the protocol corresponding to the storage space and the preset unloading strategy.
The data unloading module 304 can unload the target data to be unloaded and unload the target data to be unloaded to the target storage space based on the unloading strategy.
According to the track traffic data unloading device in the embodiment of the invention, after the target storage space corresponding to the target data to be unloaded is determined based on the configuration information of the target data to be unloaded, the unloading strategy corresponding to the target data to be unloaded is determined based on the type of the target storage space or the corresponding protocol, and then the target data to be unloaded is unloaded to the target storage space based on the unloading strategy.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor) 410, a communication Interface 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to perform a rail transit data unloading method comprising: acquiring target data to be transferred; determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred; determining a transfer strategy corresponding to target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space; and transferring the target data to be transferred to the target storage space based on the transfer strategy.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention further provides a computer program product, where the computer program product includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, a computer can execute the track traffic data unloading method provided by the above methods, and the method includes: acquiring target data to be transferred; determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred; determining a transfer strategy corresponding to target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space; and transferring the target data to be transferred to the target storage space based on the transfer strategy.
In still another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to execute the rail transit data unloading method provided by the above methods, and the method including: acquiring target data to be transferred; determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred; determining a transfer strategy corresponding to target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space; and transferring the target data to be transferred to the target storage space based on the transfer strategy.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A rail transit data unloading method is characterized by comprising the following steps:
acquiring target data to be transferred;
determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred;
determining a transfer strategy corresponding to the target data to be transferred based on the type of the target storage space or a protocol corresponding to the target storage space;
and unloading the target data to be unloaded to the target storage space based on the unloading strategy.
2. The rail transit data unloading method according to claim 1, wherein the determining, based on the type of the target storage space, an unloading strategy corresponding to the target data to be unloaded comprises:
under the condition that the type of the target storage space is MySQL or Qracle, determining that the unloading strategy corresponding to the target data to be unloaded comprises a first unloading strategy;
the first unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a Mybatis frame and a Springboot frame.
3. The rail transit data unloading method according to claim 1, wherein the determining, based on the type of the target storage space, an unloading strategy corresponding to the target data to be unloaded comprises:
under the condition that the type of the target storage space is a message queue, determining that a unloading strategy corresponding to the target data to be unloaded comprises a second unloading strategy;
and the second unloading strategy comprises the steps of acquiring a message queue actuator based on the configuration information of the target storage space, analyzing the target data to be unloaded by using JSONObject, and unloading the analyzed target data to be unloaded to the target storage space based on the message queue actuator.
4. The rail transit data unloading method according to claim 1, wherein the determining an unloading strategy corresponding to the target data to be unloaded based on the protocol corresponding to the target storage space comprises:
under the condition that the protocol corresponding to the target storage space is an HTTP (hyper text transport protocol), a UDP (user datagram protocol) or a TCP (transmission control protocol), determining that the unloading strategy corresponding to the target data to be unloaded comprises a third unloading strategy;
and the third unloading strategy comprises unloading the target data to be unloaded to the target storage space based on a protocol corresponding to the target storage space.
5. The rail transit data unloading method according to claim 1, wherein the determining, based on the type of the target storage space, an unloading strategy corresponding to the target data to be unloaded comprises:
under the condition that the type of the target storage space is ES, determining that the unloading strategy corresponding to the target data to be unloaded comprises a fourth unloading strategy;
and the fourth unloading strategy comprises the steps of acquiring an implementation class corresponding to the target storage space based on the operation type of the target storage space, and unloading the target data to be unloaded to the target storage space based on the implementation class.
6. The rail transit data unloading method according to claim 1, wherein the acquiring target data to be unloaded comprises:
acquiring original data to be transferred;
carrying out data verification on the original data to be transferred;
under the condition that the original data to be transferred and stored pass the data verification, performing data processing on the original data to be transferred and stored;
determining the original data to be transferred and stored after data processing as the target data to be transferred and stored;
wherein the data check comprises a data format check; the data processing includes data filtering.
7. The rail transit data unloading method according to any one of claims 1 to 6, wherein the unloading the target data to be unloaded to the target storage space based on the unloading strategy comprises:
and under the condition that the target storage space passes the verification, unloading the target data to be unloaded to the target storage space based on the unloading strategy.
8. A rail transit data unloading device is characterized by comprising:
the unloading data acquisition module is used for acquiring target data to be unloaded;
the storage space determining module is used for determining a target storage space corresponding to the target data to be transferred based on the configuration information of the target data to be transferred;
a transfer strategy determining module, configured to determine, based on the type of the target storage space or a protocol corresponding to the target storage space, a transfer strategy corresponding to the target data to be transferred;
and the data unloading module is used for unloading the target data to be unloaded to the target storage space based on the unloading strategy.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the rail transit data unloading method according to any one of claims 1 to 6 when executing the program.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the rail transit data unloading method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211737940.XA CN115985099A (en) | 2022-12-30 | 2022-12-30 | Rail transit data unloading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211737940.XA CN115985099A (en) | 2022-12-30 | 2022-12-30 | Rail transit data unloading method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115985099A true CN115985099A (en) | 2023-04-18 |
Family
ID=85964595
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211737940.XA Pending CN115985099A (en) | 2022-12-30 | 2022-12-30 | Rail transit data unloading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115985099A (en) |
-
2022
- 2022-12-30 CN CN202211737940.XA patent/CN115985099A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108566290B (en) | Service configuration management method, system, storage medium and server | |
| CN111245900B (en) | Distributed message sending processing system and processing method thereof | |
| CN113568686B (en) | Asynchronous processing method and device for Lua language, computer equipment and storage medium | |
| CN112835782B (en) | Interface access test method and system | |
| CN112988485A (en) | Simulation test method and device for power Internet of things equipment | |
| CN110532182B (en) | Automatic testing method and device for virtualization platform | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN112699034B (en) | Virtual login user construction method, device, equipment and storage medium | |
| CN111381940B (en) | Distributed data processing method and device | |
| CN113687858A (en) | Configuration file checking method and device, electronic equipment and storage medium | |
| CN112313627A (en) | Mapping mechanism of events to serverless function workflow instances | |
| CN112564980A (en) | Service monitoring method and system based on micro-service architecture | |
| CN117056240A (en) | Data element development and debugging method and system supporting offline Jar package | |
| CN111158654A (en) | Algorithm calling method, device, server and storage medium | |
| CN111046245A (en) | Multi-source heterogeneous data source fusion calculation method, system, equipment and storage medium | |
| CN117615057B (en) | Fault detection method, device, system, computer equipment and storage medium | |
| CN116204239A (en) | Service processing method, device and computer readable storage medium | |
| CN114610598A (en) | Test method, test device, electronic equipment and computer readable storage medium | |
| CN116974906A (en) | Code verification method, device, computer equipment and storage medium | |
| CN115985099A (en) | Rail transit data unloading method and device | |
| CN113364892B (en) | Domain name resolution method, related method, device and system for cross-multi-cluster service | |
| US10963331B2 (en) | Collecting repeated diagnostics data from across users participating in a document collaboration session | |
| CN112363774A (en) | Storm real-time task configuration method and device | |
| CN111858379A (en) | Application testing method and device | |
| CN113031960B (en) | Code compiling method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |