CN117407445B - Data storage method, system and storage medium for Internet of vehicles data platform - Google Patents
Data storage method, system and storage medium for Internet of vehicles data platform Download PDFInfo
- Publication number
- CN117407445B CN117407445B CN202311408021.2A CN202311408021A CN117407445B CN 117407445 B CN117407445 B CN 117407445B CN 202311408021 A CN202311408021 A CN 202311408021A CN 117407445 B CN117407445 B CN 117407445B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- hbase
- mysql
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000013500 data storage Methods 0.000 title claims abstract description 53
- 238000003860 storage Methods 0.000 title claims abstract description 24
- 235000006719 Cassia obtusifolia Nutrition 0.000 claims abstract description 35
- 235000014552 Cassia tora Nutrition 0.000 claims abstract description 35
- 244000201986 Cassia tora Species 0.000 claims abstract description 35
- 238000013507 mapping Methods 0.000 claims abstract description 16
- 230000006855 networking Effects 0.000 claims abstract description 5
- 238000012545 processing Methods 0.000 claims description 71
- 230000006870 function Effects 0.000 claims description 17
- 230000005540 biological transmission Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000012217 deletion Methods 0.000 description 6
- 230000037430 deletion Effects 0.000 description 6
- 238000013461 design Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000003780 insertion Methods 0.000 description 4
- 230000037431 insertion Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013499 data model Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data storage method, a system and a storage medium for a vehicle networking data platform, wherein the method comprises the following steps: step S100, setting a database connector to convert a query instruction into a corresponding query statement according to a database to be connected so as to execute a query operation; step S200, mapping the table name of MySQL into a column family of HBase, and mapping the column name of MySQL to a key of a key value pair in the column family of HBase to uniformly convert a query result set ResultSet of MySQL into a query result set ResultScanner of HBase in a mapping mode; step S300, the database connector acquires a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converts the value into a predefined JavaBean object, and returns to the client for acquiring the database query Result by using a getter method provided in the JavaBean. Therefore, the MySQL and HBase query result sets are unified, and the data storage requirements of different scales can be flexibly met.
Description
Technical Field
The invention relates to a data storage technology, in particular to a data storage method, a data storage system and a storage medium for combining two different databases of MySQL and HBase in a vehicle networking data platform.
Background
As the number of vehicles increases, so does the data storage and processing requirements. In the past, the amount of data was small and a single node was sufficient. However, as the size of vehicles increases and computing metrics becomes complex, more server resources are required to process and store data.
Therefore, how to build a data processing platform of the internet of vehicles so as to adapt to small-scale data and meet the requirement of large-scale data, so as to flexibly cope with the data storage requirements of different scales and meet the diversity of the data processing platform of the internet of vehicles is a difficult problem to be solved in the field.
Accordingly, the inventor considers that MySQL is suitable for storing small-scale data, and HBase focuses on distributed storage of massive data, so that the requirement of large-scale data storage can be met. Therefore, if the advantages of the two are combined, the above-mentioned problems can be solved, but the problem is that MySQL and HBase are two different databases, so there is a problem of inconsistency between query result sets of the two, and thus, compatibility is not achieved.
Disclosure of Invention
Therefore, the main purpose of the invention is to provide a data storage method, a system and a storage medium for a data platform of the Internet of vehicles, so as to realize the unification of MySQL and HBase query result sets and flexibly meet the data storage requirements of different scales.
In order to achieve the above object, according to a first aspect of the present invention, there is provided a data storage method for a data platform of internet of vehicles, comprising the steps of:
step S100 sets up a database connector to acquire the IP address and the connection information in the database configuration file according to the issued query instruction to determine the type MySQL or HBase of the database to be connected, and converts the query instruction into a corresponding query statement to execute the query operation.
Step S200, mapping the table name of MySQL into a column family of HBase, and mapping the column name of MySQL to a key of a key value pair in the column family of HBase to uniformly convert a query result set ResultSet of MySQL into a query result set ResultScanner of HBase in a mapping mode;
Step S300, the database connector acquires a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converts the value into a predefined JavaBean object, and returns to the client for acquiring a database query Result by using a getter method provided in the JavaBean;
step S400 writes real-time processed data into MySQL database in batches and writes streaming processed data into HBase database through Sink function of Flink.
In a possible preferred embodiment, the method for storing data of the internet of vehicles data platform further includes:
step S410 optimizes MySQL data writing, uses the Sink function of Flink, and sets intervals of time or data number by configuring the jdbcsink.sink of Flink-connector-jdbc to allow data to be written into the MySQL database in a timed batch.
In a possible preferred embodiment, the method for storing data of the internet of vehicles data platform further includes:
step S420 optimizes HBase data writing, and uses window functions in the API of the link to implement writing data into the HBase database at fixed time intervals by means of a time window.
In order to achieve the above object, corresponding to the above method, according to a second aspect of the present invention, there is also provided an internet of vehicles data platform data storage system, comprising:
The storage unit is used for storing a program comprising the steps of the data storage method of the internet of vehicles data platform, so that the database unit, the acquisition unit, the transmission unit and the processing unit can be timely adjusted and executed;
The database unit comprises a database connector, a MySQL database and an HBase database, wherein the database connector is used for converting a query instruction into a corresponding query statement according to the database to be connected so as to execute a query operation, and converting a queried MySQL query result set ResultSet into a query result set ResultScanner of the HBase in a mapping mode; the database connector is also used for acquiring a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converting the value into a predefined JavaBean object, and returning to the client for acquiring a database query Result by using a getter method provided in the JavaBean;
the acquisition unit is used for collecting vehicle data and converting the vehicle data into first data in a preset format;
The transmission unit is used for transmitting the first data to the processing unit in real time in a message queue mode;
And the processing unit is used for writing the first data processed in real time into the MySQL database in batches through the Sink of the Flink and writing the data processed in the stream into the HBase database.
In a possibly preferred embodiment, wherein the processing unit further uses a Sink function of flank, the interval of time or number of data pieces is set by configuring the jdbcssink. Sink of Flink-connector-jdbc to allow for the timed batch writing of the first data into the MySQL database.
In a possibly preferred embodiment, wherein the processing unit further uses a window function in the Flink API to enable writing of the first data into the HBase database at regular time intervals by means of a time window.
In order to achieve the above object, corresponding to the above method, according to a third aspect of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the steps of the internet of vehicles data platform data storage method as described in any one of the above.
The data storage method, the system and the storage medium of the internet of vehicles data platform provided by the invention skillfully adopt the mapping technology, unify the query result sets of HBase and MySQL, so that the internet of vehicles product platform can combine two different databases of MySQL and HBase to use, realize the balance of small-scale and large-scale data storage, and meet the data storage requirements of different scales and the data storage diversity requirement of the internet of vehicles data processing platform.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:
FIG. 1 is a schematic diagram of steps of a data storage method of an Internet of vehicles data platform according to the present invention;
FIG. 2 is a schematic diagram of database query logic in the data storage method of the Internet of vehicles data platform according to the present invention;
FIG. 3 is a schematic diagram of a MySQL database architecture in the Internet of vehicles data platform data storage method of the present invention;
fig. 4 is a schematic diagram of an HBase database architecture in the internet of vehicles data platform data storage method of the present invention;
FIG. 5 is a conceptual diagram of the overall operation flow of the Internet of vehicles data platform data storage system according to the present invention;
FIG. 6 is a conceptual diagram of data collection of the Internet of vehicles data platform data storage system of the present invention;
FIG. 7 is a conceptual diagram illustrating the flow processing of the Internet of vehicles data platform data storage system of the present invention;
Fig. 8 is a conceptual diagram of batch processing of the internet of vehicles data platform data storage system according to the present invention.
Detailed Description
In order that those skilled in the art can better understand the technical solutions of the present application, the following description will clearly and completely describe the specific technical solutions of the present application in conjunction with the embodiments to help those skilled in the art to further understand the present application. It will be apparent that the embodiments described herein are merely some, but not all embodiments of the application. It should be noted that embodiments of the present application and features of embodiments may be combined with each other by those of ordinary skill in the art without departing from the spirit of the present application and without conflicting with each other. All other embodiments, which are derived from the embodiments herein without creative effort for a person skilled in the art, shall fall within the disclosure and the protection scope of the present application.
Furthermore, the terms "first," "second," "S100," "S200," and the like in the description and in the claims and drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those described herein. Also, the terms "comprising" and "having" and any variations thereof herein are intended to cover a non-exclusive inclusion. Unless specifically stated or limited otherwise, the terms "disposed," "configured," "mounted," "connected," "coupled" and "connected" are to be construed broadly, e.g., as being either permanently connected, removably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the terms in this case will be understood by those skilled in the art in view of the specific circumstances and in combination with the prior art.
In order to enable the internet of vehicles data processing platform to flexibly cope with the data storage requirements of different scales, mySQL is considered to be suitable for storing small-scale data, and HBase is focused on the distributed storage of massive data, so that the requirement of large-scale data storage can be met, and if two databases of different types, namely HBase and MySQL, can be combined for use, the balance can be achieved in terms of meeting the performance and cost, and therefore the internet of vehicles data processing platform is better suitable for the data processing requirements of different scales.
For this purpose, as shown in fig. 1 to 4, the invention provides a data storage method for a data platform of internet of vehicles, which comprises the following steps:
Step S100 sets a database connector to convert the query instruction into a corresponding query statement according to the database to be connected to perform the query operation.
Specifically, in a preferred example, first the client connects to a database connector to issue a query instruction, the database connector obtains the IP address and connection information in the database configuration file to determine the database to be queried, and converts the database to a specific query statement according to the type of database to be connected, for example:
And judging whether the database to be queried is MySQL or HBase according to the configuration information, and generating a corresponding query language. And respectively generating corresponding SQL sentences or HBase scanning operations by configuring the date range and the vehicle ID.
When connected to the MySQL database for a query, the database connector converts the query request into an SQL statement and performs the query operation in the MySQL database.
For example, the client requests to query "data with vehicle IDs 1111, 2222, and 3333 between 2023, 1 and 2023, 1, 2", when the SQL statement ： SELECT*FROMt_vehicle_dataWHEREf_vehicle_idIN(1111,2222,3333)ANDf_date_time>='2023-01-0100:00:00'ANDf_date_time<'2023-01-0200:00:00', is to be generated and then query in the database returns MySQL as the query result set ResultSet.
When the HBase database is connected for query, the database connector generates Rowkey in the corresponding HBase table according to the vehicle ID and time.
For example, when the client requests to query "when data with 1111 ID between 2023, 1 and 2023, 1,2 is queried", it generates data with 11112023-01-01:00:00 at start Rowkey and 11112023-01-02-02:00:00 at end Rowkey, then invokes the Scan class in HBase-client package to Scan the table in HBase, obtains data corresponding to 1111 days, and returns the result as result set ResultScanner.
At this time, it is not difficult to find that, because the database result sets returned by the two database queries are inconsistent, in order to achieve consistency of the query result sets, the result set returned by MySQL needs to be converted into a format similar to that of the HBase query result set.
Step S200 maps the table name of MySQL to the column family of HBase, and corresponds the column name of MySQL to the key of the key value pair in the HBase column family, so as to uniformly convert the query result set ResultSet of MySQL into the query result set ResultScanner of HBase by mapping.
Specifically, in order to achieve consistency between MySQL and HBase query result sets, in this example, the query result set returned by MySQL is preferably converted into a format similar to the HBase query result set by a mapping means, for example, a table name of MySQL is mapped to a column group of HBase, columns in the table are corresponding one by one, and assignment is performed according to a key value pair under the HBase column group.
For example, the column f_veccle_state value in MySQL table t_veccle_data is 1, and will be mapped to a key value pair under column family VD in HBase, where key is vstat and value is 1; the column f_running_state value in MySQL table t_vehicle_data is 2, and the value is mapped into a key value pair under column group VD in HBase, wherein key is run_state, and value is 2. Through such processing, the query results of MySQL will be constructed in the form of an HBase query result set, simulating the results queried from the HBase database, so that the result sets are all converted into ResultScanner to realize consistency of the query result set, and the query result sets can be combined for unified processing in subsequent processing.
Step S300, the database connector acquires a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converts the value into a predefined JavaBean object, and returns to the client for acquiring the database query Result by using a getter method provided in the JavaBean.
Specifically, when the query result set is unified, the present example preferably converts the result set into a form of JavaBean (POJO, plain Old Java Object).
Wherein the database connector obtains values of a specific column using the Result class getValue method and converts the values into a predefined java bean object. For example, a vehicle state value is obtained by result.getvalue (bytes.tobytes ("VD"), bytes.tobytes ("vstat")) and then assigned to the vehicle state attribute java bean.setvehicle state (VEHICLESTATE). This process will be done sequentially for the different columns.
Once the JavaBeans are constructed and assigned, the connector returns these JavaBeans to the client. After the client receives the java beans, specific data can be obtained through a getter method provided in the java beans, so that the query result is processed, for example, a value java bean.
The purpose of this is to bring various advantages to converting the data into a java bean. Such as:
Object-oriented abstraction: the JavaBean abstracts the data model into objects so that business logic is more naturally expressed in code.
Readability and maintainability: the java bean promotes the readability and maintainability of the code. By using meaningful attribute names and method names, the code is easier to understand, track, and debug.
Encapsulation: the getter and setter methods of javabeans provide a unified attribute access interface. This encapsulation reduces the likelihood of direct access to the data, thereby increasing the security and hiding of the data.
Code reuse: the design of javabeans encourages reuse of objects. The same JavaBean can be used for multiple times in different business logics, so that similar data processing codes are prevented from being repeatedly written, and development efficiency is improved.
Unification of data formats: the JavaBean maintains the uniformity of the data format. When processing data from different data sources, the data is converted into a JavaBean form, which helps to standardize the format and structure of the data, so that the data processing is smoother and more consistent. In fact, this is also one of the main reasons we choose to convert data into JavaBean.
Serialization and deserialization: the javabeans implement Serializable interfaces so that they can be serialized and de-serialized in network transport, persistent storage, etc. scenarios. This not only enables persistence of the data, but also enables data to be transferred across systems. This feature becomes particularly important and convenient, especially in cluster processing scenarios.
Compatibility of data storage: by converting the data into a form of JavaBean, the results, whether queried from different types of databases, can be presented in the same JavaBean data format. This consistency facilitates data acquisition operations while providing compatibility for multiple database environments.
Therefore, through the scheme, the compatibility of data query can be realized, and the results of query from MySQL or HBase databases can be presented in the same JavaBean data format, so that the operation of data acquisition is simplified. By means of the configuration file, which database to query data from can be easily specified, compatibility of MySQL and HBase query is achieved, and logic of data acquisition is not required to be changed. The design not only improves flexibility and efficiency in the data query and acquisition stage, but also reduces the complexity of codes, and provides solid support for the stability and expandability of the data platform.
Furthermore, the basic data operation function of the database includes addition, deletion and modification besides the inquiry. The core of the scheme is to provide an efficient data storage service for the internet of vehicles data platform, wherein the data is derived from the vehicle-mounted terminal, such as collecting vehicle information at intervals of every 10 seconds or 30 seconds and writing the vehicle information into a database, and the process does not comprise data modification operation.
Therefore, the key to vehicle network data storage is to efficiently support efficient insertion, query, and deletion operations of data.
To this end, to optimize data insertion, two different databases, mySQL and HBase, may be used in combination in the preferred example for different data models to optimize the data insertion process. For example, mySQL stores data in a relational database, and HBase stores data as key-value pairs in a column-family storage mode. To adapt these different data models, mapping and conversion of the data formats may be performed. Specifically, the table name of MySQL may be converted into a column family in HBase, while the column name of MySQL is corresponding to the key of the key-value pair in the HBase column family.
Furthermore, in order to efficiently write the real-time processed internet of vehicles data into two different types of databases, namely MySQL and HBase, the characteristics of the respective databases are fully utilized to adapt to different storage requirements, and the following optimization strategies are adopted:
Step S410 is to use the Sink function of the Flink for the MySQL database, and set time or interval of data number by configuring the jdbcsink.sink of Flink-connector-jdbc so as to realize the regular batch writing of data into the MySQL database and ensure the effective storage of data.
Step S420 HBase data write optimization: for the HBase database, the writing of data at fixed time intervals can be achieved by means of a time window using a window function in the API of the link. The method can write the data processed in real time into the HBase in a batch mode, thereby ensuring continuous storage and efficient query of the data.
In terms of data deletion optimization, first, a backup of historical data must be made to ensure that the data is not lost due to the deletion operation. To avoid repeated development, the present example preferably designs custom scripts to implement backup operations for different databases.
Specifically, the backup of data is achieved by invoking a corresponding database command or tool. For example, mySQL data may be backed up using the 'mysqldump' command, while HBase data may be backed up using the HBase org.apache.hadoop.hbase.mapreduce.export command of HBase. Once the backup is complete, a script may be called to delete MySQL data, or the history data may be cleaned up by setting TTL (Time to Live) in the HBase. The design can ensure the safe backup and efficient deletion of the data, and improve the reliability of overall data management.
Through the data storage scheme, HBase and MySQL can be combined to meet the data processing requirements of different scales. Through the optimization strategies of data insertion, query and deletion, the data storage service of the Internet of vehicles data platform is effectively supported, and the high efficiency, query performance and stability of the data are ensured.
On the other hand, as shown in fig. 5 to 8, the present invention further provides a data storage system of a data platform of internet of vehicles, corresponding to the above method example, which includes:
The storage unit is used for storing a program comprising the steps of the data storage method of the internet of vehicles data platform, so that the database unit, the acquisition unit, the transmission unit and the processing unit can timely call and execute the program.
The database unit comprises a database connector, a MySQL database and an HBase database, wherein under the condition of small data volume: a single-node MySQL database is used to store small-scale data. The method is a relational database, has higher performance and reliability, and is suitable for storing small-scale data. For example, in terms of data processing, a single-node Standalone mode is deployed for the Flink, the stream processing is operated in the mode, and services such as vehicle alarming, real-time statistics of mileage per day, fence alarming and the like can be deployed in the mode; the batch processing is deployed and operated in a single-node multithreading mode, and indexes such as acceleration, rapid acceleration and deceleration times, average daily mileage and the like can be counted.
When the data amount is large: HBase is a distributed database, and each node stores a part of data by building HBase clusters. This allows the system to scale horizontally, as the amount of data increases, more nodes can be added to share the load without affecting the performance and availability of the system. The method is high in expandability and availability, suitable for processing mass data, and capable of storing the data on different nodes in a scattered mode, and further meeting the large-scale data storage requirement of data processing of the Internet of vehicles. For example, in terms of data processing, stream processing can be run in a distributed mode of a cluster by using the Flink on yarn, and the memory core number of the Flink and the parallelism of services can be freely switched according to the data quantity. Batch deployments operate in a multi-threaded mode of multiple nodes.
In order to ensure the compatibility of the two databases, the database connector is used for converting the query instruction into a corresponding query statement according to the databases to be connected so as to execute the query operation, and converting the queried MySQL query result set ResultSet into a query result set ResultScanner of HBase in a mapping mode; the database connector is further configured to obtain a value of a specific column corresponding to the vehicle information in the query Result set by using a getValue method of the Result class, to convert the value into a predefined java bean object, and return to the client to obtain the database query Result by using a getter method provided in the java bean.
The acquisition unit is used for gathering vehicle data and converting the vehicle data into first data in a preset format. In particular, the data acquisition unit of the internet of vehicles is a vital part of the internet of vehicles system, which involves collecting data from vehicle terminals and other sensors for subsequent processing and analysis. In this example, the preferred internet of vehicles data platform employs the TCP/IP protocol for data acquisition, receiving raw data from vehicle terminals and other sensors. Then, the data are parsed according to the national standard GB32960 and the partial standard JT/T808 protocols, and are converted into a readable format.
The parsed data is then converted into the first data in Protobuf format to obtain advantages such as reduced storage and transmission costs and improved data processing efficiency. Protobuf is a high-efficiency binary data serialization format, the serialization and deserialization speed is 5 times faster than that of JSON, the size is only one third of that of JSON, and the memory occupation is only one third of that of JSON.
As shown in fig. 5, after the parsed data is converted into the first data in the Protobuf format, the first data is written into the local file. The primary purpose of this is to ensure that data is not lost in the event of a failure or maintenance of the data processing components. By using the local file as a safeguard mechanism, data loss can be prevented in case of an abnormality, in particular during a crash or maintenance of the data processing components.
And the transmission unit is used for transmitting the first data to the processing unit in real time in a message queue mode. Specifically, as shown in fig. 5 to 6, the first data after being converted by the Protobuf format is pushed to Kafka. Kafka is an efficient message queuing system suitable for real-time data transmission. The design ensures that the Internet of vehicles data platform can efficiently and reliably process data in all stages of data acquisition, analysis, conversion and pushing, thereby ensuring the integrity, restorability and real-time of the data.
The message queue component is utilized to buffer the condition that the processing speeds of the production message and the consumption message are inconsistent, and the peak clipping function can be realized under the condition that the data of the production end are greatly increased so as to conveniently extract the real-time data.
The internet of vehicles data processing platform uses Kafka as a message queue component for efficient real-time data transmission. The Kafka distributed architecture is flexible and various, can be deployed by a single node or a cluster, and has strong flexibility and expandability. Kafka uses a streaming processing technology to transmit data in real time, supports high-concurrency message reading and writing, has strong adaptability to Flink, and can fully meet the real-time transmission requirement of a vehicle networking platform.
And the processing unit is used for writing the first data processed in real time into the MySQL database in batches through the Sink of the Flink and writing the data processed in the stream into the HBase database.
Specifically, as shown in fig. 7 to 8, the internet of vehicles data processing platform uses a mode of stream processing and batch processing, and real-time processing timely transmits and processes real-time data of vehicles, so as to provide real-time information and insight for vehicles and traffic management departments. The batch processing is to deeply analyze the historical data of the vehicles so as to find potential modes, trends and problems, and provide long-term analysis and monitoring for the vehicles and traffic management departments.
In the aspect of real-time processing, the Internet of vehicles data processing platform adopts a Flink technology, which is a stream processing framework, and can efficiently process real-time stream data and ensure the real-time performance and reliability of data processing.
In the case of small data volumes, the Flink may run independently on a single node using either the Local-cluster or Standalone modes. The Sink of the Flink writes the data processed in real time into the MySQL database in batches, and real-time storage and query of the data are realized by utilizing the high performance and reliability of MySQL.
In the case of large data volumes, the Flink may operate in a distributed mode of the cluster using a Flink on yarn or a Flink on 8s mode. The Sink of the link writes the data subjected to stream processing into the HBase cluster efficiently, and the advantages of the HBase in processing large data volume and high concurrency requests are fully exerted. The flexibility and scalability of the flank also makes it an ideal choice for internet of vehicles data processing.
In the aspect of off-line processing, the Internet of vehicles data processing platform adopts batch processing service to read a database, can deploy a single node or a plurality of nodes according to the size of data volume, and independently processes the data of each vehicle in a multithreading mode. And when the data volume is small, the Internet of vehicles data processing platform selects MySQL to be used as a component for data storage and processing.
MySQL is a relational database, has higher performance and reliability, supports a multithreading processing mode, fully utilizes a multi-core processor of a modern computer, and improves the efficiency and accuracy of data processing. When the data volume is large, the HBase is a distributed database, has high expandability and high availability, and utilizes the parallel processing capacity of a plurality of servers to meet the requirement of vehicle networking data processing and improve the efficiency and accuracy of data processing. Therefore, a mode of combining real-time processing and batch processing is adopted, and comprehensive and efficient data processing capacity can be realized.
Further, in a preferred example, the processing unit further uses a Sink function of flank to set an interval of time or number of data pieces by configuring a jdbcssink.sink of Flink-connector-jdbc to allow the first data to be written into the MySQL database in a timed batch.
Further, in a preferred example, the processing unit further uses a window function in the API of the link to implement writing the first data into the HBase database at regular time intervals by means of a time window.
In another aspect, the present invention also provides a computer readable storage medium having a computer program stored thereon, where the computer program, when executed by a processor, implements the steps of the internet of vehicles data platform data storage method as described in any one of the above.
In summary, the mapping technology is skillfully adopted by the data storage method, the system and the storage medium of the internet of vehicles data platform, and the query result sets of the HBase and the MySQL are unified, so that the internet of vehicles product platform can combine two different databases of the MySQL and the HBase to use, balance of small-scale and large-scale data storage is realized, and the data storage requirements of different scales and the data storage diversity requirement of the internet of vehicles data processing platform are met.
The preferred embodiments of the invention disclosed above are intended only to assist in the explanation of the invention. The preferred embodiments are not exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. The invention is to be limited only by the following claims and their full scope and equivalents, and any modifications, equivalents, improvements, etc., which fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
It will be appreciated by those skilled in the art that the system, apparatus and their respective modules provided by the present invention may be implemented entirely by logic programming method steps, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., except for implementing the system, apparatus and their respective modules provided by the present invention in a purely computer readable program code. Therefore, the system, the apparatus, and the respective modules thereof provided by the present invention may be regarded as one hardware component, and the modules included therein for implementing various programs may also be regarded as structures within the hardware component; modules for implementing various functions may also be regarded as being either software programs for implementing the methods or structures within hardware components.
Furthermore, all or part of the steps in implementing the methods of the embodiments described above may be implemented by a program, where the program is stored in a storage medium and includes several instructions for causing a single-chip microcomputer, chip or processor (processor) to execute all or part of the steps in the methods of the embodiments of the application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In addition, any combination of various embodiments of the present invention may be performed, so long as the concept of the embodiments of the present invention is not violated, and the disclosure of the embodiments of the present invention should also be considered.
Claims (7)
1. A data storage method of an Internet of vehicles data platform comprises the following steps:
Step S100, setting a database connector to acquire an IP address and connection information in a database configuration file according to a query issuing instruction so as to determine a database type MySQL or HBase to be connected, and converting the query instruction into a corresponding query statement so as to execute query operation;
Step S200, mapping the table name of MySQL into a column family of HBase, and mapping the column name of MySQL to a key of a key value pair in the column family of HBase to uniformly convert a query result set ResultSet of MySQL into a query result set ResultScanner of HBase in a mapping mode;
Step S300, the database connector acquires a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converts the value into a predefined JavaBean object, and returns to the client for acquiring a database query Result by using a getter method provided in the JavaBean;
step S400 writes real-time processed data into MySQL database in batches and writes streaming processed data into HBase database through Sink function of Flink.
2. The internet of vehicles data platform data storage method of claim 1, further comprising the steps of:
step S410 optimizes MySQL data writing, uses the Sink function of Flink, and sets intervals of time or data number by configuring the jdbcsink.sink of Flink-connector-jdbc to allow data to be written into the MySQL database in a timed batch.
3. The internet of vehicles data platform data storage method of claim 1, further comprising the steps of:
step S420 optimizes HBase data writing, and uses window functions in the API of the link to implement writing data into the HBase database at fixed time intervals by means of a time window.
4. A vehicle networking data platform data storage system, comprising:
The storage unit is used for storing a program comprising the steps of the data storage method of the internet of vehicles data platform according to any one of claims 1 to 3, so that the database unit, the acquisition unit, the transmission unit and the processing unit can be timely called and executed;
The database unit comprises a database connector, a MySQL database and an HBase database, wherein the database connector is used for converting a query instruction into a corresponding query statement according to the database to be connected so as to execute a query operation, and converting a queried MySQL query result set ResultSet into a query result set ResultScanner of the HBase in a mapping mode; the database connector is also used for acquiring a value of a specific column corresponding to the vehicle information in the query Result set by using a Result value method of a Result class, converting the value into a predefined JavaBean object, and returning to the client for acquiring a database query Result by using a getter method provided in the JavaBean;
the acquisition unit is used for collecting vehicle data and converting the vehicle data into first data in a preset format;
The transmission unit is used for transmitting the first data to the processing unit in real time in a message queue mode;
And the processing unit is used for writing the first data processed in real time into the MySQL database in batches through the Sink of the Flink and writing the data processed in the stream into the HBase database.
5. The internet of vehicles data platform data storage system of claim 4, wherein the processing unit further uses a Sink function of flank to set intervals of time or number of data pieces by configuring a jdbcssink.sink of Flink-connector-jdbc to allow for timed batch writing of the first data to the MySQL database.
6. The internet of vehicles data platform data storage system of claim 4, wherein the processing unit further uses a window function in the Flink API to enable writing the first data to the HBase database at regular intervals via a time window.
7. A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the steps of the internet of vehicles data platform data storage method according to any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311408021.2A CN117407445B (en) | 2023-10-27 | 2023-10-27 | Data storage method, system and storage medium for Internet of vehicles data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311408021.2A CN117407445B (en) | 2023-10-27 | 2023-10-27 | Data storage method, system and storage medium for Internet of vehicles data platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117407445A CN117407445A (en) | 2024-01-16 |
| CN117407445B true CN117407445B (en) | 2024-06-04 |
Family
ID=89495838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311408021.2A Active CN117407445B (en) | 2023-10-27 | 2023-10-27 | Data storage method, system and storage medium for Internet of vehicles data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117407445B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011130706A2 (en) * | 2010-04-16 | 2011-10-20 | Salesforce.Com, Inc. | Methods and systems for performing cross store joins in a multi-tenant store |
| CN106294499A (en) * | 2015-06-09 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of database data querying method and equipment |
| CN107273506A (en) * | 2017-06-19 | 2017-10-20 | 西安电子科技大学 | A kind of method of database multi-list conjunctive query |
| CN109992596A (en) * | 2019-02-25 | 2019-07-09 | 新智云数据服务有限公司 | Data processing method and device |
| WO2020015495A1 (en) * | 2018-07-16 | 2020-01-23 | 中兴通讯股份有限公司 | Method, device and apparatus for accessing heterogeneous database, plug-in and storage medium |
| CN110879812A (en) * | 2019-11-20 | 2020-03-13 | 浪潮软件股份有限公司 | Spark-based data synchronization method in e-commerce platform |
| CN111241091A (en) * | 2019-12-29 | 2020-06-05 | 南京云帐房网络科技有限公司 | Distributed column-type data storage conversion method and system for business report data |
| WO2021068549A1 (en) * | 2019-10-10 | 2021-04-15 | 苏宁易购集团股份有限公司 | Data processing method, platform and system |
| CN113821573A (en) * | 2021-08-27 | 2021-12-21 | 济南浪潮数据技术有限公司 | Mass data rapid retrieval service construction method, system, terminal and storage medium |
| CN113868306A (en) * | 2021-08-31 | 2021-12-31 | 云南昆钢电子信息科技有限公司 | Data modeling system and method based on OPC-UA specification |
| CN114153895A (en) * | 2021-11-17 | 2022-03-08 | 深圳市六度人和科技有限公司 | Construction method of real-time warehouse counting frame and real-time warehouse counting frame |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106156168B (en) * | 2015-04-16 | 2019-10-22 | 华为技术有限公司 | Across the method and across subregion inquiry unit for inquiring data in partitioned data base |
| US10121169B2 (en) * | 2015-09-16 | 2018-11-06 | Amobee, Inc. | Table level distributed database system for big data storage and query |
-
2023
- 2023-10-27 CN CN202311408021.2A patent/CN117407445B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011130706A2 (en) * | 2010-04-16 | 2011-10-20 | Salesforce.Com, Inc. | Methods and systems for performing cross store joins in a multi-tenant store |
| CN106294499A (en) * | 2015-06-09 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of database data querying method and equipment |
| CN107273506A (en) * | 2017-06-19 | 2017-10-20 | 西安电子科技大学 | A kind of method of database multi-list conjunctive query |
| WO2020015495A1 (en) * | 2018-07-16 | 2020-01-23 | 中兴通讯股份有限公司 | Method, device and apparatus for accessing heterogeneous database, plug-in and storage medium |
| CN109992596A (en) * | 2019-02-25 | 2019-07-09 | 新智云数据服务有限公司 | Data processing method and device |
| WO2021068549A1 (en) * | 2019-10-10 | 2021-04-15 | 苏宁易购集团股份有限公司 | Data processing method, platform and system |
| CN110879812A (en) * | 2019-11-20 | 2020-03-13 | 浪潮软件股份有限公司 | Spark-based data synchronization method in e-commerce platform |
| CN111241091A (en) * | 2019-12-29 | 2020-06-05 | 南京云帐房网络科技有限公司 | Distributed column-type data storage conversion method and system for business report data |
| CN113821573A (en) * | 2021-08-27 | 2021-12-21 | 济南浪潮数据技术有限公司 | Mass data rapid retrieval service construction method, system, terminal and storage medium |
| CN113868306A (en) * | 2021-08-31 | 2021-12-31 | 云南昆钢电子信息科技有限公司 | Data modeling system and method based on OPC-UA specification |
| CN114153895A (en) * | 2021-11-17 | 2022-03-08 | 深圳市六度人和科技有限公司 | Construction method of real-time warehouse counting frame and real-time warehouse counting frame |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117407445A (en) | 2024-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492040B (en) | System suitable for processing mass short message data in data center | |
| CN107038162B (en) | Real-time data query method and system based on database log | |
| CN102880503B (en) | Data analysis system and data analysis method | |
| CN111061788B (en) | Multi-source heterogeneous data conversion integration system based on cloud architecture and implementation method thereof | |
| CN109656963B (en) | Metadata acquisition method, apparatus, device and computer readable storage medium | |
| CN104699718B (en) | Method and apparatus for being rapidly introduced into business datum | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| CN103631870B (en) | System and method used for large-scale distributed data processing | |
| CN102236581B (en) | Mapping reduction method and system thereof for data center | |
| CN111381983B (en) | Lightweight message middleware system and method of virtual test target range verification system | |
| CN111400326A (en) | Smart city data management system and method thereof | |
| CN103235820B (en) | Date storage method and device in a kind of group system | |
| CN111694888A (en) | Distributed ETL data exchange system and method based on micro-service architecture | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN108446335B (en) | Heterogeneous system data extraction and unified external data exchange method based on database | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN113900810A (en) | Distributed graph processing method, system and storage medium | |
| CN104199978A (en) | System and method for realizing metadata cache and analysis based on NoSQL and method | |
| CN110825641B (en) | Micro-service application test system based on simulation data generator | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN111752920A (en) | Method, system, and storage medium for managing metadata | |
| CN113722533B (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113014618B (en) | Message processing method and system and electronic equipment | |
| CN117407445B (en) | Data storage method, system and storage medium for Internet of vehicles data platform | |
| CN110007905A (en) | A kind of generation method and system of the software development scheme based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |