CN115766867A

CN115766867A - Data processing method, device and system and electronic equipment

Info

Publication number: CN115766867A
Application number: CN202211418823.7A
Authority: CN
Inventors: 魏子涵; 孔令续; 姚雷; 郝嘉斌
Original assignee: Hangzhou Yineng Battery Management Technology Co ltd; Zhejiang Geely Holding Group Co Ltd
Current assignee: Hangzhou Yineng Battery Management Technology Co ltd; Zhejiang Geely Holding Group Co Ltd
Priority date: 2022-11-14
Filing date: 2022-11-14
Publication date: 2023-03-07

Abstract

The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, device, system, and electronic device. The method comprises the following steps: acquiring vehicle data; unpacking the vehicle data to obtain a plurality of data packets; analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in a plurality of data packets; the structure has a readable format; writing the structure of each data packet into a first type storage system; the first type of storage system is used for analyzing the condition of the vehicle; regularly storing the first type structure body in the first type storage system in a second type storage system; deleting the first structure in the first storage system; the first type of structure body is used for representing the structure body with the time of writing into the first type of storage system meeting a preset time period; the second type of storage system is used for storing binary data of the first type of structure. Vehicle data are reasonably stored through the two types of storage systems, and storage cost is saved.

Description

Data processing method, device and system and electronic equipment

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, system, and electronic device.

Background

Along with the popularization of automobiles, the attention level of corresponding vehicle safety monitoring is gradually increased. Especially, for the battery safety monitoring of new energy vehicles, it is very important for the safety of new energy vehicles.

Vehicle safety monitoring includes the management and analysis of vehicle safety, so vehicle data needs to be collected and stored. For example, battery safety monitoring includes the collection and storage of vehicle data for battery safety and battery life cycle management and analysis. In addition, since the number of vehicles is large and the real-time vehicle data of each vehicle needs to be stored in a unified real-time manner, high cost is required.

In addition, according to the policy requirements of the national ministry of industry and informatization, new energy vehicles produced by new energy vehicles and enterprises need to be equipped with vehicle-mounted monitoring terminals in the whole vehicle to obtain announcements and meet the GB/T32960 data protocol standard. The GB/T32960 protocol is a communication protocol between a vehicle-mounted terminal and a service platform, and mainly makes provisions for establishing a communication link between the vehicle-mounted terminal and the service platform, transmitting a data message format and definition thereof, uploading real-time information and the like.

For example, according to the GB/T32960 data protocol, it is required that a vehicle transmits data for a vehicle-end data up to 30 seconds, and the theoretical upper limit of the size of the transmitted data calculated from the protocol is about: 65KB, the following data can be calculated: the vehicle needs to transmit data once no more than 30 seconds at the longest, and the upper limit of the size of the data transmitted once is about: 65KB, it can be calculated that: if 1 ten thousand vehicles transmit data every day, the total storage amount occupied by the data every day is 1.74TB, and the binary data are analyzed into a human-readable format, the amount of the structure is 3 times that of the original binary data, 5.22TB is achieved, namely the number of the generated data of 1 ten thousand vehicles every day is 5.22TB, and 1905TB (365 × 5.22TB) data is stored every day after one year. Therefore, the cost of this data storage would be a significant cost to the enterprise.

Therefore, it is desirable to provide a data processing method, device, system and electronic device, which can reasonably store vehicle data; the storage cost of the vehicle data is saved.

Disclosure of Invention

The embodiment of the application provides a data processing method, a device, a system and electronic equipment, which can reasonably store vehicle data; the storage cost of the vehicle data is saved.

In a first aspect, an embodiment of the present application provides a data processing method, where the method includes:

acquiring vehicle data; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data;

analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format;

writing the structure of each data packet into a first type of storage system; the first type of storage system is used for analyzing the condition of the vehicle;

regularly storing the first-class structure bodies in the first-class storage system in a second-class storage system; deleting the first type of structure body in the first type of storage system; the first type of structure body is used for representing the structure body with the time written into the first type of storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structure.

In some optional embodiments, parsing, according to a header of each of the plurality of data packets, each field in a message body of each of the data packets into a corresponding structure, includes:

determining the message type of each data packet according to the message header of each data packet;

and analyzing each field in the message body of each data packet into a corresponding structural body according to the message type of each data packet.

In some optional embodiments, the periodically storing the first type structure in the first type storage system in the second type storage system includes:

based on the communication protocol of the vehicle data, regularly restoring the first type structure body in the first type storage system into binary data to obtain restored data;

and compressing the restored data and storing the data in the second type of storage system.

In some alternative embodiments, the vehicle data includes at least one of full vehicle data, drive motor data, fuel cell data, and engine data.

In some optional embodiments, the analyzing the vehicle condition includes at least one of vehicle battery safety monitoring and vehicle battery lifecycle analysis.

In some optional embodiments, the readable format comprises at least one of a JSON format and a CSV format.

In some optional embodiments, the message type includes at least one of login, vehicle login, real-time data, and reissue data.

In a second aspect, an embodiment of the present application provides a data processing apparatus, including:

the gateway module is used for acquiring vehicle data; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data;

the first analysis module is used for analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format;

the first storage module is used for writing the structural body of each data packet into a first type storage system, and the first type storage system is used for analyzing the condition of the vehicle;

the second storage module is used for storing the first structure in the first storage system in the second storage system at regular time and deleting the first structure in the first storage system; the first-class structure body is used for representing the structure body with the time of writing into the first-class storage system meeting a preset time period; the second type storage system is used for storing binary data corresponding to the first type structure.

In some optional embodiments, the first parsing module includes:

a type determining module, configured to determine a message type of each data packet according to the message header of each data packet;

and the field analysis module is used for analyzing each field in the message body of each data packet into a corresponding structural body according to the message type of each data packet.

In some optional embodiments, the second storage module comprises:

the restoring module is used for regularly restoring the first-class structural bodies in the first-class storage system into binary data based on the communication protocol of the vehicle data to obtain restored data;

the clearing module is used for deleting the first-type structure bodies in the first-type storage system;

and the first writing module is used for compressing the restored data and storing the compressed data in the second type of storage system.

In a third aspect, an embodiment of the present application provides a data processing system, where the system includes:

a first type storage system, a second type storage system and the data processing device;

the first type storage system is used for analyzing the condition of the vehicle based on the structural body;

and the second type of storage system is used for storing the binary data of the partial structural body obtained by the data processing device.

In a fourth aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by the processor and executes the data processing method.

In a fifth aspect, an embodiment of the present application provides a computer storage medium, where at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded by a processor and executes the data processing method.

The method comprises the steps of obtaining vehicle data; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data; analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format; writing the structure of each data packet into a first type storage system; the first type of storage system is used for analyzing the condition of the vehicle; regularly storing the first type structural body in the first type storage system in a second type storage system; deleting the first type of structure body in the first type of storage system; the first-class structure body is used for representing the structure body with the time of writing into the first-class storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structure. The vehicle data is stored and analyzed through two types of storage systems, wherein the first type of storage system is expensive and is used for storing the structure body in a readable format (namely, the structure body can be used for offline big data analysis); the second type of storage system is inexpensive and is used to store a large number of structures of binary data type that are unreadable (i.e., not conducive to human reading). The data in the readable format occupies large space, the binary data occupies small space and cannot be read directly by people, so that part of structural bodies in the first storage system with high price are restored into the binary data at regular time and stored in the second storage system with low price, the cost can be saved, and enough vehicle data can be stored. It can be understood that, when the binary data in the second type storage system needs to be analyzed, the binary data in the second type storage system is parsed into data in a readable format, and the data is written into the first type storage system. Therefore, the cost for storing the vehicle data is saved while enough vehicle data are stored and analyzed.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is an application scenario diagram of a data processing method provided in an embodiment of the present application;

fig. 2 is a schematic flowchart of a data processing method according to an embodiment of the present application;

FIG. 3 is a block diagram of a data processing system according to an embodiment of the present application;

fig. 4 is a schematic flowchart illustrating parsing of a data packet in a data processing method according to an embodiment of the present application;

fig. 5 is a hardware structure block diagram of an electronic device for implementing a data processing method according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic may be included in at least one implementation of the invention. In the description of the present invention, it should be understood that the terms "first," "second," "third," and "fourth," etc. in the description and claims of the present invention and the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

In order to better understand the present application, terms mentioned in the present application are explained below.

Tbox: namely Telematics-BOX, a data acquisition and transmission device on board the vehicle.

Enterprise level message bus: high-throughput distributed publishing and subscribing message system

The JSON format: namely JavaScript Object notification, JS Object Notation for short, which is a lightweight data exchange format. Is easy to read and write by people and can exchange data among a plurality of languages.

CSV format: comma-Separated Values, a character separation value.

A gateway: unified entry of data access in a system where unified checking of data, etc. can be performed.

Multithreading: multithreading is a concurrent execution mechanism. In short, a processor is divided into a plurality of short time slices, each time slice sequentially and alternately executes processing of each application program, and as one time slice is short, the processor is just as if serving the processor for the processor independently as one application program, so that the effect that a plurality of application programs are simultaneously performed is achieved.

Hive storage Engine: hive is a data warehouse tool used for data extraction, transformation and loading, and is an engine capable of storing, querying and analyzing large-scale data stored in Hadoop (a distributed system infrastructure).

HDFS storage system: i.e., the Hadoop Distributed File System, is a Distributed File System. The HDFS storage system can provide high-throughput data access and is suitable for application on large-scale data sets.

The OSS storage system: the Object Storage Service is a massive, safe, low-cost and high-reliability cloud Storage Service.

VIN code: namely, the Vehicle Identification Number means an automobile Identification code including information of the manufacturer, year, model, body type and code, engine code, assembly position, and the like of the automobile.

Before the data processing method according to the embodiment of the present application is described, a data processing scenario is described first.

Referring to fig. 1, fig. 1 is a view illustrating an application scenario of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 1, the electric vehicles and the enterprise platforms (enterprise platforms for short) of the electric vehicle remote service and management system communicate with each other based on an on-board data communication protocol, such as a GB/T32960 protocol. Platform exchange-based communication protocol communication is carried out between the enterprise platform and a public platform (short for the public platform) of the electric automobile remote service and management system. Specifically, the electric vehicle is provided with a vehicle-mounted terminal, such as a TBOX, and the electric vehicle collects vehicle information in real time through the vehicle-mounted terminal and sends vehicle data to the enterprise platform, for example, the vehicle data is uploaded to the enterprise platform through the vehicle-mounted terminal every 30 seconds. The enterprise platform carries out information interaction with the public platform in a bus mode.

Vehicle data is typically acquired by a data processing system through an enterprise platform for vehicle condition analysis. Specifically, the data processing system needs to store the vehicle data belonging to the binary data in a readable format for subsequent analysis based on the vehicle data. However, data in a readable format requires a large storage space of the storage system because of the necessity of using the storage system, and the storage system for storing readable data is expensive, compared to non-readable binary data.

In order to solve the above problems, the present application provides a data processing method, specifically, the present application obtains vehicle data; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data; analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format; writing the structure of each data packet into a first type of storage system; the first type of storage system is used for analyzing the condition of the vehicle; regularly storing the first-class structure bodies in the first-class storage system in a second-class storage system; deleting the first type of structure body in the first type of storage system; the first-class structure body is used for representing the structure body with the time of writing into the first-class storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structure. That is, two types of storage systems are used to store and analyze vehicle data, where the first type of storage system is expensive and is used to store structures in a readable format (i.e., the structures can be used for offline big data analysis); the second category of storage systems is inexpensive and is used to store large quantities of structures of binary data type that are unreadable (i.e., not conducive to human reading). The data in the readable format occupies large space, and the unreadable binary data occupies small space, so that part of the structural bodies in the expensive first-class storage system is restored into the binary data at regular time and stored in the cheap second-class storage system, the cost can be saved, and enough vehicle data can be stored. It can be understood that, when the binary data in the second type storage system needs to be analyzed, the binary data in the second type storage system is parsed into readable data, and the readable data is written into the first type storage system. Therefore, the cost for storing the vehicle data is saved while enough vehicle data are stored and analyzed.

A specific embodiment of a data processing method provided in an embodiment of the present application is described below, and fig. 2 is a schematic flowchart of the data processing method provided in the embodiment of the present application; the specification provides the method steps as in the examples or flowcharts, but may include more or fewer steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of sequences, and does not represent a unique order of performance. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. Specifically, as shown in fig. 2, the method may include:

s202: acquiring vehicle data; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data.

Fig. 3 is a schematic structural diagram of a data processing system according to an embodiment of the present application, where as shown in fig. 3, the data processing system includes a data processing apparatus and a data storage system. The data processing device is used for processing the vehicle data acquired from the external data source system and writing the processed vehicle data into the data storage system. The data processing apparatus comprises a load balancing service. The external data source system can be the enterprise platform, the electric automobile sends the vehicle data to the enterprise platform based on a GBT32960 protocol, and the enterprise platform sends the vehicle data to the load balancing service.

In some optional embodiments, multiple threads are used to receive vehicle data to increase the speed of data processing. The multithread processing is to improve the data processing speed by utilizing the performance of a single electronic device, however, the performance of the single electronic device always has an upper limit.

In some alternative embodiments, the load balancing service shown in fig. 3 may be used to increase the data processing speed based on the expansion of the dimension of the electronic device. Specifically, using load balancing techniques, a downstream gateway module of the load balancing service may add any electronic device. After the external data source sends the data to the load balancing service, the data is forwarded to a certain downstream electronic device, the load balancing service can detect the load processed by the downstream electronic device and send the received data to the downstream electronic device in a balanced manner, and each electronic device equally divides the processed data volume, so that the data processing performance can be greatly improved, and meanwhile, the machine expansibility is flexible. For example, the gateway module shown in FIG. 3 includes data gateway service instance 1, data gateway service instance 2, and data gateway service instance 3. Wherein each data gateway service instance processes data equally.

For example, the load balancing service shown in fig. 3 only provides one Internet Protocol (IP) address to the outside, that is, the IP of the electronic device where the load balancing service is located, and it is not necessary to display a plurality of IP addresses to an external system, which is easy to maintain and manage.

S204: analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format.

For example, as shown in fig. 3, the data processing device includes a gateway module and a message queue service, and after receiving the vehicle data of the GBT32960 protocol, the gateway module checks the vehicle data and determines the integrity of the data. Specifically, a complete GBT32960 protocol data is divided into two parts: a message header and a message body. In order to improve the processing performance of the data processing system, the gateway module analyzes a message header of the vehicle data, judges the message type according to a command unit in the message header, and if the message type is a data processing system login message, the identity verification of a login system is carried out, namely whether an account number and a password are correct or not is checked, and if the account number and the password are incorrect, the connection between the vehicle and the data processing system is disconnected. And if the message type is other non-login message types, checking the correctness and the integrity of the vehicle data according to the protocol specification. Specifically, the correctness and completeness of the vehicle data are judged through the start characters and the data lengths respectively. For example, first, it is determined whether the start symbol is "0x23"; then judging whether the length of the message body is the value of the 22 nd byte (the 22 nd byte is used for describing the length of the data unit) of the message header; and if the data passes the verification, the gateway module sends the data to the message queue service.

The gateway module can quickly process the received vehicle data to ensure that the receiving speed of the vehicle data can keep up with the speed sent by the vehicle-mounted terminal, otherwise, the vehicle data sent by the vehicle-mounted terminal will be lost.

One of the functions of the message queue service is "peak clipping", that is, after the vehicle data is sent to the message queue service, the subsequent data processing device can slowly process the vehicle data according to the processing capability of the subsequent data processing device. As shown in fig. 3, the data processing apparatus includes a data first processing module for performing data processing on the vehicle data. The message queue service sends the vehicle data to the data first processing module. For example, the message queue service shown in fig. 3 is implemented based on an enterprise-level message bus, such as a message queue service including a message queue service instance 1, a message queue service instance 2, and a message queue service instance 3, which are respectively connected to a data gateway service instance 1, a data gateway service instance 2, and a data gateway service instance 3.

Fig. 4 is a schematic flow chart of parsing a data packet in a data processing method according to an embodiment of the present application, and in some optional embodiments, in step S204, each field in a message body of each data packet is parsed into a corresponding structure according to a message header of each data packet in the multiple data packets, including:

s2041: and determining the message type of each data packet according to the message header of each data packet.

S2042: and analyzing each field in the message body of each data packet into a corresponding structural body according to the message type of each data packet.

For example, the first processing module of data in the data processing apparatus shown in fig. 3 includes a first parsing module, a first storage module, and a second storage module.

The first analysis module can be used for carrying out protocol analysis on the received data packet, for example, the first analysis module divides the data packet based on the GB/T32960 protocol and the data format into two parts for analysis, and as a complete data packet consists of a message header and a message body, the first analysis module firstly analyzes the message header, can judge the message type of the data packet according to a command unit in the message header, and calls a corresponding message body analysis method according to the message type; and then analyzing each field in the message body of each data packet into a corresponding structural body according to a message body analysis method. Specifically, in the message body parsing method, each field defined in the GB/T32960 protocol is parsed into a corresponding structure body.

S206: writing the structure of each data packet into a first type storage system; the first type of storage system is used to analyze vehicle conditions.

In some optional embodiments, the analyzing the vehicle condition comprises at least one of a vehicle battery safety monitoring and a vehicle battery lifecycle analysis.

For example, the first storage module shown in fig. 3 is used to store the structure in the HDFS storage system. HDFS storage systems belong to a first class of storage systems. Specifically, the Hive storage engine can be used for writing the structure body into the HDFS storage system.

When vehicle data is analyzed in real time, the data volume of the structure body is large, and a large storage space is occupied, for example, a binary data format designed by the GBT32960 protocol is compact, the space occupied by the structure body using the JSON format expands, and the resources and the cost of the first type storage system increase rapidly with the increase of time and access vehicles.

S208: regularly storing the first type structural body in the first type storage system in a second type storage system; deleting the first-type structure bodies in the first-type storage system; the first-class structure body is used for representing the structure body with the time of writing into the first-class storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structural body.

In some optional embodiments, the vehicle data is directly stored in the second type storage system at regular time, the vehicle data is acquired according to the requirement, and the acquired vehicle data is analyzed into a readable format and stored in the first type storage system.

Specifically, the above-described timing may be performed on a daily basis, an hourly basis, a weekly basis, or the like. The deleting of the first type structure in the first type storage system may be sampling or other regular removal of the first type structure several months/weeks/years ago, and the application is not limited herein.

For example, the second storage module shown in fig. 3 is used to store the partial structural body in the OSS storage system belonging to the second type of storage system at regular time.

In order to reduce the storage space of the storage structure and maintain the consistency between the stored data and the original vehicle data, in some optional embodiments, in step S208, the step of periodically storing the first type structure in the first type storage system in the second type storage system includes:

based on the binary protocol of the vehicle data, regularly restoring the first type structure body in the first type storage system into binary data to obtain restored data;

For example, the first processing module of data shown in fig. 3 includes a second storage module, and the second storage module is configured to execute the step S208. The second storage module comprises a restoring module, a clearing module and a first writing module.

And a restoring module, configured to execute the step S208. As shown in fig. 3, the restore module includes a timing sub-module and a restore sub-module. The timing submodule is used to determine the time to perform step S208. The restoring submodule is used for restoring the first structure bodies in the first storage system into binary data to obtain restored data. The clearing module is used for storing the restored data to a local disk file of the data processing system and clearing the structural body corresponding to the restored data from the first type of storage system. For example, there is a timing task every day to compress and pack the local disk file, wherein the data amount after compression and packing is reduced to 1/4 before compression. This will ensure that only the last specific period of data (e.g., the last 3 months of data) is stored in the first type of storage system. In the process of storing the restored data in the local disk file, the vin code corresponding to each piece of restored data and the acquisition time of the data need to be output so as to facilitate faster retrieval and positioning in the following process, the restored data written in the local disk is compressed, and the compressed data is stored in an OSS storage system with lower cost. Wherein the data volume of the structure for real-time battery alarm analysis and offline big data analysis will be 3 times the data volume of the original vehicle data. Wherein the vehicle data belongs to binary data based on GBT32960 protocol; when the structure is reduced to the reduced data, the data volume of the reduced data is reduced to 1/3 of the data volume of the structure.

In this embodiment, based on the principle that the latest structure is frequently accessed, for example, the latest structure is used to obtain the latest data for analysis, and the frequency of using the longer data is much lower, the historical data before the specific time point in the first-type storage system is periodically restored and cleared, that is, the data meeting the preset time period is periodically restored and cleared, and only the structure in the specific time period is maintained. Meanwhile, the removed part of the structural body is restored into original binary data, such as binary data conforming to the GBT32960 protocol, and the restored data is compressed and stored into a cheaper second type storage system, such as an OSS storage system, so that the vehicle data storage cost is greatly reduced.

In the above embodiment, according to the service characteristics, the binary data (i.e., the restored data) of the original GBT32960 protocol is not updated but stored, so as to meet the backtracking requirement. Specifically, the restored data is stored in a cheaper OSS storage system, and meanwhile, rapid positioning and analysis of the restored data are supported.

For example, the data processing apparatus shown in fig. 3 includes a data playback module, where the data playback module is used for writing the restored data in the second type of storage system into the first type of storage system according to the service requirement. The data playback module comprises a reading module, a second analysis module and a second writing module. The reading module is used for reading the required restored data from the second type of storage system (for example, the second type of storage system is an OSS storage system). The second analysis module is used for analyzing the read restored data so as to analyze the restored data into a readable format. The second writing module is used for storing the analyzed restored data into a first type of storage system (for example, a first type storage system HDFS storage system), or directly sending the analyzed restored data to an analysis system, for example, a real-time battery alarm analysis system.

In some alternative embodiments, the data playback module may read the restored data from the second type of storage system according to a specified time. Therefore, the vehicle data storage cost can be reduced, and the analysis and check of the historical vehicle data in business can be quickly met.

With the above-described embodiments, vehicle data is stored and analyzed by two types of storage systems, wherein the first type of storage system is expensive and the structure for storing readable formats (i.e., usable for offline big data analysis) (the structure occupies a large space and is usable for offline big data analysis); the second type of storage system is inexpensive and is used to store a large number of structures of unreadable binary data types (binary data occupies a small space). In this manner, cost savings and sufficient vehicle data may be preserved. And when the binary data in the second type storage system needs to be analyzed, the binary data in the second type storage system is analyzed into readable data, and the readable data is written into the first type storage system, namely the data playback module plays a role. Therefore, the cost for storing the vehicle data is saved while enough vehicle data are stored and analyzed.

An embodiment of the present application provides a data processing apparatus, as shown in fig. 3, the data processing apparatus includes:

the second storage module is used for regularly storing the first type structure body in the first type storage system in the second type storage system and deleting the first type structure body in the first type storage system; the second type storage system is used for storing binary data corresponding to the first type structure.

In some optional embodiments, the first parsing module includes:

In some optional embodiments, the second storage module comprises:

the restoring module is used for regularly restoring the first type structure body in the first type storage system into binary data based on the communication protocol of the vehicle data to obtain restored data;

In some optional embodiments, the data processing apparatus further includes:

and the data playback module is used for writing the restored data in the second type of storage system into the first type of storage system according to the service requirement.

An embodiment of the present application provides a data processing system, including:

the first type storage system is used for analyzing the vehicle condition based on the structural body;

and the second type of storage system is used for storing the binary data of the partial structure body obtained by the data processing device. For example, when the first type of storage system is an HDFS storage system and the second type of storage system is an OSS storage system, as shown in fig. 3, the data processing system includes a data processing apparatus and a data storage system; the data storage system comprises an HDFS storage system and an OSS storage system.

The device, system and method embodiments in the embodiments of the present application are based on the same application concept.

Fig. 5 is a hardware structure block diagram of an electronic device for implementing a data processing method according to an embodiment of the present application. The electronic device may be a server or a terminal device, and its internal structure diagram may be as shown in fig. 5. As shown in fig. 5, the electronic device 500 may have a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 510 (the processors 510 may include but are not limited to a microprocessor MPU or a Processing device such as a programmable logic device FPGA), a memory 530 for storing data, and one or more storage media 520 (e.g., one or more mass storage devices) for storing application programs 523 or data 522. Memory 530 and storage medium 520 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 520 may include one or more modules, each of which may include a series of instruction operations for the server. Still further, the central processor 510 may be configured to communicate with the storage medium 520 to execute a series of instruction operations in the storage medium 520 on the electronic device 500. The electronic device 500 may also include one or more power supplies 550, one or more wired or wireless network interfaces 550, one or more input-output interfaces 540, and/or one or more operating systems 521, such as Windows, mac OS, unix, linux, freeBSD, and the like.

The input/output interface 540 may be used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the electronic device 500. In one example, the i/o Interface 540 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the input/output interface 540 can be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

The power supply 550 may be logically coupled to the processor 510 via a power management system, such that functions of managing charging, discharging, and power consumption are performed via the power management system.

It will be understood by those skilled in the art that the structure shown in fig. 5 is only an illustration and is not intended to limit the structure of the electronic device. For example, electronic device 500 may also include more or fewer components than shown in FIG. 5, or have a different configuration than shown in FIG. 5.

Embodiments of the present application also provide a computer storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the data processing method.

Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

An embodiment of the present application further provides an electronic device, which at least includes a processor 510 and a memory 530, where at least one instruction or at least one program is stored in the memory 530, and the at least one instruction or the at least one program is loaded by the processor 510 and executes the data processing method described above.

As can be seen from the embodiments of the data processing method, the data processing device, the data processing system and the electronic device provided by the application, the vehicle data is obtained; unpacking the vehicle data to obtain a plurality of data packets; the vehicle data belongs to binary data; analyzing each field in the message body of each data packet into a corresponding structural body according to the message header of each data packet in the plurality of data packets; the structure has a readable format; writing the structure of each data packet into a first type of storage system; the first type of storage system is used for analyzing the condition of the vehicle; regularly storing the first-class structure bodies in the first-class storage system in a second-class storage system; deleting the first-type structure bodies in the first-type storage system; the first-class structure body is used for representing the structure body with the time of writing into the first-class storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structure. Namely, two types of storage systems are used to store and analyze vehicle data, wherein the first type of storage system is expensive and is used to store the structure in a readable format (i.e., the structure can be used for offline big data analysis); the second type of storage system is inexpensive and is used to store large amounts of unreadable binary data type structures. The data in the readable format occupies large space, and the binary data occupies small space and is unreadable, so that part of the structural bodies in the expensive first-class storage system are restored into the binary data at regular time and stored in the cheap second-class storage system, the cost can be saved, and enough vehicle data, such as the structural bodies, can be stored. It can be understood that, when the binary data in the second type storage system needs to be analyzed, the binary data in the second type storage system is parsed into readable data, and the readable data is written into the first type storage system. Therefore, the cost for storing the vehicle data is saved while enough vehicle data are stored and analyzed.

It should be noted that: the sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And that specific embodiments have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of data processing, the method comprising:

regularly storing the first-class structure bodies in the first-class storage system in a second-class storage system; deleting the first-type structure bodies in the first-type storage system; the first type of structure body is used for representing the structure body with the time written into the first type of storage system meeting a preset time period; the second type storage system is used for storing binary data of the first type structural body.

2. The method of claim 1, wherein parsing fields in a message body of each of the plurality of packets into corresponding structures according to a header of each of the plurality of packets comprises:

and analyzing each field in the message body of each data packet into the corresponding structural body according to the message type of each data packet.

3. The method of claim 1, wherein the timing of the storage of the first structure in the first storage system in the second storage system comprises:

based on the communication protocol of the vehicle data, regularly restoring the first-class structural bodies in the first-class storage system into binary data to obtain restored data;

4. The method of any of claims 1 to 3, wherein the vehicle data comprises at least one of full vehicle data, drive motor data, fuel cell data, and engine data.

5. The method of any one of claims 1 to 3, wherein analyzing the vehicle condition comprises at least one of vehicle battery safety monitoring and vehicle battery life cycle analysis.

6. The method of any of claims 1-3, wherein the readable format comprises at least one of a JSON format and a CSV format.

7. The method of claim 2, wherein the message type comprises at least one of login, vehicle login, real-time data, and reissue data.

8. A data processing apparatus, characterized in that the apparatus comprises:

9. A data processing system, characterized in that the system comprises:

a first type of storage system, a second type of storage system, and the data processing apparatus of claim 8;

the second type storage system is used for storing the binary data of the first type structural body sent by the data processing device.

10. An electronic device, characterized in that the electronic device comprises a processor and a memory, in which at least one instruction or at least one program is stored, which is loaded by the processor and executes the data processing method according to any one of claims 1-7.