CN115378946A - Data processing method, device and system - Google Patents

Abstract

The application discloses a data processing method, a device and a system, comprising: respectively receiving vehicle data from each target vehicle, and distributing each vehicle data to each service node; the data scheduling module acquires subscription requirements and receives target vehicle data from each service node according to the subscription requirements; the data scheduling module executes data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result; and the data scheduling module sends the processing result to a database. The method and the system solve the problem that the service system cannot realize real-time and efficient processing of the terminal data under the condition that the data amount is high in concurrency, can achieve rapid processing of high-concurrency vehicle data, and improve the processing capacity of the whole system for the data messages of the vehicle networking terminal.

Description

Data processing method, device and system

Technical Field

The present application relates to the field of communications technologies, and in particular, to a data processing method, apparatus, and system.

Background

A service system is a networked configuration of a particular technology or organization to provide services to meet the needs and desires of customers. Taking the automobile industry field as an example, a service system in the automobile industry field is generally an internet of vehicles system, which comprises three major parts: the system comprises a vehicle-mounted intelligent terminal, a cloud data processing platform and a data analysis platform. The vehicle-mounted intelligent terminal is responsible for acquiring vehicle real-time operation data and sending the vehicle real-time operation data to the cloud data processing platform, the cloud data processing platform is responsible for processing the received vehicle real-time operation data, and the data analysis platform is responsible for performing report type processing on the processed vehicle real-time operation data for a manager to check and make a decision.

However, with the rapid development of distributed timeliness, cloud technology, big data technology, and the like, various service systems have become large and complicated. When the communication quantity of the terminal equipment is large, the data volume concurrency is large, if a concurrency bottleneck occurs in the cloud data processing platform and a large amount of terminal data reported by the intelligent terminal cannot be processed in time, packet loss can be caused if the concurrency bottleneck occurs, connection interruption of the terminal equipment can be caused if the concurrency bottleneck occurs, and catastrophic influence is caused on the whole service system.

Therefore, an effective data processing method is needed to solve the requirement of the service system for efficiently processing the terminal data in real time under the condition of high data volume concurrency.

Disclosure of Invention

The application provides a data processing method, a data processing device and a data processing system, and solves the technical problem that the service system cannot realize real-time and efficient processing of terminal data under the condition of high data volume concurrence.

In one aspect, a data processing method is provided, where the method is applied to a data processing platform, where the data processing platform includes service nodes, a data scheduling module, and a database, and the method includes:

receiving vehicle data from each target vehicle;

distributing each vehicle data to each service node;

the data scheduling module acquires subscription requirements and receives target vehicle data from each service node according to the subscription requirements;

the data scheduling module executes data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result;

and the data scheduling module sends the processing result to a database.

In still another aspect, a data processing apparatus is provided, including:

the vehicle data acquisition module is used for respectively receiving vehicle data from each target vehicle;

the vehicle data distribution module is used for distributing each piece of vehicle data to each service node;

the target vehicle data receiving module is used for the data scheduling module to acquire subscription requirements and receive target vehicle data from each service node according to the subscription requirements;

the processing result acquisition module is used for controlling the data scheduling module to execute data processing operation on the target vehicle data through each thread corresponding to each service node so as to obtain each processing result;

and the processing result sending module is used for controlling the data scheduling module to send the processing result to the database.

In one possible embodiment, the target vehicle data receiving module includes:

a subscription configuration file acquisition submodule for controlling the data scheduling module to acquire subscription requirements and acquiring a subscription configuration file based on the subscription requirements; the subscription configuration file comprises a domain name of a target service node, a port of the target service node, a user name and a password and a theme of vehicle data needing subscription;

and the target vehicle data acquisition submodule is used for controlling the data scheduling module to receive the target vehicle data from each service node based on the subscription configuration file.

In one possible embodiment, the target vehicle data acquisition sub-module is configured to:

for each service node, controlling the data scheduling module to establish a receiving thread corresponding to the service node, and establishing communication connection with the service node according to the receiving thread;

and controlling the data scheduling module to receive the target vehicle data from each service node in a multi-thread manner through the receiving thread corresponding to each service node based on the subscription configuration file.

In a possible implementation manner, the processing result obtaining module is configured to:

and merging the target vehicle data and the theme of the target vehicle data according to a target format through each thread corresponding to each service node through the data scheduling module, and acquiring each processing result.

In a possible implementation, the data processing apparatus is further configured to: and controlling the data scheduling module to perform transfer caching under the lock-free queue on each processing result.

In a possible implementation manner, the processing result sending module is further configured to:

controlling the data scheduling module to classify the data of each processing result according to the data type so as to obtain a processing result set of each category;

and when the data scheduling module meets specified conditions, sending the processing result set of each category to a database.

In a possible implementation, the data processing platform further includes a load balancing module, and the data processing apparatus is further configured to:

and controlling the load balancing module to respectively receive vehicle data from each target vehicle and distribute the vehicle data to each service node in a balanced manner.

In a possible implementation, the data processing apparatus is further configured to: establishing a connection between the respective target vehicle and the respective service node based on a message queue telemetry transport protocol.

In yet another aspect, a data processing system is provided, comprising: each target vehicle, a data processing platform and a data analysis platform, wherein the data processing platform comprises a load balancing module, each service node, a message queue service module, a data scheduling module and a database;

the load balancing module is used for respectively receiving vehicle data from each target vehicle and distributing each vehicle data to each service node;

the data scheduling module is used for acquiring subscription requirements and receiving target vehicle data from each service node according to the subscription requirements;

the data scheduling module is used for executing data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result;

the data scheduling module is used for sending the processing result to a database;

and the data analysis platform is used for performing report type processing on the vehicle data in the database and acquiring a processing result.

In yet another aspect, a computer device is provided that includes a processor and a memory having stored therein at least one instruction that is loaded and executed by the processor to implement a data processing method as described herein.

In yet another aspect, a computer-readable storage medium is provided, having stored therein at least one instruction, which is loaded and executed by a processor to implement a data processing method as described herein.

The technical scheme provided by the application can comprise the following beneficial effects:

under the condition that the vehicle networking data sent by a plurality of vehicles need to be processed, the vehicle data sent by each vehicle can be respectively received through the data processing platform, and each vehicle data is distributed to each service node, and at the moment, the data scheduling module in the data processing platform executes data processing operation on target vehicle data through each thread corresponding to each service node, so that each processing result is obtained. According to the scheme, the multiple threads are set in the data scheduling module to respectively perform parallel processing on the target vehicle data sent by the multiple service nodes, so that the high-concurrency vehicle data can be quickly processed, and the processing capacity of the whole system on the vehicle networking data messages is improved.

Drawings

In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a vehicle networking system according to an exemplary embodiment.

FIG. 2 is a method flow diagram illustrating a method of data processing according to an example embodiment.

FIG. 3 is a method flow diagram illustrating a method of data processing according to an exemplary embodiment.

FIG. 4 is a diagram of a multithreading structure according to an exemplary embodiment.

Fig. 5 is a block diagram illustrating a structure of a data processing apparatus according to an exemplary embodiment.

FIG. 6 is a block diagram of a data processing system, shown in accordance with an exemplary embodiment.

FIG. 7 is a schematic diagram of a computer device shown in accordance with an exemplary embodiment.

Detailed Description

The technical solutions of the present application will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be understood that in the description of the embodiments of the present application, the term "correspond" may indicate that there is a direct correspondence or an indirect correspondence between the two, may also indicate that there is an association between the two, and may also indicate and be indicated, configured and configured, and the like.

Fig. 1 is a schematic structural diagram of a vehicle networking system according to an exemplary embodiment. The car networking system 100 includes a car-mounted intelligent terminal 110, a cloud data processing platform 120, and a data analysis platform 130.

Optionally, the vehicle-mounted intelligent terminal 110 is responsible for acquiring vehicle real-time operation data of each vehicle, so as to acquire data information such as work information, safety data, driving data, position information and the like of each vehicle, and send the acquired vehicle real-time operation data to the cloud data processing platform 120;

optionally, the cloud data processing platform 120 is responsible for processing the received vehicle real-time running data, performing processing such as filtering, secondary encapsulation, classification and caching on the vehicle real-time running data, and sending the processed vehicle real-time running data to the data analysis platform 130;

optionally, the data analysis platform 130 is responsible for performing report-type processing on the processed real-time vehicle operation data, so that a manager can check and make a decision.

Optionally, the cloud data processing platform 120 may be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server providing technical computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, big data and artificial intelligence platform, and the like.

Optionally, the system may further include a management device, where the management device is configured to manage the system (for example, manage connection states between the modules and the server, and the management device is connected to the server through a communication network. Optionally, the communication network is a wired network or a wireless network.

Optionally, the wireless or wired networks described above use standard communication techniques and/or protocols. The network is typically the internet, but may be any other network including, but not limited to, a local area network, a metropolitan area network, a wide area network, a mobile, a limited or wireless network, a private network, or any combination of virtual private networks. In some embodiments, data exchanged over the network is represented using techniques and/or formats including hypertext markup language, extensible markup language, and the like. All or some of the links may also be encrypted using conventional encryption techniques such as secure sockets layer, transport layer security, virtual private network, internet protocol security, and the like. In other embodiments, custom and/or dedicated data communication techniques may also be used in place of, or in addition to, the data communication techniques described above.

FIG. 2 is a flow chart illustrating a method of data processing according to an exemplary embodiment. The method is performed by a data processing platform. As shown in fig. 2, the data processing method may include the steps of:

in step S201, vehicle data is received from each target vehicle.

In a vehicle network application scenario, in order to provide services for networked vehicles, a service center is generally required to be set to acquire vehicle data of each target vehicle in the vehicle network. For example, when the data processing platform is the cloud data processing platform 120 in the car networking system 100 shown in fig. 1, the data processing platform may perform the receiving of the vehicle data of each target vehicle through a load balancing module in the cloud data processing platform. The load balancing module can establish communication connection with each target vehicle and acquire vehicle data sent by the target vehicles. The load balancing module is a virtual server and is used for balancing and forwarding data.

In one possible implementation, the data processing platform acquires vehicle data of each target vehicle from the vehicle-mounted intelligent terminal, and the vehicle data comprises: and data such as work information, safety data, traveling data, position information, etc. of the target vehicle.

In a possible implementation manner, the data processing platform receives the vehicle data of each target vehicle from the vehicle-mounted intelligent terminal, and simultaneously receives a theme corresponding to the vehicle data of each target vehicle. For example, the theme may be used to indicate vehicle information such as the vehicle brand, the vehicle model, and the like of the target vehicle.

And step S202, distributing each vehicle data to each service node.

In a possible implementation manner, the data processing platform includes each service node, and after the vehicle data and the theme of each target vehicle are acquired, the data processing platform distributes the vehicle data and the theme of each target vehicle to each service node in a balanced manner.

For example, when the data processing platform is the cloud data processing platform 120 in the car networking system 100 shown in fig. 1, the data processing platform may forward the received vehicle data and theme of each target vehicle to each service node through a load balancing module in the cloud data processing platform.

Step S203, the data scheduling module obtains a subscription requirement, and receives target vehicle data from each service node according to the subscription requirement.

In a possible implementation manner, after each service node receives vehicle data and a theme of each target vehicle, a data scheduling module in the data processing platform acquires a subscription requirement of a user on the vehicle data, filters the vehicle data and the theme in each service node according to the subscription requirement, and receives multiple threads of the filtered vehicle data and the filtered theme to acquire the target vehicle data and the target theme, thereby implementing subscription of messages.

Step S204, the data scheduling module executes data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result.

In a possible implementation manner, after the data scheduling module realizes subscription of the message, data merging processing operation is performed on the acquired target vehicle data and the target topic according to a target format through each thread corresponding to each service node, and a processing result is acquired.

For example, when the target theme includes the vehicle model corresponding to the target vehicle, the data merging processing operation is performed, that is, the format of the target vehicle data is converted, so that the processing result obtained after conversion includes both the target vehicle data and the target theme of the target vehicle data.

The data scheduling module can transit and cache the acquired processing result so as to facilitate the writing of subsequent data into the database. The target format can be artificially preset or automatically generated by a computer according to the requirements of clients, and is convenient for users to read.

Step S205, the data scheduling module sends the processing result to the database.

In a possible implementation manner, the data scheduling module writes the processing result into the database after acquiring the processing result, so that the data analysis platform can conveniently perform report-type processing on the vehicle data in the database, and acquire the processing result for a manager to check and make a decision.

In summary, in a case that the internet of vehicles data sent by multiple vehicles needs to be processed, the data processing platform may receive vehicle data sent by each vehicle and distribute each vehicle data to each service node, and at this time, the data scheduling module in the data processing platform performs data processing operations on target vehicle data through each thread corresponding to each service node, so as to obtain each processing result. According to the scheme, the multiple threads are set in the data scheduling module to respectively perform parallel processing on the target vehicle data sent by the multiple service nodes, so that the high-concurrency vehicle data can be rapidly processed, and the processing capacity of the whole system on the data messages of the Internet of vehicles is improved.

FIG. 3 is a method flow diagram illustrating a method of data processing according to an exemplary embodiment. The method is performed by a data processing platform. As shown in fig. 3, the data processing method may include the steps of:

in step S301, vehicle data is received from each target vehicle.

In one possible implementation, the data processing platform obtains vehicle data of each target vehicle from an on-board intelligent terminal on each vehicle, respectively, and the vehicle data includes: and data such as work information, safety data, traveling data, position information, etc. of the target vehicle.

And step S302, distributing each vehicle data to each service node.

In a possible implementation manner, the data processing platform further includes a load balancing module, and the load balancing module receives vehicle data and a theme corresponding to the vehicle data from each target vehicle, and distributes the vehicle data and the theme to each service node in a balanced manner.

Optionally, the load balancing module may be a LVS load balancing (linux virtual server), a linux virtual server, and is a virtual four-layer switch cluster system, and implements user request forwarding according to a target address and a target port, and does not generate traffic itself, and only does user request forwarding, which is currently the cluster system with the best load balancing performance, and the load balancing module implements very good scalability, and the number of nodes may increase to several thousand. Specifically, in the application, after acquiring the vehicle data and the theme of each target vehicle, the LVS load balancing (linux virtual server) distributes the vehicle data of each target vehicle to each service node in a balanced manner.

In one possible embodiment, the connection between the respective target vehicle and the respective service node is established based on a message queue telemetry transport protocol.

Optionally, the data processing platform further includes a message queue service module, where the message queue service module includes a plurality of service nodes. The method includes the steps that connection is established between each target vehicle and each service node based on a Message Queue Telemetry Transport (MQTT), the Message Queue Telemetry Transport (MQTT) is a lightweight protocol based on a publish/subscribe (publish/subscribe) mode, and the protocol is established on a TCP/IP protocol.

Optionally, MQTT service software adopted by each service node in the Message queue service module is mosquitto, which is a Message agent and queue software that implements an open source of a Message queue Telemetry Transport protocol (MQTT), and provides a lightweight Message push mode that supports publishable/subscribeable Message push mode. In the application, the mosquitto provides the message publishing and message subscribing services, that is, after each service node receives the vehicle data of each target vehicle, the service node provides the message publishing function for the received data in a multi-level theme format and provides the message subscribing function for the subsequent data scheduling module, the application of the multi-level theme reduces and plans the types of the vehicle data, and the time required by classified scheduling is shortened.

The multi-level theme format in the application can be published by adopting a message theme format of 'U/vehicle enterprise name/vehicle type/vehicle VIN/vehicle-mounted terminal unique code/component type/data type', the message theme format is a layer-by-layer progressive relation, and can be simultaneously matched through multiple layers of wildcards, and the meaning of each level of representation is shown in table 1:

TABLE 1

Step S303, the data scheduling module obtains a subscription requirement, and obtains a subscription profile based on the subscription requirement.

In one possible embodiment, the subscription profile includes a domain name of the target service node, a port of the target service node, a username password, and a topic of the vehicle data to which the subscription is required.

Optionally, after each service node receives the vehicle data and the theme of the target vehicle, the data scheduling module obtains a subscription requirement of the user on the vehicle data, and then obtains a subscription configuration file according to the subscription requirement.

Step S304, the data scheduling module receives target vehicle data from the service nodes based on the subscription profile.

In a possible implementation manner, for each service node, the data scheduling module establishes a receiving thread corresponding to the service node, and establishes a communication connection with the service node according to the receiving thread;

and the data scheduling module receives the target vehicle data from each service node in a multithread mode through the receiving thread corresponding to each service node based on the subscription configuration file.

Optionally, as shown in the schematic view of the multithreading structure shown in fig. 4, after the data scheduling module acquires the subscription configuration file, the data scheduling module separately establishes a receiving thread corresponding to the service node for each service node, establishes a communication connection with the service node according to the receiving thread, and after all receiving threads are established and the communication connection is realized, the data scheduling module filters vehicle data and topics in each service node based on the subscription configuration file, and receives the filtered vehicle data and topics from each service node in a multithreading manner through the receiving threads corresponding to each service node, so as to acquire target vehicle data and target topics, thereby implementing the subscription of the data scheduling module for messages based on the subscription configuration file.

Optionally, when the data scheduling module subscribes a message based on the subscription profile, it is necessary to subscribe the "U/#" topic of each service node first, and then implement subscription of vehicle data based on the topic. Wherein, "U/#" represents receiving all upstream data, and # represents a multi-level wildcard that can match a multi-level topic, i.e., a multi-level wildcard can match all subset topics that match the previous topic hierarchy of the wildcard.

Step S305, the data scheduling module performs data processing operations on the target vehicle data through each thread corresponding to each service node to obtain each processing result.

In a possible implementation manner, the data scheduling module performs merging processing on the target vehicle data and the theme of the target vehicle data according to a target format through each thread corresponding to each service node, so as to obtain each processing result.

In a possible implementation manner, the data scheduling module performs transfer caching on each processing result under a lock-free queue.

Optionally, the data scheduling module is used as a consumer of data, after the subscription of the message based on the subscription configuration file is implemented, the data scheduling module analyzes the subscribed information (i.e., the target vehicle data and the target theme) to obtain specific vehicle data and theme, and after the analysis is implemented, the data scheduling module merges and processes the analyzed vehicle data and theme according to a target format through each thread corresponding to each service node to obtain each processing result.

The analyzed target vehicle data and the target theme are merged according to a target format, each processing result (namely, the final subscription message) is obtained, namely, information is added in the analyzed target vehicle data based on the target theme, and the analyzed vehicle data is converted into the target format to obtain the target subscription message. The additional information is: the name of the vehicle, the classification of the vehicle type, the VIN of the vehicle, the unique code of the terminal, the type of the component, the type of the uploaded data and the timestamp of the received event.

Optionally, after obtaining each processing result, the data scheduling module performs temporary transfer caching on the processing result, as shown in fig. 4, a lock-free queue is used when performing temporary transfer caching, the lock-free queue is a queue in a lock-free state, and locking is not required when a plurality of threads operate data simultaneously, because locking/unlocking is an action that consumes system resources, and the use of the lock-free queue can reduce the time for caching and reduce the consumption of system resources.

And step S306, the data scheduling module sends the processing result to a database.

In a possible implementation manner, the data scheduling module classifies data of each processing result (i.e. the final subscription message) according to data type;

and write the data processing result (i.e. the final subscription message) under the same data type into the database once.

In a possible implementation manner, the data scheduling module performs data classification on each processing result according to a data type to obtain a processing result set of each category;

and when the data scheduling module meets specified conditions, sending the processing result set of each category to a database.

Further, the specified conditions may include: one is that according to the period of processing result (final subscription message) acquisition, the system automatically sends the processing result set (final subscription message set) of each category to the database according to the category after a certain period of time; one is that when the data scheduling module stores a certain number of processing results (i.e. final subscription messages), the processing results are automatically sent to the database according to categories; one is to send the processing result sets of the respective categories (i.e. the final subscription message set) to the database when the system receives a specific sending instruction.

Optionally, after obtaining the target subscription message, the data scheduling module writes the target subscription message into a database, where the database may be a Redis database, and the Redis database is an open-source high-performance memory type database, and may cache and store the target subscription message obtained by the data scheduling module, and finally be called by the data analysis platform.

Optionally, when the data scheduling module writes the target subscription message into the database, a Pipeline technology (Pipeline) may be used to improve system throughput, that is, the client (i.e., the data scheduling module) sends a plurality of requests to the server (database) at one time, and does not need to wait for a reply of the request during the process, and finally reads the result, that is, the subscription message set meeting the specified condition is sent to the database.

For example, for a read-write model of 'once data reporting- > write once Redis database- > read once write result', the operation of the database is too frequent, which may quickly result in too high system delay and low throughput, and may not meet the target. By optimizing the service codes, a large number of Redis database operations are combined, the throughput of the system can be obviously improved by once writing after the data of the same type are accumulated, and the problem of low efficiency of the Redis database written once is solved.

Through practical verification, the efficiency of writing 10 pieces of data at a time is 7 times that of writing 1 piece of data at a time; the efficiency of writing 100 pieces of data at a time is 30 times that of writing 1 piece of data at a time.

In summary, in a case that the internet of vehicles data sent by multiple vehicles needs to be processed, the data processing platform may receive vehicle data sent by each vehicle and distribute each vehicle data to each service node, and at this time, the data scheduling module in the data processing platform performs data processing operations on target vehicle data through each thread corresponding to each service node, so as to obtain each processing result. According to the scheme, the multiple threads are set in the data scheduling module to respectively perform parallel processing on the target vehicle data sent by the multiple service nodes, so that the high-concurrency vehicle data can be quickly processed, and the processing capacity of the whole system on the vehicle networking data messages is improved.

Fig. 5 is a block diagram showing a structure of a data processing apparatus according to an exemplary embodiment, the data processing apparatus including:

a vehicle data acquisition module 501, configured to receive vehicle data from each target vehicle;

a vehicle data distribution module 502 for distributing each vehicle data to each service node;

a target vehicle data receiving module 503, configured to acquire a subscription requirement by the data scheduling module, and receive target vehicle data from each service node according to the subscription requirement;

a processing result obtaining module 504, configured to control the data scheduling module to perform data processing operations on the target vehicle data through each thread corresponding to each service node, so as to obtain each processing result;

and a processing result sending module 505, configured to control the data scheduling module to send the processing result to the database.

In one possible implementation, the target vehicle data receiving module 503 includes:

a subscription configuration file acquisition submodule for controlling the data scheduling module to acquire subscription requirements and acquiring a subscription configuration file based on the subscription requirements; the subscription configuration file comprises a domain name of a target service node, a port of the target service node, a user name and a password and a theme of vehicle data needing subscription;

and the target vehicle data acquisition submodule is used for controlling the data scheduling module to receive the target vehicle data from each service node based on the subscription configuration file.

In one possible embodiment, the target vehicle data acquisition sub-module is configured to:

for each service node, controlling the data scheduling module to establish a receiving thread corresponding to the service node, and establishing communication connection with the service node according to the receiving thread;

and controlling the data scheduling module to receive the target vehicle data from each service node in a multi-thread manner through the receiving thread corresponding to each service node based on the subscription configuration file.

In a possible implementation manner, the processing result obtaining module 504 is configured to:

and merging the target vehicle data and the theme of the target vehicle data according to a target format through each thread corresponding to each service node through the data scheduling module, and acquiring each processing result.

In a possible implementation, the data processing apparatus is further configured to: and controlling the data scheduling module to perform transfer caching under the lock-free queue on each processing result.

In a possible implementation manner, the processing result sending module 505 is further configured to:

controlling the data scheduling module to classify the processing results according to data types so as to obtain a processing result set of each category;

and when the data scheduling module meets specified conditions, sending the processing result set of each category to a database.

In a possible implementation, the data processing platform further includes a load balancing module, and the data processing apparatus is further configured to:

and controlling the load balancing module to respectively receive the vehicle data from each target vehicle and distribute the vehicle data to each service node in a balanced manner.

In a possible implementation, the data processing apparatus is further configured to: a connection is established between the respective target vehicle and the respective service node based on a message queue telemetry transport protocol.

In summary, in a case that the internet of vehicles data sent by multiple vehicles needs to be processed, the data processing platform may receive vehicle data sent by each vehicle and distribute each vehicle data to each service node, and at this time, the data scheduling module in the data processing platform performs data processing operations on target vehicle data through each thread corresponding to each service node, so as to obtain each processing result. According to the scheme, the multiple threads are set in the data scheduling module to respectively perform parallel processing on the target vehicle data sent by the multiple service nodes, so that the high-concurrency vehicle data can be quickly processed, and the processing capacity of the whole system on the vehicle networking data messages is improved.

FIG. 6 is a block diagram illustrating a data processing system including: each target vehicle, a data processing platform and a data analysis platform, wherein the data processing platform comprises a load balancing module, each service node, a data scheduling module and a database;

the load balancing module is used for respectively receiving vehicle data from each target vehicle and distributing each vehicle data to each service node;

the data scheduling module is used for acquiring subscription requirements and receiving target vehicle data from each service node according to the subscription requirements;

the data scheduling module is used for executing data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result;

the data scheduling module is used for sending the processing result to a database;

and the data analysis platform is used for performing report type processing on the vehicle data in the database and acquiring a processing result.

In summary, when the vehicle networking data sent by a plurality of vehicles needs to be processed, the data processing platform may receive vehicle data sent by each vehicle, and distribute each vehicle data to each service node, where the data scheduling module in the data processing platform performs a data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result. According to the scheme, the multiple threads are set in the data scheduling module to respectively perform parallel processing on the target vehicle data sent by the multiple service nodes, so that the high-concurrency vehicle data can be rapidly processed, and the processing capacity of the whole system on the data messages of the Internet of vehicles is improved.

Fig. 7 is a schematic diagram of a computer device according to an exemplary embodiment of the present application, the computer device including a memory and a processor, the memory being used for storing a computer program, and the computer program, when executed by the processor, implementing the data processing method described above.

The processor may be a Central Processing Unit (CPU). The Processor may also be other general purpose Processor, digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or a combination thereof.

The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the methods of the embodiments of the present invention. The processor executes various functional applications and data processing of the processor by executing non-transitory software programs, instructions and modules stored in the memory, that is, the method in the above method embodiment is realized.

The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor, and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the processor via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

In an exemplary embodiment, a computer readable storage medium is also provided for storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps of the above method. For example, the computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (10)

1. A data processing method is applied to a data processing platform, the data processing platform comprises service nodes, a data scheduling module and a database, and the method comprises the following steps:

receiving vehicle data from each target vehicle;

distributing each vehicle data to each service node;

the data scheduling module acquires subscription requirements and receives target vehicle data from each service node according to the subscription requirements;

the data scheduling module executes data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result;

and the data scheduling module sends the processing result to a database.

2. The method of claim 1, wherein the data scheduling module obtains subscription requirements and receives target vehicle data from the service nodes according to the subscription requirements, and comprises:

the data scheduling module acquires subscription requirements and acquires subscription configuration files based on the subscription requirements; the subscription configuration file comprises a domain name of a target service node, a port of the target service node, a user name and a password and a theme of vehicle data needing subscription;

the data scheduling module receives target vehicle data from the respective service nodes based on the subscription profile.

3. The method of claim 2, wherein the data scheduling module receives target vehicle data from the respective service nodes based on the subscription profile, comprising:

aiming at each service node, the data scheduling module establishes a receiving thread corresponding to the service node and establishes communication connection with the service node according to the receiving thread;

and the data scheduling module receives the target vehicle data from each service node in a multithreading manner through the receiving thread corresponding to each service node based on the subscription configuration file.

4. The method of claim 3, wherein the data scheduling module performs data processing operations on the target vehicle data through respective threads corresponding to respective service nodes to obtain respective processing results, comprising:

and the data scheduling module combines the target vehicle data and the theme of the target vehicle data according to a target format through each thread corresponding to each service node to obtain each processing result.

5. The method of claim 4, further comprising: and the data scheduling module performs transfer caching under the lock-free queue on each processing result.

6. The method of claim 5, wherein the data scheduling module sends the processing result to a database, comprising:

the data scheduling module classifies the processing results according to data types to obtain processing result sets of various categories;

and when the data scheduling module meets specified conditions, sending the processing result set of each category to a database.

7. The method of any of claims 1-6, wherein the data processing platform further comprises a load balancing module, the method further comprising:

the load balancing module receives vehicle data from each target vehicle and distributes the vehicle data to each service node in a balanced manner.

8. The method of claim 7, further comprising: and establishing connection between each target vehicle and each service node based on a message queue telemetry transmission protocol.

9. A data processing apparatus, comprising:

the vehicle data acquisition module is used for respectively receiving vehicle data from each target vehicle;

the vehicle data distribution module is used for distributing each piece of vehicle data to each service node;

the target vehicle data receiving module is used for controlling the data scheduling module to acquire subscription requirements and receiving target vehicle data from each service node according to the subscription requirements;

the processing result acquisition module is used for controlling the data scheduling module to execute data processing operation on the target vehicle data through each thread corresponding to each service node so as to obtain each processing result;

and the processing result sending module is used for controlling the data scheduling module to send the processing result to the database.

10. A data processing system, comprising: each target vehicle, a data processing platform and a data analysis platform, wherein the data processing platform comprises a load balancing module, each service node, a message queue service module, a data scheduling module and a database;

the load balancing module is used for respectively receiving vehicle data from each target vehicle and distributing each vehicle data to each service node;

the data scheduling module is used for acquiring subscription requirements and receiving target vehicle data from each service node according to the subscription requirements;

the data scheduling module is used for executing data processing operation on the target vehicle data through each thread corresponding to each service node to obtain each processing result;

the data scheduling module is used for sending the processing result to a database;

and the data analysis platform is used for performing report type processing on the vehicle data in the database and acquiring a processing result.

Images