CN115543584B - Data processing method, device, equipment and medium - Google Patents
Data processing method, device, equipment and medium Download PDFInfo
- Publication number
- CN115543584B CN115543584B CN202211487203.9A CN202211487203A CN115543584B CN 115543584 B CN115543584 B CN 115543584B CN 202211487203 A CN202211487203 A CN 202211487203A CN 115543584 B CN115543584 B CN 115543584B
- Authority
- CN
- China
- Prior art keywords
- data
- output
- preset
- processing
- processing template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/4401—Bootstrapping
- G06F9/442—Shutdown
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a device, equipment and a medium, which relate to the field of vehicle-road cooperation, and the method comprises the following steps: determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; and starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template. The invention improves the usability, maintainability and expansibility of the data input and output model and reduces the labor time cost.
Description
Technical Field
The invention relates to the field of vehicle-road cooperation, in particular to a data processing method, device, equipment and medium.
Background
Currently, external system data and internal system data often encounter various problems in the access process, such as possible inconsistency of transmission protocols, possible inconsistency of formats corresponding to transmission, possible inconsistency of fields of contents, and the like. For these inconsistencies, much manpower and time are often required to be invested to correct the inconsistencies, wherein the invested cost and the inconsistencies present a certain proportional relationship, and the data access among different systems also faces challenges in the aspects of late expansibility and maintainability.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method, apparatus, device, and medium, so as to solve the problem that data of an internal and external system consumes more manpower and material resources in an access process.
According to a first aspect, an embodiment of the present invention provides a data processing method, including:
determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one to one, and the second threads correspond to the output sources one to one;
starting the first thread, acquiring to-be-processed data corresponding to an input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; and the preset output source processing template required by the corresponding output source is stored in each second thread.
With reference to the first aspect, in a first implementation manner of the first aspect, the starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data specifically includes:
starting the number of the input sources to determine the first threads with the corresponding number, and determining the preset input source processing template and the preset data processing template corresponding to the first threads;
acquiring data to be processed corresponding to the input source based on the preset input source processing template, and processing the data to be processed based on the preset data processing template to obtain the output data; the preset input source processing template comprises an input mode, an input source address loading mode and an input source data format, and the preset data processing template comprises a data analysis mode, a data mapping relation and an output format;
and determining that the preset input source processing template and the preset data processing template in the first thread are executed within a first preset time, and closing the corresponding first thread.
With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data further specifically includes:
and determining that the preset input source processing template and the preset data processing template in the first thread are not completely executed within a first preset time, and releasing the first thread.
With reference to the first implementation manner of the first aspect, in a third implementation manner of the first aspect, the acquiring, based on the preset input source processing template, to-be-processed data corresponding to the input source, and processing, based on the preset data processing template, the to-be-processed data to obtain the output data specifically includes:
determining a corresponding input adapter based on the input mode;
collecting sample data based on an input adapter, an input source address loading mode and an input source data format;
determining a corresponding processing adapter based on a data analysis mode, and analyzing the sample data based on the processing adapter;
and processing the analyzed sample data into output data based on the data mapping relation and the output format.
With reference to the first implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template specifically includes:
starting the second threads with the number corresponding to the number of the output sources, and determining the preset output source processing template corresponding to the second threads;
distributing the output data to the corresponding output source based on the preset output source processing template; the preset output source processing template comprises an output mode and an output source address loading mode;
and determining that the preset output source processing template in the second thread is completely executed within a second preset time, and closing the corresponding second thread.
With reference to the fourth implementation manner of the first aspect, in the fifth implementation manner of the first aspect, the starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template further specifically includes:
and determining that the preset output source processing template in the second thread is not completely executed within a second preset time, and releasing the second thread.
With reference to the fourth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the distributing the output data to the corresponding output source based on the preset output source processing template specifically includes:
determining a corresponding output adapter based on the output mode;
encapsulating the output data based on the output adapter;
and distributing the packaged output data to the corresponding output source based on the output source address loading mode.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, where the apparatus includes:
a processing preparation module for determining the number of at least one input source and the number of output sources, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one to one, and the second threads correspond to the output sources one to one;
the first processing module is used for starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
the second processing module is used for starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; each second thread is stored with the preset output source processing template required by the corresponding output source
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of any of the data processing methods described above when executing the program.
In a fourth aspect, the present invention further provides a non-transitory computer readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of the above.
According to the data processing method, the data processing device, the data processing equipment and the data processing medium, when an external system or third-party data is accessed, the corresponding first thread is configured for each input source, the corresponding second thread is configured for each output source, the preset input source processing template and the preset data processing template which are needed by the corresponding input source are stored in each first thread, the preset output source processing template which is needed by the corresponding output source is stored in each second thread, the first thread acquires sample data and processes the sample data to obtain output data based on the preset input source processing template and the preset data processing template, and the second thread distributes the output data to the corresponding output source based on the preset output source processing template.
Drawings
The features and advantages of the present invention will be more clearly understood by reference to the accompanying drawings, which are illustrative and not to be construed as limiting the invention in any way, and in which:
FIG. 1 is a flow chart illustrating a data processing method provided by the present invention;
FIG. 2 is a schematic flow chart of step S20 of the data processing method provided by the present invention;
FIG. 3 is a second schematic flowchart of step S20 of the data processing method provided in the present invention;
FIG. 4 is a flow chart illustrating step S22 of the data processing method provided by the present invention;
FIG. 5 is a flow chart showing one of the steps S30 in the data processing method provided by the present invention;
fig. 6 shows a second schematic flowchart of step S30 in the data processing method provided by the present invention;
fig. 7 shows a schematic flow chart of step S32 in the data processing method provided by the present invention;
FIG. 8 is a schematic diagram of a data processing apparatus according to the present invention;
fig. 9 shows a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The vehicle-road cooperation means that advanced wireless communication, new-generation internet and other technologies are adopted, dynamic real-time information interaction of vehicles and vehicles is carried out in all directions, active safety control of vehicles and road cooperative management are carried out on the basis of full-time dynamic traffic information collection and fusion, effective cooperation of people and vehicles is fully realized, traffic safety is guaranteed, traffic efficiency is improved, and therefore a safe, efficient and environment-friendly road traffic system is formed. The technical points of vehicle and road cooperation are three points: firstly, the cooperation of a man-vehicle-road system is forced; secondly, emphasizing regional large-scale networking joint control; thirdly, emphasizes the information interaction by utilizing the multi-mode traffic network. The vehicle-road cooperation is a result of fusion of information technology and two industries of automobiles and transportation, at present, data of each automobile is split no matter the research and development of single-automobile intelligence or road construction is carried out, the intelligence is limited intelligence, data generated by most of automobiles cannot be effectively adopted, the vehicle-road cooperation is characterized in that the automobiles, roads, people and clouds are effectively combined, the data generated by each automobile can be flexibly applied, the problem of operation encountered by a user is effectively solved, and meanwhile, a reasonable solution can be provided for various traffic conditions generated on the roads. The cooperation of the vehicle and the road is the core of intelligent transportation and intelligent high speed.
In the current V2X scheme, an on-board module, a roadside module, and a network facility need to be deployed, and the relationship among the on-board module, the roadside module, and the network facility is: the vehicle-mounted module in the vehicle identifies and collects vehicle data, the collected vehicle data are sent to the road side module or the network facility, the road side module collects and generates vehicle condition information on a road, the vehicle data and the vehicle condition data information are forwarded to the network facility, the network facility receives the removed data and the vehicle condition data, the data and the vehicle condition data are subjected to comprehensive analysis and cooperative processing and then fed back to the road side module, and the road side module forwards the data and the vehicle condition data to the vehicle-mounted module in the corresponding vehicle, so that corresponding vehicle-road cooperation is realized.
However, the manufacturers of the on-board module, the roadside module, and the network facility may belong to different manufacturers, and for a certain manufacturer, data belonging to relevant devices manufactured by the manufacturer is internal system data, and data not belonging to relevant devices manufactured by the manufacturer is external system data. During the access process, the external system data and the internal system data often encounter various problems, such as possible inconsistency of transmission protocols, possible inconsistency of corresponding transmission formats, possible inconsistency of content fields, and the like. To correct these inconsistencies, much labor and time are often required. The cost and inconsistency of the investment present a certain proportional relationship, which causes the data access between different systems to face a challenge in the later period of expansibility and maintainability.
For example, prior art solutions typically deal with data processing distribution for one-to-one mathematical models. In the implementation mode of part of schemes, hard coding is adopted, or data mapping without forwarding or flexible forwarding is realized, and in addition, the data are collected by means of a big data assembly and enter a data pool for unified processing.
The data processing method of the present invention is described below with reference to fig. 1, and includes the steps of:
s10, determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of the input sources and a corresponding number of second threads based on the number of the output sources.
The number n of the input sources and the number m of the output sources are not less than 1, namely, the method is an n-to-m mathematical model input-output processing distribution scheme, so that the method not only can be used for external system or third-party data access, but also can be used for data migration, data backup, data acquisition and the like.
In the embodiment of the present invention, a pipeline (pipeline) is formed based on a first thread and a second thread, where the number of the first threads is n and the number of the second threads is m. Wherein, the corresponding input source can be determined by the address of the input source, and the corresponding output source can be determined by the address of the output source.
S20, starting first threads, acquiring data to be processed corresponding to an input source based on a preset input source processing template, processing the data to be processed based on the preset data processing template, and obtaining output data.
And S30, starting second threads, and distributing output data to corresponding output sources based on preset output source processing templates.
As an optional implementation manner in the embodiment of the present invention, the preset input source processing template and the preset data processing template in different first threads are not completely the same, the preset output source processing template in different second threads is not completely the same, after the corresponding processing is completed, each first thread obtains one copy of output data, then all the output data are aggregated, and corresponding processing is performed through each second thread, so as to distribute the output data to the corresponding output source.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in different first threads are not completely the same, the preset output source processing templates in all second threads are all the same, after the corresponding processing is completed, each first thread obtains one piece of output data, then all the output data are aggregated, and corresponding processing is performed through each second thread, the processing manners in the second threads are all the same, and finally, the output data are distributed to the corresponding output sources.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in all the first threads are the same, the preset output source processing templates in different second threads are not completely the same, and since the data capture and data processing manners represented by the first threads are the same, related processing is performed based on the same data capture and data processing manners, one piece of output data is also obtained corresponding to each first thread, then all the output data are aggregated, corresponding processing is performed through each second thread, and finally, the output data are distributed to the corresponding output source.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in all the first threads are the same, and the preset output source processing templates in all the second threads are the same, because the data capture and data processing manners represented by the first threads are the same, the related processing may be performed based on the same data capture and data processing manners, one piece of output data may also be obtained corresponding to each first thread, then all the output data are aggregated and corresponding processing is performed through each second thread, the processing manners in the second threads are all the same, and finally the output data are distributed to the corresponding output sources.
In the embodiment of the present invention, the preset input source processing template includes an input mode, an input source address loading mode, and an input source data format, the preset data processing template includes a data parsing mode, a data mapping relationship, and an output format, and the preset output source processing template includes an output mode and an output source address loading mode. More specifically, the input mode can support multiple modes, such as text, http, kafka, websocket, active collection, passive input and the like, and provides a custom development and expansion spi mode; the data mapping relation supports user interface customized mapping, and comprises heterogeneous field processing, customized fields, mock data rules, xml and json output format conversion, extraction and aggregation of n input sources and the like when preset data processing templates in all first threads are consistent.
It should be noted that the preset input source processing template, the preset data processing template and the output source processing module may be configured by the user, and store the relevant contents in the database.
The data processing method is characterized in that when an external system or third-party data is accessed, various inconsistent problems are solved, a corresponding first thread is configured for each input source, a corresponding second thread is configured for each output source, a preset input source processing template and a preset data processing template which are required by the corresponding input source are stored in each first thread, a preset output source processing template which is required by the corresponding output source is stored in each second thread, the first thread acquires sample data based on the preset input source processing template and the preset data processing template and processes the sample data to obtain output data, and the second thread distributes the output data to the corresponding output source based on the preset output source processing template.
The data processing method of the present invention is described below with reference to fig. 2, where step S20 specifically includes:
s21, starting the number of the input sources to determine a first thread with a corresponding number, and determining a preset input source processing template and a preset data processing template corresponding to the first thread.
S22, acquiring data to be processed corresponding to the input source based on the preset input source processing template, and processing the data to be processed based on the preset data processing template to obtain output data.
S23, determining that the preset input source processing template and the preset data processing template in the first thread are executed within the first preset time, and closing the corresponding first thread.
In the embodiment of the present invention, when the preset input source processing template and the preset data processing template in one first thread are executed within the specified first preset time, which indicates that the first thread is working normally, the corresponding first thread is closed after working normally.
The data processing method of the present invention is described below with reference to fig. 3, and step S20 further specifically includes:
s21, starting the number of the input sources to determine a first thread with a corresponding number, and determining a preset input source processing template and a preset data processing template corresponding to the first thread.
And S22, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data.
And S24, determining that the preset input source processing template and the preset data processing template in the first thread are not completely executed within the first preset time, and releasing the first thread.
In the embodiment of the present invention, when the preset input source processing template and the preset data processing template in the first thread are not completely executed within the specified first preset time, it indicates that the first thread cannot normally operate currently, and the method actively releases the input source processing template and the preset data processing template after the first preset time is exceeded. It should be noted that, when it is determined that a certain first thread cannot normally operate, the method may analyze a cause causing an exception, such as a program operation failure or other system causes, and also generate an alarm message, where the alarm message includes a number of the first thread, a corresponding exception of the first thread, and a cause causing an exception of the corresponding first thread.
The data processing method of the present invention is described below with reference to fig. 4, where step S22 specifically includes:
and S221, determining a corresponding input adapter based on the input mode. In the embodiment of the invention, the input adapter is an adapter in the forms of text, http, kafka, websocket, active collection, passive input and the like.
S222, collecting sample data based on the input adapter, the input source address loading mode and the input source data format.
And S223, determining a corresponding processing adapter based on the data analysis mode, and analyzing the sample data based on the processing adapter. In the embodiment of the present invention, the processing adapter is an adapter in the form of json, xml, or the like.
And S224, processing the analyzed sample data into output data based on the data mapping relation and the output format.
The data processing method of the present invention is described below with reference to fig. 5, where step S30 specifically includes:
s31, starting the number of the output sources to determine a corresponding number of second threads, and determining a preset output source processing template corresponding to the second threads.
And S32, distributing the output data to the corresponding output source based on the preset output source processing template.
And S33, determining that the preset output source processing template in the second thread is executed within second preset time, and closing the corresponding second thread.
In the embodiment of the present invention, when the preset output source processing template in a second thread is executed within the specified second preset time, which indicates that the second thread is working normally, the corresponding second thread is closed after working normally.
The data processing method of the present invention is described below with reference to fig. 6, and step S30 further specifically includes:
s31, starting the number of the output sources to determine a corresponding number of second threads, and determining a preset output source processing template corresponding to the second threads.
And S32, distributing the output data to the corresponding output source based on the preset output source processing template.
And S34, determining that the preset output source processing template in the second thread is not completely executed within the second preset time, and releasing the second thread.
In the embodiment of the present invention, when the preset output source processing template in the second thread is not completely executed within the specified second preset time, which indicates that the second thread cannot normally operate currently, the method actively releases the second thread after the second preset time is exceeded. It should be noted that, similarly, when it is determined that a certain second thread cannot work normally, the method may analyze a cause causing an abnormality, such as a program operation failure or another system cause, and may also generate alarm information, where the alarm information includes a number of the first thread, a corresponding abnormality of the first thread, and a cause causing an abnormality of the corresponding second thread.
The data processing method of the present invention is described below with reference to fig. 7, where step S32 specifically includes:
s321, based on the output mode, a corresponding output adapter is determined. In the embodiment of the invention, the output adapter is an adapter in the forms of http, kafka, websocket and the like.
And S322, packaging the output data based on the output adapter.
And S323, distributing the encapsulated output data to a corresponding output source based on the output source address loading mode, namely the target address of the output source.
The data processing device provided by the invention is described below, and the data processing device described below and the data processing method described above can be referred to correspondingly.
The data processing apparatus of the present invention is described below with reference to fig. 8, and includes the steps of:
the processing preparation module 10 is configured to determine the number of at least one input source and the number of at least one output source, and determine a corresponding number of first threads based on the number of input sources and determine a corresponding number of second threads based on the number of output sources.
The number n of the input sources and the number m of the output sources are not less than 1, namely, the device is an n-to-m mathematical model input-output processing distribution scheme, so that the device not only can be used for external system or third-party data access, but also can be used for data migration, data backup, data acquisition and the like.
In the embodiment of the present invention, a pipeline (pipeline) is formed based on a first thread and a second thread, where the number of the first threads is n and the number of the second threads is m. Wherein, the corresponding input source can be determined by the address of the input source, and the corresponding output source can be determined by the address of the output source.
The first processing module 20 is configured to start a first thread, acquire to-be-processed data corresponding to an input source based on a preset input source processing template, and process the to-be-processed data based on the preset data processing template to obtain output data.
The second processing module 30 is configured to start a second thread, and distribute output data to a corresponding output source based on a preset output source processing template, where in the embodiment of the present invention, each second thread stores a preset output source processing template required by the corresponding output source.
As an optional implementation manner in the embodiment of the present invention, the preset input source processing template and the preset data processing template in different first threads are not completely the same, the preset output source processing template in different second threads is not completely the same, after the corresponding processing is completed, each first thread obtains one copy of output data, then all the output data are aggregated, and corresponding processing is performed through each second thread, so as to distribute the output data to the corresponding output source.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in different first threads are not completely the same, the preset output source processing templates in all second threads are all the same, after the corresponding processing is completed, each first thread obtains one copy of output data, then all the output data are aggregated, and corresponding processing is performed through each second thread, the processing manners in the second threads are all the same, and finally, the output data are distributed to the corresponding output sources.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in all the first threads are the same, the preset output source processing templates in different second threads are not completely the same, and since the data capture and data processing manners represented by the first threads are the same, related processing is performed based on the same data capture and data processing manners, one piece of output data is also obtained corresponding to each first thread, then all the output data are aggregated, corresponding processing is performed through each second thread, and finally, the output data are distributed to the corresponding output source.
As another optional implementation manner in the embodiment of the present invention, the preset input source processing templates and the preset data processing templates in all the first threads are the same, and the preset output source processing templates in all the second threads are the same, because the data capture and data processing manners represented by the first threads are the same, the related processing may be performed based on the same data capture and data processing manners, one piece of output data may also be obtained corresponding to each first thread, then all the output data are aggregated and corresponding processing is performed through each second thread, the processing manners in the second threads are all the same, and finally the output data are distributed to the corresponding output sources.
In the embodiment of the present invention, the preset input source processing template includes an input mode, an input source address loading mode, and an input source data format, the preset data processing template includes a data parsing mode, a data mapping relationship, and an output format, and the preset output source processing template includes an output mode and an output source address loading mode. More specifically, the input mode can support multiple modes, such as text, http, kafka, websocket, active collection, passive input and the like, and provides a custom development and expansion spi mode; the data mapping relation supports user interface customized mapping, and comprises heterogeneous field processing, customized fields, mock data rules, xml and json output format conversion, extraction and aggregation of n input sources and the like when preset data processing templates in all the first threads are consistent.
It should be noted that the preset input source processing template, the preset data processing template and the output source processing module may be configured by the user, and store the relevant contents in the database.
The data processing device is used for solving various inconsistent problems when an external system or third-party data is accessed, a corresponding first thread is configured for each input source, a corresponding second thread is configured for each output source, a preset input source processing template and a preset data processing template which are required by the corresponding input source are stored in each first thread, a preset output source processing template which is required by the corresponding output source is stored in each second thread, the first thread acquires sample data based on the preset input source processing template and the preset data processing template and processes the sample data to obtain output data, and the second thread distributes the output data to the corresponding output source based on the preset output source processing template.
Fig. 9 illustrates a physical structure diagram of an electronic device, and as shown in fig. 9, the electronic device may include: a processor (processor) 410, a communication Interface 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may invoke logical commands in the memory 430 to perform a data processing method comprising:
determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one to one, and the second threads correspond to the output sources one to one;
starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; and the preset output source processing template required by the corresponding output source is stored in each second thread.
In addition, the logic commands in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic commands are sold or used as a separate medium. Based on such understanding, the technical solution of the present invention may be essentially or partially contributed to by the prior art, or may be embodied in a form of a software medium, which is stored in a storage medium and includes a plurality of commands for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program medium, the computer program medium comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program, when executed by a processor, being capable of executing the data processing method provided by the above methods, the method comprising:
determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one by one, and the second threads correspond to the output sources one by one;
starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; and the preset output source processing template required by the corresponding output source is stored in each second thread.
In still another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the data processing method provided by the above methods, the method including:
determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one to one, and the second threads correspond to the output sources one to one;
starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; and the preset output source processing template required by the corresponding output source is stored in each second thread.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above technical solutions may be embodied in the form of a software medium which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes commands for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A data processing method is characterized in that the method is applied to an on-board module, a road side module and a network facility, wherein the on-board module in a vehicle identifies and collects vehicle data and sends the collected vehicle data to the road side module or the network facility, the road side module collects and generates vehicle condition information on a road and forwards the vehicle data and the vehicle condition data information to the network facility, the network facility receives the vehicle data and the vehicle condition data, carries out comprehensive analysis and cooperative processing and then feeds back the vehicle data and the vehicle condition data to the road side module, and the road side module forwards the vehicle data and the vehicle condition data to the on-board module in the corresponding vehicle, and the method comprises the following steps:
determining the number of at least one input source and the number of at least one output source, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one to one, and the second threads correspond to the output sources one to one;
starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; the preset output source processing template required by the corresponding output source is stored in each second thread;
the starting of the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data specifically include:
starting the number of the input sources to determine the first threads with the corresponding number, and determining the preset input source processing template and the preset data processing template corresponding to the first threads;
acquiring data to be processed corresponding to the input source based on the preset input source processing template, and processing the data to be processed based on the preset data processing template to obtain the output data; the preset input source processing template comprises an input mode, an input source address loading mode and an input source data format, and the preset data processing template comprises a data analysis mode, a data mapping relation and an output format;
determining that the preset input source processing template and the preset data processing template in the first thread are executed within a first preset time, and closing the corresponding first thread;
the starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template specifically includes:
starting the second threads with the number corresponding to the number of the output sources, and determining the preset output source processing template corresponding to the second threads;
distributing the output data to the corresponding output source based on the preset output source processing template; the preset output source processing template comprises an output mode and an output source address loading mode;
and determining that the preset output source processing template in the second thread is completely executed within a second preset time, and closing the corresponding second thread.
2. The data processing method according to claim 1, wherein the starting the first thread, collecting to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data further specifically comprises:
and determining that the preset input source processing template and the preset data processing template in the first thread are not completely executed within a first preset time, and releasing the first thread.
3. The data processing method according to claim 1, wherein the acquiring, based on the preset input source processing template, the to-be-processed data corresponding to the input source, and processing, based on the preset data processing template, the to-be-processed data to obtain the output data specifically includes:
determining a corresponding input adapter based on the input mode;
collecting sample data based on an input adapter, an input source address loading mode and an input source data format;
determining a corresponding processing adapter based on a data analysis mode, and analyzing the sample data based on the processing adapter;
and processing the analyzed sample data into output data based on the data mapping relation and the output format.
4. The data processing method according to claim 1, wherein the starting of the second thread and the distribution of the output data to the corresponding output source based on a preset output source processing template further specifically include:
and determining that the preset output source processing template in the second thread is not executed within a second preset time, and releasing the second thread.
5. The data processing method according to claim 1, wherein the distributing the output data to the corresponding output source based on the preset output source processing template specifically includes:
determining a corresponding output adapter based on the output mode;
encapsulating the output data based on the output adapter;
and distributing the packaged output data to the corresponding output source based on the output source address loading mode.
6. The device is applied to an on-board module, a road side module and a network facility, wherein the on-board module in a vehicle identifies and collects vehicle data and sends the collected vehicle data to the road side module or the network facility, the road side module collects and generates vehicle condition information on a road and forwards the vehicle data and the vehicle condition data information to the network facility, the network facility receives the vehicle data and the vehicle condition data, performs comprehensive analysis and cooperative processing on the vehicle data and the vehicle condition data and then feeds the vehicle data and the vehicle condition data back to the road side module, and the road side module forwards the vehicle condition information to the on-board module in the corresponding vehicle, and the device comprises:
a processing preparation module for determining the number of at least one input source and the number of output sources, and determining a corresponding number of first threads based on the number of input sources and a corresponding number of second threads based on the number of output sources; the first threads correspond to the input sources one by one, and the second threads correspond to the output sources one by one;
the first processing module is used for starting the first thread, acquiring to-be-processed data corresponding to the input source based on a preset input source processing template, and processing the to-be-processed data based on the preset data processing template to obtain output data; the preset input source processing template and the preset data processing template required by the corresponding input source are stored in each first thread;
the second processing module is used for starting the second thread, and distributing the output data to the corresponding output source based on a preset output source processing template; the preset output source processing template required by the corresponding output source is stored in each second thread;
the first processing module is specifically configured to:
starting the first threads with the number corresponding to the number of the input sources, and determining the preset input source processing template and the preset data processing template corresponding to the first threads;
acquiring data to be processed corresponding to the input source based on the preset input source processing template, and processing the data to be processed based on the preset data processing template to obtain the output data; the preset input source processing template comprises an input mode, an input source address loading mode and an input source data format, and the preset data processing template comprises a data analysis mode, a data mapping relation and an output format;
determining that the preset input source processing template and the preset data processing template in the first thread are executed within a first preset time, and closing the corresponding first thread;
the second processing module is specifically configured to:
starting the second threads with the quantity corresponding to the quantity determination of the output sources, and determining the preset output source processing template corresponding to the second threads;
distributing the output data to the corresponding output source based on the preset output source processing template; the preset output source processing template comprises an output mode and an output source address loading mode;
and determining that the preset output source processing template in the second thread is completely executed within a second preset time, and closing the corresponding second thread.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the data processing method according to any of claims 1 to 5 are implemented when the processor executes the program.
8. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211487203.9A CN115543584B (en) | 2022-11-25 | 2022-11-25 | Data processing method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211487203.9A CN115543584B (en) | 2022-11-25 | 2022-11-25 | Data processing method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115543584A CN115543584A (en) | 2022-12-30 |
| CN115543584B true CN115543584B (en) | 2023-04-07 |
Family
ID=84719735
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211487203.9A Active CN115543584B (en) | 2022-11-25 | 2022-11-25 | Data processing method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115543584B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104111983B (en) * | 2014-06-30 | 2017-12-19 | 中国科学院信息工程研究所 | A kind of open multi-source data acquiring system and method |
| CN107636602A (en) * | 2017-07-14 | 2018-01-26 | 深圳市元征科技股份有限公司 | A kind of data processing method and data processing equipment |
| CN111210884B (en) * | 2019-12-31 | 2024-04-26 | 创业慧康科技股份有限公司 | Clinical medical data acquisition method, device, medium and equipment |
| CN112131289A (en) * | 2020-08-17 | 2020-12-25 | 武汉旷视金智科技有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2022
- 2022-11-25 CN CN202211487203.9A patent/CN115543584B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115543584A (en) | 2022-12-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109164783B (en) | Vehicle diagnosis method, apparatus, device, and medium | |
| CN107656520B (en) | CAN bus data analysis method and computer readable storage medium | |
| CN112115604B (en) | Combined simulation method and system for Modelica model and Simulink model and electronic equipment | |
| CN115277882B (en) | CAN message database establishment method and device, vehicle-mounted electronic equipment and storage medium | |
| EP4312199A1 (en) | Method and device for remotely diagnosing vehicle faults, vehicle and computer storage medium | |
| CN111478944B (en) | Vehicle-mounted terminal and vehicle CAN protocol adaptation method, device, equipment and medium | |
| CN111737023A (en) | Vehicle-mounted event processing method, cloud server and computer-readable storage medium | |
| CN110941251A (en) | Production control method, device, equipment and medium based on digital twin body | |
| CN113467771B (en) | Model-based industrial edge cloud collaboration system and method | |
| CN115878681A (en) | Method and device for acquiring automatic driving data, storage medium and electronic device | |
| CN114401207B (en) | Communication abnormal terminal equipment positioning method and device and electronic equipment | |
| CN115543584B (en) | Data processing method, device, equipment and medium | |
| CN113726624A (en) | Method and device for acquiring CAN (controller area network) signal by TBOX (tunnel boring machine) device | |
| CN113836454A (en) | Content display page display method, device, medium and equipment | |
| CN114021332A (en) | Debugging method, device, equipment and storage medium | |
| CN116522871B (en) | Data format conversion method, device, equipment and storage medium | |
| CN112835598A (en) | Automobile ECU (electronic control Unit) flashing method and system and computer readable storage medium | |
| CN115237101A (en) | Debugging method and intelligent debugging device for motor train unit and debugging system for motor train unit | |
| CN114817671A (en) | Log analysis method and system for air conditioner Internet of things | |
| CN114430421A (en) | Method and system for automatically generating alarm rules based on function sets of various vehicle types | |
| CN110866614B (en) | GSP-based transparent operation and maintenance method and system for intelligent substation automation equipment | |
| CN111895618A (en) | Cloud platform data visualization control method and device and storage medium | |
| CN111782475A (en) | Distributed log access method and system based on command mode | |
| CN112395695A (en) | Method and system for establishing simulation scene in real time | |
| CN110727457A (en) | Component management method, device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |