CN112732788A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112732788A CN112732788A CN202110032338.5A CN202110032338A CN112732788A CN 112732788 A CN112732788 A CN 112732788A CN 202110032338 A CN202110032338 A CN 202110032338A CN 112732788 A CN112732788 A CN 112732788A
- Authority
- CN
- China
- Prior art keywords
- data
- measurement
- control data
- data processing
- time sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 15
- 238000003672 processing method Methods 0.000 title abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 131
- 238000005259 measurement Methods 0.000 claims abstract description 100
- 238000000034 method Methods 0.000 claims abstract description 52
- 230000008878 coupling Effects 0.000 claims abstract description 36
- 238000010168 coupling process Methods 0.000 claims abstract description 36
- 238000005859 coupling reaction Methods 0.000 claims abstract description 36
- 230000008569 process Effects 0.000 claims abstract description 31
- 238000004590 computer program Methods 0.000 claims description 13
- 238000004458 analytical method Methods 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 20
- 230000006870 function Effects 0.000 description 13
- 238000009826 distribution Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000007405 data analysis Methods 0.000 description 5
- 239000000872 buffer Substances 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Abstract
The invention discloses a data processing method, a data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring measurement and control data from a plurality of systems; analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data; clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship; based on a dynamic load balancing strategy, sending data without coupling relation to different data processing nodes so that the data processing nodes process the data in parallel; and receiving the data processing results of the data processing nodes, and outputting the data processing results in sequence according to the time sequence information. The invention can improve the accuracy and the high efficiency of data parallel processing.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
With the continuous advance of aerospace engineering in China, the density of aerospace tasks is increasingly higher, flight measurement and control tasks are increasingly complex, and the data downlink code rate is increased by orders of magnitude. In order to ensure that the measurement and control system can quickly judge and accurately control the flight process state and provide high requirements for the real-time performance of data processing, the measurement and control system is limited by the performance and resources of computer hardware, and in order to improve the processing performance, a parallel processing technology is generally adopted.
The system composition scale of modern aerospace engineering is very large, the interior of the system is divided into a plurality of subsystems according to services, the subsystems are composed of a plurality of subsystems, and the subsystems are closely linked when performing cooperative work, so that the downlink measurement and control data of the systems have strong coupling and time sequence characteristics.
At present, when data parallel processing is designed in a large-scale space measurement and control application software system, in order to solve the problems of time sequence and data coupling, the association relationship between data sources is often manually identified, a static data distribution strategy is established, and then parallel processing of load balancing is performed.
However, the manual maintenance cost is high, the difficulty is high, the required period is long, and the efficiency is low; in addition, due to human difference, the professional nature of the personnel and other factors, the identification errors and the missing items can be caused, and wrong input is generated for later static load balancing information configuration; in addition, due to human factors and limitations, in static data allocation, the granularity of data allocation is coarse, and the processing performance of a hardware system is difficult to give full play.
Disclosure of Invention
In view of the above, the present invention provides a data processing method, an apparatus, an electronic device and a storage medium to solve at least one of the above-mentioned problems.
According to a first aspect of the present invention, there is provided a data processing method, the method comprising:
acquiring measurement and control data from a plurality of systems;
analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data;
clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship;
based on a dynamic load balancing strategy, sending data without coupling relation to different data processing nodes so that the data processing nodes process the data in parallel;
and receiving the data processing results of the data processing nodes, and outputting the data processing results in sequence according to the time sequence information.
According to a second aspect of the present invention, there is provided a data processing apparatus, the apparatus comprising:
the data acquisition unit is used for acquiring measurement and control data from a plurality of systems;
the analysis unit is used for analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data;
the clustering unit is used for clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship;
the data sending unit is used for sending the data without the coupling relation to different data processing nodes based on a dynamic load balancing strategy so as to enable the data processing nodes to process the data in parallel;
and the data output unit is used for receiving the data processing results of the data processing nodes and outputting the data processing results in sequence according to the time sequence information.
According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method when executing the program.
According to a fourth aspect of the invention, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
According to the technical scheme, the acquired measurement and control data are analyzed according to the preset configuration information, the dependency relationship in the measurement and control data is identified, the time sequence information of the measurement and control data is acquired, then the measurement and control data are clustered according to the dependency relationship of the measurement and control data, data without coupling relationship are generated, then the data without coupling relationship are sent to different data processing nodes based on a dynamic load balancing strategy, the data processing nodes process the data in parallel, and when the data processing results of the data processing nodes are received, the data processing results can be output in sequence according to the time sequence information.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow diagram of a data processing method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of data clustering according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a process for writing parsing results based on index numbers according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating an output parsing result according to an embodiment of the present invention;
FIG. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary block diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, a static data distribution strategy is established by manually identifying the incidence relation between data sources, and then parallel processing of load balancing is carried out. However, the efficiency and accuracy of manual identification are low, and more errors are generated in the later data processing. Based on the above, the embodiment of the invention provides a data processing scheme, which fully considers the characteristics of data dependence and time sequence constraint, does not need manual identification, and can ensure the correctness and the high efficiency of data parallel processing. Embodiments of the present invention are described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the method including:
And 102, analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data.
103, clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship.
And 104, sending the data without the coupling relation to different data processing nodes based on the dynamic load balancing strategy so that the data processing nodes process the data in parallel.
And 105, receiving the data processing results of the data processing nodes, and outputting the data processing results in sequence according to the time sequence information.
The method comprises the steps of analyzing obtained measurement and control data according to preset configuration information, identifying the dependency relationship in the measurement and control data and obtaining time sequence information of the measurement and control data, clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship, sending the data without coupling relationship to different data processing nodes based on a dynamic load balancing strategy, processing the data by the data processing nodes in parallel, outputting the data processing results in sequence according to the time sequence information when the data processing results of the data processing nodes are received, and compared with the prior art, the method provided by the embodiment of the invention can identify the dependency relationship in the measurement and control data without manual identification, so that the accuracy and the efficiency of data parallel processing can be improved.
In an embodiment, for step 103, a topological directed graph of the measurement and control data may be generated according to the dependency relationship of the measurement and control data; and then, clustering the data with the dependency relationship based on a topological sorting algorithm and a topological directed graph to generate data without coupling relationship.
In the specific implementation process, the data dependency-based topological sorting algorithm can ensure that the depended data and the depended source packet are processed firstly, the data and source packet with dependency precondition are processed later, the source packet with dependency constraint is processed serially, and the source packet without dependency constraint is processed parallelly, and the main process comprises the following steps: relationship description, data clustering and topological sorting, and the three processes are respectively described below.
(1) Description of relationships
And mapping the data and the dependency relationship thereof to the graph to form a directed graph, wherein the data is mapped to a vertex in the graph, and the dependency relationship is mapped to an edge on the graph.
Defining (dependency), and assuming the data set as D, the data dependency R is defined as follows:
Defining (dependency graph), assuming that D is a data set and R is a data dependency, G ═ V, E > is defined as follows:
Where V represents the set of vertices of the graph, E represents the set of edges,
indicating that there is one edge e that is present,
representing elements of the data set D other than a,
representing elements in the representation data set D other than b.
(2) Data clustering
According to the dependency relationship shown in fig. 2, the data with dependency are merged, and then classified according to the data merging process, wherein the classification process is shown in fig. 2.
(3) Topological ordering
From the perspective of fig. 2, if the vertex a has an outgoing edge e ═ a, b >, the data corresponding to the point a needs to be calculated after the data corresponding to the point b is calculated, that is, the vertex with an incoupling degree of not 0 is calculated before the vertex with an incoupling degree of 0. Therefore, the dependency sorting problem is converted into a topological sorting problem on the directed graph, and each time a vertex on the graph is scanned, the vertex with the in-degree of 0 is moved into the parallel queue.
And the sorting algorithm decides whether to move the vertex into the parallel set of the calculation according to whether the in-degree of the vertex is 0 or not, and if so, the vertex is moved out of the current graph until no vertex exists in the graph (sorting is finished) or the out-degrees of the vertices are all >0 (with a ring, sorting is unsuccessful). The specific algorithm is as follows:
in a specific implementation process, after the time sequence information of the measurement and control data is acquired in step 102, a data index number may be added to each data in the measurement and control data according to the time sequence information based on a time sequence indexing technology.
For step 105, the data processing results can be output in order according to the data index number.
To ensure data consistency, in step 102, the measurement and control data are analyzed sequentially according to a time sequence during real-time analysis (i.e., analysis). By a time sequence indexing technology, a data index number is established, and the problem of parallel processing time sequence is solved by adopting an analysis result fast writing and reading mechanism.
(1) Defining data index numbers
Three elements defining the data index number as shown below: data source identification, virtual channel, frame number:
DATA：{SOURCE，VCID，FRAMEID}
in the data preprocessing step (i.e., the analysis operation in step 102), an index number is added to each data frame according to the data sequence.
(2) Fast writing and reading of the parsed result based on the index number (i.e., the data processing result of step 105)
Fig. 3 shows a fast writing process based on the index number parsing result, when the parsing result is written, a data index tree is established according to the index number, the parsing result is inserted in the leaf node of the index tree by using an insertion sorting method, and the data writing process supports concurrent writing.
(3) Resolution result readout support concurrent readout
Fig. 4 is a schematic diagram of reading out all the analysis results after two times of concurrent reading, and as shown in fig. 4, first, an initial reading flag is set for each reading thread, and each time a flag position result is read out, the reading flag is deleted from the index tree and updated.
The parallel processing scheme for the strongly-coupled data fully considers the characteristics of data dependence and time sequence constraint, and ensures the correctness and the high efficiency of the data parallel processing. The topological sorting algorithm based on the data dependency relationship solves the problem of parallel processing node distribution during data dependency, processes depended data and source packets firstly, processes the data and source packets with dependency preconditions and then processes the data and source packets with dependency relationship restrictions, serially processes the source packets with dependency relationship restrictions, and processes the source packets without dependency relationship restrictions in parallel, thereby improving the processing performance. Meanwhile, in order to ensure data consistency, the measurement and control data are analyzed in sequence according to time sequence during real-time analysis, a data index number is established through a time sequence indexing technology, and the problem of parallel processing time sequence is solved by adopting an analysis result fast writing and reading mechanism.
Based on similar inventive concepts, the embodiment of the present invention further provides a data processing apparatus, which is preferably used for implementing the flow of the data processing method.
Fig. 5 is a block diagram of the data processing apparatus, and as shown in fig. 5, the data processing apparatus includes: data acquisition unit 1, analysis unit 2, clustering unit 3, data transmission unit 4 and data output unit 5, wherein:
the data acquisition unit 1 is used for acquiring measurement and control data from a plurality of systems;
the analysis unit 2 is used for analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data;
the clustering unit 3 is used for clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship;
the data sending unit 4 is used for sending the data without the coupling relation to different data processing nodes based on a dynamic load balancing strategy so as to enable the data processing nodes to process the data in parallel;
and the data output unit 5 is used for receiving the data processing results of the data processing nodes and outputting the data processing results in sequence according to the time sequence information.
The analysis unit 2 analyzes the acquired measurement and control data according to the preset configuration information, identifies the dependency relationship in the measurement and control data and acquires the time sequence information of the measurement and control data, and then, the clustering unit 3 carries out clustering operation on the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship, then the data sending unit 4 sends the data without coupling relationship to different data processing nodes based on a dynamic load balancing strategy, each data processing node processes the data in parallel, when receiving the data processing results of the respective data processing nodes, the data output unit 5 may output the data processing results in order according to the timing information, compared to the prior art, according to the embodiment of the invention, the dependency relationship in the measurement and control data can be identified without manual identification, so that the accuracy and the efficiency of data parallel processing can be improved.
Specifically, the clustering unit includes: the directed graph generation module and the clustering module: the directed graph generating module is used for generating a topological directed graph of the measurement and control data according to the dependency relationship of the measurement and control data; and the clustering module is used for clustering the data with the dependency relationship based on a topological sorting algorithm and a topological directed graph so as to generate data without coupling relationship.
In one embodiment, the apparatus further includes: and the data index number adding unit is used for adding data index numbers to each piece of data in the measurement and control data according to the time sequence information based on a time sequence index technology.
Correspondingly, the data output unit is specifically configured to: and outputting the data processing results in sequence according to the data index numbers.
For specific execution processes of the units and the modules, reference may be made to the description in the foregoing method embodiments, and details are not described here again.
In practical operation, the units and the modules may be combined or may be singly arranged, and the present invention is not limited thereto.
For a better understanding of embodiments of the present invention, an example is given below.
FIG. 6 is an exemplary architecture diagram of a data processing apparatus, as shown in FIG. 6, comprising: data analysis module, data distribution module, parallel processing module, order preserving module, wherein:
and the data analysis module is used for automatically finishing the extraction of the strong coupling dependency relationship and the time sequence information in the measurement and control data according to the preset configuration information and describing the extracted information into a data structure which can be identified by a computer.
The configuration information here includes: the parameter description information, the calculation formula, the precondition and the like required by the processing of the space measurement and control data.
In a specific implementation process, when the data analysis module processes the temperature type parameter a, a reference voltage parameter B needs to be used for calculation, and in configuration information, a calculation formula is described by using the parameter B; for another example, when the data analysis module processes the parameter C, the parameter C is processed only in a certain flight phase of the space measurement and control, the flight phase is usually described by using a certain parameter D, and the precondition in the configuration information of the parameter C includes a certain condition of the parameter D. When a certain parameter X has the above two conditions, the data analysis module needs to perform dependency analysis, and a general method is to perform regular expression matching on each parameter information to be processed, match whether processing conditions of other parameters Y exist in configuration information, if so, X depends on Y, and during actual processing, the processing of X is completed after Y is required to be processed based on timing requirements.
And the data distribution module is used for sequencing, clustering and distributing the data in sequence by using a topological sequencing algorithm according to the analyzed data dependency relationship, so that the accuracy of parallel processing on the premise of ensuring the data dependency relationship is ensured.
And the parallel processing module is used for distributing the data which are subjected to clustering distribution to different nodes for processing by using a dynamic load balancing strategy, so that the data processing performance is improved.
And the order-preserving module is used for inputting the processed data in a mode of parallel writing and sequential reading of results according to the extracted time sequence information, so that the data is ensured to enter first.
The example device of the embodiment of the invention provides a parallel processing architecture for strong coupling data, after external measurement and control data are received, the dependency relationship in the data can be analyzed according to preset configuration information, time sequence index information in the data is extracted, after the data are analyzed and extracted, a data distribution module carries out topological sorting and clustering division on the data to form a data block which can be independent and has no coupling relationship, then a parallel processing module carries out dynamic distribution on the processed data block according to the processing load condition to complete data parallel processing, an order preserving module writes data results which are processed in parallel into a data index tree structure in parallel and is responsible for data to be issued outwards, and the time sequence is guaranteed according to the index tree structure when the data are issued.
The present embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, and the like, but is not limited thereto. In this embodiment, the electronic device may be implemented by referring to the above method embodiment and the data processing apparatus embodiment, and the contents thereof are incorporated herein, and repeated descriptions are omitted.
Fig. 7 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention. As shown in fig. 7, the electronic device 600 may include a central processor 100 and a memory 140; the memory 140 is coupled to the central processor 100. Notably, this diagram is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the data processing functions may be integrated into the central processor 100. The central processor 100 may be configured to control as follows:
And 102, analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data.
103, clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship.
And 104, sending the data without the coupling relation to different data processing nodes based on the dynamic load balancing strategy so that the data processing nodes process the data in parallel.
And 105, receiving the data processing results of the data processing nodes, and outputting the data processing results in sequence according to the time sequence information.
As can be seen from the above description, in the electronic device provided in the embodiment of the present application, the acquired measurement and control data is analyzed according to the preset configuration information, the dependency relationship in the measurement and control data is identified, and the time sequence information of the measurement and control data is acquired, and then, clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship, then sending the data without coupling relationship to different data processing nodes based on a dynamic load balancing strategy, processing the data by the data processing nodes in parallel, when receiving the data processing result of each data processing node, compared with the prior art, the data processing method and the device can identify the dependency relationship in the measurement and control data without manual identification, thereby improving the accuracy and the efficiency of data parallel processing.
In another embodiment, the data processing apparatus may be configured separately from the central processor 100, for example, the data processing apparatus may be configured as a chip connected to the central processor 100, and the data processing function is realized by the control of the central processor.
As shown in fig. 7, the electronic device 600 may further include: communication module 110, input unit 120, audio processing unit 130, display 160, power supply 170. It is noted that the electronic device 600 does not necessarily include all of the components shown in fig. 7; furthermore, the electronic device 600 may also comprise components not shown in fig. 7, which may be referred to in the prior art.
As shown in fig. 7, the central processor 100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, the central processor 100 receiving input and controlling the operation of the various components of the electronic device 600.
The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 100 may execute the program stored in the memory 140 to realize information storage or processing, etc.
The input unit 120 provides input to the cpu 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display an object to be displayed, such as an image or a character. The display may be, for example, an LCD display, but is not limited thereto.
The memory 140 may be a solid state memory such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 140 may also be some other type of device. Memory 140 includes buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage section 142, and the application/function storage section 142 is used to store application programs and function programs or a flow for executing the operation of the electronic device 600 by the central processing unit 100.
The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging application, address book application, etc.).
The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132 to implement general telecommunications functions. Audio processor 130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 130 is also coupled to the central processor 100, so that recording on the local can be enabled through a microphone 132, and so that sound stored on the local can be played through a speaker 131.
Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the data processing method.
In summary, the embodiment of the present invention provides an order-preserving scheme for parallel processing of measurement and control data, and aims to solve the problems of result disorder and processing errors caused by strong data source coupling and time sequence characteristics during parallel processing of data. The embodiment of the invention designs a serial-parallel combined processing architecture, realizes data serial processing with dependency relationship, does not have dependent source packet parallel processing, and exerts parallel processing performance to the maximum extent. Meanwhile, the embodiment of the invention provides a dynamic allocation strategy for data dependence, measurement and control data with a coupling relation are subjected to relevance division, dependent data are arranged in dependence data for pretreatment, and data without the dependence relation can be processed in parallel. In addition, the embodiment of the invention designs and processes the order-preserving output of the result, and realizes the consistency of the result and the input sequence.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings. The many features and advantages of the embodiments are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the embodiments which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the embodiments of the invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope thereof.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method of data processing, the method comprising:
acquiring measurement and control data from a plurality of systems;
analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data;
clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship;
based on a dynamic load balancing strategy, sending data without coupling relation to different data processing nodes so that the data processing nodes process the data in parallel;
and receiving the data processing results of the data processing nodes, and outputting the data processing results in sequence according to the time sequence information.
2. The method of claim 1, wherein clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship comprises:
generating a topological directed graph of the measurement and control data according to the dependency relationship of the measurement and control data;
and clustering the data with the dependency relationship based on a topological sorting algorithm and a topological directed graph to generate data without coupling relationship.
3. The method of claim 1, wherein after obtaining timing information of the measurement and control data, the method further comprises:
and based on a time sequence indexing technology, adding a data index number to each piece of data in the measurement and control data according to the time sequence information.
4. The method of claim 3, wherein outputting data processing results in sequence according to the timing information comprises:
and outputting the data processing results in sequence according to the data index numbers.
5. A data processing apparatus, characterized in that the apparatus comprises:
the data acquisition unit is used for acquiring measurement and control data from a plurality of systems;
the analysis unit is used for analyzing the measurement and control data according to preset configuration information so as to identify the dependency relationship in the measurement and control data and acquire the time sequence information of the measurement and control data;
the clustering unit is used for clustering the measurement and control data according to the dependency relationship of the measurement and control data to generate data without coupling relationship;
the data sending unit is used for sending the data without the coupling relation to different data processing nodes based on a dynamic load balancing strategy so as to enable the data processing nodes to process the data in parallel;
and the data output unit is used for receiving the data processing results of the data processing nodes and outputting the data processing results in sequence according to the time sequence information.
6. The apparatus of claim 5, wherein the clustering unit comprises:
the directed graph generating module is used for generating a topological directed graph of the measurement and control data according to the dependency relationship of the measurement and control data;
and the clustering module is used for clustering the data with the dependency relationship based on a topological sorting algorithm and a topological directed graph so as to generate data without coupling relationship.
7. The apparatus of claim 5, further comprising:
and the data index number adding unit is used for adding data index numbers to each piece of data in the measurement and control data according to the time sequence information based on a time sequence index technology.
8. The apparatus of claim 7, wherein the data output unit is specifically configured to:
and outputting the data processing results in sequence according to the data index numbers.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 4 are implemented when the processor executes the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110032338.5A CN112732788A (en) | 2021-01-11 | 2021-01-11 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110032338.5A CN112732788A (en) | 2021-01-11 | 2021-01-11 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112732788A true CN112732788A (en) | 2021-04-30 |
Family
ID=75590291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110032338.5A Pending CN112732788A (en) | 2021-01-11 | 2021-01-11 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112732788A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156782A (en) * | 2011-04-14 | 2011-08-17 | 国电南瑞科技股份有限公司 | Graph theory-based formula parallel calculation management method for electric power system |
| CN102902590A (en) * | 2012-09-03 | 2013-01-30 | 南京师范大学 | Parallel digital terrain analysis-oriented massive DEM (Digital Elevation Model) deploying and scheduling method |
| CN105426410A (en) * | 2015-11-02 | 2016-03-23 | 东软集团股份有限公司 | Data acquisition system and analytic method for same |
| CN110245010A (en) * | 2019-06-21 | 2019-09-17 | 广州小鹏汽车科技有限公司 | A kind of data processing method and device |
| US20200073712A1 (en) * | 2018-08-30 | 2020-03-05 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method, apparatus, device and medium for processing topological relation of tasks |
-
2021
- 2021-01-11 CN CN202110032338.5A patent/CN112732788A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156782A (en) * | 2011-04-14 | 2011-08-17 | 国电南瑞科技股份有限公司 | Graph theory-based formula parallel calculation management method for electric power system |
| CN102902590A (en) * | 2012-09-03 | 2013-01-30 | 南京师范大学 | Parallel digital terrain analysis-oriented massive DEM (Digital Elevation Model) deploying and scheduling method |
| CN105426410A (en) * | 2015-11-02 | 2016-03-23 | 东软集团股份有限公司 | Data acquisition system and analytic method for same |
| US20200073712A1 (en) * | 2018-08-30 | 2020-03-05 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method, apparatus, device and medium for processing topological relation of tasks |
| CN110245010A (en) * | 2019-06-21 | 2019-09-17 | 广州小鹏汽车科技有限公司 | A kind of data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109769141B (en) | Video generation method and device, electronic equipment and storage medium | |
| CN109947426B (en) | Application program generation method and device and electronic equipment | |
| CN110377462B (en) | Interface testing method and device and terminal equipment | |
| CN109815448B (en) | Slide generation method and device | |
| CN110543301B (en) | Method and device for generating jenkins code file | |
| CN109408468A (en) | Document handling method and device calculate equipment and storage medium | |
| CN115048254B (en) | Simulation test method, system, equipment and readable medium for data distribution strategy | |
| CN112463634A (en) | Software testing method and device under micro-service architecture | |
| CN102426567A (en) | Graphical editing and debugging system of automatic answer system | |
| CN111367895A (en) | Data migration method and device | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN111078573A (en) | Test message generation method and device | |
| CN112860264B (en) | Method and device for reconstructing abstract syntax tree | |
| CN113746840A (en) | JSON format message generation method and device | |
| CN112732788A (en) | Data processing method and device, electronic equipment and storage medium | |
| WO2023071566A1 (en) | Data processing method and apparatus, computer device, computer-readable storage medium, and computer program product | |
| CN109857576A (en) | RPC interface call method and device | |
| CN113419957A (en) | Rule-based big data offline batch processing performance capacity scanning method and device | |
| CN113360411A (en) | Code coverage rate detection method and device based on parallel variation test | |
| CN112905461A (en) | Method and device for executing automatic interface test case | |
| CN113760680A (en) | Method and device for testing system pressure performance | |
| CN108415981B (en) | Data dimension generation method, device, equipment and computer readable storage medium | |
| CN115208960B (en) | Protocol switching method and related equipment | |
| CN111526034A (en) | Route centralized management method, device, electronic device and storage medium | |
| CN111552479A (en) | Mirror image file compiling method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |