CN114138861A - Multi-source heterogeneous data processing method, device and system - Google Patents
Multi-source heterogeneous data processing method, device and system Download PDFInfo
- Publication number
- CN114138861A CN114138861A CN202111393333.1A CN202111393333A CN114138861A CN 114138861 A CN114138861 A CN 114138861A CN 202111393333 A CN202111393333 A CN 202111393333A CN 114138861 A CN114138861 A CN 114138861A
- Authority
- CN
- China
- Prior art keywords
- data
- heterogeneous
- processing
- different
- normalization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 91
- 238000006243 chemical reaction Methods 0.000 claims abstract description 81
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000010606 normalization Methods 0.000 claims abstract description 32
- 230000011664 signaling Effects 0.000 claims abstract description 26
- 238000004458 analytical method Methods 0.000 claims abstract description 21
- 238000005259 measurement Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims abstract description 15
- 238000004590 computer program Methods 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 12
- 230000002457 bidirectional effect Effects 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 23
- 230000006870 function Effects 0.000 description 23
- 238000004891 communication Methods 0.000 description 17
- 238000010248 power generation Methods 0.000 description 7
- 239000000872 buffer Substances 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012351 Integrated analysis Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 238000003892 spreading Methods 0.000 description 1
- 238000011425 standardization method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention discloses a multi-source heterogeneous data processing method, a device and a system, wherein the method comprises the following steps: receiving heterogeneous data from different data sources, the heterogeneous data comprising: data of different source devices and different data types; analyzing the heterogeneous data according to a preset analysis rule; performing a normalization conversion process on the parsed data based on a predetermined normalization rule, the normalization conversion process including: data code conversion processing and remote signaling remote measurement value conversion processing; and sending the data after the standardization processing to a data platform so as to be convenient for subsequent use. By the invention, the efficiency of data application can be improved.
Description
Technical Field
The invention relates to the field of data processing, in particular to a multi-source heterogeneous data processing method, device and system.
Background
In recent years, big data technology is rapidly developed, application of new technology in the field of new energy gradually becomes a hotspot of industrial research, and how to apply the big data technology to assist new energy transformation becomes a research focus. And the characteristics of multi-source isomerism, uneven data quality, non-uniform standard, isolated data island and the like of new energy data at the present stage provide challenges for application of new energy big data. The data application aims to realize that the data of internal and external low-density low-value data are uniformly extracted, cleaned and stored in a warehouse, mined, analyzed and produced to form high-density high-value knowledge, so that services are provided for internal and external systems. This is achieved on the premise of reliability, safety, real-time, stability and standardization of data acquisition and transmission.
The power generation equipment in the large-scale new energy station is various in type and quantity, correspondingly, the matched power transmission and transformation equipment and control equipment are various, and meanwhile, a large amount of unstructured data such as pictures and character records exist. The new energy stations are numerous, the manufacturers are numerous, the models are different, the data standards are different, and great challenges are provided for application of new energy big data.
At present, all-around, whole-process and high-precision monitoring of each power generation device is realized in most power generation stations, a large amount of multi-source heterogeneous data is accumulated, the data are dispersed in each monitoring system, great difficulty is brought to subsequent data processing and conversion work, and the data application efficiency is reduced.
Disclosure of Invention
Accordingly, the present invention is directed to a method, an apparatus, and a system for processing multi-source heterogeneous data to solve at least one of the above-mentioned problems.
According to a first aspect of the present invention, there is provided a multi-source heterogeneous data processing method, the method comprising:
receiving heterogeneous data from different data sources, the heterogeneous data comprising: data of different source devices and different data types;
analyzing the heterogeneous data according to a preset analysis rule;
performing a normalization conversion process on the parsed data based on a predetermined normalization rule, the normalization conversion process including: data code conversion processing and remote signaling remote measurement value conversion processing;
and sending the data after the standardization processing to a data platform so as to be convenient for subsequent use.
According to a second aspect of the present invention, there is provided a multi-source heterogeneous data processing apparatus, the apparatus comprising:
a data receiving unit, configured to receive heterogeneous data from different data sources, where the heterogeneous data includes: data of different source devices and different data types;
the analysis unit is used for analyzing the heterogeneous data according to a preset analysis rule;
a normalization conversion unit configured to perform normalization conversion processing on the parsed data based on a predetermined normalization rule, the normalization conversion processing including: data code conversion processing and remote signaling remote measurement value conversion processing;
and the data sending unit is used for sending the data after the standardization processing to a data platform so as to be convenient for subsequent use.
According to a third aspect of the invention, there is provided a multi-source heterogeneous data processing system, the system comprising: according to the multi-source heterogeneous data processing device and the data platform, the data platform receives and stores the data after standardized processing from the multi-source heterogeneous data processing device, so that the data can be used later.
According to a fourth aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the program.
According to a fifth aspect of the invention, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
According to the technical scheme, the received heterogeneous data is analyzed according to the preset analysis rule, then the analyzed data is subjected to standardized conversion processing based on the preset standardized rule, and the standardized data is sent to the data platform so as to be used subsequently, therefore, the acquisition of multi-source heterogeneous data and the conversion and access of standardized data can be realized, and the data application efficiency can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a block diagram of a multi-source heterogeneous data processing system according to an embodiment of the present invention;
fig. 2 is a block diagram of the structure of a multi-source heterogeneous data processing apparatus 1 according to an embodiment of the present invention;
FIG. 3 is an exemplary diagram of a multi-source heterogeneous data processing system according to an embodiment of the invention;
FIG. 4 is a block diagram of an exemplary structure of a standardized data interface according to an embodiment of the invention;
FIG. 5 is a diagram of an example of data encoding according to an embodiment of the present invention;
FIG. 6 is a diagram of an example of a fault code of a fault class test point of an electronic control system according to an embodiment of the present invention;
FIG. 7 is a flow diagram of a method of multi-source heterogeneous data processing according to an embodiment of the present invention;
fig. 8 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Because a large amount of scattered multi-source heterogeneous data exist in the current power generation station, the efficiency of data application is reduced. Based on the scheme, the multi-source heterogeneous data processing scheme can achieve acquisition of multi-source heterogeneous data and conversion and access of standardized data, and therefore the data application efficiency can be improved. Embodiments of the present invention are described in detail below with reference to the accompanying drawings.
Fig. 1 is a block diagram of a multi-source heterogeneous data processing system according to an embodiment of the present invention, as shown in fig. 1, the system includes: the multi-source heterogeneous data processing device comprises a multi-source heterogeneous data processing device 1 and a data platform 2, wherein the multi-source heterogeneous data processing device 1 is used for standardizing received multi-source heterogeneous data, and the data platform is used for receiving and storing the data which are subjected to the standardized processing from the multi-source heterogeneous data processing device so as to be convenient for subsequent use. The multi-source heterogeneous data processing apparatus 1 is described in detail below.
Fig. 2 is a block diagram of a multi-source heterogeneous data processing apparatus 1, as shown in fig. 2, which includes: a data receiving unit 11, a parsing unit 12, a normalization conversion unit 13, and a data sending unit 14, wherein:
a data receiving unit 11, configured to receive heterogeneous data from different data sources, where the heterogeneous data includes: data of different source devices, different data types.
In actual operation, the data receiving unit may receive heterogeneous data from different data sources based on a unidirectional, bidirectional switching pattern.
And the analysis unit 12 is configured to perform analysis processing on the heterogeneous data according to a predetermined analysis rule. Through the analysis processing, the corresponding conversion rule of the data can be determined.
A normalization conversion unit 13, configured to perform normalization conversion processing on the parsed data based on a predetermined normalization rule, where the normalization conversion processing includes: data code conversion processing and remote signaling telemetry value conversion processing.
Specifically, the normalization conversion unit includes: the remote signaling telemetry value conversion system comprises a data coding conversion module and a remote signaling telemetry value conversion module, wherein:
the data code conversion module is used for carrying out data code conversion processing on the analyzed data according to the data source equipment and the data type based on a preset coding rule;
and the remote signaling and remote measuring value conversion module is used for carrying out remote signaling and remote measuring value conversion processing on the analyzed data according to the accessed remote signaling and remote measuring value data format.
In one embodiment, the heterogeneous data includes: data of different fault class measurement points. The remote signaling and remote measuring value conversion module can perform remote signaling and remote measuring value conversion processing on the analyzed data according to remote signaling and remote measuring value data formats from different fault measuring points based on a preset fault coding rule.
And the data sending unit 14 is used for sending the standardized data to a data platform so as to facilitate subsequent use.
The analysis unit 12 analyzes the heterogeneous data received by the data receiving unit 11 according to a preset analysis rule, then the standardization conversion unit 13 performs standardization conversion processing on the analyzed data based on a preset standardization rule, and the data sending unit 14 sends the standardized data to a data platform so as to be used subsequently.
For a better understanding of the present invention, embodiments of the present invention are described in detail below in connection with the exemplary system shown in FIG. 3.
In this example, the different data sources are from a new energy site, and a data fusion application can be realized through the system of this example, which is the basis for building a new energy big data platform.
Referring to fig. 3, a schematic diagram of data interaction between the data center and the new energy station (i.e., station 1, station n in the figure) is shown, in this structure, a high-reliability data acquisition device is utilized to fully acquire all production operation data of the new energy station to a standardized data interface of the data center (having the function of the multi-source heterogeneous data processing device). Meanwhile, the system can realize the bidirectional interaction of the data center and the station information, is convenient for the station to receive the service guide information processed by the data center, and realizes the equipment state monitoring, the safety production management, the production management control and optimization and the like of the station.
The standardized data interface collects data of all parties and then sends the data to the data platform, and the normal operation of the standardized data interface is the basis and the premise of the normal operation of the data platform. The method relates to the safety of the system, the accuracy and the real-time performance of data and the stable operation of the whole system.
Fig. 4 is a block diagram of an example structure of a standardized data interface, as shown in fig. 4, which includes: data communication module, data write in module, data analysis module, data conversion module, data buffer module, data reading module, controller, time synchronization module and storage module, wherein:
the data communication module can have wired and wireless communication functions at the same time, supports 4G/5G network communication, can be set according to the requirements of an access data source, and is convenient for enhancing the adaptation scene of a data interface. The module can adopt a multi-process mode or an independent service program mode, simultaneously synchronously collects multiple data sources, and each link in the remote data uploading process is independent in sending and receiving, so that the coupling and dependency among communication programs can be reduced, the execution efficiency is improved, and the safety risk is reduced. According to the specific requirements of data acquisition, the data center can be set to switch between a unidirectional transmission mode and a bidirectional interaction mode, so that bidirectional interaction between the data center and the station information is realized.
The data writing module is used for receiving data, the data writing module and the data analysis module are integrated with the current mainstream communication protocol analysis rule, after the data is accessed, the data analysis module can analyze the data according to a data format, automatically judge the communication protocol used by the data analysis module, analyze the data by utilizing the integrated analysis rule, determine a data conversion standard, provide a programmable function aiming at a special and individual communication protocol, realize the analysis of the data rule by programming, and realize the functions of being compatible with various communication protocols, various data acquisition types, unlimited label point quantity, high-speed data acquisition frequency, fault processing, log recording and the like aiming at a source system adopting the individual communication protocol. For some data which cannot be automatically collected, such as production logs, operation analysis reports, photo data and the like, the data writing module provides a manual input function.
In order to ensure the safety and the continuity of data acquisition and data conversion, the data cache module is configured and the storage module is also configured. Once the network interruption occurs and the data transmission connection is interrupted, the collected data can be stored for a certain time, data caching is carried out, interface program data can form cache files with a certain size and quantity, when the connection is recovered, the cache files can be continuously transmitted out through the connection, the data are ensured not to be lost, and the safety of the data can be effectively improved by adopting a full-file cache mode.
After data analysis, the data conversion module is used for realizing standardized conversion of data, and the data conversion module can perform an intelligent coding function and an intelligent conversion function of remote signaling remote measurement values on the acquired data measurement points, and the intelligent coding function and the intelligent conversion function of the remote signaling remote measurement values are respectively described below.
(1) The intelligent data coding function is used for coding according to the type of data source equipment, the source of a data station and the type of data, and can also be used for editing coding rules.
The intelligent data coding refers to coding of data measuring points, and specifically comprises two parts, wherein the codes count 20 bits, and the codes are coded in a mode of combining Arabic numerals and English letters. As shown in fig. 5, the encoded data includes: the information of the region, the transformer substation and the station includes the relevant information of the unit, and the coding of the relevant information of the unit is described in detail below.
1) Data source coding requirements
The data source number of the data occupies 1 bit of code, and a corresponding code value is selected from the following table 1 according to the data access source system, wherein table 1 shows an example of the data source number value.
| Encoding a value | Description of the |
| 1 | Monitoring system of power generation equipment (SCADA) |
| 2 | AVC system |
| 3 | AGC system |
| 4 | Power prediction |
| 5 | SVG |
| 6 | PMU |
| 7 | Stand-alone information |
| 8 | In-station monitoring |
| 9 | Equipment body |
TABLE 1
2) Device type coding requirements
The device type of the data occupies 1 bit of code, and the corresponding code value is selected from the following table 2 according to the access data device.
| Value of | Description of the |
| 1 | Wind |
| 2 | Photovoltaic power generation equipment |
TABLE 2
3) Device numbering
The device coding of the data occupies 3-bit coding, the number is numbered from the Arabic numeral 001, the Arabic numeral coding is maximum to 999, then the coding is carried out by adopting a combination mode of letters and Arabic numerals, the encoding is carried out according to the sequence of capitals from the lowest position, and if the capitals of the lowest position are used up, the encoding is carried out by one bit higher than 1, and the cycle is repeated. The high order is carried to the letter, and the low order is recycled from 012 · 9A B · Y Z. For example, 999, 00A, 00B … 00Z, 01A …, the device code should be consistent with the number of the device ledger provided.
4) Type of signal
The signal type of the data, which occupies 1 bit of code, selects the corresponding code value from table 3 below according to the type of the access data signal.
TABLE 3
5) Part numbering
The part number of the data occupies 1 bit number, corresponding coding values are selected from the following tables 4 and 5 according to the part accessed by the accessed data measuring point, and the wind turbine generator part (see table 4) and the photovoltaic power generation equipment part (see table 5) are respectively coded. Components such as data access are not shown in the table below, and the spreading and configuration of the code values is performed empirically.
TABLE 4
| Value of | Description of the invention |
| 0 | Non-component information (status information, etc.) |
| 1 | Inverter with a |
| 2 | Collection flow box |
| 3 | Photovoltaic string |
| 4 | Weather station |
TABLE 5
6) Type of measuring point
The measuring point type number of the data occupies 1 bit for coding, and a corresponding coding value is selected from the following table 6 according to the type of the accessed data measuring point.
| Value of | Description of the |
| 1 | Classes of |
| 2 | Data class |
| 3 | Status class |
TABLE 6
7) Point code
And measuring point coding of the data occupies 4-bit coding, numbering is carried out from the Arabic numerals 0001, the Arabic numerals code is maximum to 9999, then coding is carried out in a mode of combining letters and the Arabic numerals, the encoding is carried out according to the sequence of capitals from the lowest position, and if the capitals of the lowest position are used up, the encoding is carried out by 1 bit higher, and the cycle is carried out. The high order is carried to the letter, and the low order is recycled from 012 · 9A B · Y Z. The measuring points with different equipment numbers are coded respectively.
(2) The intelligent conversion function of the remote signaling remote measurement value can realize the standardized conversion of the remote signaling quantity and the fault information, and can edit according to the format of the access data.
The following description takes the formulation rule of the fault class measurement point value as an example.
1) The fault measuring points are classified and numbered according to the components, the fault measuring points of each component are coded from 0001, the number of the fault measuring points of each component is not limited, and a unit fault coding table can be formulated according to the fault measuring points
2) The coding rule can contain all faults of the unit, and shutdown faults and alarms are distinguished;
3) the value of each fault measurement point is a maximum 32-bit binary number, which represents the corresponding 32 fault status information, 1 indicates that a fault occurs, and 0 indicates that no fault occurs.
The calculation rule of the fault point measurement value is as follows:
a. according to the sequence of a unit fault coding table, every 32 fault messages of each type of component form a measuring point value, the measuring point value is less than 32 bits, and the last measuring point value of the component is formed according to the residual quantity.
b. The binary digits are represented from low to high, and are compiled from component 0001 according to a unit fault code table.
c. Each fault word starts from a fault code of 0001 of each component according to a unit fault code table, and the 32-bit binary digit is formed by fault states of every 32 fault codes from low order to high order.
d. When the measured point value is uploaded, the binary digits are converted into decimal digits.
Fig. 6 is a diagram illustrating an example of fault codes of a fault class measurement point of an electronic control system, and as shown in fig. 6, a fault word 1 of the electronic control system indicates that a component is first 32-bit fault information of the electronic control system, and a fault word 2 (not shown) of the electronic control system indicates that a component is 33 th-64 th-bit fault information of the electronic control system.
As described above, the embodiment of the present invention provides a data standardized data interface and a coding method applied to new energy, which solve the problems of nonstandard data format and low application efficiency in the new energy data center construction process, and achieve standardization of data access and improve data application efficiency by standardized acquisition of data.
Based on similar inventive concepts, the embodiment of the present invention further provides a multi-source heterogeneous data processing method, which is preferably used for implementing the functions of the multi-source heterogeneous data processing apparatus.
Fig. 7 is a flowchart of the multi-source heterogeneous data processing method, as shown in fig. 7, the method includes:
In actual operation, heterogeneous data from different data sources may be received based on a unidirectional, bidirectional handoff pattern.
And 702, analyzing the heterogeneous data according to a preset analysis rule.
Specifically, based on a preset encoding rule, data encoding conversion processing may be performed on the analyzed data according to the data source device and the data type; and carrying out telesignaling telemetering value conversion processing on the analyzed data according to the accessed telesignaling telemetering value data format.
In one embodiment, the heterogeneous data includes: data of different fault class measurement points. The analyzed data can be subjected to remote signaling and remote measuring value conversion processing according to remote signaling and remote measuring value data formats from different fault measuring points based on a preset fault coding rule.
The received heterogeneous data is analyzed according to the preset analysis rule, then the analyzed data is subjected to standardized conversion processing based on the preset standardized rule, and the standardized data is sent to the data platform so as to be used subsequently, therefore, the acquisition of multi-source heterogeneous data and the conversion and access of standardized data can be realized, and the efficiency of data application can be improved.
For the specific execution process of the above steps, reference may be made to the description in the above system embodiment, and details are not described here.
The present embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, and the like, but is not limited thereto. In this embodiment, the electronic device may be implemented by referring to the above method embodiment and the embodiment of the multi-source heterogeneous data processing apparatus/system, and the contents thereof are incorporated herein, and repeated descriptions are omitted.
Fig. 8 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention. As shown in fig. 8, the electronic device 600 may include a central processor 100 and a memory 140; the memory 140 is coupled to the central processor 100. Notably, this diagram is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the multi-source heterogeneous data processing functionality may be integrated into central processor 100. The central processor 100 may be configured to control as follows:
receiving heterogeneous data from different data sources, the heterogeneous data comprising: data of different source devices and different data types;
analyzing the heterogeneous data according to a preset analysis rule;
performing a normalization conversion process on the parsed data based on a predetermined normalization rule, the normalization conversion process including: data code conversion processing and remote signaling remote measurement value conversion processing;
and sending the data after the standardization processing to a data platform so as to be convenient for subsequent use.
As can be seen from the above description, according to the electronic device provided in the embodiment of the present application, the received heterogeneous data is analyzed according to the predetermined analysis rule, then the analyzed data is subjected to the standardized conversion processing based on the predetermined standardization rule, and the standardized data is sent to the data platform, so as to be used subsequently, thereby achieving the acquisition of the multi-source heterogeneous data and the conversion and access of the standardized data, and improving the efficiency of data application.
In another embodiment, the multi-source heterogeneous data processing apparatus/system may be configured separately from the central processing unit 100, for example, the multi-source heterogeneous data processing apparatus/system may be configured as a chip connected to the central processing unit 100, and the multi-source heterogeneous data processing function is realized by the control of the central processing unit.
As shown in fig. 8, the electronic device 600 may further include: communication module 110, input unit 120, audio processing unit 130, display 160, power supply 170. It is noted that the electronic device 600 does not necessarily include all of the components shown in FIG. 8; furthermore, the electronic device 600 may also comprise components not shown in fig. 8, which may be referred to in the prior art.
As shown in fig. 8, the central processor 100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, the central processor 100 receiving input and controlling the operation of the various components of the electronic device 600.
The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 100 may execute the program stored in the memory 140 to realize information storage or processing, etc.
The input unit 120 provides input to the cpu 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display an object to be displayed, such as an image or a character. The display may be, for example, an LCD display, but is not limited thereto.
The memory 140 may be a solid state memory such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 140 may also be some other type of device. Memory 140 includes buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage section 142, and the application/function storage section 142 is used to store application programs and function programs or a flow for executing the operation of the electronic device 600 by the central processing unit 100.
The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging application, address book application, etc.).
The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132 to implement general telecommunications functions. Audio processor 130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 130 is also coupled to the central processor 100, so that recording on the local can be enabled through a microphone 132, and so that sound stored on the local can be played through a speaker 131.
Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the above-mentioned multi-source heterogeneous data processing method.
To sum up, the embodiment of the present invention provides a standardized acquisition interface device and a data standardization method for acquiring multi-source heterogeneous data in a new energy scene, and realizes standardized access of data from different manufacturers, different stations, different devices, and different sources through a standardized data interface, thereby improving the efficiency of data application.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings. The many features and advantages of the embodiments are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the embodiments which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the embodiments of the invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope thereof.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (11)
1. A multi-source heterogeneous data processing method, characterized in that the method comprises:
receiving heterogeneous data from different data sources, the heterogeneous data comprising: data of different source devices and different data types;
analyzing the heterogeneous data according to a preset analysis rule;
performing a normalization conversion process on the parsed data based on a predetermined normalization rule, the normalization conversion process including: data code conversion processing and remote signaling remote measurement value conversion processing;
and sending the data after the standardization processing to a data platform so as to be convenient for subsequent use.
2. The method of claim 1, wherein receiving heterogeneous data from different data sources comprises:
heterogeneous data from different data sources is received based on a unidirectional, bidirectional switching pattern.
3. The method of claim 1, wherein performing a normalization transformation process on the parsed data based on a predetermined normalization rule comprises:
performing data coding conversion processing on the analyzed data according to data source equipment and data types based on a preset coding rule;
and carrying out telesignaling telemetering value conversion processing on the analyzed data according to the accessed telesignaling telemetering value data format.
4. The method of claim 3, wherein the heterogeneous data comprises: the telecommand telemetering value conversion processing of the analyzed data according to the accessed telecommand telemetering value data format of the data of different fault measuring points comprises the following steps:
and carrying out telesignalling telemetering value conversion processing on the analyzed data according to the telesignalling telemetering value data formats from different fault measuring points based on a preset fault coding rule.
5. A multi-source heterogeneous data processing apparatus, the apparatus comprising:
a data receiving unit, configured to receive heterogeneous data from different data sources, where the heterogeneous data includes: data of different source devices and different data types;
the analysis unit is used for analyzing the heterogeneous data according to a preset analysis rule;
a normalization conversion unit configured to perform normalization conversion processing on the parsed data based on a predetermined normalization rule, the normalization conversion processing including: data code conversion processing and remote signaling remote measurement value conversion processing;
and the data sending unit is used for sending the data after the standardization processing to a data platform so as to be convenient for subsequent use.
6. The apparatus of claim 5, wherein the data receiving unit is specifically configured to:
heterogeneous data from different data sources is received based on a unidirectional, bidirectional switching pattern.
7. The apparatus of claim 5, wherein the normalization conversion unit comprises:
the data code conversion module is used for carrying out data code conversion processing on the analyzed data according to the data source equipment and the data type based on a preset coding rule;
and the remote signaling and remote measuring value conversion module is used for carrying out remote signaling and remote measuring value conversion processing on the analyzed data according to the accessed remote signaling and remote measuring value data format.
8. The apparatus of claim 7, wherein the heterogeneous data comprises: the remote signaling and remote measuring value conversion module is specifically used for the data of different fault measuring points:
and carrying out telesignalling telemetering value conversion processing on the analyzed data according to the telesignalling telemetering value data formats from different fault measuring points based on a preset fault coding rule.
9. A multi-source heterogeneous data processing system, the system comprising: the multi-source heterogeneous data processing apparatus of any of claims 5 to 8, and a data platform to receive and store the standardized processed data from the multi-source heterogeneous data processing apparatus for subsequent use.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 4 are implemented when the processor executes the program.
11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111393333.1A CN114138861A (en) | 2021-11-23 | 2021-11-23 | Multi-source heterogeneous data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111393333.1A CN114138861A (en) | 2021-11-23 | 2021-11-23 | Multi-source heterogeneous data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114138861A true CN114138861A (en) | 2022-03-04 |
Family
ID=80391072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111393333.1A Pending CN114138861A (en) | 2021-11-23 | 2021-11-23 | Multi-source heterogeneous data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114138861A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383742A (en) * | 2023-06-05 | 2023-07-04 | 深圳普菲特信息科技股份有限公司 | Rule chain setting processing method, system and medium based on feature classification |
| CN116451994A (en) * | 2023-03-23 | 2023-07-18 | 国网浙江省电力有限公司 | Site operation safety control method based on energy Internet service system |
| CN116579300A (en) * | 2023-07-14 | 2023-08-11 | 安徽华云安科技有限公司 | Automatic conversion method and device for multi-source heterogeneous data |
| CN117171534A (en) * | 2023-11-03 | 2023-12-05 | 济南二机床集团有限公司 | Multi-source heterogeneous data acquisition method, system, device and medium for numerical control machine tool |
-
2021
- 2021-11-23 CN CN202111393333.1A patent/CN114138861A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116451994A (en) * | 2023-03-23 | 2023-07-18 | 国网浙江省电力有限公司 | Site operation safety control method based on energy Internet service system |
| CN116451994B (en) * | 2023-03-23 | 2024-01-05 | 国网浙江省电力有限公司 | Site operation safety control method based on energy Internet service system |
| CN116383742A (en) * | 2023-06-05 | 2023-07-04 | 深圳普菲特信息科技股份有限公司 | Rule chain setting processing method, system and medium based on feature classification |
| CN116383742B (en) * | 2023-06-05 | 2023-08-11 | 深圳普菲特信息科技股份有限公司 | Rule chain setting processing method, system and medium based on feature classification |
| CN116579300A (en) * | 2023-07-14 | 2023-08-11 | 安徽华云安科技有限公司 | Automatic conversion method and device for multi-source heterogeneous data |
| CN117171534A (en) * | 2023-11-03 | 2023-12-05 | 济南二机床集团有限公司 | Multi-source heterogeneous data acquisition method, system, device and medium for numerical control machine tool |
| CN117171534B (en) * | 2023-11-03 | 2024-03-19 | 济南二机床集团有限公司 | Multi-source heterogeneous data acquisition method, system, device and medium for numerical control machine tool |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114138861A (en) | Multi-source heterogeneous data processing method, device and system | |
| CN111585344B (en) | Substation intelligent checking method and device based on total station IED simulation | |
| CN103905834B (en) | The method and device of audio data coding form conversion | |
| CN102118293A (en) | Method for compressing and storing communication messages | |
| CN110769002A (en) | LabVIEW-based message analysis method, system, electronic device and medium | |
| CN110728834A (en) | WAMS measurement data compression transmission method based on Beidou short message | |
| CN102447478B (en) | Encoding method and system for accident rule digital logic design of nuclear power plant | |
| CN105404472A (en) | Method and apparatus for compressing storage space of log time data | |
| CN112884120A (en) | Graph neural network representation architecture | |
| CN106056227B (en) | Intelligent substation service tracking method based on IEC61850 standard | |
| CN113010473B (en) | Method and equipment for editing YAML file | |
| CN103036877A (en) | Device and method for code generation of coding and decoding based on threshold limit value (TLV) form protocol | |
| CN103684812A (en) | Remote equipment management method and device | |
| CN116244202A (en) | Automatic performance test method and device | |
| CN114726380B (en) | Monitoring data lossless compression method, device, equipment and readable storage medium | |
| CN102131161B (en) | Method, device and system for encoding short message | |
| JP4821287B2 (en) | Structured document encoding method, encoding apparatus, encoding program, decoding apparatus, and encoded structured document data structure | |
| CN112015726B (en) | User activity prediction method, system and readable storage medium | |
| CN114221312B (en) | Power distribution network protection rapid setting and checking method based on topological relation | |
| CN114448775B (en) | Equipment fault information processing method and device, electronic equipment and storage medium | |
| CN113077063B (en) | Defect management method and device for power transformation equipment based on voice and image recognition | |
| CN112905464B (en) | Application running environment data processing method and device | |
| CN112232026A (en) | Meteorological field data conversion method suitable for HYSPLIT atmospheric diffusion model | |
| CN105678637A (en) | Data parsing system and application method thereof for vehicle charging posts | |
| CN102497555B (en) | High definition encoder Chinese support method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |