CN114138856A - Efficient data acquisition and data analysis processing system - Google Patents

Efficient data acquisition and data analysis processing system Download PDF

Info

Publication number
CN114138856A
CN114138856A CN202111207167.1A CN202111207167A CN114138856A CN 114138856 A CN114138856 A CN 114138856A CN 202111207167 A CN202111207167 A CN 202111207167A CN 114138856 A CN114138856 A CN 114138856A
Authority
CN
China
Prior art keywords
data
module
acquisition
engine
data acquisition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202111207167.1A
Other languages
Chinese (zh)
Inventor
王正勤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Industry Technical College
Original Assignee
Guangdong Industry Technical College
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Industry Technical College filed Critical Guangdong Industry Technical College
Priority to CN202111207167.1A priority Critical patent/CN114138856A/en
Publication of CN114138856A publication Critical patent/CN114138856A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2465Query processing support for facilitating data mining operations in structured databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/26Visual data mining; Browsing structured data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • G06F16/285Clustering or classification
    • G06F16/287Visualization; Browsing

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Arrangements For Transmission Of Measured Signals (AREA)

Abstract

The invention belongs to the technical field of data processing, in particular to a high-efficiency data acquisition and data analysis processing system, which provides a scheme that the data acquisition system comprises an equipment data acquisition end, a local server and mobile communication equipment, the equipment data acquisition end comprises a data acquisition module and a data detection module, the data acquisition module comprises a streaming acquisition engine, a batch acquisition engine, a CDC acquisition engine and a network crawler acquisition engine, and the high-efficiency data analysis processing system comprises a data acquisition module, a data type acquisition module, a data set construction module, a storage relationship construction module, a structured data platform, a reliability judgment module and a machine identification judgment module. The invention collects the network data of each device, and can respectively collect various types of power grid data from each device by using various data collection engines with different collection modes, so as to meet the requirement of an external analysis system on comprehensive analysis of the data.

Description

Efficient data acquisition and data analysis processing system
Technical Field
The invention relates to the technical field of data processing, in particular to a system with high-efficiency data acquisition and data analysis and processing functions.
Background
Data (Data) is a representation of facts, concepts or instructions that can be processed by either manual or automated means. After the data is interpreted and given a certain meaning, it becomes information. Data processing (data processing) is the collection, storage, retrieval, processing, transformation, and transmission of data.
The basic purpose of data processing is to extract and derive valuable, meaningful data for certain people from large, possibly chaotic, unintelligible amounts of data. Data processing is the basic link of system engineering and automatic control. Data processing is throughout various fields of social production and social life. The development of data processing technology and the breadth and depth of its application have greatly influenced the progress of human society development.
The current data acquisition has the following defects: the type of the power grid data which can be collected by the existing data collection module is single, and the requirement of an external analysis system for comprehensively analyzing the power grid data is difficult to meet.
Disclosure of Invention
Based on the technical problems that the type of power grid data which can be acquired by the conventional data acquisition module provided in the background art is single and the requirement of an external analysis system for comprehensively analyzing the power grid data is difficult to meet, the invention provides the high-efficiency data acquisition and data analysis processing system.
The invention provides a high-efficiency data acquisition system which comprises an equipment data acquisition end, a local server and mobile communication equipment, wherein the equipment data acquisition end comprises a data acquisition module and a data detection module, the data detection module is used for detecting data, the data acquisition module comprises a streaming acquisition engine, a batch acquisition engine, a CDC acquisition engine and a web crawler acquisition engine, for the network data of each equipment, the data acquisition module acquires real-time data by using the streaming acquisition engine, acquires offline data by using the batch acquisition engine, acquires real-time change data by using the CDC acquisition engine, and acquires the network data by using the web crawler acquisition engine;
the device data acquisition end is used for being electrically connected with a device, the device data acquisition end is used for acquiring on-off state data and early warning alarm state data of the device, the device data acquisition end is used for establishing communication with the local server, the local server is used for establishing communication with the control terminal, network data of each device are acquired, and various data acquisition engines with different acquisition modes can be used for acquiring various types of power grid data from each device respectively so as to meet the requirement of an external analysis system for comprehensively analyzing the data.
Preferably, the local server comprises a data transceiver module and a data processing module, and the data processing module comprises an analysis unit and a generation unit, wherein the analysis unit is used for identifying the type of each equipment data acquisition terminal establishing communication with the local server; the generating unit is used for generating an identification ID for each analyzed equipment data acquisition end.
Preferably, the streaming acquisition engine comprises a distributed real-time big data processing framework for real-time computation, and the distributed real-time big data processing framework is a Storm streaming framework.
Preferably, the device data acquisition end further comprises a first processing unit and a first wireless communication unit; the first processing unit is connected with at least one sensor; the first processing unit is used for receiving the collected data collected by the sensor and wirelessly transmitting the collected data to the edge computing gateway through the first wireless communication unit.
The invention provides a high-efficiency data analysis processing system which comprises a data acquisition module, a data type acquisition module, a data set construction module, a storage relation construction module, a structured data platform, a reliability judgment module and a machine identification judgment module, wherein the data acquisition module is used for acquiring example data corresponding to a target database;
the data type acquisition module is used for determining the data type stored in the target database according to the example data;
the data set construction module is used for constructing data sets, each data set corresponds to each target database one by one, the data sets comprise data stored in the corresponding target database, and the data sets take data types stored in the corresponding target databases as names of the data sets.
Preferably, the storage relationship building module is configured to build a storage relationship, where the storage relationship includes a corresponding relationship between a data type and a storage address corresponding to each data set, and the data analysis processing system further includes a second storage location changing module, where the second storage location changing module is configured to compare the visualization times of each data set in a preset time period with a preset low-time threshold, obtain a data set whose visualization times are less than or equal to the preset low-time threshold, obtain a second data set in the data set whose visualization times are less than or equal to the preset low-time threshold, transfer and store the obtained second data set from a local place to the cloud, and update the storage relationship.
Preferably, the structured data platform comprises a classification module and a storage module; the classification module performs data extraction, data summarization, data sorting, data processing, data analysis and data classification on data in the relational database resources according to types to generate data used by a computational analysis engine; the unstructured data platform is an unstructured data management platform and comprises a data acquisition unit, a data storage unit and a data processing unit which are connected in sequence; the files are classified through the unstructured data platform, labels of classification, attribute features, keywords and the like are marked, and an index based on the classification, the attribute features and the keyword labels is constructed.
Preferably, the reliability judging module judges the accuracy of the identification result of the radio signal, and if the judgment result is correct, the judgment result output module outputs the correct identification result of the radio signal, and if the judgment result is wrong, the prompting module gives a prompt.
Preferably, the machine identification judgment module judges the identification result of the intelligent learning system, and if the judgment result is correct, the result reference module outputs the identification result, and if the judgment result is wrong, the error reporting and recording module records the wrong result.
The beneficial effects of the invention are as follows:
1. this kind has high-efficient data acquisition and data analysis processing system, through being provided with the data acquisition module, the data acquisition module includes STREAMING collection engine, gathers the engine in batches, CDC collection engine and web crawler collection engine, gathers the network data of every equipment, gathers polytype electric wire netting data from every equipment respectively with regard to the different multiple data acquisition engines of usable collection mode to satisfy the demand that external analysis system carries out comprehensive analysis to data.
2. This a have high-efficient data acquisition and data analysis processing system, the state information of data acquisition module collection equipment, and send local server department, make local server handle data and send control terminal department, thereby make the user can look over the real-time status of equipment through control terminal, and send control command to local server through control terminal, local server receives feedback data acquisition module behind control terminal's the control command, data acquisition module handles and carries out after receiving the control command that local server department sent, thereby can control equipment through data acquisition module, set up like this, can control a plurality of equipment simultaneously through a control terminal, thereby be favorable to promoting data acquisition and the managerial efficiency to a plurality of equipment.
3. According to the high-efficiency data acquisition and data analysis processing system, the storage relation construction module, the structured data platform, the credibility judgment module and the machine identification judgment module are arranged, the first data sets are transferred from the cloud end to be stored locally, the requirement can be met, when the data sets need to be acquired for visualization in the follow-up process, the data sets can be acquired from the local quickly and safely, the acquisition speed and efficiency are guaranteed, the possibility of acquisition failure is reduced, and the data safety is improved.
The parts of the device not involved are the same as or can be implemented using prior art.
Drawings
FIG. 1 is a schematic diagram of a high-efficiency data acquisition system according to the present invention;
fig. 2 is a schematic structural diagram of a system with efficient data analysis and processing according to the present invention.
Detailed Description
The technical solution of the present patent will be described in further detail with reference to the following embodiments.
Reference will now be made in detail to embodiments of the present patent, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present patent and are not to be construed as limiting the present patent.
In the description of this patent, it is to be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in the orientations and positional relationships indicated in the drawings for the convenience of describing the patent and for the simplicity of description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are not to be considered limiting of the patent.
In the description of this patent, it is noted that unless otherwise specifically stated or limited, the terms "mounted," "connected," and "disposed" are to be construed broadly and can include, for example, fixedly connected, disposed, detachably connected, disposed, or integrally connected and disposed. The specific meaning of the above terms in this patent may be understood by those of ordinary skill in the art as appropriate.
Referring to fig. 1, a system with high efficiency for data acquisition, comprising an equipment data acquisition end, a local server and a mobile communication device, wherein the equipment data acquisition end comprises a data acquisition module and a data detection module, the data detection module is used for detecting data, the data acquisition module comprises a streaming acquisition engine, a batch acquisition engine, a CDC acquisition engine and a webcrawler acquisition engine, for network data of each equipment, the data acquisition module acquires real-time data by using the streaming acquisition engine, acquires offline data by using the batch acquisition engine, acquires real-time change data by using the CDC acquisition engine, and acquires network data by using the webcrawler acquisition engine;
the device data acquisition end is used for being electrically connected with the device, acquiring the on-off state data and the early warning state data of the device, establishing communication with the local server, and establishing communication with the control terminal;
through being provided with the data acquisition module, the data acquisition module includes STREAMING collection engine, gathers engine, CDC collection engine and web crawler collection engine in batches, gathers the network data of every equipment, gathers polytype electric wire netting data from every equipment respectively with regard to the different multiple data acquisition engines of usable collection mode to satisfy the demand that external analysis system carries out comprehensive analysis to data.
Referring to fig. 1, in the present invention, a local server includes a data transceiver module and a data processing module, and the data processing module includes an analysis unit and a generation unit, the analysis unit is used for identifying the device type of each device data acquisition end that establishes communication with the local server; the generating unit is used for generating an identification ID for each analyzed equipment data acquisition end, the stream type acquisition engine comprises a distributed real-time big data processing framework for real-time calculation, and the distributed real-time big data processing framework is a Storm stream type processing framework;
the local server receives the control instruction of the control terminal and feeds back the control instruction to the data acquisition module, the data acquisition module receives the control instruction sent by the local server and then processes and executes the control instruction, so that the equipment can be controlled through the data acquisition module, the control terminal can control a plurality of pieces of equipment simultaneously through the control terminal, and the data acquisition and management efficiency of the plurality of pieces of equipment is improved.
Referring to fig. 1, in the present invention, the device data acquisition end further includes a first processing unit and a first wireless communication unit; the first processing unit is connected with at least one sensor; the first processing unit is used for receiving the collected data collected by the sensor and wirelessly transmitting the collected data to the edge computing gateway through the first wireless communication unit.
Referring to fig. 2, the system for analyzing and processing data with high efficiency includes a data acquisition module, a data type acquisition module, a data set construction module, a storage relationship construction module, a structured data platform, a reliability judgment module and a machine identification judgment module, where the data acquisition module is used to acquire example data corresponding to a target database;
the data type acquisition module is used for determining the data type stored in the target database according to the example data;
and the data set construction module is used for constructing data sets, each data set corresponds to each target database one by one, each data set comprises data stored in the corresponding target database, and the data types stored in the corresponding target databases are used as the names of the data sets in the data sets.
Referring to fig. 2, in the present invention, the storage relationship building module is configured to build a storage relationship, where the storage relationship includes a corresponding relationship between a data type and a storage address corresponding to each data set, and the data analysis processing system further includes a second storage location changing module, where the second storage location changing module is configured to compare visualization times of each data set in a preset time period with a preset low-time threshold, obtain a data set whose visualization times are less than or equal to the preset low-time threshold, obtain a second data set in the data set whose visualization times are less than or equal to the preset low-time threshold, transfer and store the obtained second data set from a local place to a cloud, and update the storage relationship;
the first data sets are transferred from the cloud to be stored locally, so that the requirement can be met, when the data sets need to be acquired again for visualization subsequently, the data sets can be acquired from the local quickly and safely, the acquisition speed and efficiency are guaranteed, the possibility of acquisition failure is reduced, and the data safety is improved.
Referring to fig. 2, in the present invention, the structured data platform includes a classification module and a storage module; the classification module performs data extraction, data summarization, data sorting, data processing, data analysis and data classification on data in the relational database resources according to types to generate data used by a computational analysis engine; the unstructured data platform is an unstructured data management platform and comprises a data acquisition unit, a data storage unit and a data processing unit which are connected in sequence; the files are classified through the unstructured data platform, labels of classification, attribute features, keywords and the like are marked, and an index based on the classification, the attribute features and the keyword labels is constructed.
Referring to fig. 2, in the present invention, the reliability judging module judges the accuracy of the recognition result of the radio signal, the judging result output module outputs the correct recognition result of the radio signal if the reliability judging module is correct, the prompt module gives a prompt if the reliability judging module is wrong, the machine recognition judging module judges the recognition result of the intelligent learning system, the result reference module outputs the recognition result if the reliability judging module is correct, and the error reporting recording module records the error result if the reliability judging module is wrong.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.

Claims (9)

1. A high-efficiency data acquisition system comprises an equipment data acquisition end, a local server and mobile communication equipment, and is characterized in that the equipment data acquisition end comprises a data acquisition module and a data detection module, the data detection module is used for detecting data, the data acquisition module comprises a streaming acquisition engine, a batch acquisition engine, a CDC acquisition engine and a web crawler acquisition engine, network data of each equipment is acquired, the data acquisition module acquires real-time data by using the streaming acquisition engine, acquires offline data by using the batch acquisition engine, acquires real-time change data by using the CDC acquisition engine, and acquires the network data by using the web crawler acquisition engine;
the device data acquisition end is used for being electrically connected with a device, the device data acquisition end is used for acquiring on-off state data and early warning alarm state data of the device, the device data acquisition end is used for establishing communication with the local server, and the local server is used for establishing communication with the control terminal.
2. The system of claim 1, wherein the local server comprises a data transceiver module and a data processing module, and the data processing module comprises an analysis unit and a generation unit, and the analysis unit is configured to perform device type identification on each device data acquisition end that establishes communication with the local server; the generating unit is used for generating an identification ID for each analyzed equipment data acquisition end.
3. The system of claim 1, wherein the streaming engine comprises a distributed real-time big data processing framework for real-time computation, and the distributed real-time big data processing framework is a Storm streaming framework.
4. The system for efficient data collection and data analysis and processing according to claim 1, wherein the device data collection end further comprises a first processing unit and a first wireless communication unit; the first processing unit is connected with at least one sensor; the first processing unit is used for receiving the collected data collected by the sensor and wirelessly transmitting the collected data to the edge computing gateway through the first wireless communication unit.
5. A high-efficiency data analysis processing system comprises a data acquisition module, a data type acquisition module, a data set construction module, a storage relation construction module, a structured data platform, a reliability judgment module and a machine identification judgment module, and is characterized in that the data acquisition module is used for acquiring example data corresponding to a target database;
the data type acquisition module is used for determining the data type stored in the target database according to the example data;
the data set construction module is used for constructing data sets, each data set corresponds to each target database one by one, the data sets comprise data stored in the corresponding target database, and the data sets take data types stored in the corresponding target databases as names of the data sets.
6. The system according to claim 5, wherein the storage relationship building module is configured to build a storage relationship, the storage relationship includes a correspondence between a data type and a storage address corresponding to each data set, the data analysis processing system further includes a second storage location changing module, the second storage location changing module is configured to compare the visualization times of each data set in a preset time period with a preset low-time threshold, obtain a data set whose visualization times is less than or equal to the preset low-time threshold, obtain a second data set of the data sets whose visualization times is less than or equal to the preset low-time threshold, transfer and store the obtained second data set from the local place to the cloud, and update the storage relationship.
7. The system of claim 6, wherein the structured data platform comprises a classification module, a storage module; the classification module performs data extraction, data summarization, data sorting, data processing, data analysis and data classification on data in the relational database resources according to types to generate data used by a computational analysis engine; the unstructured data platform is an unstructured data management platform and comprises a data acquisition unit, a data storage unit and a data processing unit which are connected in sequence; the files are classified through the unstructured data platform, labels of classification, attribute features, keywords and the like are marked, and an index based on the classification, the attribute features and the keyword labels is constructed.
8. The system as claimed in claim 7, wherein the reliability determining module determines accuracy of the identification result of the radio signal, the determination result output module outputs the correct identification result of the radio signal if the accuracy is correct, and the prompt module issues a prompt if the accuracy is wrong.
9. The system of claim 8, wherein the machine recognition and determination module determines the recognition result of the intelligent learning system, and if the recognition result is correct, the result reference module outputs the recognition result, and if the recognition result is wrong, the error recording module records the wrong result.
CN202111207167.1A 2021-10-15 2021-10-15 Efficient data acquisition and data analysis processing system Withdrawn CN114138856A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111207167.1A CN114138856A (en) 2021-10-15 2021-10-15 Efficient data acquisition and data analysis processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111207167.1A CN114138856A (en) 2021-10-15 2021-10-15 Efficient data acquisition and data analysis processing system

Publications (1)

Publication Number Publication Date
CN114138856A true CN114138856A (en) 2022-03-04

Family

ID=80395056

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111207167.1A Withdrawn CN114138856A (en) 2021-10-15 2021-10-15 Efficient data acquisition and data analysis processing system

Country Status (1)

Country Link
CN (1) CN114138856A (en)

Similar Documents

Publication Publication Date Title
CN103700031A (en) Electric power warning information publishing method in regulation and control integration mode
CN102819239A (en) Intelligent fault diagnosis method of numerical control machine tool
CN110807460B (en) Transformer substation intelligent patrol system based on image recognition and application method thereof
CN113419492A (en) NB-IoT industrial pollutant monitoring system based on multi-source heterogeneous sensor data fusion
CN110930262A (en) Emergency electric power emergency disposal field global monitoring information interaction support preposition service system and method
CN107918629A (en) The correlating method and device of a kind of alarm failure
CN109245305A (en) A kind of photovoltaic plant automatic early-warning cloud platform and system
CN114689965A (en) Power equipment online monitoring method and system based on embedded intelligent sensor
CN113516565A (en) Intelligent alarm processing method and device for power monitoring system based on knowledge base
CN113157521A (en) Monitoring method and monitoring system for whole life cycle of block chain
CN112362102A (en) Monitoring system for immediately monitoring agricultural production environment
CN112907929A (en) Environment-friendly monitoring system and method based on electricity utilization information
CN114138856A (en) Efficient data acquisition and data analysis processing system
CN107133669A (en) Track traffic synthetic monitoring intelligent short message alarm method
CN115221221B (en) Frequency converter operating efficiency intelligent supervision system based on data analysis
CN116596725A (en) System and method for managing one-network of non-waste city based on block chain technology
CN115905348A (en) Industrial electricity utilization abnormity early warning method and system of multi-source heterogeneous data
US11954955B2 (en) Method and system for collecting and monitoring vehicle status information
CN114584585A (en) Industrial equipment self-diagnosis system and method based on Internet of things
CN113129160A (en) Electric power communication network inspection method based on equipment state perception and intellectualization
CN111831680A (en) Electric power data processing system based on big data platform
CN115858505B (en) Data processing method for identifying state abnormality of frequency converter
CN211718887U (en) Computer information evidence obtaining and analyzing system
CN114546519B (en) Industrial control safety data acquisition system and method
CN210375190U (en) Novel mechanical state monitoring device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20220304