CN113032379B - Distribution network operation and inspection-oriented multi-source data acquisition method - Google Patents
Distribution network operation and inspection-oriented multi-source data acquisition method Download PDFInfo
- Publication number
- CN113032379B CN113032379B CN202110282493.2A CN202110282493A CN113032379B CN 113032379 B CN113032379 B CN 113032379B CN 202110282493 A CN202110282493 A CN 202110282493A CN 113032379 B CN113032379 B CN 113032379B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- source
- interface
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distribution network operation and inspection oriented multi-source data acquisition method, which adopts a micro-service framework to provide data information required to be synchronized through a target system and comprises the following steps: s1, the data source system actively or passively pushes data information; s2, after the target system receives the request, the following steps are carried out: s21, for different data receiving modes, providing different analysis data models respectively for data analysis; s22, caching data; s23, caching by adopting an asynchronous multithreading processing queue; s24, establishing a mapping relation between the synchronous field and the database field, keeping the field attributes consistent, and mapping data; and S25, persisting the data in a database table to complete data synchronization. According to the invention, through establishment of the acquisition standard of multi-source data, the access flexibility of the multi-source data under multiple platforms is improved, the data quality problem is reduced, the system application capability is improved, the inspection system application capability is improved, the equipment study and judgment accuracy is ensured, and the reliability of the net rack is improved.
Description
Technical Field
The invention relates to the technical field of power distribution networks, in particular to a distribution network operation and inspection multi-source data acquisition method.
Background
The distribution network is arranged at the end of the whole power grid and is a window facing the society of power enterprises, the operation management of the distribution network is directly related to thousands of households, and the social responsibility and the influence are huge. With the continuous development of society, higher and higher requirements are put forward on lean management of distribution networks. Due to the comprehensive development of the construction of the intelligent power distribution room, a large amount of secondary monitoring equipment data and data (such as production business data, equipment ledger data, project material data and the like) of the original data support platform are subjected to correlation analysis.
Due to the influence of the stage and the technical property of each business system in constructing and implementing the data management system and other economic and human factors, a large amount of business data adopting different storage modes are accumulated, the adopted data management systems are quite different from simple file databases to complex network databases, and therefore heterogeneous data sources are formed. At present, the access is realized through technical modes such as HTTP, Webservice, files and the like. However, at present, multi-source heterogeneous data cannot be accessed quickly and accurately, and meanwhile, a multi-source data access system can generate a concurrency problem, so that the processing stability is influenced, and the data quality is influenced.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a distribution network operation and inspection oriented multi-source data acquisition method, which aims to solve the problems that the processing stability and the data quality are influenced due to the fact that the concurrency quantity is generated in the conventional multi-source data access mode, improve the multi-source data access flexibility under multiple platforms, reduce the data quality problem, improve the application capability of an inspection system, ensure the accuracy of system data analysis and improve the reliability of a net rack through the establishment of multi-source data acquisition specifications.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows.
The distribution network operation and inspection multi-source data acquisition method adopts a micro-service framework and provides data information required to be synchronized through a target system, and comprises the following steps:
s1, the data source system actively or passively pushes data information;
s2, after the target system receives the request, the following steps are carried out:
s21, for different data receiving modes, providing different analysis data models respectively for data analysis;
s22, caching data;
s23, adopting asynchronous multithread processing queue buffer;
s24, establishing a mapping relation between the synchronous field and the database field, keeping the field attributes consistent, and mapping data;
and S25, persisting the data in a database table to complete data synchronization.
According to the technical scheme, when the data source system actively pushes the data information, the data source system calls a web service interface of the target system through the transit system, and the data information is pushed in a character string mode.
Further optimizing the technical scheme, the data information access process specifically comprises the following steps:
s10, splitting the data access service according to the actual requirement, and deploying a plurality of data processing services;
s20, presetting an interface protocol, and handing over a corresponding interface key to a data pushing platform, wherein the data pushing platform receives service pushing data to an inlet of multi-source data, and the inlet receiving service provides a unified inlet service interface for acquiring a data source according to configuration;
s30, after the pushed data enters the entry receiving service, the data enters the database to be stored in a special cache table through the safety check of the interface secret key, and the cache ID is sent to the queue cache service;
s40, extracting the data to be processed from the database by the buffer ID identification in the multiple data processing service request queue service, and separating and serving the two functions of the received data and the processed data through the queue buffer service forwarding identification;
and S50, reading the cache from the database according to the identifier, verifying whether the numerical characters of the cache meet the interface specification through data inspection, and converting the numerical characters of the cache into database table objects through data structure model mapping.
According to the technical scheme, when the data source system pushes data information passively, the data source system provides a data query interface, and the target system issues a timing task to call the query interface regularly to acquire synchronous data.
Further optimizing the technical scheme, the data information access process specifically comprises the following steps:
an interface protocol is configured in advance, an interface is accessed at regular time to obtain data through system task scheduling, the obtained unstructured data are preprocessed and converted, the preprocessed and converted data are cached in a database subsequently, and identification information is sent to a queue service;
and the data processing service reads information from the cache according to the identification, and automatically acquires and stores the data through the steps of data model conversion, numerical value verification and database processing.
The technical scheme is further optimized, and the data text transmission mode is that a data source system provides data in a file mode; and the target system issues a timing task and reads the file from the text server at regular time.
Due to the adoption of the technical scheme, the technical progress of the invention is as follows.
The invention applies the main data management idea to manage multi-source heterogeneous data, thereby ensuring the system coordination, the reuse and the universality of service data and the correctness of the data. The acquisition standard based on the multi-source data is established, the multi-source data access flexibility under multiple platforms can be improved, the data quality problem is reduced, the system application capacity is improved, the inspection system application capacity is improved, and the equipment study and judgment accuracy is ensured. Through the collection standard of multisource data, improved data transmission ability, improved data quality, improved the reliability of rack.
The method can access multi-source heterogeneous data in real time, quickly and accurately; designing a multi-source data acquisition format, carrying out object analysis according to data sources, and formulating an access data template; a technical implementation mode is established, and a data transmission technical route is standardized; a data interaction mechanism is designed, effective verification and feedback are carried out on problem data, and the data problem can be found and solved.
When the synchronous data volume of the system is large, the interface response speed can be effectively improved through the data caching technology, the data is cached firstly, then the rest flow is processed by the asynchronous thread, the synchronous performance of the system is improved, and the running stability of the system is ensured.
The invention ensures the key of the synchronization accuracy through a data mapping technology, establishes the mapping relation between the synchronization field and the database field, and keeps the consistency of the field attributes. And finally, persisting the data in a database table to complete the synchronous process.
The invention constructs diversified data analysis models. The data transmission form between systems is different, and some forms are character string form or file form. Different data receiving modes are provided with different analytical data models respectively, the method can be widely applied to a multi-data source system, and the expandability principle of the interface is met.
The queue cache service of the present invention is responsible for managing the distribution of data access cache identities. Meanwhile, the cache ID identifications in the multiple data processing service request queue services extract the data to be processed from the database, and the received data and the processed data can be separated and respectively serviced through the queue cache service forwarding identifications. The entrance receiving service business is light, the data processing service supports expansion as required, the problem of concurrency generated by a multi-source data access system is solved, and the processing stability is ensured.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a flow chart illustrating the deployment of multiple data processing services according to actual requirements.
Detailed Description
The invention will be described in further detail below with reference to the figures and specific examples.
An intelligent logistics business code generation system based on cross-platform technology is disclosed, which is combined with fig. 1 to fig. 2, adopts a micro-service framework, and provides data information required to be synchronized through a target system, and comprises the following steps:
and S1, the data source system actively or passively pushes data information.
S2, after receiving the request, the target system starts to complete data synchronization after a series of processing such as data analysis, data caching, queue processing caching, data mapping, data persistence and the like, and the method specifically comprises the following steps:
and S21, providing different analysis data models for different data receiving modes, and analyzing the data.
And S22, data caching is carried out. The system has larger synchronous data volume, the data cache can effectively improve the response speed of the interface, the data is cached firstly, and then the rest flow is processed by an asynchronous thread, so that the synchronous performance of the system is improved, and the running stability of the system is ensured.
And S23, adopting asynchronous multithreading processing queue buffer. Asynchronous multithreading queue caching ensures data security, and each thread processes data of one queue independently.
And S24, data mapping is the key for ensuring the synchronization accuracy, a mapping relation between the synchronization field and the database field is established, the field attributes are kept consistent, and data mapping is carried out.
And S25, persisting the data in a database table, and completing the synchronization process.
When the data source system actively pushes the data information, the data source system calls a webservice interface of the target system through the transfer system, and pushes the data information in character strings.
When the data source system actively pushes data information, the data information access process specifically comprises the following steps:
and S10, splitting the data access service according to actual requirements, and deploying a plurality of data processing services, including a data entry receiving service, a queue cache service and a data processing service.
S20, presetting an interface protocol, and handing over a corresponding interface key to a data push platform (application), wherein the data push platform (application) receives service push data to an entrance of multi-source data through a fixed format, the entrance receiving service provides a uniform entrance service interface for acquiring a data source according to configuration, and the source mode comprises HTTP, Webservice, files and the like.
S30, after the pushed data enters the entry receiving service, the data enters the database to be stored in the special cache table through the safety check of the interface secret key, and the cache ID is sent to the queue cache service.
And S40, the queue buffer service is responsible for managing the distribution of the data access buffer identification. Meanwhile, the cache ID identifications in the multiple data processing service request queue services extract the data to be processed from the database, and the queue cache service forwarding identifications can separate and respectively serve two functions of receiving data and processing data. The entrance receiving service business is light, the data processing service supports expansion as required, the problem of concurrency generated by a multi-source data access system is solved, and the processing stability is ensured.
S50, the multi-source data processing service automatically requests the queue service for the cache identification, after obtaining the cache identification, reads the cache from the database according to the identification, firstly, the data inspection is carried out, whether the numerical characters of the cache accord with the interface specification is verified, and then, the cache is mapped by the data structure model and converted into the database table object. And according to the actual service, performing addition, deletion and modification operation on the table object, and triggering a subsequent function through Java reflection.
Besides the above data access by adopting other platform (application) push modes, the multi-source data access service also supports an active pull mode to acquire data. When the data source system pushes data information passively, the data source system provides a data query interface, the target system issues a timing task to call the query interface at regular time, synchronous data are obtained, and the follow-up process is consistent with the processing flow of the target system.
When the data source system passively pushes data information, the data information access process specifically includes the following steps:
an interface protocol is configured in advance, the interface is accessed regularly to obtain data through system task scheduling, the obtained unstructured data (such as a text file) are preprocessed and converted, the preprocessed and converted data are cached in a database, and identification information is sent to a queue service.
The data processing service is the same as a conventional access mode, reads information from the cache according to the identification, and automatically acquires and stores the data through the steps of data model conversion, numerical value verification and database processing.
The data text transmission is in a form that a data source system provides data in a file mode. And the target system issues a timing task and reads the file from the text server at a timing. The flow after the target system analyzes the obtained data is consistent with the above.
Before the steps are carried out, an analysis model and a mapping model are required to be constructed.
The invention constructs diversified data analysis models, and the transmission data forms among systems are different, namely, the data analysis models are in a character string form or a file form. Different data receiving modes are provided, different analytical data models are provided, the method can be widely applied to a multi-data-source system, and the expandability principle of the interface is met.
The steps of the invention for configuring the mapping model are as follows:
and establishing a universal unified webservice interface. According to the requirement of interface universality, a target system issues a webservcie interface by adopting a soap protocol, unifies and standardizes the general fields of the interface, and provides the fields for each heterogeneous system to call. The webservice interface has high stability and safety. The data interface has higher stability and safety, can improve the operating efficiency of the system, and effectively protects the message safety of the user.
And establishing a mapping model which is convenient to configure. When data of a data source system is docked, the data to be received is various, and at this time, for the various data, a target system needs to be adapted and mapped with the data of each heterogeneous system. The data and the database field relation are accessed through the mapping of the field relation, and the effect of accurately receiving the data is achieved.
The invention applies the main data management idea to manage multi-source heterogeneous data, thereby ensuring the system coordination, the reuse and the universality of service data and the correctness of the data. The acquisition standard based on multi-source data is established, so that the data access flexibility can be improved, the data quality problem is reduced, the system application capacity is improved, and the equipment studying and judging accuracy is ensured. Through the acquisition standard of multi-source data, the data transmission capability is improved, and the data quality is improved.
When the synchronous data volume of the system is large, the interface response speed can be effectively improved through the data caching technology, the data is cached firstly, then the rest flow is processed by the asynchronous thread, the synchronous performance of the system is improved, and the running stability of the system is ensured.
The invention ensures the key of the synchronization accuracy through a data mapping technology, establishes the mapping relation between the synchronization field and the database field, and keeps the consistency of the field attributes. And finally, persisting the data in a database table to finish the synchronous process.
The invention constructs diversified data analysis models. The data transmission form between systems is different, and some forms are character string form or file form. Different data receiving modes are provided with different analytical data models respectively, the method can be widely applied to a multi-data source system, and the expandability principle of the interface is met.
Claims (4)
1. The distribution network operation and inspection multi-source data acquisition method is characterized in that a micro-service framework is adopted, and data information required to be synchronized is provided through a target system, and the method comprises the following steps:
s1, the data source system actively or passively pushes data information;
when the data source system actively pushes data information, the data source system calls a webservice interface of a target system through a transfer system, and pushes the data information in a character string manner;
when the data source system actively pushes data information, the data information access process specifically comprises the following steps:
s10, splitting the data access service according to actual requirements, and deploying a plurality of data processing services;
s20, presetting an interface protocol, and handing over a corresponding interface key to a data pushing platform, wherein the data pushing platform receives service pushing data to an inlet of multi-source data, and the inlet receiving service provides a unified inlet service interface for acquiring a data source according to configuration;
s30, after the pushed data enter the entrance to receive service, the data enter the database to be stored in the special cache table through the safety check of the interface secret key, and the cache ID is sent to the queue cache service;
s40, extracting the data to be processed from the database by the buffer ID identification in the multiple data processing service request queue service, and separating and serving the two functions of the received data and the processed data through the queue buffer service forwarding identification;
s50, reading the cache from the database according to the identification, verifying whether the cache numerical characters meet the interface specification through data inspection, and converting into a database table object through data structure model mapping;
s2, after the target system receives the request, the following steps are carried out:
s21, providing different analysis data models for different data receiving modes respectively, and analyzing the data;
s22, caching data;
s23, adopting asynchronous multithread processing queue buffer;
s24, establishing a mapping relation between the synchronous field and the database field, keeping the field attributes consistent, and mapping data;
and S25, persisting the data in a database table to complete data synchronization.
2. The distribution network operation and maintenance oriented multi-source data acquisition method as claimed in claim 1, wherein when the data source system passively pushes data information, the data source system provides a data query interface, and the target system issues a timing task to regularly call the query interface to acquire synchronous data.
3. The distribution network operation and maintenance oriented multi-source data acquisition method according to claim 2, wherein the data information access process specifically comprises the following steps:
an interface protocol is configured in advance, an interface is accessed regularly to obtain data through system task scheduling, the obtained unstructured data are preprocessed and converted, the preprocessed and converted data are cached in a database, and identification information is sent to a queue service;
and the data processing service reads information from the cache according to the identification, and automatically acquires and stores the data through the steps of data model conversion, numerical value verification and database processing.
4. The distribution network operation and inspection oriented multi-source data acquisition method as claimed in claim 1, wherein the data text transmission is in a form that a data source system provides data in a file mode; and the target system issues a timing task and reads the file from the text server at regular time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110282493.2A CN113032379B (en) | 2021-03-16 | 2021-03-16 | Distribution network operation and inspection-oriented multi-source data acquisition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110282493.2A CN113032379B (en) | 2021-03-16 | 2021-03-16 | Distribution network operation and inspection-oriented multi-source data acquisition method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113032379A CN113032379A (en) | 2021-06-25 |
| CN113032379B true CN113032379B (en) | 2022-07-22 |
Family
ID=76471134
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110282493.2A Active CN113032379B (en) | 2021-03-16 | 2021-03-16 | Distribution network operation and inspection-oriented multi-source data acquisition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032379B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835786B (en) * | 2021-09-30 | 2023-04-28 | 四川新网银行股份有限公司 | Data docking system, method and computer readable storage medium |
| CN117076574B (en) * | 2023-10-16 | 2024-01-23 | 北京持安科技有限公司 | Method and device capable of arranging multiple data sources for synchronous aggregation of data |
| CN117171534B (en) * | 2023-11-03 | 2024-03-19 | 济南二机床集团有限公司 | Multi-source heterogeneous data acquisition method, system, device and medium for numerical control machine tool |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729366A (en) * | 2017-09-08 | 2018-02-23 | 广东省建设信息中心 | A kind of pervasive multi-source heterogeneous large-scale data synchronization system |
| US10545893B1 (en) * | 2019-01-11 | 2020-01-28 | Arm Limited | Interrupt controller and method of operation of an interrupt controller |
| CN111784130A (en) * | 2020-06-19 | 2020-10-16 | 国电南瑞南京控制系统有限公司 | Power distribution network Internet of things management system, data processing method and storage medium |
| CN111930529A (en) * | 2020-10-09 | 2020-11-13 | 上海富友支付服务股份有限公司 | Data synchronization method, module and system based on message queue and micro-service |
| CN112131286A (en) * | 2020-11-26 | 2020-12-25 | 畅捷通信息技术股份有限公司 | Data processing method and device based on time sequence and storage medium |
| CN112269820A (en) * | 2020-10-30 | 2021-01-26 | 浪潮云信息技术股份公司 | Method and platform for realizing data batch synchronization based on CMSP |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239505B (en) * | 2014-09-12 | 2015-10-07 | 何苗 | The directed real time bidirectional of situationization pushes System of Synchronous Processing |
| CA3099814C (en) * | 2018-05-11 | 2021-11-30 | Imagine Communications Corp. | Distributed ledger platform for computing applications |
| CN112100265A (en) * | 2020-09-17 | 2020-12-18 | 博雅正链(北京)科技有限公司 | Multi-source data processing method and device for big data architecture and block chain |
-
2021
- 2021-03-16 CN CN202110282493.2A patent/CN113032379B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729366A (en) * | 2017-09-08 | 2018-02-23 | 广东省建设信息中心 | A kind of pervasive multi-source heterogeneous large-scale data synchronization system |
| US10545893B1 (en) * | 2019-01-11 | 2020-01-28 | Arm Limited | Interrupt controller and method of operation of an interrupt controller |
| CN111784130A (en) * | 2020-06-19 | 2020-10-16 | 国电南瑞南京控制系统有限公司 | Power distribution network Internet of things management system, data processing method and storage medium |
| CN111930529A (en) * | 2020-10-09 | 2020-11-13 | 上海富友支付服务股份有限公司 | Data synchronization method, module and system based on message queue and micro-service |
| CN112269820A (en) * | 2020-10-30 | 2021-01-26 | 浪潮云信息技术股份公司 | Method and platform for realizing data batch synchronization based on CMSP |
| CN112131286A (en) * | 2020-11-26 | 2020-12-25 | 畅捷通信息技术股份有限公司 | Data processing method and device based on time sequence and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113032379A (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113032379B (en) | Distribution network operation and inspection-oriented multi-source data acquisition method | |
| CN109756364B (en) | Micro-service performance optimization system and analysis method based on log analysis | |
| CN105045820B (en) | Method for processing video image information of high-level data and database system | |
| CN107818120B (en) | Data processing method and device based on big data | |
| CN105138661B (en) | A kind of network security daily record k-means cluster analysis systems and method based on Hadoop | |
| US8380192B2 (en) | Method and apparatus for managing profile information of mobile terminal in mobile communications system | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN108073625B (en) | System and method for metadata information management | |
| US8224834B2 (en) | Database message analysis support technique | |
| CN110609782B (en) | Micro-service optimization system and method based on big data | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN103092866B (en) | Data monitoring method and supervising device | |
| CN108228683A (en) | A kind of distributed intelligence electric network data analysis platform based on cloud computing | |
| CN102026228B (en) | Statistical method and equipment for communication network performance data | |
| CN111143468B (en) | Multi-database data management method based on MPP distributed technology | |
| CN111913933A (en) | Power grid historical data management method and system based on unified support platform | |
| CN101021916A (en) | Business process analysis method | |
| CN107153679B (en) | Extraction statistical method and system for semi-structured big data | |
| CN112306992A (en) | Big data platform based on internet | |
| CN113468340B (en) | Construction system and construction method of industrial knowledge graph | |
| CN112687267A (en) | Internet of things data semantic processing system | |
| CN108959041A (en) | Method, server and the computer readable storage medium that information is sent | |
| EP2442235A1 (en) | Method adapting to mobile search interface, search server and system thereof | |
| KR101878291B1 (en) | Big data management system and management method thereof | |
| CN102291469B (en) | Individual oriented acquisition cloud service system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |