CN115550257A - Method for realizing data routing processing between power distribution type systems and storage medium - Google Patents
Method for realizing data routing processing between power distribution type systems and storage medium Download PDFInfo
- Publication number
- CN115550257A CN115550257A CN202211264425.4A CN202211264425A CN115550257A CN 115550257 A CN115550257 A CN 115550257A CN 202211264425 A CN202211264425 A CN 202211264425A CN 115550257 A CN115550257 A CN 115550257A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- routing
- service
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 55
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000006243 chemical reaction Methods 0.000 claims abstract description 7
- 238000004140 cleaning Methods 0.000 claims abstract description 7
- 230000006855 networking Effects 0.000 claims abstract description 5
- 230000008569 process Effects 0.000 claims description 9
- 230000010354 integration Effects 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000012550 audit Methods 0.000 claims description 3
- 230000000903 blocking effect Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013499 data model Methods 0.000 claims description 3
- 230000026676 system process Effects 0.000 claims description 3
- 230000009467 reduction Effects 0.000 abstract description 2
- 238000000926 separation method Methods 0.000 abstract 1
- 238000003066 decision tree Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/74—Address processing for routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/14—Routing performance; Theoretical aspects
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method and a storage medium for realizing data routing processing among electric power distributed systems, which comprises the steps of receiving data by an API gateway, distributing data by a data router, cleaning data by a data cleaner, processing peak clipping and valley filling data and the like, and realizes networking of the data by the data router, so that data flow among the distributed service systems is the same as that of the data flow among the systems, the routing flow logic and the service logic of the data are separated, and the stable operation of the service systems is effectively ensured. The invention realizes the separation of the strategies of data circulation and conversion from the application logic codes of the application system, can separate complex and redundant business rules from the whole supported business system, and realizes the reusability of the architecture and the components. The invention can adapt to massive data request and processing by asynchronous processing, data shunting, decoupling, peak clipping and valley filling, and solves the problems of performance reduction, downtime and the like of processing mass data in a peak period.
Description
Technical Field
The invention relates to a method and a storage medium for realizing data routing processing among power distributed systems, in particular to a method and a storage medium for realizing that a large amount of data are accurately transferred and processed among systems through a series of operations and rules among systems distributed and deployed in a power enterprise and helping the power enterprise to improve the efficiency of data processing.
Background
In the construction of the information system of the power enterprise, different technical frameworks are adopted by different units to construct each business information system, the process is relatively independent, and the operation departments and the operation environments of each system form a system. However, the service data of the power enterprise often needs to be completed by the cooperation of a plurality of service systems, but the deployment mode of each system is multi-level deployment of network, province and city, so that the data flow and processing between the systems are difficult, and the data in different regions and different systems are difficult to flow and process well. At present, when data interaction is needed between systems, a hard coding mode is usually adopted, and direct point-to-point integration is performed through modes such as WebService and gRPC, and the cost of development and operation and maintenance is increased through the integration mode. And the availability and maintainability of each service information system is also reduced.
Under the background, research work related to data routing of the power enterprises needs to be carried out, research and design are carried out on various links of data architecture, data organization, data distribution, data sharing, data transfer, data application and the like of the power enterprises, and a method for realizing data routing processing among power distributed systems is provided, so that data transfer and processing in different systems are efficiently processed, and stable operation of each business information system is ensured.
Disclosure of Invention
The invention aims to provide a method and a storage medium for realizing data routing processing among power distributed systems, which are applied to data circulation and processing among different systems.
Aiming at the purposes, the technical scheme provided by the invention is as follows:
a method for realizing data routing processing among electric power distributed systems is characterized in that a computer program for realizing the data routing method is embedded in an application program or implanted on a chip, so that strategies of data circulation and conversion are separated from application logic codes of an application system, and the processing method can be realized through configuration. The method can separate the complex and redundant business rules from the whole supported business system, and can realize the reusability of the architecture and the components.
The invention comprises the following steps:
step 101, receiving data by the API gateway (asynchronous processing):
using a unified data gateway to receive service data, supporting modes such as WebService, restful, gRPC and the like, and supporting a configuration center: apollo, nacos, support registry: eureka, nacos. The interface form between different systems is compatible to the maximum extent. The high performance gateway is implemented using FastHttp, exceeding Nginx in performance. Due to the adoption of an asynchronous data processing mode, the waiting and the blocking of a service system can not be caused. On the other hand, the point-to-point integration mode is changed into a star-type integration mode, and the availability, maintainability and safety among systems are improved. Meanwhile, for a distributed deployment system, the system deployment is carried out in a distributed mode, like networking data by using a router. The specific process comprises the following steps:
step 1: the service system receives a request of a user, and receives service data request information input by the user.
Step 2: and the service system writes the data to be circulated into the message queue.
And 3, step 3: and the processing program forwards the data in the message queue to an API gateway of the data routing service for processing.
Step 102, the data router distributes data (data splitting):
the processing of the data (discarding, storing the database, or forwarding other routing nodes) is determined based on the routing table. And combining the routing table, wherein the data content of the data transmitted by the service system needs to contain host names of other routing nodes, and the routing table finds the API gateway address of the corresponding routing node by matching the host names. And if the data routing is not finished, performing abnormal data log recording for audit analysis. And distributing the data to one or N data routing centers for processing according to the routing table. If the current data processing center is consistent with the current routing node, performing subsequent processing; and if the host name is inconsistent with the current routing node, forwarding the data to the corresponding routing node, and continuously processing the data by the next routing node.
The method comprises the following specific steps:
step 1: the routing center acquires service data to be transferred from the message queue;
and 2, step: inquiring a routing table, and acquiring an interface address and an interface type through a service system identifier in service data;
step 3, if the output interface is found out, forwarding the data through a specific transponder;
and 4, step 4: if no exit interface is found, the data is recorded and the source system is informed of the abnormal condition.
Step 103, data cleaner cleans data (decouples):
the processing logic and pressure of the business system are simplified by the washer. Because the service systems of all the nodes have differences due to the characteristics of the attributes, the data cleaning work is separated from the application logic by adopting the rule engine, so that the adaptability is higher, the stability of the system is higher, and on the other hand, the service systems do not need to perform complex logic check and conversion work to reduce the pressure of the service systems. Based on the goal, a set of business processing rules is designed according to the grammar of the decision tree, and the specific steps are as follows:
step 1: determining a service domain to which the data message belongs;
step 2: determining a data model to which the data message belongs;
and 3, step 3: determining the classification of the message according to the characteristics of the message, and finding out a suitable data converter;
and 4, step 4: converting the data of the data message according to the configured conversion rule;
and 5: and outputting the target message.
Step 104, data processing (peak clipping and valley filling):
certain peak clipping and valley filling operations are realized through the previous steps, but the risk that the pressure of a service system is increased sharply and downtime is caused due to the fact that a large amount of data are increased sharply in the peak period of morning and evening is avoided. This is done using Kafka as the message queue.
Step 1: the data processor acquires data to be processed by the service system from the data cleaning queue;
step 2: the processor processes the data to the database or distributes the data to the secondary queue again;
and 3, step 3: and the service system processes the data according to the processing capacity, and sequentially acquires the data from the queue for processing.
The present invention also provides a computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the method of the present invention for implementing inter-power distributed system data routing processing.
The invention has the advantages of
The invention realizes the networking of data through the data router, so that the data flow among the service systems deployed in a distributed way is the same as the flow among the systems, the routing flow logic and the service logic of the data are separated, and the stable operation of the service systems is effectively ensured. Meanwhile, the method can adapt to massive data requests and processing through asynchronous processing, data distribution, decoupling, peak clipping and valley filling, and solves the problems that performance reduction, downtime and the like occur when a business system needs to process massive circulation data in a peak period.
Drawings
FIG. 1: a data routing gateway deployment map.
FIG. 2 is a schematic diagram: an operational schematic diagram for asynchronously processing received data.
FIG. 3: and converting and classifying the field of the to-be-handled state by using a rule engine.
FIG. 4: and converting the time field in the to-do list by using a rule engine.
FIG. 5 is a schematic view of: and (4) a peak clipping and valley filling data processing flow chart.
Detailed Description
A method for realizing data routing processing among electric power distributed systems comprises the following steps:
step 101, receiving data by the API gateway (asynchronous processing):
as shown in fig. 1 and fig. 2, a unified data gateway is used to receive service data, support modes such as WebService, restful, and gRPC, and support a configuration center: apollo, nacos, support registry: eureka, nacos. The interface form between different systems is compatible to the maximum extent. The high performance gateway is implemented using FastHttp, exceeding Nginx in performance. Due to the adoption of an asynchronous data processing mode, the waiting and the blocking of a service system can not be caused. On the other hand, the point-to-point integration mode is changed into a star-type integration mode, and the availability, maintainability and safety among systems are improved. Meanwhile, for the distributed deployment system, the system deployment is also performed in a distributed manner, like the data networking is performed by using a router.
The method comprises the following specific steps:
step 1: the service system receives the request of the user, and the service system receives the service data request information input by the user.
And 2, step: and the service system writes the data to be circulated into the message queue.
And 3, step 3: and the processing program forwards the data in the message queue to an API gateway of the data routing service for processing.
Step 102, the data router distributes data (data splitting):
the processing of the data (discarding, storing the database, or forwarding other routing nodes) is determined based on the routing table.
And combining the routing table, wherein the data content of the data transmitted by the service system needs to contain host names of other routing nodes, and the routing table finds the API gateway address of the corresponding routing node by matching the host names. And if the data routing is not finished, performing abnormal data log recording for audit analysis. And the data can be distributed to one or N data routing centers for processing according to the routing table. If the current data processing center is consistent with the current routing node, the subsequent processing is carried out; and if the host name is inconsistent with the current routing node, forwarding the data to the corresponding routing node, and continuously processing the data by the next routing node.
The method comprises the following specific steps:
step 1: the routing center acquires service data to be transferred from the message queue;
and 2, step: inquiring a routing table, and acquiring an interface address and an interface type through a service system identifier in service data;
step 3, if the output interface is found out, forwarding the data through a specific transponder;
and 4, step 4: if no exit interface is found, the data is recorded and the source system is informed of the abnormal condition.
An example of a routing table rule is as follows:
| business system identification | Outlet interface | Interface type |
| GZ_SC | http://0.0.0.0/api/interface | REST |
| YN_YX | http://0.0.0.1/api/interface | REST |
| GZ01_SC | http://0.0.0.2/api/interface | REST |
| HN_SC | http://0.0.0.3/api/interface | REST |
Step 103, data washer washing data (decoupling):
the processing logic and pressure of the business system are simplified by the washer. Because the service systems of all the nodes have differences due to the characteristics of the attributes, the data cleaning work is separated from the application logic by adopting the rule engine, so that the adaptability is higher, the stability of the system is higher, and on the other hand, the service systems do not need to perform complex logic check and conversion work to reduce the pressure of the service systems. Based on this goal, a set of business process rules is designed according to the syntax of the decision tree, as shown in fig. 3 and 4.
The method comprises the following specific steps:
step 1: determining a service domain to which the data message belongs;
step 2: determining a data model to which the data message belongs;
and step 3: determining the classification of the message according to the characteristics of the message, and finding out a suitable data converter;
and 4, step 4: converting the data of the data message according to the configured conversion rule;
and 5: and outputting the target message.
Step 104, data processing (peak clipping and valley filling):
referring to fig. 5, a certain peak clipping and valley filling operation is already realized through the previous steps, but the risk that a great amount of data is increased sharply to cause a sudden increase in the pressure of a business system and cause downtime is also avoided. This is done using Kafka as the message queue.
The method comprises the following specific steps:
step 1: the data processor acquires data to be processed by the service system from the data cleaning queue;
step 2: the processor processes the data to the database or distributes the data to the secondary queue again;
and 3, step 3: and the service system processes the data according to the processing capacity, and sequentially acquires the data from the queue for processing.
Claims (7)
1. A method for realizing data routing processing between electric power distributed systems is characterized by comprising the following steps:
step 101, the api gateway receives data, including:
the method uses a uniform data gateway to receive service data, supports the modes of WebService, restful and gRPC, and supports a configuration center comprising the following steps: apollo, nacos, supporting the registry comprising: eureka, nacos;
the interface form between different systems is compatible; implementing a high performance gateway using FastHttp; the adopted asynchronous data processing mode is used for avoiding causing the waiting and the blocking of a service system;
the point-to-point integration mode is changed into a star-shaped integration mode, and the usability, maintainability and safety among systems are improved;
the system deployment is carried out in a distributed mode, namely, the networking of data is carried out by using a router;
step 102, the data router distributes data, including:
determining processing of the data based on the routing table;
combining with a routing table, for data transmitted by a service system, the data content requires to contain host names of other routing nodes, and the routing table finds an API gateway address of a corresponding routing node by matching the host names; if the data routing is not finished, abnormal data log recording is carried out for audit analysis;
distributing the data to one or N data routing centers according to the routing table for processing;
if the current data processing center is consistent with the current routing node, the subsequent processing is carried out;
if the host name is inconsistent with the current routing node, forwarding the data to the corresponding routing node, and continuously processing the data by the next routing node;
step 103, the data cleaner cleans data, including:
the processing logic and pressure of the business system are simplified through the cleaner;
separating the data cleaning work from the application logic by adopting a rule engine;
step 104, peak clipping and valley filling data processing, including:
kafka is used as a message queue to avoid data surge during peak hours in the morning and evening.
2. The method of claim 1, wherein the step 101 further comprises:
(1) The service system receives a request of a user, and receives service data request information input by the user;
(2) The service system writes the data to be transferred into a message queue;
(3) And the processing program forwards the data in the message queue to an API gateway of the data routing service for processing.
3. The method of claim 1, wherein the step 102 further comprises:
(1) The routing center acquires service data to be transferred from the message queue;
(2) Inquiring a routing table, and acquiring an interface address and an interface type through a service system identifier in service data;
(3) If the out interface is found, forwarding the data through a specific repeater;
(4) If no exit interface is found, the data is recorded and the source system is informed of the abnormal condition.
4. The method of claim 1, wherein the step 103 further comprises:
(1) Determining a service domain to which the data message belongs;
(2) Determining a data model to which the data message belongs;
(3) Determining the classification of the message according to the characteristics of the message, and finding out a suitable data converter;
(4) Converting the data of the data message according to the configured conversion rule;
(5) And outputting the target message.
5. The method of claim 1, wherein the step 104 further comprises:
(1) The data processor acquires data to be processed by the service system from the data cleaning queue;
(2) The processor processes the data to the database or distributes the data to the secondary queue again;
(3) And the service system processes the data according to the processing capacity, and sequentially acquires the data from the queue for processing.
6. The method of claim 3, wherein the rules of the routing table are:
。
7. A computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the method of implementing a power distributed inter-system data routing process of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211264425.4A CN115550257A (en) | 2022-10-17 | 2022-10-17 | Method for realizing data routing processing between power distribution type systems and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211264425.4A CN115550257A (en) | 2022-10-17 | 2022-10-17 | Method for realizing data routing processing between power distribution type systems and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115550257A true CN115550257A (en) | 2022-12-30 |
Family
ID=84735980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211264425.4A Pending CN115550257A (en) | 2022-10-17 | 2022-10-17 | Method for realizing data routing processing between power distribution type systems and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115550257A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130208592A1 (en) * | 2010-08-06 | 2013-08-15 | Bejing Qiantang Network Technology Company, Ltd. | Traffic-control-based data transmission method and communication system |
| CN104811475A (en) * | 2015-03-27 | 2015-07-29 | 深圳市华运国际物流有限公司 | Enterprise service bus middleware based on Restful technology |
| CN106411905A (en) * | 2016-10-10 | 2017-02-15 | 青海帝特斯软件开发有限公司 | Method for implementing distributed architecture through soft routing |
| CN108092884A (en) * | 2017-11-23 | 2018-05-29 | 南京邮电大学 | A kind of wireless access gateway system and application process |
| CN110865997A (en) * | 2019-11-08 | 2020-03-06 | 国网四川省电力公司电力科学研究院 | Online identification method for hidden danger of power system equipment and application platform thereof |
| CN113923149A (en) * | 2020-07-09 | 2022-01-11 | 阿里巴巴集团控股有限公司 | Network access method, device, network system, electronic equipment and storage medium |
| CN114866479A (en) * | 2022-04-24 | 2022-08-05 | 中电万维信息技术有限责任公司 | Service gateway-based service fusion method |
-
2022
- 2022-10-17 CN CN202211264425.4A patent/CN115550257A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130208592A1 (en) * | 2010-08-06 | 2013-08-15 | Bejing Qiantang Network Technology Company, Ltd. | Traffic-control-based data transmission method and communication system |
| CN104811475A (en) * | 2015-03-27 | 2015-07-29 | 深圳市华运国际物流有限公司 | Enterprise service bus middleware based on Restful technology |
| CN106411905A (en) * | 2016-10-10 | 2017-02-15 | 青海帝特斯软件开发有限公司 | Method for implementing distributed architecture through soft routing |
| CN108092884A (en) * | 2017-11-23 | 2018-05-29 | 南京邮电大学 | A kind of wireless access gateway system and application process |
| CN110865997A (en) * | 2019-11-08 | 2020-03-06 | 国网四川省电力公司电力科学研究院 | Online identification method for hidden danger of power system equipment and application platform thereof |
| CN113923149A (en) * | 2020-07-09 | 2022-01-11 | 阿里巴巴集团控股有限公司 | Network access method, device, network system, electronic equipment and storage medium |
| CN114866479A (en) * | 2022-04-24 | 2022-08-05 | 中电万维信息技术有限责任公司 | Service gateway-based service fusion method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102375837B (en) | Data acquiring system and method | |
| CN108334557B (en) | Aggregated data analysis method and device, storage medium and electronic equipment | |
| CN101330431B (en) | Method and system for storing instant information | |
| CN102859961A (en) | Distributed video transcode system with adaptive file processing | |
| CN103944974A (en) | Method for processing protocol messages, method for processing faults of controllers and related device | |
| CN114710571B (en) | Data packet processing system | |
| CN111210340A (en) | Automatic task processing method and device, server and storage medium | |
| CN106354870B (en) | A kind of method and apparatus of data load | |
| CN108881651A (en) | Data processing method, device and equipment of call platform and storage medium | |
| CN103248636A (en) | Offline download system and method | |
| CN115550257A (en) | Method for realizing data routing processing between power distribution type systems and storage medium | |
| CN106790489A (en) | Parallel data loading method and system | |
| CN113656502A (en) | Data synchronization method, system, electronic device and storage medium | |
| CN111352726A (en) | Streaming data processing method and device based on containerized micro-service | |
| CN110750362A (en) | Method and apparatus for analyzing biological information, and storage medium | |
| CN110795423A (en) | Data extraction method for rapid cleaning and conversion | |
| US10944846B2 (en) | Distribution schedule creation method and distribution schedule creation apparatus | |
| CN116149814A (en) | KAFKA-based data persistence task distributed scheduling method and system | |
| CN112765109B (en) | Queue type data storage analysis method and system | |
| CN115344633A (en) | Data processing method, device, equipment and storage medium | |
| KR101526344B1 (en) | System and method for processing similar emails | |
| CN114896222A (en) | Log data processing method and device, computer equipment and system | |
| CN111401819B (en) | Intersystem data pushing method and system | |
| CN107239369A (en) | A kind of database filing standby system and method | |
| CN112783892A (en) | Chained task execution engine realized through event-driven model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |