CN117390021A - Data transparent transmission collision library system and method based on strategy mode and proxy mode - Google Patents
Data transparent transmission collision library system and method based on strategy mode and proxy mode Download PDFInfo
- Publication number
- CN117390021A CN117390021A CN202311334283.9A CN202311334283A CN117390021A CN 117390021 A CN117390021 A CN 117390021A CN 202311334283 A CN202311334283 A CN 202311334283A CN 117390021 A CN117390021 A CN 117390021A
- Authority
- CN
- China
- Prior art keywords
- data
- transparent
- mode
- transparent transmission
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 title claims abstract description 92
- 238000004458 analytical method Methods 0.000 claims abstract description 75
- 230000008569 process Effects 0.000 claims abstract description 61
- 238000012545 processing Methods 0.000 claims abstract description 52
- 238000006243 chemical reaction Methods 0.000 claims abstract description 34
- 238000012544 monitoring process Methods 0.000 claims abstract description 30
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000013211 curve analysis Methods 0.000 claims description 3
- 238000011144 upstream manufacturing Methods 0.000 description 20
- 238000003032 molecular docking Methods 0.000 description 16
- 238000013461 design Methods 0.000 description 9
- 238000012423 maintenance Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention relates to the technical field of transparent transmission data transmission, and particularly discloses a data transparent transmission collision library system and a method based on a strategy mode and a proxy mode, wherein the method comprises the steps of receiving original transparent transmission data, acquiring a data structure of the original transparent transmission data, and generating a structure record table; updating an analysis strategy based on the structure record table at regular time, and converting the original transparent data into uniform format data based on the analysis strategy; monitoring the conversion process of the original transparent transmission data in real time, and carrying out secondary updating on the analysis strategy according to the monitoring result; and constructing a transparent channel of the analysis strategy based on the data frequency item. The invention realizes the flexibility and expandability of data processing by using the strategy mode. Different data formats can correspond to different policy classes, so that compatible processing of various data formats is achieved.
Description
Technical Field
The invention relates to the technical field of transparent data transmission, in particular to a data transparent collision library system and method based on a strategy mode and a proxy mode.
Background
The prior art has certain compatibility limitations when processing transparent transmission data in different formats. Because different upstream systems may employ different data formats and communication protocols, additional parsers or format conversions are required, increasing complexity in system development and maintenance.
For expanding the functions of the system and accessing new upstream and downstream data transmission collision base channels, the prior art needs to carry out a great deal of modification and adjustment on the system flow, and has larger workload and lower efficiency. This limits the flexibility and scalability of the system in rapidly changing business environments.
Because the prior art often requires writing a large amount of code and logic when processing a complex pass-through crash library system, the maintainability of the system is reduced. Long-term maintenance and improvement of the system may be faced with difficulties and risks.
The prior art may present security risks in the data transmission collision process. If the transparent data is not sufficiently checked and filtered, malicious attack or illegal access may be caused, resulting in risks such as data leakage or system paralysis.
In summary, the prior art has objective drawbacks in terms of compatibility, expansibility, maintenance difficulty, safety, and the like.
Disclosure of Invention
The invention aims to provide a data transparent transmission collision library system and method based on a strategy mode and a proxy mode, so as to solve the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions:
a data transparent library collision method based on a policy mode and a proxy mode, the method comprising:
receiving original transparent transmission data, obtaining a data structure of the original transparent transmission data, and generating a structure record table; the structure record table comprises a data structure item and a data frequency item;
updating an analysis strategy based on the structure record table at regular time, and converting the original transparent data into uniform format data based on the analysis strategy;
monitoring the conversion process of the original transparent transmission data in real time, and carrying out secondary updating on the analysis strategy according to the monitoring result;
and constructing a transparent channel of the analysis strategy based on the data frequency item.
As a further scheme of the invention: the step of receiving the original transparent transmission data, obtaining the data structure of the original transparent transmission data and generating the structure record list comprises the following steps:
establishing a connection channel with a downstream system, and inserting a data enhancement node into the connection channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
receiving original transparent transmission data based on a connecting channel, and processing the original transparent transmission data through a data enhancement node;
acquiring a data structure of original transparent transmission data, recording the occurrence times of various data structures, and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
As a further scheme of the invention: the step of periodically updating the analysis strategy based on the structure record table and converting the original transparent data into the uniform format data based on the analysis strategy comprises the following steps:
counting the data times of each data structure and the variation interval thereof;
counting the number of data according to the fluctuation interval to obtain a frequency curve;
analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
updating an analysis strategy when the data characteristics accord with preset data conditions;
and converting the original transparent data into uniform format data based on the latest analysis strategy.
As a further scheme of the invention: the step of monitoring the conversion process of the original transparent data in real time and carrying out secondary updating on the analysis strategy according to the monitoring result comprises the following steps:
monitoring the conversion process of original transparent transmission data in real time, and calculating the resource consumption;
comparing the resource consumption according to the similarity of the data structure, and calculating a consumption difference value;
marking an abnormal conversion process according to the consumption difference value and the similarity;
and carrying out secondary updating on the analysis strategy of the abnormal conversion process.
As a further scheme of the invention: the step of constructing the transparent transmission channel of the analysis strategy based on the data frequency item comprises the following steps:
reading a data frequency item in a preset time;
when the data times reach a preset time threshold, a transparent channel corresponding to the analysis strategy is constructed, and one transparent channel corresponds to one analysis strategy.
The technical scheme of the invention also provides a data transparent transmission collision library system based on the strategy mode and the proxy mode, which comprises the following steps:
the record table generation module is used for receiving the original transparent transmission data, acquiring a data structure of the original transparent transmission data and generating a structure record table; the structure record table comprises a data structure item and a data frequency item;
the format unification module is used for updating the analysis strategy based on the structure record table at regular time and converting the original transparent data into unified format data based on the analysis strategy;
the strategy updating module is used for monitoring the conversion process of the original transparent transmission data in real time and carrying out secondary updating on the analysis strategy according to the monitoring result;
and the channel creation module is used for constructing a transparent channel of the analysis strategy based on the data frequency item.
As a further scheme of the invention: the record table generation module includes:
the node inserting unit is used for establishing a connecting channel with a downstream system and inserting a data enhancement node into the connecting channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
the data processing unit is used for receiving the original transparent transmission data based on the connecting channel and processing the original transparent transmission data through the data enhancement node;
the frequency counting unit is used for acquiring the data structure of the original transparent transmission data, recording the occurrence frequency of various data structures and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
As a further scheme of the invention: the format unification module comprises:
the statistics unit is used for counting the data times of each data structure and the variation interval of the data times;
the frequency analysis unit is used for counting the frequency of the data according to the fluctuation interval to obtain a frequency curve;
the curve analysis unit is used for analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
the strategy updating unit is used for updating the analysis strategy when the data characteristics accord with preset data conditions;
and the execution unit is used for converting the original transparent transmission data into uniform format data based on the latest analysis strategy.
As a further scheme of the invention: the policy updating module comprises:
the monitoring and calculating unit is used for monitoring the conversion process of the original transparent transmission data in real time and calculating the resource consumption;
the comparison calculation unit is used for comparing the consumption of the resources according to the similarity of the data structures and calculating a consumption difference value;
a process marking unit for marking an abnormal conversion process according to the consumption difference and the similarity;
and the updating execution unit is used for carrying out secondary updating on the analysis strategy in the abnormal conversion process.
As a further scheme of the invention: the channel creation module includes:
the data reading unit is used for reading the data frequency items in the preset time;
and the construction execution unit is used for constructing a transparent channel corresponding to the analysis strategy when the data frequency reaches a preset frequency threshold value, and one transparent channel corresponds to one analysis strategy.
Compared with the prior art, the invention has the beneficial effects that: the invention realizes the flexibility and expandability of data processing by using the strategy mode. Different data formats can correspond to different policy classes, so that compatible processing of various data formats is achieved. By using the proxy mode, proxy management of system functions is achieved. Proxy objects may insert new functions, add restrictions, etc. to better control access and operation to real system functions. By using the template method mode, the unification and standardization of the data transparent transmission flow are realized. The flow of data transparent transmission is defined in the abstract class, and a plurality of methods requiring subclass realization are reserved, so that the sequence and consistency of the flow are ensured. By using the factory mode, the capability of quickly docking new upstream and downstream through-transmission collision warehouse channels is realized. The specific factory class can create corresponding channel objects according to different upstream channels, and uniformly manage the upstream channels and the process of butting new channels.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following description will briefly introduce the drawings that are needed in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are only some embodiments of the present invention.
FIG. 1 is a flow diagram of a method for data transparent collision base based on policy mode and agent mode.
Fig. 2 is a data transparent flow.
FIG. 3 is a block diagram of the constituent architecture of a data transparent collision base system based on a policy mode and a proxy mode.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, in an embodiment of the present invention, a method for data transparent collision between a policy mode and a proxy mode includes:
receiving original transparent transmission data, obtaining a data structure of the original transparent transmission data, and generating a structure record table; the structure record table comprises a data structure item and a data frequency item;
updating an analysis strategy based on the structure record table at regular time, and converting the original transparent data into uniform format data based on the analysis strategy;
monitoring the conversion process of the original transparent transmission data in real time, and carrying out secondary updating on the analysis strategy according to the monitoring result;
and constructing a transparent channel of the analysis strategy based on the data frequency item.
The scheme in the prior art is generally designed for a data transmission collision library system and mainly comprises the following steps:
1. and (3) data receiving: the system receives data transparent requests from upstream via different communication protocols (e.g., HTTP, TCP/IP, etc.). These requests may be encoded in different formats (e.g., json, XML, etc.).
2. Data analysis: the received data needs to be analyzed, key information in the data is extracted, and the data is organized and stored according to a data model in the system. This step typically requires writing parsers for different data formats or the use of specialized parsing tools.
3. And (3) data processing: the parsed data needs to undergo a series of processing operations including data cleaning, verification, conversion, calculation, etc. Depending on the traffic requirements, data screening, filtering, or aggregation operations may also be required.
4. And (3) data storage: the processed data needs to be stored in a database or other persistent storage medium for subsequent query and analysis operations. The storage scheme may be selected according to a specific scenario, such as a relational database, a NoSQL database, etc.
5. And (3) data transmission: the system may need to transmit the processed data to a downstream system or other business module. This step typically involves the conversion of the data format and the adaptation of the data exchange protocol to ensure that the data is properly received and processed by the downstream system.
The scheme of the prior art has been widely applied in the field of data transmission collision libraries, but has some limitations. For example, for data transparent transmission in different formats, different resolvers are required to be written or special resolving tools are used, so that development and maintenance difficulties are increased; for expanding the functions of the system and docking new upstream and downstream data transmission collision channels, a large amount of system flow modification is needed, and the workload is large and the efficiency is low. Therefore, the technical scheme provided by the patent aims at solving the limitations and reaching the industry top level.
In an example of the technical scheme of the invention, the system comprises a transparent data receiving module, a transparent data processing module and an upstream data transparent channel docking module, and the specific description is as follows:
the transparent data receiving module has the capability of enhancing the unified system function, adopts the design of combining the strategy mode and the proxy mode, and ensures that the data receiving and processing process of the system is more flexible, expandable and safe due to the combined design of the strategy mode and the proxy mode of the transparent data receiving module. Through unified system function enhancement, the system can provide efficient and reliable transparent transmission data processing capacity, meet different service requirements and ensure stable operation of the system.
The transparent data processing module is designed by combining a strategy mode and a template method mode, is used for processing transparent data in different formats, and can be changed minimally when processing different transparent types, and the design by combining the strategy mode and the template method mode of the transparent data processing module enables the module to process transparent data in different formats efficiently and flexibly, and can be changed minimally when processing different transparent types. This provides convenience for the scalability and maintainability of the system, while also guaranteeing the unification and specification of the pass-through data processing process.
The upstream data transparent channel docking module is designed by combining a strategy mode and a factory mode, is used for realizing docking with different transparent channels, can select corresponding strategies for processing according to requirements, and can select corresponding strategies for processing according to different transparent channels by adopting the design by combining the strategy mode and the factory mode, and unified management of strategy objects is realized. Thus, when the transparent channel is newly added, modified or deleted, only the corresponding strategy class and configuration need to be newly added, modified or deleted, and the codes of other parts do not need to be modified. The design can improve the expansibility and maintainability of the system, so that the docking process of the transmission channel is more flexible and controllable.
As shown in fig. 2, the interaction relationship between the three modules is as follows:
1. the transparent transmission data receiving module transmits the received transparent transmission data to the transparent transmission data processing module
2. The transparent data processing module is used for further processing transparent data and transmitting the transparent data to the upstream data transparent channel docking module
3. And the upstream data transparent transmission channel docking module calls a specific channel to finish data transparent transmission.
As a preferred embodiment of the present invention, the step of receiving the original transparent data, obtaining the data structure of the original transparent data, and generating the structure record table includes:
establishing a connection channel with a downstream system, and inserting a data enhancement node into the connection channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
receiving original transparent transmission data based on a connecting channel, and processing the original transparent transmission data through a data enhancement node;
acquiring a data structure of original transparent transmission data, recording the occurrence times of various data structures, and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
In an example of the technical solution of the present invention, the function of the transparent data receiving module is described as follows:
and (3) data receiving: and the transparent transmission data receiving module acquires original transparent transmission data by establishing connection with a downstream system. Through application of the proxy mode, the module can perform interception, recording, verification and other function enhancement on the data receiving process, and ensure the safety and reliability of the data.
Data parsing strategy: the pass-through data receiving module defines a series of parsing strategies according to different data formats, and the strategies comprise parsing algorithms and rules. Through the application of the strategy mode, the system can select a proper analysis strategy according to configuration or parameters when in operation, and flexibly process the transparent transmission data in different formats.
And (3) conversion of analysis results: and the transparent data receiving module converts the analyzed data into a uniform internal data structure. In this process, the use of proxy mode allows enhanced system functions such as data format verification, data conversion, data merging, etc.
System function enhancement: the transparent data receiving module intercepts and enhances the data receiving process through the proxy mode, and can add additional functional modules such as log record, performance monitoring, exception handling and the like. Therefore, the system can realize unified system function enhancement while receiving data, and the maintainability and expandability of the system are improved.
Further, regarding the transparent data processing module, the following is specifically described:
policy mode: the transparent transmission data processing module adopts a strategy mode to process transparent transmission data in different formats. It defines an abstract policy interface, which each specific data processing policy class implements and is responsible for handling transparent data in a specific format. According to the required processing strategy, the module can select the corresponding strategy to process according to the data format. The new transparent transmission type processing only needs to add a specific strategy class and appoints the strategy in the configuration, and the existing code is not required to be modified.
Template method mode: and the transparent transmission data processing module controls the transparent transmission process of the data by using a template method mode. It defines an abstract template method that normalizes and encapsulates the entire process of processing transparent data and defines some variable steps or hook methods for implementation by specific policy classes. In this way, in the data processing process, the template method can call each step or the hook method according to a predefined sequence, so as to ensure that the processing flow of the transparent data is kept uniform.
Minimum modification principle: the design of the transparent transmission data processing module considers the minimum modification principle when processing different transparent transmission types. By adopting the strategy mode and the template method mode, a new transparent transmission type is newly added, and the corresponding processing strategy class is only needed to be realized, and the strategy is designated in the configuration. Because the module has already defined the processing flow and specification, need not to modify the core code of the module when adding different transmission types to process, in order to achieve the goal of minimum change.
The design of combining the strategy mode and the template method mode of the transparent transmission data processing module enables the module to efficiently and flexibly process transparent transmission data in different formats, and the module is minimally changed when processing different transparent transmission types. This provides convenience for the scalability and maintainability of the system, while also guaranteeing the unification and specification of the pass-through data processing process.
As a preferred embodiment of the present invention, the step of periodically updating the parsing policy based on the structure record table, and converting the original transparent data into the unified format data based on the parsing policy includes:
counting the data times of each data structure and the variation interval thereof;
counting the number of data according to the fluctuation interval to obtain a frequency curve;
analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
updating an analysis strategy when the data characteristics accord with preset data conditions;
and converting the original transparent data into uniform format data based on the latest analysis strategy.
In an example of the technical scheme of the present invention, a specific description is given to an updating process of an analysis policy, where the analysis policy corresponds to each data structure, and when the number of occurrences of data of a certain data structure is large and the variation situation of the data of the certain data structure is severe (reflected by derivative characteristics), the analysis policy is updated.
As a preferred embodiment of the technical scheme of the present invention, the step of monitoring the conversion process of the original transparent transmission data in real time and performing secondary updating on the analysis strategy according to the monitoring result includes:
monitoring the conversion process of original transparent transmission data in real time, and calculating the resource consumption;
comparing the resource consumption according to the similarity of the data structure, and calculating a consumption difference value;
marking an abnormal conversion process according to the consumption difference value and the similarity;
and carrying out secondary updating on the analysis strategy of the abnormal conversion process.
The above content further limits the updating process of the analysis strategy, and the principle is that the conversion process of the original transparent transmission data is monitored in real time, so that how much resources are consumed, namely the resource consumption, can be calculated; on this basis, if the data structures of the two data are similar, the resource consumption of the two data should be similar under the same better analysis strategy, and if the resource consumption of any analysis strategy is higher, the corresponding analysis strategy is not excellent enough, and at this time, the corresponding analysis strategy needs to be updated.
As a preferred embodiment of the technical scheme of the present invention, the step of constructing the transparent channel of the parsing strategy based on the data frequency item includes:
reading a data frequency item in a preset time;
when the data times reach a preset time threshold, a transparent channel corresponding to the analysis strategy is constructed, and one transparent channel corresponds to one analysis strategy.
The above-mentioned content defines the upstream data transparent transmission module, which is specifically as follows:
the upstream data transparent channel docking module is designed by combining a strategy mode and a factory mode, is used for docking with different transparent channels, and can select corresponding strategies for processing according to requirements. The following is a description of the module:
policy mode: the upstream data transparent channel docking module adopts a strategy mode to process different transparent channels. It defines an abstract policy interface, which each specific policy class implements and is responsible for interfacing with a specific transparent channel. According to the required transparent transmission channel, the module can select a corresponding butt joint mode for processing according to a strategy. Therefore, no matter the transparent transmission channel is added, modified or deleted, only the corresponding strategy class is added, modified or deleted, and codes of other parts are not affected.
Factory mode: the upstream data transparent channel docking module adopts a factory mode to uniformly manage the creation of the strategy object. It defines a factory class that is responsible for selecting the corresponding policy object based on the configuration or parameters and returning it to the caller. The factory class can dynamically create different strategy objects according to the needs, so that unified management and flexible switching of strategies are realized.
By adopting the design of combining the strategy mode and the factory mode, the upstream data transparent channel docking module can select corresponding strategies for processing according to different transparent channels, and unified management of strategy objects is realized. Thus, when the transparent channel is newly added, modified or deleted, only the corresponding strategy class and configuration need to be newly added, modified or deleted, and the codes of other parts do not need to be modified. The design can improve the expansibility and maintainability of the system, so that the docking process of the transmission channel is more flexible and controllable.
Referring to fig. 3, in an embodiment of the present invention, a data transparent collision library system 10 based on a policy mode and a proxy mode, where the system 10 includes:
the record table generating module 11 is configured to receive original transparent transmission data, acquire a data structure of the original transparent transmission data, and generate a structure record table; the structure record table comprises a data structure item and a data frequency item;
the format unifying module 12 is configured to update an parsing policy at regular time based on the structure record table, and convert the original transparent data into unified format data based on the parsing policy;
the strategy updating module 13 is used for monitoring the conversion process of the original transparent transmission data in real time and carrying out secondary updating on the analysis strategy according to the monitoring result;
the channel creation module 14 is configured to construct a transparent channel of the parsing strategy based on the data frequency item.
Further, the record table generating module 11 includes:
the node inserting unit is used for establishing a connecting channel with a downstream system and inserting a data enhancement node into the connecting channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
the data processing unit is used for receiving the original transparent transmission data based on the connecting channel and processing the original transparent transmission data through the data enhancement node;
the frequency counting unit is used for acquiring the data structure of the original transparent transmission data, recording the occurrence frequency of various data structures and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
Specifically, the format unification module 12 includes:
the statistics unit is used for counting the data times of each data structure and the variation interval of the data times;
the frequency analysis unit is used for counting the frequency of the data according to the fluctuation interval to obtain a frequency curve;
the curve analysis unit is used for analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
the strategy updating unit is used for updating the analysis strategy when the data characteristics accord with preset data conditions;
and the execution unit is used for converting the original transparent transmission data into uniform format data based on the latest analysis strategy.
Further, the policy updating module 13 includes:
the monitoring and calculating unit is used for monitoring the conversion process of the original transparent transmission data in real time and calculating the resource consumption;
the comparison calculation unit is used for comparing the consumption of the resources according to the similarity of the data structures and calculating a consumption difference value;
a process marking unit for marking an abnormal conversion process according to the consumption difference and the similarity;
and the updating execution unit is used for carrying out secondary updating on the analysis strategy in the abnormal conversion process.
Still further, the channel creation module 14 includes:
the data reading unit is used for reading the data frequency items in the preset time;
and the construction execution unit is used for constructing a transparent channel corresponding to the analysis strategy when the data frequency reaches a preset frequency threshold value, and one transparent channel corresponds to one analysis strategy.
The technical scheme of the invention has the beneficial effects that:
flexibility and scalability of data processing: through the application of the strategy mode, corresponding strategies can be selected for processing according to different data formats, so that compatibility of various data formats is realized. This enables the system to accommodate changing data formats and can quickly expand and support more data formats by adding new policy classes.
Proxy management of system functions: by application of the proxy mode, the proxy object can proxy the system functions, insert new functions, add restrictions, etc. This provides better control and expansion of the system functionality while also enhancing the safety and maintainability of the system.
Unification and standardization of data transparent transmission flow: by applying the template method mode, the flow of data transparent transmission is defined, and some methods requiring subclass realization are reserved. The method ensures that the flow of data transparent transmission has unified sequence and consistency, simplifies the development and maintenance processes, and improves the stability and reliability of the system.
Fast docking new upstream and downstream transparent collision warehouse channel: by applying the factory mode, the quick docking capability of the upstream channel and the downstream channel is realized. The specific factory class creates corresponding channel objects according to different upstream channels, and uniformly manages the upstream channels and the process of butting new channels. The system can quickly adapt to different channel requirements, reduces sensitivity to external system variation, and improves expandability and flexibility of the system.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (10)
1. A method for data transparent collision base based on a policy mode and a proxy mode, the method comprising:
receiving original transparent transmission data, obtaining a data structure of the original transparent transmission data, and generating a structure record table; the structure record table comprises a data structure item and a data frequency item;
updating an analysis strategy based on the structure record table at regular time, and converting the original transparent data into uniform format data based on the analysis strategy;
monitoring the conversion process of the original transparent transmission data in real time, and carrying out secondary updating on the analysis strategy according to the monitoring result;
and constructing a transparent channel of the analysis strategy based on the data frequency item.
2. The method for data transparent collision base based on policy mode and agent mode according to claim 1, wherein the step of receiving original transparent data, obtaining a data structure of the original transparent data, and generating a structure record table comprises:
establishing a connection channel with a downstream system, and inserting a data enhancement node into the connection channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
receiving original transparent transmission data based on a connecting channel, and processing the original transparent transmission data through a data enhancement node;
acquiring a data structure of original transparent transmission data, recording the occurrence times of various data structures, and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
3. The method for data transparent collision base based on policy mode and proxy mode according to claim 1, wherein the step of updating the parsing policy based on the structure record table at regular time and converting the original transparent data into uniform format data based on the parsing policy comprises:
counting the data times of each data structure and the variation interval thereof;
counting the number of data according to the fluctuation interval to obtain a frequency curve;
analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
updating an analysis strategy when the data characteristics accord with preset data conditions;
and converting the original transparent data into uniform format data based on the latest analysis strategy.
4. The method for data transparent collision library based on policy mode and agent mode according to claim 1, wherein the step of monitoring the conversion process of the original transparent data in real time and performing secondary update on the analysis policy according to the monitoring result comprises:
monitoring the conversion process of original transparent transmission data in real time, and calculating the resource consumption;
comparing the resource consumption according to the similarity of the data structure, and calculating a consumption difference value;
marking an abnormal conversion process according to the consumption difference value and the similarity;
and carrying out secondary updating on the analysis strategy of the abnormal conversion process.
5. The method for data transparent collision base based on the policy mode and the agent mode according to claim 1, wherein the step of constructing transparent channels of the resolution policy based on the data frequency item comprises:
reading a data frequency item in a preset time;
when the data times reach a preset time threshold, a transparent channel corresponding to the analysis strategy is constructed, and one transparent channel corresponds to one analysis strategy.
6. A policy mode and agent mode based data transparent library system, the system comprising:
the record table generation module is used for receiving the original transparent transmission data, acquiring a data structure of the original transparent transmission data and generating a structure record table; the structure record table comprises a data structure item and a data frequency item;
the format unification module is used for updating the analysis strategy based on the structure record table at regular time and converting the original transparent data into unified format data based on the analysis strategy;
the strategy updating module is used for monitoring the conversion process of the original transparent transmission data in real time and carrying out secondary updating on the analysis strategy according to the monitoring result;
and the channel creation module is used for constructing a transparent channel of the analysis strategy based on the data frequency item.
7. The policy mode and agent mode based data transparent collision library system of claim 6, wherein the record table generation module comprises:
the node inserting unit is used for establishing a connecting channel with a downstream system and inserting a data enhancement node into the connecting channel; the data enhancement node is used for intercepting, recording and verifying original transparent transmission data;
the data processing unit is used for receiving the original transparent transmission data based on the connecting channel and processing the original transparent transmission data through the data enhancement node;
the frequency counting unit is used for acquiring the data structure of the original transparent transmission data, recording the occurrence frequency of various data structures and generating a structure record table;
in the process of acquiring the data structure of the original transparent data, the performance parameters of the channel are monitored in real time and a log is generated.
8. The policy mode and agent mode based data transparent collision library system of claim 6, wherein the format unification module comprises:
the statistics unit is used for counting the data times of each data structure and the variation interval of the data times;
the frequency analysis unit is used for counting the frequency of the data according to the fluctuation interval to obtain a frequency curve;
the curve analysis unit is used for analyzing the frequency curve to obtain data characteristics; the data features include multi-order derivative features;
the strategy updating unit is used for updating the analysis strategy when the data characteristics accord with preset data conditions;
and the execution unit is used for converting the original transparent transmission data into uniform format data based on the latest analysis strategy.
9. The policy mode and agent mode based data transparent collision library system of claim 6, wherein the policy update module comprises:
the monitoring and calculating unit is used for monitoring the conversion process of the original transparent transmission data in real time and calculating the resource consumption;
the comparison calculation unit is used for comparing the consumption of the resources according to the similarity of the data structures and calculating a consumption difference value;
a process marking unit for marking an abnormal conversion process according to the consumption difference and the similarity;
and the updating execution unit is used for carrying out secondary updating on the analysis strategy in the abnormal conversion process.
10. The policy mode and agent mode based data transparent collision library system of claim 6, wherein the channel creation module comprises:
the data reading unit is used for reading the data frequency items in the preset time;
and the construction execution unit is used for constructing a transparent channel corresponding to the analysis strategy when the data frequency reaches a preset frequency threshold value, and one transparent channel corresponds to one analysis strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311334283.9A CN117390021A (en) | 2023-10-14 | 2023-10-14 | Data transparent transmission collision library system and method based on strategy mode and proxy mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311334283.9A CN117390021A (en) | 2023-10-14 | 2023-10-14 | Data transparent transmission collision library system and method based on strategy mode and proxy mode |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117390021A true CN117390021A (en) | 2024-01-12 |
Family
ID=89469569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311334283.9A Pending CN117390021A (en) | 2023-10-14 | 2023-10-14 | Data transparent transmission collision library system and method based on strategy mode and proxy mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117390021A (en) |
-
2023
- 2023-10-14 CN CN202311334283.9A patent/CN117390021A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4729262B2 (en) | Location awareness architecture and systems | |
| JP3289605B2 (en) | Hardware resource management module common method | |
| US9717011B2 (en) | Event management in telecommunications networks | |
| CN111712813A (en) | Intelligent preprocessing of multi-dimensional time series data | |
| US20080059436A1 (en) | Providing a visual representation of a sub-set of a visual program | |
| US7523492B2 (en) | Secure gateway with proxy service capability servers for service level agreement checking | |
| CN110880146A (en) | Block chain chaining method, device, electronic equipment and storage medium | |
| US7130898B2 (en) | Mechanism for facilitating invocation of a service | |
| WO2017007981A1 (en) | Action correlation framework | |
| US11552868B1 (en) | Collect and forward | |
| CN102090039B (en) | A method of performing data mediation, and an associated computer program product, data mediation device and information system | |
| CN112732731B (en) | Method and device for consuming service data, electronic equipment and readable storage medium | |
| CN116991929A (en) | Micro-service system based on big hospital data | |
| CN117390021A (en) | Data transparent transmission collision library system and method based on strategy mode and proxy mode | |
| CN106844156B (en) | WCF communication message interception method in intelligent building integrated management platform system | |
| CN114936249A (en) | File analysis and conversion method, device, equipment and readable storage medium | |
| CN1932760B (en) | System and method for execution of application program | |
| Hormann et al. | Parsing and extracting features from opc unified architecture in industrial environments | |
| da Silva Rocha et al. | Aggregating data center measurements for availability analysis | |
| KR20210066595A (en) | Edge Device for Interworking with Heterogeneous Legacy Protocols in Smart Factory | |
| KR102441348B1 (en) | Method for generating standard message and transceiving tactical data in tactical data link using standard message dynamic define algorithm | |
| CN117971834A (en) | Collaborative storage method and system based on blockchain | |
| LOPEZ-MUÑOZ | Unified Namespaces in Industrial IoT architectures. | |
| CN113259233A (en) | DDS-based automation gateway of heterogeneous communication protocol | |
| CN117331617A (en) | Data interaction method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |