CN114401202A - Data cycle monitoring method and storage medium - Google Patents

Data cycle monitoring method and storage medium Download PDF

Info

Publication number
CN114401202A
CN114401202A CN202111491566.5A CN202111491566A CN114401202A CN 114401202 A CN114401202 A CN 114401202A CN 202111491566 A CN202111491566 A CN 202111491566A CN 114401202 A CN114401202 A CN 114401202A
Authority
CN
China
Prior art keywords
data
packet
processing
log
identification code
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111491566.5A
Other languages
Chinese (zh)
Inventor
赵立
倪时龙
李仕镇
叶建军
翁斌
韩晓光
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Gemean Beijing Information Technology Co ltd
Original Assignee
Gemean Beijing Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Gemean Beijing Information Technology Co ltd filed Critical Gemean Beijing Information Technology Co ltd
Priority to CN202111491566.5A priority Critical patent/CN114401202A/en
Publication of CN114401202A publication Critical patent/CN114401202A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/04Processing captured monitoring data, e.g. for logfile generation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00Packet switching elements
    • H04L49/90Buffering arrangements
    • H04L49/9057Arrangements for supporting packet reassembly or resequencing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

A data cycle monitoring method and storage medium, wherein the method includes the following steps, when the upper end establishes the data packet, produce the correspondent identification code of the said data packet, record the said identification code; recombining the data packet to obtain a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet; the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end; and tracing the life cycle of the processing packet according to the log. The method and the device can generate the unique identification code corresponding to the data packet when the data packet is created at the upper end, can also generate the identification code of the corresponding displacement to the recombined processing packet when the recombination is required, and can trace the source of the life cycle of the processing packet according to the log of the identification code when the data at the upper end is transmitted to the lower end. Finally, the effect of looking up the source of the data packet is achieved.

Description

Data cycle monitoring method and storage medium
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and a storage medium for tracing and monitoring data during data transmission.
Background
For a cross-network, cross-border and cross-domain data security transmission system, how to ensure the security and stability of a data transmission process is a problem of great concern, a log mode is generally adopted to record the processing result of data in the whole transmission process, transmission logs among different areas are synchronized, and then the logs are connected in series through data packet names, so that the whole data transmission is monitored. In fact, in some service scenarios, a single data packet is not transmitted, sometimes a plurality of data packets need to be processed, aggregated and then transmitted according to service requirements, and sometimes a large data packet needs to be split and then transmitted respectively to ensure transmission efficiency and performance. In this case, the data transmission process of the conventional log record will have a "fault" phenomenon, and the life cycle and transmission process of the data packet cannot be completely tracked.
Disclosure of Invention
Therefore, it is necessary to provide a method capable of monitoring generation and subsequent change of data, so as to meet the requirement of tracing the source of the data transmitted by the network.
To achieve the above object, the inventors provide a data cycle monitoring method, comprising the steps of,
when a data packet is created at an upper end, generating an identification code corresponding to the data packet, and recording the identification code;
recombining the data packet to obtain a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet;
the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end;
and tracing the life cycle of the processing packet according to the log.
In a specific embodiment, the method further comprises the step of displaying the life cycle of the processing packet according to the log.
Specifically, the step of recombining the data packets includes integrating a plurality of data packets to obtain a processing packet for combining a plurality of data packets.
Specifically, the step of reconstructing the data packet includes splitting the data packet to obtain a processing packet after splitting the data packet.
Furthermore, the log is also used for recording the state and the abnormal information of the data packet or the processing packet, and the data transmission efficiency is evaluated according to the log.
A data cycle monitoring storage medium storing a computer program which when executed performs the steps of:
when a data packet is created at an upper end, generating an identification code corresponding to the data packet, and recording the identification code;
when the data packet is recombined, obtaining a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet;
the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end;
and tracing the life cycle of the processing packet according to the log.
In particular, the computer program when executed further performs a step comprising presenting the life cycle of the processing package according to the log.
Specifically, the computer program, when executed, performs reassembly of the data packets further includes a step of integrating a plurality of data packets to obtain a processing packet for merging the plurality of data packets.
Specifically, the computer program, when executed, performs reassembly of the data packet, and further includes a step of splitting the data packet to obtain a processing packet after splitting the data packet.
Furthermore, the log is also used for recording the state and the abnormal information of the data packet or the processing packet, and the data transmission efficiency is evaluated according to the log.
Through the scheme, the unique identification code corresponding to the data packet can be generated when the data packet is created at the upper end, the identification code corresponding to the displacement can be generated for the recombined processing packet when the data packet needs to be recombined, and when the data at the upper end is transmitted to the lower end, the life cycle of the processing packet can be traced according to the log of the identification code. Finally, the effect of looking up the source of the data packet is achieved.
Drawings
FIG. 1 is a flow chart of a data cycle monitoring method according to an embodiment;
FIG. 2 is a flow chart illustrating data cycle monitoring according to an embodiment;
FIG. 3 is a schematic diagram of a data package according to an embodiment;
FIG. 4 is a diagram illustrating packet splitting according to an embodiment;
fig. 5 is a schematic diagram of a data cycle monitoring storage medium according to an embodiment.
Detailed Description
To explain technical contents, structural features, and objects and effects of the technical solutions in detail, the following detailed description is given with reference to the accompanying drawings in conjunction with the embodiments.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or related to other embodiments specifically defined. In principle, in the present application, the technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution as long as there is no technical contradiction or conflict.
Unless defined otherwise, technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the use of relational terms herein is intended only to describe particular embodiments and is not intended to limit the present application.
In the description of the present application, the term "and/or" is a expression for describing a logical relationship between objects, meaning that three relationships may exist, for example a and/or B, meaning: there are three cases of A, B, and both A and B. In addition, the character "/" herein generally indicates that the former and latter associated objects are in a logical relationship of "or".
In this application, terms such as "first" and "second" are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
Without further limitation, in this application, the use of "including," "comprising," "having," or other similar expressions in phrases and expressions of "including," "comprising," or "having," is intended to cover a non-exclusive inclusion, and such expressions do not exclude the presence of additional elements in a process, method, or article that includes the recited elements, such that a process, method, or article that includes a list of elements may include not only those elements but also other elements not expressly listed or inherent to such process, method, or article.
As is understood in the examination of the guidelines, the terms "greater than", "less than", "more than" and the like in this application are to be understood as excluding the number; the expressions "above", "below", "within" and the like are understood to include the present numbers. In addition, in the description of the embodiments of the present application, "a plurality" means two or more (including two), and expressions related to "a plurality" similar thereto are also understood, for example, "a plurality of groups", "a plurality of times", and the like, unless specifically defined otherwise.
In the description of the embodiments of the present application, spatially relative expressions such as "central," "longitudinal," "lateral," "length," "width," "thickness," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used, and the indicated orientations or positional relationships are based on the orientations or positional relationships shown in the specific embodiments or drawings and are only for convenience of describing the specific embodiments of the present application or for the convenience of the reader, and do not indicate or imply that the device or component in question must have a specific position, a specific orientation, or be constructed or operated in a specific orientation and therefore should not be construed as limiting the embodiments of the present application.
Unless specifically stated or limited otherwise, the terms "mounted," "connected," "secured," and "disposed" used in the description of the embodiments of the present application are to be construed broadly. For example, the connection can be a fixed connection, a detachable connection, or an integrated arrangement; it can be a mechanical connection, an electrical connection, or a communication connection; they may be directly connected or indirectly connected through an intermediate; which may be communication within two elements or an interaction of two elements. Specific meanings of the above terms in the embodiments of the present application can be understood by those skilled in the art to which the present application pertains in accordance with specific situations.
In the embodiment shown in fig. 1, which is a flow chart of a data cycle monitoring method, it can be seen that the method includes the steps of,
s1, when a data packet is created at the upper end, generating an identification code corresponding to the data packet, and recording the identification code;
s2, the data packet is recombined to obtain a recombined processing packet, an identification code corresponding to the processing packet is generated, and the identification code corresponding to the processing packet is recorded;
s3, the upper end transmits the processing packet to the lower end, and the log recorded by the upper end and the log of the lower end are synchronized;
s4, tracing the life cycle of the processing packet according to the log.
In the scheme, the upper end can possibly generate data in the working process, when transmission is needed, data packets can be generated according to a transmission protocol, the upper end can generate an identification code corresponding to each data packet, the data packets can be searched and traced according to the identification codes, log recording can be performed when the upper end generates the data packets, and log information can include one or more of data processing time, unique identification codes, time consumption, states, exceptions and the like. When the data packets need to be combined or split, the data packets can be integrated or split according to requirements to obtain processing packets, and when the processing packets are generated, log recording can be performed, wherein log recording information can include one or more of generation time, unique identification codes, time consumption, states, exceptions and the like of the processing packets. In some embodiments, the data packet or the processing packet may be data or a corresponding service component, etc., and the content of the data packet is determined according to the logic of packetization. Through the scheme, the unique identification code corresponding to the data packet can be generated when the data packet is created at the upper end, the corresponding unique identification code can be generated for the recombined processing packet when the data packet needs to be recombined, and when the data at the upper end is transmitted to the lower end, the life cycle of the processing packet can be traced according to the log of the identification code. Finally, the effect of looking up the source of the data packet is achieved.
In other specific embodiments, as shown in fig. 2, the method further includes a step of displaying, by S5, a life cycle of the processing package according to the log. One or more of the generation time, the unique identification code, the elapsed time, the status, the abnormality, and the like of the processing packet are recorded in the log. The presentation interface may present the information and, in a further embodiment, the data packet from which the processing packet originated may also be displayed. It should be noted that the processing packet received by the lower end may also be a new data packet, and the associated reassembly data that is passed to its own lower end may also be recorded as a new processing packet. Therefore, the life cycle display can include a multi-stage recombination process, and each recombined new processing packet has a corresponding number, so that records such as creation time of the earliest data can be traced back and the like until the upper-level data packet is continuously displayed according to the log. Through the scheme, the user can visually obtain the display of the contents such as the packaging process of the data packet, the user can extract data as required, and the user experience is improved.
In the specific embodiment shown in fig. 3, the step of reassembling the data packets includes integrating a plurality of data packets, so as to obtain a processing packet combining a plurality of data packets, S21. When the merged processing packet is generated, log recording may also be performed, and the log recording information may include one or more of the synthesis time of the processing packet, the unique identification code corresponding to the processing packet, time consumption, status, exception, and the like. In some embodiments, the data packet or the processing packet may be data or a corresponding service component, and the content of the merged processing packet is determined according to the logic of different merged packets. Through the scheme, the unique identification code corresponding to the data packet can be generated when the data packet is created at the upper end, the corresponding unique identification code can be generated for the merged processing packet when merging is needed, and when the data at the upper end is transmitted to the lower end, the life cycle of the merged processing packet can be traced according to the log of the identification code. Finally, the effect of looking up the source of the data packet is achieved.
In the specific embodiment shown in fig. 4, the step of reassembling the data packet includes splitting the data packet, and obtaining a processed packet after splitting the data packet, in S22. When the split processing packet is generated, log recording may also be performed, and the log recording information may include one or more of synthesis time of the processing packet, a unique identification code corresponding to the processing packet, time consumption, a state, an exception, and the like. In some embodiments, the data packet or the processing packet may be data or a corresponding service component, and the content of the split processing packet is determined according to the logic of different combined packets. Through the scheme, when the data packet is created at the upper end, the unique identification code corresponding to the data packet can be generated, when the data packet needs to be split, the corresponding unique identification code can be generated for the split processing packet, and when the data at the upper end is transmitted to the lower end, the life cycle of the split processing packet can be traced according to the log of the identification code. Finally, the effect of looking up the source of the data packet is achieved.
In some other further embodiments, the log is further used for recording status and exception information of the data packets or the processing packets, and performing data transmission efficiency evaluation according to the log. In some embodiments, a step may also be performed of verifying integrity of the received data packet or the processed packet. Data encryption can also be performed when the data packet is created or generated, and data verification can be performed in the subsequent process. If the subsequent data verification is illegal, the data transmission process can be determined to be abnormal. In a further embodiment, if a plurality of processing packets obtained after a plurality of transmissions have a traceability relationship with the same initial data packet, and a plurality of processing packets have a phenomenon of data transmission abnormality, the efficiency of data transmission after multi-stage transmission can be determined according to the proportion of the processing packets with data transmission abnormality in the processing packets having the same traceability relationship. In other embodiments, after the processing packets are subjected to a predetermined finishing procedure, such as being legally destroyed and deleted, the processing packets are regarded as the life cycle of the data, and the proportion of the abnormal processing packets, which are illegal because of being unusable or verified, is calculated, so that the efficiency of the data transmission after the multi-stage transmission of the whole system can be determined. The transmission efficiency of data transmission in a data transmission period is evaluated by the above-described scheme. The technical effect of evaluating the data processing accuracy of the system is achieved.
In other specific embodiments, in order to simplify the operation of the log burying point and standardize the log output, we use the idea of AOP (aspect-oriented programming) to add a pre-method and a post-method to each packet processing process without changing the logic of the original data transmission process processing method. For this, we first define a standardized log proxy object, and the log information includes data processing time, unique batch number, time consumption, status, exception and other information; secondly, performing componentization splitting on the whole data transmission process, splitting each data processing component (or service), enabling the components to inherit the same parent class, and completely handing the log processing of the prepositive method and the postpositive method to the parent class; finally, all data processing components can complete the log point burying operation only by transmitting some necessary parameters; and finally, carrying out statistical analysis on the data transmission logs, comprehensively evaluating the quality and efficiency of data exchange, and comprehensively monitoring the life cycle of the data from point to surface and from surface to point.
In the embodiment shown in fig. 5, the present application further discloses a data cycle monitoring storage medium 5 storing a computer program which, when executed, performs the steps of:
when a data packet is created at an upper end, generating an identification code corresponding to the data packet, and recording the identification code;
when the data packet is recombined, obtaining a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet;
the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end;
and tracing the life cycle of the processing packet according to the log.
Through the storage medium, when a data packet is created at an upper end, a unique identification code corresponding to the data packet can be generated, when the data packet needs to be recombined, a corresponding displacement identification code can be generated for the recombined processing packet, and when the data at the upper end is transmitted to a lower end, the life cycle of the processing packet can be traced according to a log of the identification code. Finally, the effect of looking up the source of the data packet is achieved.
In particular, the computer program when executed further performs a step comprising presenting the life cycle of the processing package according to the log.
Specifically, the computer program, when executed, performs reassembly of the data packets further includes a step of integrating a plurality of data packets to obtain a processing packet for merging the plurality of data packets.
Specifically, the computer program, when executed, performs reassembly of the data packet, and further includes a step of splitting the data packet to obtain a processing packet after splitting the data packet.
Furthermore, the log is also used for recording the state and the abnormal information of the data packet or the processing packet, and the data transmission efficiency is evaluated according to the log.
It should be noted that, although the above embodiments have been described herein, the invention is not limited thereto. Therefore, based on the innovative concepts of the present invention, the technical solutions of the present invention can be directly or indirectly applied to other related technical fields by making changes and modifications to the embodiments described herein, or by using equivalent structures or equivalent processes performed in the content of the present specification and the attached drawings, which are included in the scope of the present invention.

Claims (10)

1. A data cycle monitoring method is characterized by comprising the following steps,
when a data packet is created at an upper end, generating an identification code corresponding to the data packet, and recording the identification code;
recombining the data packet to obtain a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet;
the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end;
and tracing the life cycle of the processing packet according to the log.
2. The data cycle monitoring method of claim 1, further comprising the step of exposing a life cycle of the processing package according to the log.
3. The data cycle monitoring method of claim 1, wherein the step of reassembling the data packets includes the step of integrating a plurality of data packets to obtain a processing packet that merges a plurality of data packets.
4. The data cycle monitoring method of claim 1, wherein the step of reassembling the data packets includes splitting the data packets to obtain the split processing packets.
5. The data cycle monitoring method of claim 1, wherein the log is further used for recording status and exception information of the data packets or the processing packets, and performing data transmission efficiency evaluation according to the log.
6. A data cycle monitoring storage medium, storing a computer program that, when executed, performs the steps of:
when a data packet is created at an upper end, generating an identification code corresponding to the data packet, and recording the identification code;
when the data packet is recombined, obtaining a recombined processing packet, generating an identification code corresponding to the processing packet, and recording the identification code corresponding to the processing packet;
the upper end transmits the processing packet to the lower end, and synchronizes the log recorded by the upper end with the log of the lower end;
and tracing the life cycle of the processing packet according to the log.
7. The data cycle monitoring storage medium of claim 6, wherein the computer program when executed further performs the step of exposing a life cycle of the processing package from the log.
8. The data cycle monitoring storage medium of claim 6, wherein the computer program when executed performs the step of reorganizing the data packets further comprising the step of integrating a plurality of data packets to obtain a processing packet that merges a plurality of data packets.
9. The data cycle monitoring storage medium of claim 6, wherein the computer program when executed performs reassembly of the data packets further comprising the step of splitting the data packets to obtain the processed packets after splitting the data packets.
10. The data cycle monitoring storage medium of claim 6, wherein the log is further configured to record status and exception information of the data packets or the processing packets, and perform data transmission efficiency evaluation according to the log.
CN202111491566.5A 2021-12-08 2021-12-08 Data cycle monitoring method and storage medium Pending CN114401202A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111491566.5A CN114401202A (en) 2021-12-08 2021-12-08 Data cycle monitoring method and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111491566.5A CN114401202A (en) 2021-12-08 2021-12-08 Data cycle monitoring method and storage medium

Publications (1)

Publication Number Publication Date
CN114401202A true CN114401202A (en) 2022-04-26

Family

ID=81226546

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111491566.5A Pending CN114401202A (en) 2021-12-08 2021-12-08 Data cycle monitoring method and storage medium

Country Status (1)

Country Link
CN (1) CN114401202A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120159098A1 (en) * 2010-12-17 2012-06-21 Microsoft Corporation Garbage collection and hotspots relief for a data deduplication chunk store
CN104778420A (en) * 2015-04-24 2015-07-15 广东电网有限责任公司信息中心 Method for establishing safety management view of full life cycle of unstructured data
CN105721193A (en) * 2014-12-05 2016-06-29 方正国际软件(北京)有限公司 Method and device for system information monitoring
CN113065127A (en) * 2021-02-24 2021-07-02 山东英信计算机技术有限公司 Database protection method, system and medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120159098A1 (en) * 2010-12-17 2012-06-21 Microsoft Corporation Garbage collection and hotspots relief for a data deduplication chunk store
CN105721193A (en) * 2014-12-05 2016-06-29 方正国际软件(北京)有限公司 Method and device for system information monitoring
CN104778420A (en) * 2015-04-24 2015-07-15 广东电网有限责任公司信息中心 Method for establishing safety management view of full life cycle of unstructured data
CN113065127A (en) * 2021-02-24 2021-07-02 山东英信计算机技术有限公司 Database protection method, system and medium

Similar Documents

Publication Publication Date Title
EP3373516B1 (en) Method and device for processing service calling information
Wen et al. A novel approach for process mining based on event types
Amyot et al. User requirements notation: the first ten years, the next ten years
Maggi et al. Monitoring business constraints with linear temporal logic: An approach based on colored automata
Van der Aalst et al. Replaying history on process models for conformance checking and performance analysis
van der Aalst et al. Process mining: A two-step approach using transition systems and regions
CN110377336A (en) Interface document generation method, device, computer equipment and storage medium
CN106657192B (en) Method and equipment for presenting service calling information
US10664382B2 (en) System and method for tool chain data capture through parser for empirical data analysis
CN101819551B (en) Method and system for recording service interaction data of enterprise message system
Philipps et al. Model-based test case generation for smart cards
Andrés et al. Formal passive testing of timed systems: Theory and tools
US20150286555A1 (en) System and method for converting the business processes to test-centric activity diagrams
Van der Aalst Federated process mining: exploiting event data across organizational boundaries
CN108460068A (en) Method, apparatus, storage medium and the terminal that report imports and exports
CN110413507A (en) System detection method, device, computer equipment and storage medium
Al-Far et al. Measuring impact score on confidentiality, integrity, and availability using code metrics
Wilberg et al. Data analytics in product development: Implications from expert interviews
Benharrat et al. Constraint-based oracles for timed distributed systems
US11860858B1 (en) Decoding distributed ledger transaction records
Pan et al. A new reliability evaluation method for networks with imperfect vertices using BDD
CN114401202A (en) Data cycle monitoring method and storage medium
Losavio et al. Reference architecture design using domain quality view
KR20220115859A (en) Edge table representation of the process
Wang et al. Scenario-based method for business process analysis and improvement in SOA

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination