CN111447170A

CN111447170A - Data processing method and system, computer system and computer readable medium

Info

Publication number: CN111447170A
Application number: CN201910047350.6A
Authority: CN
Inventors: 刘爱慧
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-01-17
Filing date: 2019-01-17
Publication date: 2020-07-24
Anticipated expiration: 2039-01-17
Also published as: CN111447170B

Abstract

The present disclosure provides a data processing method, including: acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server; determining an acquisition strategy corresponding to a transmission protocol based on at least the transmission protocol; processing data to be acquired based on an acquisition strategy to obtain target data conforming to a specified format; and storing the target data by the target server. In addition, the present disclosure also discloses a data processing system, a computer system and a computer readable medium.

Description

Data processing method and system, computer system and computer readable medium

Technical Field

The present disclosure relates to the field of data processing, and more particularly, to a data processing method and system, a computer system, and a computer readable medium.

Background

Currently, data transmission protocols on networks mainly adopt HTTP and direct socket (TCP) (HTTP is also based on TCP, but is still considered as two types in terms of use), and based on the two transmission protocols, various companies perform various data encapsulation and develop various data protocols (also referred to as frames in the disclosure and separated from the transmission protocols), so that although the protocols are consistent, the data formats of transmission are different, and even the same company has different frames. In the scenes that transmitted data need to be collected for request playback, data analysis and the like, different requirements naturally occur for data collection monitoring and joint debugging tests according to different protocols and frames, different processing needs to be performed, for example, different transmission modes are obviously required, and the intermediate transmission data collection modes are different due to different frames and even different protocols used by the same frame under the same protocol.

However, in implementing the concept of the present disclosure, the inventors found that at least the following problems exist in the related art: in the related art, it takes a lot of work in the case where data of all protocols needs to be collected, analyzed and used, such as the requirement of testing.

In view of the above problems in the related art, no effective solution has been proposed at present.

Disclosure of Invention

In view of the above, the present disclosure provides a data processing method and system, a computer system and a computer readable storage medium.

One aspect of the present disclosure provides a data processing method, including: acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server; determining an acquisition strategy corresponding to the transmission protocol at least based on the transmission protocol; processing the data to be acquired based on the acquisition strategy to obtain target data conforming to a specified format; and storing the target data by the target server.

According to an embodiment of the present disclosure, the determining, based on at least the transmission protocol, an acquisition policy corresponding to the transmission protocol includes: detecting whether the transmission protocol is based on an HTTP (hyper text transport protocol); under the condition that the transmission protocol is not based on an HTTP (hyper text transport protocol), acquiring a data format of the data to be transmitted; detecting whether the data format meets a preset condition or not to obtain a detection result; and determining an acquisition strategy corresponding to the transmission protocol at least based on the detection result.

According to an embodiment of the present disclosure, the data to be collected includes transmission data and return data, and the determining, based on at least the detection result, a collection policy corresponding to the transmission protocol includes: under the condition that the detection result shows that the data format meets the preset condition, inserting a first interceptor into the client or inserting the first interceptor into the server; and determining the sending data and the return data acquired by the first interceptor as an acquisition strategy corresponding to the transmission protocol.

According to an embodiment of the present disclosure, the determining, based on at least the detection result, an acquisition policy corresponding to the transmission protocol includes: under the condition that the detection result shows that the data format does not meet the preset condition, inserting a second interceptor and a third interceptor into the client; and determining the sending data acquired by the second interceptor and the return data acquired by the third interceptor as the acquisition strategy corresponding to the transmission protocol.

According to an embodiment of the present disclosure, the specified format includes a string format, and the processing the data to be collected based on the collection policy to obtain the target data conforming to the specified format includes: and analyzing the data to be acquired to obtain the target data which accords with the character string format.

Another aspect of the present disclosure provides a data processing system comprising: the acquisition module is used for acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server; the determining module is used for determining an acquisition strategy corresponding to the transmission protocol based on the transmission protocol; the processing module is used for processing the data to be acquired based on the acquisition strategy to obtain target data conforming to a specified format; and the storage module is used for storing the target data through the target server.

According to an embodiment of the present disclosure, the determining module includes: a first detection unit, configured to detect whether the transport protocol is based on an HTTP transport protocol; an obtaining unit, configured to obtain a data format of the to-be-transmitted data when the transmission protocol is not based on an HTTP transmission protocol; the second detection unit is used for detecting whether the data format meets a preset condition or not so as to obtain a detection result; and a determining unit, configured to determine, based on the detection result, an acquisition policy corresponding to the transmission protocol.

According to an embodiment of the present disclosure, the determining unit includes: a first inserting subunit, configured to insert a first interceptor at the client or insert a first interceptor at the server, when the detection result indicates that the data format meets a preset condition; and a first determining subunit, configured to determine, as the acquisition policy corresponding to the transmission protocol, the transmission data and the return data acquired by the first interceptor.

According to an embodiment of the present disclosure, the determining unit includes: the second inserting subunit is used for inserting a second interceptor and a third interceptor into the client under the condition that the detection result shows that the data format does not meet the preset condition; and a second determining subunit, configured to determine, as an acquisition policy corresponding to the transmission protocol, the transmission data acquired by the second interceptor and the return data acquired by the third interceptor.

According to an embodiment of the present disclosure, the specified format includes a string format, and the processing the data to be collected based on the collection policy to obtain the target data conforming to the specified format includes: and the analysis module is used for analyzing the data to be acquired so as to obtain the target data which accords with the character string format.

Another aspect of the disclosure provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any of the above.

Another aspect of the present disclosure provides a computer system comprising: one or more processors; a storage device to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of the above.

According to the embodiment of the disclosure, at least based on the transmission protocol, the acquisition strategy corresponding to the transmission protocol is determined, the data to be acquired based on the acquisition strategy is processed to obtain the target data conforming to the specified format, and the target data is stored by the target server, so that the technical problem that a large amount of work is consumed under the condition that data of all protocols and frames need to be acquired, analyzed and used in the related art can be at least partially overcome, the data of different protocols can be intensively acquired and processed, and are intensively stored and read, so that a unified platform and service customization can be built based on the data.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

FIG. 1A schematically illustrates an exemplary architecture of a data processing method and data processing system according to an embodiment of the present disclosure;

FIG. 1B schematically illustrates an application scenario of a data processing method and a data processing system according to an embodiment of the present disclosure;

FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;

fig. 3A schematically illustrates a flow chart for determining an acquisition policy corresponding to a transmission protocol based at least on the transmission protocol, in accordance with an embodiment of the present disclosure;

fig. 3B schematically shows a flow chart of data processing in the related art;

FIG. 3C schematically shows a flow diagram of data processing according to an embodiment of the disclosure;

fig. 3D schematically illustrates a flow diagram for determining an acquisition policy corresponding to a transmission protocol based at least on the transmission protocol, according to another embodiment of the present disclosure;

FIG. 3E schematically shows a data processing process according to another embodiment of the present disclosure;

FIG. 3F schematically illustrates a flow diagram for determining an acquisition policy corresponding to a transmission protocol based at least on the transmission protocol, in accordance with yet another embodiment of the present disclosure;

FIG. 3G schematically shows a data processing process according to yet another embodiment of the present disclosure;

FIG. 4 schematically shows a block diagram of a data processing system according to an embodiment of the present disclosure;

FIG. 5A schematically illustrates a block diagram of a determination module according to an embodiment of the disclosure;

FIG. 5B schematically shows a block diagram of a determination unit according to an embodiment of the present disclosure;

FIG. 5C schematically shows a block diagram of a determining unit according to another embodiment of the present disclosure; and

FIG. 6 schematically shows a block diagram of a computer system suitable for implementing the data processing method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

The present disclosure provides a data processing method, including: acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server; determining an acquisition strategy corresponding to a transmission protocol based on at least the transmission protocol; processing data to be acquired based on an acquisition strategy to obtain target data conforming to a specified format; and storing the target data by the target server.

FIG. 1A schematically illustrates an exemplary architecture 100 of a data processing method and data processing system according to embodiments of the disclosure. It should be noted that fig. 1A is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1A, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The data processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in FIG. 1A are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 1B schematically illustrates an application scenario of a data processing method and a data processing system according to an embodiment of the present disclosure.

The data processing method and the data processing system of the embodiment of the disclosure are suitable for any application scene needing to collect and process data transmitted in a network. For example, the transmitted data may be collected, the collected data may be used for requested playback, data analysis, and the like, and particularly, the data for different protocols and frameworks may be collected and processed centrally, and stored and read centrally.

It is to be noted that the present disclosure relates to terms including:

HTTP: is a client and server side standard for requests and responses (TCP).

SOCKET: two programs on the network exchange data via a bidirectional communication link, one end of which is called a SOCKET.

RPC: rpc (remote Procedure call), a remote Procedure call, is a protocol that requests services from a remote computer program over a network without knowledge of the underlying network technology.

As shown in fig. 1B, in the figure, data a and data B respectively represent different data transmission frames, according to the data processing method provided by the present disclosure, different data sources, such as HTTP, a SOCKET (TCP) or B SOCKET data, can capture the transmitted data by a method provided by a proxy mode or a Remote Procedure Call (RPC) architecture itself, and analyze the transmitted data (to obtain HTTP request data or deserialization, etc.) to obtain readable data, which is uploaded to a data access platform through a unified data access interface, so as to be stored and displayed conveniently.

Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the method includes operations S210 to S240. Wherein:

in operation S210, a transmission protocol of data to be acquired is acquired.

In operation S220, an acquisition policy corresponding to a transmission protocol is determined based on at least the transmission protocol.

In operation S230, data to be collected, which is collected based on the collection policy, is processed to obtain target data conforming to a specified format.

In operation S240, target data is stored by the target server.

According to the embodiment of the disclosure, the data to be collected is data transmitted between the client and the server. The method comprises the steps of determining an acquisition strategy corresponding to a transmission protocol at least according to the acquired transmission protocol of data to be acquired, determining an acquisition strategy corresponding to the transmission protocol based on a transmission frame of the data to be acquired, and determining an acquisition strategy corresponding to the transmission protocol based on the transmission protocol and the transmission frame of the data to be acquired, wherein the disclosure is not limited.

According to the embodiment of the disclosure, different acquisition modes can be adopted for different protocols and/or frameworks, but the finally collected data is serialized content, such as JSON character strings can be uniformly adopted, and even for different RPC frameworks, the finally collected data can be converted into the same fixed format. And finally, uploading the collected data content to a specified server through a uniform data access interface of the storage terminal. The target server may be a WEB server, and may read data from the database or store data in the database.

It should be noted that the same protocol or the same framework may have different acquisition implementations, and the implemented programming languages may be different, and may even be implemented at the API level of the operating system. What this scheme wants to protect is that the collection of multiple protocols and/or frameworks supports and data format concentrate this method itself.

According to the embodiment of the disclosure, target data conforming to a specified format can be uploaded through a unified data access interface and uploaded to a unified data access platform, such as a target server, in a data application scene, the target data can be used as a data provider, when an agent receives a request, response data of the same request can be directly read out through the unified interface and serialized, and input data which is completely consistent with the captured input data can be obtained; the data can be delivered to the network through the proxy service, the response request initiator plays a role of a data stub, so that the response can be obtained even if the corresponding service does not exist or is unavailable, and the data can be recorded for verification of the data or for calling of a tracking interface and the like.

Specifically, the storage end is a standard WEB service, provides a REST interface for uploading request data, the acquisition end may be implemented by using different languages, but the data format is uniform, the acquisition end completes acquisition and directly calls the following interface for uploading, and the storage end is unrelated to the protocol framework.

According to an embodiment of the present disclosure, the definition of the interface:

url：/api/bridge/collect

parameters of the interface:

type: data transfer protocol, http/rpc.

Framework: a frame. http is empty, RPC fills in a different RPC frame.

Serialize: the serialization scheme, if any, is written on, such as hessian, msgpack, protobuf, etc. of JSF.

Request: request content, HTTP or RPC request data.

Response: response content, response data of HTTP or RPC.

Responding:

code: response code, 0 is successful.

Msg: and responding to the description, and if the response is wrong, determining that the response is wrong content.

Data: the specific response data.

The uploaded content will be saved to the database for subsequent reading.

It can be understood that the above application scenarios are based on a unified data access platform, and therefore, the same data access platform can simultaneously meet the use requirements of different protocols and frameworks.

According to the embodiment of the disclosure, the acquisition strategy corresponding to the transmission protocol is determined at least based on the transmission protocol, the data to be acquired based on the acquisition strategy is processed to obtain the target data conforming to the specified format, and the target data is stored by the target server, so that the technical problem that a large amount of work is consumed under the condition that the data of all protocols and frames need to be acquired, analyzed and used in the related art can be at least partially overcome, the data of different protocols can be intensively acquired and processed, and are intensively stored and read, so that a unified platform and service customization can be built based on the data.

The method shown in fig. 2 is further described with reference to fig. 3A-3G in conjunction with specific embodiments.

Fig. 3A schematically illustrates a flow chart for determining an acquisition policy corresponding to a transport protocol based on at least the transport protocol according to an embodiment of the present disclosure.

As shown in fig. 3A, the aforementioned operation S220 (determining an acquisition policy corresponding to a transmission protocol based on at least the transmission protocol) includes operations S311 to S314. Wherein:

in operation S311, it is detected whether the transport protocol is an HTTP-based transport protocol.

In operation S312, in a case where the transmission protocol is not the HTTP-based transmission protocol, a data format of the data to be transmitted is acquired.

In operation S313, it is detected whether the data format satisfies a preset condition to obtain a detection result.

In operation S314, an acquisition policy corresponding to the transmission protocol is determined based on at least the detection result.

Specifically, the HTTP data comprises UR L and the like, and the transmission contents are generally plain texts (under the condition of no account opening and compression), and are directly extracted and spliced into formatted contents to be stored.

In particular, the collection and processing of data is done in the proxy server. In the process of processing the proxy server (may use netty first), firstly, proxy setting is performed on the http client to point to the address and port of our own proxy server, then, after the proxy service receives the request content, the content generally has its own format, and then it is converted into the following fixed format.

Requesting:

the abnormal response result is directly discarded, and the format finally uploaded to the storage end is the format.

According to the embodiment of the disclosure, under the condition that the transmission protocol is not HTTP, the acquisition strategy corresponding to the transmission protocol is determined according to the detection result of whether the data format meets the preset condition, so that the acquisition of various data formats can be realized.

Fig. 3B schematically shows a flow chart of data processing in the related art.

As shown in fig. 3B, a typical HTTP transmission method generally includes the following steps:

and (4) capturing data by using an HTTP proxy, and storing and displaying the data. And the tool like fiddler can replay the request based on the captured (or collected) data, i.e. send a request to the server side as it is and acquire the content of the response. This is a typical scenario for data capture and use. The contents of the data are request headers (request headers), request bodies (request bodies), response headers (response headers), and response bodies (response bodies).

It should be noted that fig. 3B shows only a typical HTTP scenario, and obviously, different frames of different protocols will have different implementations, which are not described herein again, but it can be understood that the whole process and the usage scenario are different.

Fig. 3C schematically shows a flow diagram of data processing according to an embodiment of the present disclosure.

As shown in fig. 3C, for different data sources, such as HTTP or TCP data, the transmitted data is captured by a proxy method or a method provided by the RPC architecture itself, and the transmitted data is parsed (parsed into HTTP request data or deserialized, etc.), so as to obtain readable data for being stored and displayed conveniently.

Fig. 3D schematically illustrates a flow diagram for determining an acquisition policy corresponding to a transmission protocol based at least on the transmission protocol, according to another embodiment of the disclosure.

As shown in fig. 3D, the aforementioned operation S314 (the data to be collected includes the transmission data and the return data, and the collection policy corresponding to the transmission protocol is determined based on at least the detection result) includes operations S321 and S322. Wherein:

in operation S321, in a case that the detection result indicates that the data format satisfies the preset condition, a first interceptor is inserted at the client or inserted at the server.

In operation S322, the acquisition of the transmission data and the return data by the first interceptor is determined as the acquisition policy corresponding to the transmission protocol.

According to the embodiment of the disclosure, JSF is taken as an example, the JSF collection can be realized by directly inserting a Filter into a client or a server without an agent.

A schematic diagram of a data processing process according to another embodiment of the disclosure is schematically shown in conjunction with fig. 3E. As shown in fig. 3E, a custom Filter can be inserted into a FilterChain of JSF, and in this Filter, before sending, the contents called by the request, i.e., the interface name and the interface parameters, are read and then stored in a fixed format, which can be a JSON method:

and then, after the RPC request is sent, the result of the RPC request call can be obtained, and the result can be directly stored, wherein the result is a serialized object or a null value.

Likewise, the results of the call exception are also directly discarded.

By the embodiment of the disclosure, the acquisition of the transmission data and the return data acquired by the first interceptor is determined as the acquisition strategy corresponding to the transmission protocol, and the acquisition of various data formats can be realized.

Fig. 3F schematically illustrates a flow diagram for determining an acquisition policy corresponding to a transmission protocol based at least on the transmission protocol according to yet another embodiment of the disclosure.

As shown in fig. 3F, the aforementioned operation S314 (the data to be collected includes the transmission data and the return data, and the collection policy corresponding to the transmission protocol is determined based on at least the detection result) includes operations S331 and S332. Wherein:

in operation S331, in case that the detection result indicates that the data format does not satisfy the preset condition, a second interceptor and a third interceptor are inserted at the client.

In operation S332, it is determined that the transmission data acquired by the second interceptor and the return data acquired by the third interceptor are the acquisition policies corresponding to the transmission protocol.

According to the embodiment of the disclosure, the underlying transport of Dubbo is based on netty, and collection can use a custom Handler to intercept content.

A schematic diagram of a data processing process according to another embodiment of the disclosure is schematically shown in conjunction with fig. 3G. As shown in fig. 3G, a sending Handler and a receiving Handler are respectively inserted into the client of the Dubbo, and the interface call and the returned data after the call are respectively collected. However, unlike JSF, the Dubbo transmission is relatively original data, and there may be a case of packet transmission whether transmitting or receiving, and it needs to be determined by itself and then spliced. And storing the finally obtained request and the calling result in the same JSF format.

The calling request contents include an interface name and interface parameters, and are stored in a fixed format, which can be a JSON mode:

By the embodiment of the disclosure, the acquisition of the sending data acquired by the second interceptor and the return data acquired by the third interceptor are determined as the acquisition strategies corresponding to the transmission protocol, so that the acquisition of various data formats can be realized.

According to an embodiment of the present disclosure, the specified format includes a character string format, and the aforementioned operation S230 (processing the data to be collected based on the collection policy to obtain the target data conforming to the specified format) includes parsing the data to be collected to obtain the target data conforming to the character string format.

It is understood that the string format includes, but is not limited to, JSON manner, and different string formats may be selected according to different data storage requirements and rules, as long as the string format is readable data, so as to facilitate storage and display.

Through the embodiment of the disclosure, the data to be acquired, which is acquired based on the acquisition strategy, is subjected to serialization and/or deserialization processing to obtain the target data conforming to the character string format, so that the same data access platform can simultaneously meet the use requirements of different protocols.

According to the embodiment of the disclosure, when the client (the consumption end) and the server (the providing end) perform data interaction, transparent transmission is performed through the proxy. The agent captures the send and response data and passes it to the serialization and deserialization tools (which may be implemented in different languages, and this disclosure is not limited specifically), and finally obtains a readable data. And uploading the data to a storage server and a database of the interface by using a uniform interface.

It should be noted that, proxy and serialization and deserialization include, but are not limited to, the following implementation manners:

the proxy can be implemented by netty, either TCP or HTTP, without considering the case of HTTPs encryption, since the scenario involved is generally in the development and debug phase and does not require HTTPs mode. The content of a request can be obtained by the proxy (possibly the request is too long and divided into a plurality of packets).

The first one is that HTTP is relatively simple, generally carries UR L and the like, and the transmission content is generally plaintext (without account opening compression), and the contents are directly extracted and spliced into a formatted content for storage.

And the second method comprises the following steps: the transmission of TCP or SOCKET is relatively complex. In the following, the JSF is taken as an RPC framework for example, the used transport sub-protocols include hessian, msgpack, protobuf and the like, and after the input content, that is, the content read by the agent, is serialized, different deserialization and serialization modes need to be respectively customized for the different protocols.

The JSF carries the serialization and deserialization modes of the message, and the code of the framework band can be reused, because the method is a necessary step for both a consuming end and a providing end of the JSF, and the JSF receives the complete serialized content of the other end and then deserializes the content locally. But for the agent, the content does not need to be converted into the corresponding JAVA instance, but only needs to be interpreted and converted into the content before the JAVA instance, and the format of the content is relatively fixed and is also the content which can be provided for people to read. Similarly, other RPC frames also have their own serializing and deserializing devices, and can be directly multiplexed. Finally, the content after the deserialization interpretation is stored in a fixed format for later use. Even when the agent records the data transmitted by hessian, the pile data can be provided by msgpack when the pile data is provided, and the serialization put into the agent is converted into msgpack.

The present disclosure differs from the existing methods and tools in that firstly, the protocols and frameworks of the client and the server can be various, and secondly, on the nodes behind the agent, there will be a plurality of different collection and processing nodes for different purposes, but finally, data access is performed through a unified interface.

FIG. 4 schematically shows a block diagram of a data processing system according to an embodiment of the present disclosure.

As shown in fig. 4, the data processing 400 includes an obtaining module 410, a determining module 420, a processing module 430, and a storing module 440. Wherein:

the obtaining module 410 is configured to obtain a transmission protocol of data to be collected, where the data to be collected is data transmitted between a client and a server.

The determining module 420 is configured to determine, based on the transmission protocol, an acquisition policy corresponding to the transmission protocol.

And the processing module 430 is configured to process the data to be acquired, which is acquired based on the acquisition policy, so as to obtain target data conforming to a specified format.

A storage module 440, configured to store the target data through the target server.

Fig. 5A schematically illustrates a block diagram of a determination module according to an embodiment of the present disclosure.

As shown in fig. 5A, the determining module 420 includes a first detecting unit 511, an obtaining unit 512, a second detecting unit 513 and a determining unit 514. Wherein:

a first detecting unit 511, configured to detect whether the transport protocol is an HTTP-based transport protocol.

An obtaining unit 512, configured to obtain a data format of the data to be transmitted when the transmission protocol is not based on the HTTP transmission protocol.

A second detecting unit 513, configured to detect whether the data format satisfies a preset condition to obtain a detection result.

A determining unit 514, configured to determine, based on the detection result, an acquisition policy corresponding to the transmission protocol.

Fig. 5B schematically illustrates a block diagram of determining a cell according to an embodiment of the disclosure.

As shown in fig. 5B, the determining unit 514 includes a first inserting sub-unit 521 and a first determining sub-unit 522. Wherein:

and the first inserting subunit 521 is configured to insert a first interceptor at the client or insert the first interceptor at the server, if the detection result indicates that the data format meets the preset condition.

A first determining subunit 522, configured to determine, as the acquisition policy corresponding to the transmission protocol, the transmission data and the return data acquired by the first interceptor.

Fig. 5C schematically shows a block diagram of determining a cell according to another embodiment of the present disclosure.

As shown in fig. 5C, the determination unit 514 includes a second insertion sub-unit 531 and a second determination sub-unit 532. Wherein:

and the second inserting subunit 531 is configured to insert a second interceptor and a third interceptor in the client if the detection result indicates that the data format does not meet the preset condition.

A second determining subunit 532, configured to determine that the acquisition of the sending data by the second interceptor and the acquisition of the return data by the third interceptor are the acquisition policies corresponding to the transmission protocol.

According to an embodiment of the present disclosure, the processing module 430 is further configured to parse the data to be collected to obtain target data conforming to a string format.

Any one or more of the modules, units, and sub-units according to the embodiments of the present disclosure may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a programmable logic array (P L a), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations, or may be implemented at least partially as a computer program module that, when executed, may perform the corresponding function.

For example, any of the obtaining module 410, the determining module 420, the processing module 430, and the storage module 440 may be combined into one module, or any one of the modules may be split into multiple modules, or at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module according to embodiments of the present disclosure, at least one of the obtaining module 410, the determining module 420, the processing module 430, and the storage module 440 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a programmable logic array (P L A), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable manner of integrating or packaging the circuits, or any one or a suitable combination of any of these, or at least one of the obtaining module 410, the determining module 420, the processing module 430, and the storage module 440 may be implemented at least in part as a computer program module that executes a computer program when executed by the computer.

FIG. 6 schematically illustrates a block diagram of a computer system suitable for implementing the object processing method and system according to an embodiment of the present disclosure. The computer system illustrated in FIG. 6 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.

As shown in fig. 6, a computer system 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. Processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 601 may also include onboard memory for caching purposes. Processor 601 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.

In the RAM603, various programs and data necessary for the operation of the system 600 are stored. The processor 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or RAM 603. It is to be noted that the program may also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in one or more memories.

According to AN embodiment of the present disclosure, the system 600 may further include AN input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604. the system 600 may further include one or more of AN input section 606 including a keyboard, a mouse, and the like, AN output section 607 including a device such as a Cathode Ray Tube (CRT), a liquid crystal display (L CD), and the like, and a speaker, a storage section 608 including a hard disk, and the like, and a communication section 609 including a network interface card such as a L AN card, a modem, and the like, the communication section 609 performs communication processing via a network such as the Internet, a drive 610 is also connected to the I/O interface 605 as necessary, a removable medium 611 such as a magnetic disk, AN optical disk, a magneto-optical disk, a semiconductor memory, and the like, is mounted on the drive 610 as necessary, so that a computer program read therefrom is mounted into the storage section 608 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program, when executed by the processor 601, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 602 and/or RAM603 described above and/or one or more memories other than the ROM 602 and RAM 603.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data processing, comprising:

acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server;

determining an acquisition strategy corresponding to the transmission protocol based on at least the transmission protocol;

processing the data to be acquired based on the acquisition strategy to obtain target data conforming to a specified format; and

storing the target data by a target server.

2. The method of claim 1, wherein the determining, based at least on the transmission protocol, an acquisition policy corresponding to the transmission protocol comprises:

detecting whether the transmission protocol is based on an HTTP transmission protocol;

under the condition that the transmission protocol is not based on an HTTP transmission protocol, acquiring a data format of the data to be transmitted;

detecting whether the data format meets a preset condition or not to obtain a detection result; and

and determining an acquisition strategy corresponding to the transmission protocol at least based on the detection result.

3. The method of claim 2, wherein the data to be collected comprises transmit data and return data, and the determining, based at least on the detection result, a collection policy corresponding to the transmission protocol comprises:

under the condition that the detection result shows that the data format meets the preset condition, inserting a first interceptor into the client or inserting the first interceptor into the server; and

and determining the sending data and the return data acquired by the first interceptor as the acquisition strategy corresponding to the transmission protocol.

4. The method of claim 2, wherein the determining, based at least on the detection, an acquisition policy corresponding to the transmission protocol comprises:

under the condition that the detection result shows that the data format does not meet the preset condition, inserting a second interceptor and a third interceptor into the client; and

and determining the sending data acquired by the second interceptor and the return data acquired by the third interceptor as the acquisition strategy corresponding to the transmission protocol.

5. The method of claim 1, wherein the specified format comprises a string format, and the processing the data to be collected based on the collection policy to obtain the target data conforming to the specified format comprises:

and analyzing the data to be acquired to obtain the target data conforming to the character string format.

6. A data processing system comprising:

the acquisition module is used for acquiring a transmission protocol of data to be acquired, wherein the data to be acquired is data transmitted between a client and a server;

the determining module is used for determining an acquisition strategy corresponding to the transmission protocol based on the transmission protocol;

the processing module is used for processing the data to be acquired based on the acquisition strategy to obtain target data conforming to a specified format; and

and the storage module is used for storing the target data through the target server.

7. The system of claim 6, wherein the determining module comprises:

a first detection unit, configured to detect whether the transport protocol is an HTTP-based transport protocol;

the acquisition unit is used for acquiring the data format of the data to be transmitted under the condition that the transmission protocol is not based on an HTTP (hyper text transport protocol);

the second detection unit is used for detecting whether the data format meets a preset condition or not so as to obtain a detection result; and

and the determining unit is used for determining an acquisition strategy corresponding to the transmission protocol based on the detection result.

8. The system of claim 7, wherein the determining unit comprises:

the first inserting subunit is used for inserting a first interceptor into the client or inserting the first interceptor into the server under the condition that the detection result shows that the data format meets the preset condition; and

and the first determining subunit is used for determining the sending data and the return data acquired by the first interceptor as the acquisition strategy corresponding to the transmission protocol.

9. The system of claim 7, wherein the determining unit comprises:

the second inserting subunit is used for inserting a second interceptor and a third interceptor into the client under the condition that the detection result shows that the data format does not meet the preset condition; and

and the second determining subunit is configured to determine, as the acquisition policy corresponding to the transmission protocol, the transmission data acquired by the second interceptor and the return data acquired by the third interceptor.

10. The system of claim 6, wherein the specified format comprises a string format, and the processing the data to be collected based on the collection policy to obtain the target data conforming to the specified format comprises:

and the analysis module is used for analyzing the data to be acquired so as to obtain the target data conforming to the character string format.

11. A computer system, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 5.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 5.