CN116708219A

CN116708219A - DPI platform-based data acquisition method and device

Info

Publication number: CN116708219A
Application number: CN202310817904.2A
Authority: CN
Inventors: 邹学强; 马璐
Original assignee: National Computer Network and Information Security Management Center
Current assignee: National Computer Network and Information Security Management Center
Priority date: 2023-07-05
Filing date: 2023-07-05
Publication date: 2023-09-05

Abstract

The application discloses a data acquisition method based on a DPI platform, which can determine a data acquisition strategy according to service requirements; then, determining resource support parameters required by the DPI platform according to the data acquisition strategy; then, judging whether the resource supporting parameter is overloaded; and then, if the resource supporting parameter is overload, determining the effectiveness of the data acquisition strategy through an optimal resource occupation reduction algorithm. According to the DPI platform-based data acquisition method, the data acquisition strategy is flexibly formulated according to the service requirements, so that the data acquisition strategy is effective, the data is acquired according to the data acquisition strategy, and the service diversity requirements of the current network service analysis department are met.

Description

DPI platform-based data acquisition method and device

Technical Field

The application belongs to a data acquisition technology of a DPI system, and particularly relates to a data acquisition method and device based on a DPI platform.

Background

With the increasing of 5G users, the mobile internet network traffic is huge in scale, and related services of mobile internet traffic data are more and more, so that the flexibility of the traffic acquisition and distribution system based on the DPI (Deep Packet Inspection ) system is one of the major problems to be solved. The DPI data acquisition technology with high elasticity and the message acquisition technology based on a rule engine are used for realizing flexible, dynamic and accurate data acquisition according to service requirements. However, the existing mobile internet DPI acquisition platform can only support full acquisition of a signaling plane and a user plane, the acquisition method is single, flexible load acquisition of most protocols is not supported, and the service diversity requirement of the existing network service analysis department is difficult to be completely met.

Disclosure of Invention

In order to solve the defects of the prior art, the application provides a DPI platform-based data acquisition method and device, and aims to provide flexible acquisition data according to service requirements and service diversity requirements of a hidden-foot-up network service analysis department.

The technical effects to be achieved by the application are realized by the following scheme:

in a first aspect, the present application provides a data acquisition method based on a DPI platform, the method comprising:

determining a data acquisition strategy according to service requirements;

determining resource support parameters required by the DPI platform according to the data acquisition strategy;

judging whether the resource supporting parameters are overloaded;

if the resource supporting parameter is overload, determining the effectiveness of a data acquisition strategy through an optimal resource occupation reduction algorithm;

the data acquisition strategy comprises a data acquisition type, data acquisition parameters, a data acquisition path, a data acquisition flow and strategy life cycle management.

Optionally, a plurality of the data acquisition policies are run in the DPI platform;

before the step of determining the resource support parameters required at the DPI platform according to the data acquisition strategy, the method comprises:

Performing policy conflict detection on a plurality of data acquisition policies, wherein the policy conflict detection content comprises detection data acquisition paths and data type coverage areas;

and if the data acquisition strategies have conflict, determining the effectiveness of the data acquisition strategies according to the priority of the data acquisition strategies.

Optionally, the performing policy conflict detection on the plurality of data acquisition policies includes:

comparing the data acquisition paths and the data type coverage areas of any two data acquisition strategies, and if the data acquisition paths and the data type coverage areas of the two data acquisition strategies are inconsistent, judging that the two data acquisition strategies have no conflict.

the resource supporting parameters at least comprise computing power, memory, storage and bandwidth of the CPU; the judging whether the resource supporting parameter is overloaded comprises the following steps:

comparing the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies in the DPI platform with those of the CPU in the DPI platform, and judging that the resource support parameter is overloaded if one of the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies in the DPI platform exceeds the DPI platform.

Optionally, if the resource supporting parameter is overload, determining, by an optimal resource occupation reduction algorithm, the effectiveness of the data acquisition policy includes:

if the resource supporting parameter is overload, the data acquisition strategy is adjusted according to the resource parameter of the DPI platform and the optimal resource occupation reduction algorithm, and the updated data acquisition strategy is obtained;

and running the updated data acquisition strategy to enable the updated data acquisition strategy to be effective.

Optionally, the optimal resource occupation reduction algorithm includes:

acquiring the priority sequence of a plurality of data acquisition strategies, and arranging the plurality of data acquisition strategies according to the priority sequence;

determining at least one data acquisition strategy arranged in the priority sequence in front as a 1 st data acquisition strategy group which takes effect preferentially according to the plurality of data acquisition strategies arranged in the priority sequence and the calculation power, the memory, the storage and the bandwidth of a CPU in the DPI platform;

determining at least one data acquisition strategy arranged in front in the priority order as a 2 nd data acquisition strategy group which takes effect preferentially according to the calculation power, memory, storage and bandwidth of a CPU in the DPI platform of the rest data acquisition strategies;

The data acquisition policy group 1 and the data acquisition policy group 1 are not overloaded with the total resource support parameters, and the data acquisition policy group 1 are sequentially operated on the DPI platform;

the data acquisition strategy comprises the following steps:

after the data acquisition exceeds the hit flow, strategy aging is carried out according to the sequence from low to high of the strategy sequence number so as to meet the requirement that the hit flow is in a specified range, and conversely, when the flow is reduced to a certain degree, a flow recovery mechanism is started to carry out strategy recovery, and automatic regulation of the flow is completed;

strategy first-in first-out, missed strategy, reject already hit strategy first, need to consider whether the strategy hits, reject the strategy that misses; or (b)

The data acquisition strategy comprises the following steps:

after the data acquisition exceeds the hit flow, all strategies are carried out in a mode of returning only the first packet to meet the requirement that the hit flow is in a specified range in order to reduce the number of the hit flow, and conversely, when the flow is reduced to a certain degree, a flow recovery mechanism is started to recover the strategies, and the automatic regulation of the flow is completed.

Optionally, after the step of determining the data acquisition policy according to the service requirement, the method includes:

receiving and/or translating the data acquisition strategy, wherein the translating means that information of the data acquisition strategy is converted into a strategy instruction which can be executed by the DPI platform, and translating an application scene of the data acquisition strategy comprises acquiring network layer original message data, transmission layer original message data, application layer original message data and log data of specific conditions.

Optionally, the data acquisition process includes:

uploading the existing data acquisition strategy, and requesting the data acquisition strategy determined according to the service requirement;

the existing data acquisition strategy is found to be the same as the data acquisition strategy determined according to the service requirement;

and requesting the data acquisition strategy determined according to the service requirement again to acquire an updated data acquisition strategy.

Optionally, if the resource supporting parameter is overload, determining, by the optimal resource occupation reduction algorithm, the validating step of the data acquisition policy includes:

transmitting, filtering, managing and detecting the quality of the data acquired by applying the data acquisition strategy;

And distributing the data subjected to transmission, filtering, management and quality detection to each data receiving and distributing system.

In a second aspect, the present application provides a DPI platform based data acquisition apparatus, the apparatus comprising:

the first determining unit is used for determining a data acquisition strategy according to the service requirement;

a second determining unit, configured to determine, according to the data acquisition policy, a resource support parameter required at the DPI platform;

the judging unit is used for judging whether the resource supporting parameter is overloaded or not;

the execution unit is used for determining the effectiveness of the data acquisition strategy through an optimal resource occupation reduction algorithm if the resource support parameter is overloaded;

In a third aspect, the present application provides a readable medium comprising execution instructions which, when executed by a processor of an electronic device, perform the method according to any of the first aspects.

In a fourth aspect, the present application provides an electronic device comprising a processor and a memory storing execution instructions, the processor performing the method according to any one of the first aspects when executing the execution instructions stored in the memory.

The application has the following advantages:

the data acquisition method based on the DPI platform can determine a data acquisition strategy according to service requirements; then, determining resource support parameters required by the DPI platform according to the data acquisition strategy; then, judging whether the resource supporting parameter is overloaded; and then, if the resource supporting parameter is overload, determining the effectiveness of the data acquisition strategy through an optimal resource occupation reduction algorithm. The DPI platform-based data acquisition method flexibly establishes the data acquisition strategy according to the service requirement, so that the data acquisition strategy is effective, the data is acquired according to the data acquisition strategy, and the service diversity requirement of the service analysis department of the current network is hidden.

Drawings

In order to more clearly illustrate the embodiments of the application or the prior art solutions, the drawings which are used in the description of the embodiments or the prior art will be briefly described below, it being obvious that the drawings in the description below are only some of the embodiments described in the present application, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.

Fig. 1 is a flowchart of a data acquisition method based on a DPI platform according to an embodiment of the application;

fig. 2 is a second flow chart of a data acquisition method based on a DPI platform according to an embodiment of the application;

FIG. 3 is a flow chart illustrating a data acquisition process according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an Internet acquisition platform according to an embodiment of the application;

fig. 5 is a schematic structural diagram of a DPI platform-based data acquisition device according to an embodiment of the application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The existing mobile internet DPI (Deep Packet Inspection ) acquisition platform can only support full acquisition of a signaling plane and a user plane, the acquisition method is single, flexible load acquisition of most protocols is not supported, and service diversity requirements of an existing network service analysis department are difficult to fully meet.

In view of the above, the present application provides a data acquisition method and apparatus based on a DPI platform, which aims to provide flexible acquisition of data according to service requirements, and to hide the service diversity requirements of the service analysis department of the existing network.

Non-limiting embodiments of the present application are described in detail below with reference to the attached drawing figures.

Fig. 1 is a schematic flow chart of a data acquisition method based on a DPI platform according to an embodiment of the application. As can be seen from the drawings, the DPI platform-based data acquisition method includes step S02, step S04, step S06, and step S08.

Step S02: determining a data acquisition strategy according to service requirements;

step S04: determining resource support parameters required by the DPI platform according to the data acquisition strategy;

step S06: judging whether the resource supporting parameters are overloaded;

step S08: and if the resource supporting parameter is overload, determining the effectiveness of the data acquisition strategy through an optimal resource occupation reduction algorithm. The validation of the data acquisition policy refers to running the data acquisition policy in the DPI platform to acquire the required data.

The data acquisition strategy is determined according to the service requirement, so that the data can be flexibly acquired. The data acquisition strategy comprises a data acquisition type, data acquisition parameters, a data acquisition path, a data acquisition flow and strategy life cycle management. Illustratively, the data type includes at least one of network layer original message data, transport layer original message data-IP (Internet Protocol ) five tuple, transport layer original message data-specific location payload, application layer original message data, log data of specific condition, and the like. The data acquisition parameters comprise at least one of calculation power, storage, broadband, acquisition time and the like required for acquiring data, and the data acquisition path refers to which network route or which network channel is used for acquiring the data. The data acquisition flow refers to a step performed during acquisition, for example, at least one step of signal handshake before acquisition, data acquisition policy reporting, response policy request message, and the like. The policy lifecycle management is based on a management method, supports the management of the whole lifecycle such as loading, unloading, execution time length and the like of all the data acquisition policies in the management and control equipment, and is used for the flow of the data acquired by the data acquisition policies in the whole lifecycle. The overall process of loading to unloading from a data acquisition strategy is automated, typically by organizing the data acquisition strategy into various layers according to a specific method, and automatically moving data from one layer to another based on those critical conditions. The method supports automatic loading after the creation and the change of the data acquisition strategy and automatic unloading after the deletion, supports the change of the quick processing data acquisition strategy, submits the change of the data acquisition strategy and automatically adjusts the change of the data acquisition strategy, actively evaluates the influence of the change of the data acquisition strategy, pre-analyzes the influence of business data streams generated by the change of the newly built or changed data acquisition strategy through agility on resources, and reduces confusion and complexity of the data acquisition strategy by utilizing the existing data acquisition strategy and objects as much as possible.

Illustratively, the DPI platform has an intelligent policy checking mechanism, and is configured to comb the data acquisition policy, optimize the hidden data acquisition policy, the redundant data acquisition policy, the empty data acquisition policy, and the like, so as to avoid service risks and security risks caused by the non-compliant data acquisition policy, and timely clear the temporarily opened or expired data acquisition policy, thereby avoiding redundancy of the data acquisition policy.

In the embodiment of the application, the data is acquired in the DPI platform through the data acquisition strategy, and the data at least comprises network layer original message data, transmission layer original message data-IP quintuple, transmission layer original message data-specific position load, application layer original message data, log data of specific conditions and the like.

In summary, the data acquisition method based on the DPI platform flexibly formulates a data acquisition policy according to the service requirement, so that the data acquisition policy is effective, and the service diversity requirement of the service analysis department of the current network is hidden.

In an embodiment, a plurality of said data acquisition strategies are run in said DPI platform. As shown in fig. 2, in step S04, the DPI platform-based data acquisition method includes step S031 and step S032 before determining the resource support parameters required by the DPI platform according to the data acquisition policy.

Step S031: performing policy conflict detection on a plurality of data acquisition policies, wherein the policy conflict detection content comprises detection data acquisition paths and data type coverage areas;

step S032: and if the data acquisition strategies have conflict, determining the effectiveness of the data acquisition strategies according to the priority of the data acquisition strategies.

It will be appreciated that the DPI platform may run multiple data acquisition strategies simultaneously, but not in the event of a conflict. If the multiple data acquisition strategies conflict, which means that the data acquisition paths of at least two data acquisition strategies in the multiple data acquisition strategies are the same, if two data acquisition strategies simultaneously acquire data through one path, the broadband of the path is fixed, so that one data acquisition strategy can be supported to acquire data, and if two or more data acquisition strategies simultaneously acquire data through the path, the path is blocked, or the acquired data parameters are disordered, so that the data acquisition failure is caused. Therefore, before the DPI platform runs a plurality of data acquisition strategies simultaneously, whether the data acquisition strategies collide or not is detected, and normal running of the DPI platform is ensured. Similarly, if the data acquired in the plurality of data acquisition strategies is repeated, the data is acquired repeatedly, or the memory storing the data is accessed simultaneously, so that the frequency of accessing the memory is high, and the access is blocked. It is necessary to detect whether the coverage of the data type required to be acquired by the plurality of data acquisition strategies is repeated. If a plurality of data acquisition strategies have conflicts, the priority of the data acquisition strategies determines the effectiveness of the data acquisition strategies. Illustratively, the validation times of the plurality of data acquisition policies may be arranged according to the priorities of the data acquisition policies, avoiding the plurality of data acquisition policies from running simultaneously.

In one embodiment, step S031, performing policy conflict detection on the plurality of data acquisition policies includes step S0311.

Step S0311: comparing the data acquisition paths and the data type coverage areas of any two data acquisition strategies, and if the data acquisition paths and the data type coverage areas of the two data acquisition strategies are inconsistent, judging that the two data acquisition strategies have no conflict.

In the above embodiment, if there are conflicts in the plurality of data acquisition strategies, this indicates that the data acquisition paths of at least two data acquisition strategies in the plurality of data acquisition strategies are identical or the coverage areas of the data types are coincident. To distinguish whether the data acquisition paths with the two data acquisition strategies are identical or whether the coverage of the data types with the two data acquisition strategies overlap. And comparing the data acquisition paths and the data type coverage areas of any two data acquisition strategies to prevent the data acquisition paths of at least two data acquisition strategies in the plurality of data acquisition strategies from being identical or the data type coverage areas from being overlapped.

In one embodiment, a plurality of data acquisition strategies are run in the DPI platform, and the resource support parameters include at least CPU computational power, memory, storage, and bandwidth. Step S06, judging whether the resource support parameter is overloaded, including step S061.

Step S061: comparing the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies on the DPI platform with the resource parameters of the DPI platform, namely comparing the calculation power, the memory, the storage and the bandwidth of the CPU in the DPI platform, and judging that the resource support parameters are overloaded if one of the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies on the DPI platform exceeds the DPI platform.

It can be appreciated that the data acquisition strategy requires a certain CPU power, memory, storage and bandwidth to operate smoothly or successfully on the DPI platform when the DPI platform is operating. If the CPU computing power, memory, storage and bandwidth of the DPI platform cannot be fully covered by the requirements of the data acquisition policy, the data acquisition policy cannot be run on the DPI platform or is easy to be blocked when the DPI platform is running.

In an embodiment, step S08, if the resource supporting parameter is overload, determines that the data acquisition policy is effective through the optimal resource occupation reduction algorithm, including step S081 and step S082.

Step S081: if the resource supporting parameter is overload, the data acquisition strategy is adjusted according to the resource parameter of the DPI platform and the optimal resource occupation reduction algorithm, and the updated data acquisition strategy is obtained;

step S082: and running the updated data acquisition strategy to enable the updated data acquisition strategy to be effective.

It can be known that when the resource parameters of the DPI platform cannot be hidden from the resource parameters of the data acquisition policy, that is, the data acquisition policy cannot smoothly or smoothly run on the DPI platform. At this time, the data acquisition strategy needs to be adjusted according to the resource parameters of the DPI platform and the optimal resource occupation reduction algorithm, so that the updated data acquisition strategy is suitable for running on the DPI platform.

In one embodiment, the optimal resource occupation reduction algorithm includes a step a, a step B, and a step C.

Step A: acquiring the priority sequence of a plurality of data acquisition strategies, and arranging the plurality of data acquisition strategies according to the priority sequence;

and (B) step (B): determining at least one data acquisition strategy arranged in the priority sequence in front as a 1 st data acquisition strategy group which takes effect preferentially according to the plurality of data acquisition strategies arranged in the priority sequence and the calculation power, the memory, the storage and the bandwidth of a CPU in the DPI platform;

Step C: and determining at least one data acquisition strategy arranged in front in the priority sequence as a 2 nd data acquisition strategy group which takes effect preferentially according to the rest data acquisition strategies and the calculation power, memory, storage and bandwidth of a CPU in the DPI platform.

The DPI platform is used for sequentially running the 1 st data acquisition strategy group and the 1 st data acquisition strategy group.

In an embodiment, the plurality of data acquisition policies includes a data acquisition policy a, a data acquisition policy B, a data acquisition policy C, a data acquisition policy D, a data acquisition policy E, a data acquisition policy F, and a data acquisition policy G. And arranging a plurality of data acquisition strategies according to the priority order into a data acquisition strategy A, a data acquisition strategy C, a data acquisition strategy D, a data acquisition strategy B, a data acquisition strategy F, a data acquisition strategy G and a data acquisition strategy E. And determining at least one data acquisition strategy arranged in the priority sequence as a 1 st data acquisition strategy group { data acquisition strategy A, data acquisition strategy C and data acquisition strategy D } which takes effect preferentially according to the plurality of data acquisition strategies arranged in the priority sequence and the computational power, memory, storage and bandwidth of a CPU in the DPI platform. And determining at least one data acquisition strategy arranged in the front in the priority sequence as a 2 nd data acquisition strategy group { data acquisition strategy B and data acquisition strategy F } which take effect preferentially according to the rest data acquisition strategies and the calculation power, memory, storage and bandwidth of a CPU in the DPI platform, wherein the 3 rd data acquisition strategy group { data acquisition strategy G and data acquisition strategy E }. The total resource supporting parameters in the 1 st data acquisition strategy group { data acquisition strategy A, data acquisition strategy C, data acquisition strategy D }, the 2 nd data acquisition strategy group { data acquisition strategy B, data acquisition strategy F } and the 3 rd data acquisition strategy group { data acquisition strategy G, data acquisition strategy E } are not overloaded, the sum of the resource supporting parameters of the data acquisition strategy A, data acquisition strategy C and data acquisition strategy D is not overloaded, the sum of the resource supporting parameters of the data acquisition strategy B and data acquisition strategy F is not overloaded, and the sum of the resource supporting parameters of the data acquisition strategy G and data acquisition strategy E is not overloaded, namely, the resource parameters of the DPI platform are not exceeded. The 1 st data acquisition policy group { data acquisition policy A, data acquisition policy C, data acquisition policy D }, the 2 nd data acquisition policy group { data acquisition policy B, data acquisition policy F } and the 3 rd data acquisition policy group { data acquisition policy G, data acquisition policy E } are sequentially operated on the DPI platform.

The data acquisition strategy conflict is mainly reflected on the priority of the data acquisition strategy, and the data acquisition strategy conflict detection of 5 scenes, namely, network layer original message data, transmission layer original message data-IP five-tuple, transmission layer original message data-specific position load, application layer original message data and log data under specific conditions, can be classified into two major categories, namely, original message data strategy conflict detection and log data strategy conflict detection, and the system design carries out strategy conflict detection based on a correlation and priority dual scheme, namely, when a new data acquisition strategy arrives, the correlation and the priority of the new data acquisition strategy and the strategy which is in effect are judged and compared.

In terms of data acquisition strategies, different data acquisition strategies are realized by adopting different matching, IP rules are matched by using trie trees (dictionary trees), and service application IDs are matched by using hash tables, so that the rule matching efficiency is improved;

in the running process of the system, along with the change of the issued data acquisition strategy and the change of the access flow data, the hit flow can be caused to fluctuate, and when the hit flow exceeds the index range required by the specification, the system adopts the flow discarding data acquisition strategy to protect in order not to influence the running of the whole system, and the current data acquisition strategy comprises two modes:

The strategy aging mechanism is used for performing strategy aging according to the sequence from low to high after the data acquisition exceeds the hit flow so as to meet the requirement that the hit flow is in a specified range, and conversely, when the flow is reduced to a certain degree, the flow recovery mechanism is started to perform strategy recovery, and the automatic regulation of the flow is completed;

strategy first-in first-out, missed strategy, reject already hit strategy first, need to consider whether the strategy hits, reject the strategy that misses;

and after the data acquisition exceeds the hit flow, all rules are carried out in a mode of only returning the first packet so as to meet the requirement that the hit flow is in a specified range, and conversely, when the flow is reduced to a certain degree, a flow recovery mechanism is started to recover the rules and complete automatic regulation of the flow.

The flow acquisition may be performed in any of the above ways.

In an embodiment, step S02, after determining a data acquisition policy according to a service requirement, the data acquisition method based on the DPI platform includes:

the DPI platform receives and/or translates the data acquisition strategy, wherein the translation refers to converting information of the data acquisition strategy into a strategy instruction which can be executed by the DPI platform, and translating an application scene of the data acquisition strategy comprises acquiring network layer original message data, transmission layer original message data, application layer original message data and log data of specific conditions.

In one embodiment, as shown in fig. 3, the data acquisition process includes step 2, step 4, and step 6.

Step 2: uploading the existing data acquisition strategy, and requesting the data acquisition strategy determined according to the service requirement;

step 4: the existing data acquisition strategy is found to be the same as the data acquisition strategy determined according to the service requirement;

step 6: and requesting the data acquisition strategy determined according to the service requirement again to acquire an updated data acquisition strategy.

In an embodiment, in step S08, if the resource support parameter is overload, after determining that the data acquisition policy is effective by the optimal resource occupation reduction algorithm, the data acquisition method based on the DPI platform includes step S09 and step S10.

Step S09: transmitting, filtering, managing and detecting the quality of the data acquired by applying the data acquisition strategy;

step S10: and distributing the data subjected to transmission, filtering, management and quality detection to each data receiving and distributing system.

In an embodiment, the DPI platform-based data acquisition method is applied to the internet. As shown in fig. 4, the internet collection platform is provided with a data sharing management platform and a DPI platform. The data sharing management platform comprises a strategy management subsystem and a data receiving and distributing subsystem. The DPI platform comprises novel management and control equipment and a data reporting interface. The first determining unit in the data sharing management platform determines a data acquisition strategy according to the service requirement, and the strategy management subsystem issues and distributes the data acquisition strategy to the DPI platform, wherein the data acquisition strategy comprises a log data acquisition strategy, a transmission layer original message data strategy and the like under specific conditions. The novel management and control equipment of the DPI platform receives the data acquisition strategy, a translation unit of the novel management and control equipment translates the data acquisition strategy, a detection unit of the novel management and control equipment detects the data acquisition strategy, the novel management and control equipment executes path selection of the data acquisition strategy, and the managed strategy life cycle. After the DPI platform obtains the required data according to the data acquisition strategy, a data reporting interface of the DPI platform subscribes, transmits, filters, manages and detects the quality of the data, and sends the data to a data receiving and distributing subsystem, and the data receiving and distributing subsystem receives and verifies the data.

When the internet acquisition platform is accessed to the service interface, the interaction frequency and the transmission information size of the acquired data are fully considered, and the performance consumption of the internet acquisition platform is reduced as much as possible. According to the data transmission frequency and the file capacity, the method can be divided into the following two types of data interaction modes: non-real-time large-scale data such as business log data or original code stream data, large capacity and low frequency; it is proposed to take a "Web service+ftp" mode, web Service being a Web Service, FTP being FTP (File Transfer Protocol ) being one of the protocols in the TCP/IP protocol suite. And generating a file after acquisition, sending FTP information of the file through a Web Service protocol, and logging in an FTP server to obtain the file according to the FTP information content after the upper application system receives a Web Service request. Real-time data with smaller flow, such as real-time alarm, position change, real-time tracking and the like, has small capacity and high frequency; the data transmission frequency is higher, a certain requirement is met on real-time performance, and the data is suggested to be directly transmitted by adopting a faster interaction protocol (such as MQ, socket and the like).

The Internet collection platform supports data subscription and release, is provided with a data subscription and release center, and realizes centralized management of data classification, data release and subscription information. The file interface is mainly used for providing batch access of original code stream, a large amount of XDR (External Data Representation, external data) basic data and detail data for the elastic acquisition management platform by the DPI system, and the elastic acquisition management platform can acquire the XDR basic data and the detail data in a subscription mode. The message interface mainly supports the DPI system to provide data query capability for the elastic acquisition management platform. The elastic acquisition management platform sends an original code stream and an XDR detail data query strategy to the DPI system.

The file interface uses FTP/SFTP protocol to report XDR, it can send the file produced by producer to consumer in real time, high-efficiency and various modes, and its function can meet the requirement of most projects, and can upload the existing file or file produced in real time. And supporting various uploading modes such as load balancing, priority and the like. Support compression uploading, support archiving and storing files in a destination server according to a specified mode, and the like.

The file interface can simultaneously execute uploading tasks of a plurality of items, and uploading threads are multithreaded thread groups in order to ensure uploading efficiency. To ensure that a plurality of uploading threads uniformly execute uploading tasks of a plurality of projects (maximally utilizing bandwidth), and simultaneously, upload logs are output to databases such as Mysql, redis and the like in real time, an abnormal mode is adopted, program coupling is low, even if a network is not smooth, or a Mysql server is down, the upload logs are stored in a cache, data loss is not caused, and the work of the uploading threads is not influenced completely.

All policies of collecting flow and log issued by the collecting platform directly influence the data inflow and outflow conditions of the system, and strict monitoring needs to be carried out aiming at the resource occupation conditions, and the policies mainly comprise platform monitoring, service monitoring, performance panels, policy intelligent analysis and statistical report forms. The platform monitoring is used for completing monitoring on the running state of the system, such as monitoring on the CPU occupation/load state, storage occupation condition, data reporting bandwidth resource, queue load/performance occupied by the key module and the like of the system. The service monitoring is used for completing monitoring of the running state of the service encapsulated by the system, such as calling frequency analysis of the service, data quantity analysis of the service, response time analysis of the service, monitoring condition of the service and the like. The performance instrument panel macroscopically controls the overall performance running condition of the platform in an instrument panel mode and provides a guiding basis for macroscopic decision. The intelligent analysis of the strategy is to pre-judge the data flow in advance by mining and analyzing the business data customized by the acquisition strategy, analyze the related resources, judge the system performance and key bottlenecks, and simultaneously support the analysis of the influence scope of the independent acquisition strategy. The statistical report forms are displayed in a multi-dimensional report form mode by extracting, cleaning and counting the customized data of the acquisition strategies, and the data quantity of each acquisition strategy and the influence conditions on system performance and the like are visually displayed.

The internet performs data acquisition according to the data acquisition flow, firstly, the DPI platform uploads the existing data acquisition strategy to the data sharing management platform, and the data sharing management platform sends a data acquisition strategy request message to the DPI platform. If the DPI platform translates and discovers that the same strategy is executing, the DPI platform replies a strategy response message, the data sharing management platform deletes the data acquisition strategy request message, and the DPI platform deletes the strategy response message. And the data sharing management platform sends a data acquisition strategy request message to the DPI platform again, and if the DPI platform translates that the same strategy is not found to be executed, the DPI platform replies a strategy response message and uploads the strategy response message to the updated strategy data of the data sharing management platform.

The application also provides a data acquisition device based on the DPI platform, as shown in fig. 5, the device comprises:

Optionally, a plurality of the data acquisition strategies are run in the DPI platform, the apparatus comprising:

the detection unit is used for carrying out policy conflict detection on a plurality of data acquisition policies, and the policy conflict detection content comprises a detection data acquisition path and a data type coverage area;

Optionally, the detection unit is configured to: comparing the data acquisition paths and the data type coverage areas of any two data acquisition strategies, and if the data acquisition paths and the data type coverage areas of the two data acquisition strategies are inconsistent, judging that the two data acquisition strategies have no conflict.

Optionally, the judging unit is configured to: comparing the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies in the DPI platform with those of the CPU in the DPI platform, and judging that the resource support parameter is overloaded if one of the calculation power, the memory, the storage and the bandwidth of the CPU required by the data acquisition strategies in the DPI platform exceeds the DPI platform.

Optionally, the execution unit is configured to: if the resource supporting parameter is overload, the data acquisition strategy is adjusted according to the resource parameter of the DPI platform and the optimal resource occupation reduction algorithm, and the updated data acquisition strategy is obtained;

Optionally, the execution unit is configured to: acquiring the priority sequence of a plurality of data acquisition strategies, and arranging the plurality of data acquisition strategies according to the priority sequence;

Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. At the hardware level, the electronic device comprises a processor, optionally an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 6, but not only one bus or type of bus.

And the memory is used for storing the execution instruction. In particular, a computer program that executes instructions may be executed. The memory may include memory and non-volatile storage and provide the processor with instructions and data for execution.

In one possible implementation manner, the processor reads the corresponding execution instruction from the nonvolatile memory into the memory and then executes the execution instruction, and may also acquire the corresponding execution instruction from other devices, so as to form a data acquisition method based on the DPI platform on a logic level. The processor executes the execution instructions stored in the memory to implement the DPI platform-based data acquisition method provided in any embodiment of the application by executing the execution instructions.

The method performed by the data acquisition method based on the DPI platform according to the embodiment of the application shown in fig. 6 may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The embodiment of the application also provides a readable medium, wherein the readable storage medium stores an execution instruction, and when the stored execution instruction is executed by a processor of electronic equipment, the electronic equipment can be enabled to execute the data acquisition method based on the DPI platform provided by any embodiment of the application, and the method is particularly used for executing the data acquisition method based on the DPI platform.

The electronic device described in the foregoing embodiments may be a computer.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware aspects.

The embodiments of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment is mainly described in the differences from the other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. A data acquisition method based on a DPI platform, the method comprising:

determining a data acquisition strategy according to service requirements;

judging whether the resource supporting parameters are overloaded;

2. The DPI platform based data acquisition method according to claim 1, wherein a plurality of said data acquisition strategies are run in said DPI platform;

3. The DPI platform based data acquisition method according to claim 2, wherein said performing policy conflict detection on a plurality of said data acquisition policies comprises:

4. The DPI platform based data acquisition method according to claim 1, wherein a plurality of said data acquisition strategies are run in said DPI platform;

5. The DPI platform based data acquisition method according to claim 4, wherein if the resource support parameter is overload, determining the validation of the data acquisition policy by an optimal resource occupancy reduction algorithm includes:

6. The DPI platform based data acquisition method according to claim 5, wherein the optimal resource occupancy reduction algorithm comprises:

the data acquisition strategy comprises the following steps:

The data acquisition strategy comprises the following steps:

7. The DPI platform-based data acquisition method according to claim 1, wherein after the determining a data acquisition policy step according to a traffic demand, the method comprises:

8. The DPI platform based data acquisition method according to claim 3, wherein said data acquisition procedure includes:

9. The DPI platform-based data acquisition method according to claim 1, wherein the determining the validation step of the data acquisition policy by the optimal resource occupancy reduction algorithm if the resource support parameter is overloaded comprises:

10. A DPI platform based data acquisition device, the device comprising:

11. A readable medium comprising execution instructions which, when executed by a processor of an electronic device, perform the method of any of claims 1-9.

12. An electronic device comprising a processor and a memory storing execution instructions, the processor performing the method of any of claims 1-9 when the processor executes the execution instructions stored in the memory.