CN116032929A - Data processing system, method and equipment - Google Patents

Data processing system, method and equipment Download PDF

Info

Publication number
CN116032929A
CN116032929A CN202310323795.9A CN202310323795A CN116032929A CN 116032929 A CN116032929 A CN 116032929A CN 202310323795 A CN202310323795 A CN 202310323795A CN 116032929 A CN116032929 A CN 116032929A
Authority
CN
China
Prior art keywords
data packet
data
distribution control
processing module
distribution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310323795.9A
Other languages
Chinese (zh)
Other versions
CN116032929B (en
Inventor
王野
杨光
姚怡东
石磊
郝文杰
徐悦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202310323795.9A priority Critical patent/CN116032929B/en
Publication of CN116032929A publication Critical patent/CN116032929A/en
Application granted granted Critical
Publication of CN116032929B publication Critical patent/CN116032929B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00Reducing energy consumption in communication networks
    • Y02D30/50Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The application provides a data processing system, a data processing method and data processing equipment, wherein the system comprises client equipment deployed at a cloud end, data distribution equipment and a plurality of back-end server groups, the data distribution equipment comprises a first processing module and a second processing module, and the back-end server groups comprise a plurality of servers for processing the same service. The client device is used for sending the data packet to be distributed to the data distribution device; the data distribution device is used for matching a distribution control strategy associated with the data packet through the first processing module and transmitting the data packet to the target back-end server group based on the distribution control strategy; if the first processing module is not matched with the distribution control strategy associated with the data packet, the second processing module learns the distribution control strategy of the data packet so as to guide the first processing module to transmit the data packet. In the scheme, the first processing module and the second processing module which are relatively independent are divided, so that the overall distribution efficiency of the data distribution equipment can be improved.

Description

Data processing system, method and equipment
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a data processing system, method, and device.
Background
With the continuous development of computer networks and cloud technologies, various cloud service systems have grown, and more enterprise users choose to transmit services through the cloud, so as to realize efficient, highly flexible and highly reliable service requirements.
At present, the cloud can support data transmission of multiple services, so that access data are distributed to a back-end cloud server, and response data are transmitted back from the back-end cloud server, however, the data distribution efficiency of the existing transmission scheme is low.
Disclosure of Invention
The embodiment of the application provides a data processing system, a data processing method and data processing equipment, and data distribution efficiency is improved.
A first aspect of an embodiment of the present application provides a data processing system, including a client device deployed at a cloud end, a data distribution device, and a plurality of backend server groups, where the data distribution device includes a first processing module and a second processing module, and the backend server groups include a plurality of servers for processing the same service;
the client device is configured to: transmitting a data packet to be distributed to the data distribution equipment;
the data distribution device is used for: matching, by the first processing module, a distribution control policy associated with the data packet, and transmitting the data packet to a target backend server group based on the distribution control policy; and if the first processing module is not matched with the distribution control strategy associated with the data packet, learning the distribution control strategy of the data packet through the second processing module so as to guide the first processing module to transmit the data packet.
In an embodiment of the first aspect of the present application, the first processing module configures the preset packet detection algorithm; the first processing module is specifically configured to:
acquiring metadata of the data packet through the preset packet detection algorithm;
transmitting the data packet to a target back-end server group based on a distribution control policy associated with the data packet if the metadata matches the distribution control policy;
if the distribution control policy associated with the data packet is not matched based on the metadata, the data packet and the metadata are sent to the second processing module to cause the second processing module to learn the distribution control policy of the data packet.
In an embodiment of the first aspect of the present application, the preset packet detection algorithm configured in the first processing module includes a hash algorithm, and a hash flow table is further configured in the first processing module, where the hash flow table includes an association relationship between a hash string and a distribution control policy;
the first processing module is specifically configured to: acquiring a first hash character string of the data packet through the hash algorithm, wherein the first hash character string is metadata of the data packet;
If the first hash character string is matched in the Ha Xiliu table, acquiring a distribution control strategy associated with the first hash character string, and transmitting the data packet to a target back-end server group based on the distribution control strategy;
and if the first hash character string is not matched in the Ha Xiliu table, the first processing module sends the data packet and the first hash character string to the second processing module so that the second processing module learns a distribution control strategy of the data packet.
In one embodiment of the first aspect of the present application, the second processing module is specifically configured to:
and determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet.
In one embodiment of the first aspect of the present application, the second processing module includes a packet monitoring module and a distribution learning module;
the data packet monitoring module is used for: the validity of the data packet is checked by acquiring quintuple information of the data packet, and if the data packet is determined to be legal, the data packet is sent to the distribution learning module;
the distribution learning module is used for: and determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet.
In one embodiment of the first aspect of the present application, the distribution learning module is specifically configured to:
acquiring application layer data of the data packet, wherein the application layer data comprises services of the data packet;
obtaining a second hash character string of the application layer data of the data packet by using a hash algorithm;
and taking the distribution control strategy associated with the second hash character string as the distribution control strategy of the data packet.
In one embodiment of the first aspect of the present application, the second processing module further includes a server monitoring module, where the server monitoring module is configured to: load information and/or health state information of the target server group are obtained;
the distribution learning module is further configured to: and determining at least one server from the target back-end server group based on the distribution control strategy of the data packet and the load information and/or the health state information of the target back-end server group.
In an embodiment of the first aspect of the present application, the first processing module is further configured to:
acquiring a distribution control strategy of a plurality of data packets transmitted in a preset period, wherein the plurality of data packets are data packets of the same service;
counting the use frequency of the distribution control strategy of the data packets;
Adjusting the priority of the distribution control strategy based on the frequency of use;
and transmitting a plurality of new data packets to be distributed through the priority of the adjusted distribution control strategy from high to low.
In one embodiment of the first aspect of the present application, the distribution control policy includes at least one of: transmitting the transmission path information, protocol information, flow control information and flow metering information of the data packet.
A second aspect of the embodiments of the present application provides a data processing method, applied to a data distribution device, where the method includes:
receiving a data packet to be distributed from a client device;
acquiring metadata of the data packet through a preset packet detection algorithm;
transmitting the data packet to a target back-end server group based on a distribution control policy associated with the data packet if the data packet matches the distribution control policy based on the metadata;
if the distribution control policy associated with the data packet is not matched based on the metadata, determining the distribution control policy of the data packet by parsing the application layer data of the data packet.
A third aspect of the embodiments of the present application provides a data processing method, applied to a data distribution device, the method including:
Receiving a live broadcast data packet to be distributed, wherein the live broadcast data packet comprises live broadcast service data uploaded by a main broadcasting terminal device or a spectator terminal device;
acquiring metadata of the live broadcast data packet through a preset packet detection algorithm;
transmitting the live data packet to a target live server group based on a distribution control policy associated with the live data packet if the metadata matches the distribution control policy based on the metadata;
if the metadata does not match the distribution control policy associated with the live data packet, determining the distribution control policy of the live data packet by parsing the application layer data of the live data packet.
A fourth aspect of the embodiments of the present application provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor to cause the electronic device to perform the method according to the second aspect of the present application or the method according to the third aspect.
A fifth aspect of embodiments of the present application provides a computer readable storage medium having stored thereon a computer program for execution by a processor to perform the method according to the second aspect of the present application, or the method according to the third aspect.
A sixth aspect of embodiments of the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method of the second aspect of the present application, or the method of the third aspect.
A seventh aspect of embodiments of the present application provides a chip comprising a processor for invoking a computer program in a memory to perform a method as described in the second aspect of the present application, or a method as described in the third aspect.
The embodiment of the application provides a data processing system, a data processing method and data processing equipment, wherein the system comprises client equipment deployed at a cloud end, data distribution equipment and a plurality of back-end server groups, the data distribution equipment comprises a first processing module and a second processing module, and the back-end server groups comprise a plurality of servers for processing the same service. The client device is used for sending the data packet to be distributed to the data distribution device; the data distribution device is used for matching a distribution control strategy associated with the data packet through the first processing module and transmitting the data packet to the target back-end server group based on the distribution control strategy; if the first processing module is not matched with the distribution control strategy associated with the data packet, the second processing module learns the distribution control strategy of the data packet so as to guide the first processing module to transmit the data packet. In the scheme, the first processing module and the second processing module which are relatively independent are divided, so that the overall distribution efficiency of the data distribution equipment can be improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of a data processing system according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a first processing module according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a second processing module according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of a data distribution device according to an embodiment of the present application;
fig. 6 is a flow chart of a data processing method according to an embodiment of the present application;
fig. 7 is a schematic flow chart of a data processing method according to an embodiment of the present application;
fig. 8 is a flow chart of a data processing method according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", and the like are used to distinguish the same item or similar items having substantially the same function and effect. For example, the first processing module and the second processing module are merely for distinguishing between different modules in the data distribution device, and are not limited in their order, and similar descriptions are given below: a first hash string and a second hash string. It will be appreciated by those of skill in the art that the words "first," "second," and the like do not limit the amount and order of execution, and that the words "first," "second," and the like do not necessarily differ.
In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one item (seed/number)" or the like means any combination of these items, including any combination of single item (seed/number) or plural items (seed/number). For example, at least one (seed/seed) of a, b or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.
It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "such as" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. It is to be appreciated that the use of words such as "exemplary" or "such as" are intended to present related concepts in a concrete fashion.
The following is a brief description of the terms of art to which the embodiments of the present application relate.
First, mongoDB, a database based on distributed file storage, written in the C++ language, is intended to provide an extensible high-performance data storage solution for Web applications.
Second, redis (Remote Dictionary Server), a remote dictionary service, is an open-source application programming interface API written in ANSI C language, supporting networks, log-type, key-Value databases that can be memory-based or persistent, and provides multiple languages.
Third, the RESTful API is an architectural style of an Application Program Interface (API) that requests access to or use of data using the hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP).
Fourth, the available area is an area where the cloud server has independent power and network in the same area, for example, in the eastern China 1 area, where there are a plurality of available areas. In the embodiment of the application, the available area can be provided with data distribution equipment so as to realize data package distribution of the area.
In order to facilitate understanding of the solution, the application scenario and the inventive concept of the present application will be explained first.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application. The data processing system provided by the application can be applied to an application scene shown in fig. 1. As shown in fig. 1, the application scenario includes: a client device 101, a data distribution device 102, and a plurality of backend server groups. The client device 101 may comprise a base station device or a core network device, such as a 5G base station or a 5G core network element. The data distribution device 102 may be a data distribution device deployed on a public cloud, private cloud, edge cloud, or hybrid cloud. The back-end server group includes a plurality of servers for processing the same service, and different back-end server groups in fig. 1 handle different services. For example, in fig. 1, a plurality of servers in the back-end server group 1 are used to process live traffic, a plurality of servers in the back-end server group 2 are used to process cloud game traffic, and a plurality of servers in the back-end server group 3 are used to process autopilot traffic. The plurality of servers of the back-end server group can be distributed deployed servers on public cloud, private cloud, edge cloud or hybrid cloud, the plurality of servers can be located in the same or different available areas, and the plurality of servers can be mutually isolated and independent.
Illustratively, in the live broadcast field, the client device 101 receives a service request sent by the anchor device, where the service request includes live broadcast data to be uploaded collected by the anchor device, the client device 101 generates a data packet to be distributed based on the service request, and sends the data packet to the data distribution device, where a service of the data packet is a live broadcast service. The data distribution device 102 analyzes and processes the data packet, and transmits the data packet to a back-end server group corresponding to the live broadcast service. The data distribution device 102 may select one or more servers of the backend server group to which the live service corresponds. In one example, the data distribution device 102 may select one or more servers from the back-end server group that perform well based on the operation of each server of the current back-end server group, and transmit data packets to the one or more servers. If the server of the transmission path is found to be faulty, the data distribution device 102 may reselect an appropriate server from the back-end server group, so as to reduce the loss or delay of the data packet caused by the device fault.
It should be noted that, in addition to the application scenario of the data processing system provided in the embodiment of the present application, the application scenario is applicable to the distribution of live broadcast service data packets in the live broadcast field, and is also applicable to the distribution of data packets of any cloud service such as the distribution of data packets of game services in the cloud game field, the distribution of data packets reported by vehicle-mounted devices in the autopilot field, and the like.
It should be noted that, the data packet to be distributed may be an upstream data packet from the client device to the back-end server group, or may be a downstream data packet from the back-end server group to the client device. For example, in the live broadcast field, the data packet to be distributed may be an uplink live broadcast data packet from the client device to the live broadcast server, or may be a downlink live broadcast data packet from the live broadcast server to the client device. The distribution processing principle is similar whether the data packet is an uplink data packet or a downlink data packet, and the following embodiments take the uplink data packet as an example for scheme description.
At present, the data distribution equipment deployed based on the above scene has the problem of low data distribution efficiency. In this regard, the embodiment of the application proposes a data processing system, where a data distribution device in the data processing system may configure a first processing module and a second processing module, and through the first processing module, a portion of data packets may be quickly distributed, and the portion of data packets may be directly based on a preset packet detection algorithm to quickly match a distribution control policy. And for the data packet which cannot be matched with the distribution control strategy quickly, the second processing module is used for determining the distribution control strategy of the data packet, and finally the first processing module is used for completing the distribution of the data packet. Based on the parallel processing of the two processing modules, the distribution efficiency of the data distribution equipment can be improved, and the experimental data display distribution efficiency is improved by at least 40%. The data distribution equipment is configured with distribution control strategies of different services and functions, so that the distribution of data packets of different services can be met.
In addition, the data distribution equipment is provided with a server monitoring module, the server monitoring module continuously monitors the running condition of the back-end server group by taking a single server in the back-end server group as granularity, realizes dynamic and accurate fine granularity flow scheduling, and avoids distributing data packets to the overload or unavailable back-end servers.
The data distribution equipment simultaneously supports the distribution of the uplink and downlink data packets, and realizes the common management of the uplink and downlink data.
The data processing system provided in the present application is described in detail below with reference to specific embodiments. It should be noted that, the technical solution provided in the embodiments of the present application may include some or all of the following, and the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.
FIG. 2 is a schematic diagram of a data processing system according to an embodiment of the present application. As shown in fig. 2, the data processing system of the present embodiment includes a client device 101 deployed in the cloud, a data distribution device 102, and a plurality of back-end server groups, where the data distribution device 102 includes a first processing module 1021 and a second processing module 1022, and one back-end server group includes a plurality of servers for processing the same service.
In this embodiment, the client device 101 may be a network device deployed on a public cloud, a private cloud, an edge cloud, or a hybrid cloud, such as a base station device or a core network device. The data distribution device 102 may be a device deployed on a public cloud, private cloud, edge cloud, or hybrid cloud. The plurality of servers in the back-end server set may be servers deployed on public cloud, private cloud, edge cloud, or hybrid cloud.
It should be noted that, the data distribution device 102 may be a virtual device on the cloud, such as a virtual machine or a container, instead of a physical device, for transmitting data packets on a virtual network. The back-end server group is also called a Service Set (Service Set), which is a group of Service instances that have the same Service type and are interchangeable, and these Service instances share the same Service data. Servers in the back-end server group are also known as Service instances (Service Instance), which are instances that provide back-end services (responsible for handling and responding to client requests), such as virtual compute server on cloud (ECS, elastic Compute Service) instances or container instances where applications are deployed.
The client device 101 is configured to: the data packet to be distributed is transmitted to the data distribution device 102.
The data distribution device 102 is configured to: matching, by the first processing module 1021, a distribution control policy associated with the data packet, and transmitting the data packet to the target backend server group based on the distribution control policy; if the first processing module 1021 does not match the distribution control policy associated with the data packet, the second processing module 1022 learns the distribution control policy of the data packet to instruct the first processing module 1021 to transmit the data packet.
For example, as shown in fig. 2, if the service of the data packet is a live service, the data distribution device may be quickly matched to a distribution control policy of the data packet by the first processing module 1021, and transmit the data packet to the back-end server group 1 based on the distribution control policy, where the back-end server group 1 includes a plurality of servers for processing the live service. If the first processing module 1021 does not match the distribution control policy of the data packet, the first processing module 1021 sends the data packet to the second processing module 1022, the second processing module 1022 learns the distribution control policy of the data packet, and the first processing module 1021 transmits the data packet to the back-end server group 1 based on the distribution control policy of the data packet learned by the second processing module 1022.
It should be noted that, the functions of the first processing module include fast matching the distribution control policy of the data packet without parsing the data packet, and executing data transmission. The function of the second processing module includes parsing the data packet to learn a distribution control policy for the data packet. In this embodiment, the first processing module and the second processing module are independent from each other, and when the first processing module is not matched with the data packet 1, the second processing module learns the distribution control policy of the data packet 1, and at the same time, the first processing module may execute the distribution control policy of the matched data packet 2 and the transmission of the data packet 2, instead of waiting for the second processing module to return to the distribution control policy of the data packet 1 all the time, so that the data distribution efficiency of the data distribution device may be improved.
In this embodiment, the distribution control policy of the data packet includes at least one of the following: transmission path information, protocol information, flow control information, and flow metering information of the transmission packet.
The transmission path information includes, but is not limited to, transmission path information for transmitting the data packet to the target server group, transmission path information for redirecting the data packet, transmission path information for rewriting the data packet, transmission path information for mirroring the data packet. The redirection is that the data distribution device determines that the bandwidth of the data packet is greater than a threshold value, or the destination address of the data packet is illegal, and the data distribution device returns a response message to the client device to inform the client device to compress the data packet, or modify the destination address of the data packet. The rewriting refers to that after the data distribution device receives a data packet, relevant data is written in the data packet, and then the data packet is forwarded. Mirroring refers to the simultaneous forwarding of data packets of a production environment to a server of a back-end testing environment.
The protocol information includes backend protocols that need to be satisfied when transmitting the data packets to the backend server set, which illustratively include, for example, HTTP, hypertext transfer security protocols (Hypertext Transfer Protocol Secure, HTTPs), hypertext transfer protocol version 2 (HTTP/2), gRPC, and the like.
The flow control information includes a bandwidth requirement for transmitting the data packet of the current service to the back-end server group, for example, a maximum bandwidth for transmitting the data packet of the current service to the back-end server group is 5M.
The traffic metering information includes a number requirement or a total number requirement of data packets transmitting the current traffic. For example, the maximum number of live service data packets transmitted during the preset period is 1000 data packets, or the total number of live service data packets transmitted during the preset period is at most 10G.
Based on the above distribution control policy, in one possible implementation manner, after obtaining the distribution control policy (including the determination of the first processing module itself or the indication of the second processing module) of the data packet, the first processing module of the data distribution device transmits the data packet to the target back-end server group based on the transmission path information in the distribution control policy of the data packet, where the target back-end server group is used to process the service of the data packet. In particular, the data distribution device may transmit the data packet to one or more servers of the target backend server group. And transmitting the data packet to a plurality of servers of the target back-end server group, so that single-point faults of the servers can be avoided, and the reliability of data transmission is improved.
In summary, the data processing system provided in this embodiment includes a client device disposed at a cloud end, a data distribution device, and a plurality of back-end server groups, where the data distribution device includes a first processing module and a second processing module, and the back-end server groups include a plurality of servers for processing the same service. The client device is used for sending the data packet to be distributed to the data distribution device; the data distribution device is used for matching a distribution control strategy associated with the data packet through the first processing module and transmitting the data packet to the target back-end server group based on the distribution control strategy; if the first processing module is not matched with the distribution control strategy associated with the data packet, the second processing module learns the distribution control strategy of the data packet so as to guide the first processing module to transmit the data packet. In the scheme, the first processing module and the second processing module which are relatively independent are divided, so that the overall distribution efficiency of the data distribution equipment can be improved.
In an alternative embodiment, the first processing module configures a preset packet detection algorithm, and the processing procedure of the first processing module is described in detail below.
In one possible implementation, after receiving a data packet sent from a client device, the first processing module of the data distribution device directly obtains metadata of the data packet through a preset packet detection algorithm without parsing the data packet. If the first processing module matches, in the lookup table, a distribution control policy associated with the metadata of the data packet based on the metadata of the data packet, the data packet is transmitted to the target back-end server group based on the matched distribution control policy. The lookup table comprises metadata of data packets of various services and distribution control strategies associated with the metadata.
According to the embodiment, the metadata of the data packet is obtained, and the proper distribution control strategy is quickly matched based on the metadata, so that the application layer data of the data packet is not required to be analyzed, and the distribution efficiency of the data packet is improved.
It should be noted that, the data packet sent by the client device is usually an encrypted data packet, and the packet detection algorithm directly performs an operation on the encrypted data packet to obtain metadata of the data packet, where the metadata of the data packet is usually character string data, and the metadata is encrypted data.
In another possible embodiment, if the first processing module of the data distribution device does not match the distribution control policy associated with the metadata of the data packet in the lookup table based on the metadata of the data packet, the data packet needs to be sent to the second processing module of the data distribution device, and the second processing module determines the distribution control policy of the data packet by parsing the application layer data of the data packet (a process of learning the distribution control policy). The application layer data of the data packet is plaintext data, and the application layer data comprises the service type of the data packet.
Optionally, the preset packet detection algorithm includes a hash algorithm. Correspondingly, the lookup table is a hash flow table, and the hash flow table comprises hash character strings of data packets of various services and distribution control strategies associated with the hash character strings.
It should be noted that, in this embodiment, the preset packet detection algorithm is not limited specifically, and other algorithms may be used to obtain metadata of the data packet, so as to construct a corresponding lookup table, improve the query efficiency, and implement a distribution control policy for quickly obtaining the data packet.
Illustratively, the first processing module of the data distribution device obtains the metadata of the data packet 1, and if the metadata does not match the distribution control policy associated with the data packet 1, the data packet 1 and the metadata of the data packet 1 are sent to the second processing module of the data distribution device, so that the second processing module parses the application layer data of the data packet 1 to determine the distribution control policy of the data packet 1. When the second processing module parses the data packet 1, the first processing module may obtain the metadata of the next data packet, for example, the data packet 2, and if the metadata of the data packet 3 matches the distribution control policy, the first processing module may first perform forwarding of the data packet 2, instead of waiting for the processing result of the data packet 1 by the second processing module, so that the overall execution speed of the data distribution apparatus may be improved.
The data distribution apparatus of the present embodiment, the first processing module is configured to acquire metadata of a data packet to be distributed, so as to be able to quickly match to a distribution control policy associated with the metadata of the data packet, and speed up transmission of the data packet. If the appropriate distribution control strategy cannot be matched, the data packet enters the second processing module. The second processing module is used for analyzing the application layer data of the data packet and matching a proper distribution control strategy based on the application layer data of the data packet. Compared with the second processing module, the processing speed of the first processing module is faster, and the overall distribution efficiency of the data distribution equipment can be improved by dividing the first processing module and the second processing module which are relatively independent. In addition, the second processing module analyzes the data packet to learn the distribution control strategy of the data packet, and the process does not influence the first processing module to obtain the metadata of other data packets in parallel, so that the data distribution efficiency of the data distribution equipment can be further improved.
Based on the above embodiments, fig. 3 is a schematic diagram of a first processing module according to an embodiment of the present application. As shown in fig. 3, the preset packet detection algorithm configured in the first processing module includes a hash algorithm, and the first processing module is further configured with a hash flow table, where the hash flow table includes an association relationship between a hash string and a distribution control policy.
The first processing module is specifically configured to obtain a first hash string of the data packet through a hash algorithm, where the first hash string is metadata of the data packet; if the first hash character string is matched in the Ha Xiliu table, acquiring a distribution control strategy associated with the first hash character string, and transmitting the data packet to a target back-end server group based on the distribution control strategy; if the first hash string is not matched in the Ha Xiliu table, the first processing module sends the data packet and the first hash string to the second processing module, so that the second processing module determines a distribution control policy of the data packet by analyzing application layer data of the data packet.
In an alternative embodiment, the first processing module further includes a transmission execution module, a flow control execution module, and a flow metering execution module. The transmission execution module can be used for transmitting the data packet meeting the back-end protocol requirement to the target back-end server group based on the transmission path information and the protocol information in the distribution control strategy of the data packet. The flow control execution module is used for controlling the bandwidth of the transmission execution module for transmitting the data packet based on the flow control information in the distribution control strategy of the data packet. The flow metering execution module is used for controlling the quantity or the size of the data packets transmitted by the transmission execution module based on the flow metering information in the distribution control strategy of the data packets. By means of the mutual matching of the execution modules in the first processing module, scheduling performance of the data distribution equipment is improved, and therefore data transmission efficiency and accuracy are improved.
On the basis of several of the above embodiments, fig. 4 is a schematic diagram of a second processing module according to an embodiment of the present application. As shown in fig. 4, the second processing module in the data distribution device includes a data packet listening module and a distribution learning module. The data packet monitoring module is used for verifying the validity of the data packet by acquiring quintuple information of the data packet, and if the data packet is determined to be legal, the data packet is sent to the distribution learning module. The distribution learning module is used for determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet. It should be appreciated that if the packet listening module determines that the packet is not legitimate, the packet will be discarded and an alarm triggered, such as sending an alarm message to the client device.
In this embodiment, the five-tuple information of the data packet includes a source internet protocol (Internet Protocol, IP) address, a source port, a destination IP address, a destination port, and a transport layer protocol of the data packet.
Optionally, transport layer protocols for the data packets include, but are not limited to, stream control transport protocol (Stream Control Transmission Protocol, STCP), transmission control protocol (Transmission Control Protocol, TCP), user datagram protocol (User Datagram Protocol, UDP), and the like.
Specifically, the data packet monitoring module may perform validity check on the data packet through the following several embodiments.
In an alternative embodiment, the data packet monitoring module acquires a destination port of the data packet, and if the destination port is forbidden, determines that the data packet is illegal; if the destination port is not disabled, determining the data packet method.
In an alternative embodiment, the data packet monitoring module acquires the destination IP address of the data packet, and if the destination IP address is forbidden, determines that the data packet is illegal; if the destination IP address is not forbidden, determining the data package method.
In an alternative embodiment, the data packet monitoring module acquires a destination port and a destination IP address of the data packet, and if at least one of the destination port and the destination IP address is forbidden, determines that the data packet is illegal; and if the destination port and the destination IP address are not forbidden, determining that the data package is legal.
The data packet monitoring module is configured in the second processing module so as to check the legality of the data packet, prevent malicious attack of external equipment and improve the safety of data transmission of the data distribution equipment.
It should be noted that, the data packet monitoring module may also use other policies to filter illegal data packets, which is not limited in any way in this embodiment of the present application.
In an optional embodiment, the distribution learning module is specifically configured to obtain application layer data of a data packet, where the application layer data includes a service of the data packet; obtaining a second hash character string of the application layer data of the data packet by using a hash algorithm; and taking the distribution control strategy associated with the second hash character string as the distribution control strategy of the data packet.
Specifically, the distribution learning module may analyze the data packet by using a transport layer protocol in the five-tuple information of the data packet, to obtain application layer data of the data packet. The distribution learning module stores hash character strings of application layer data of data packets of various services and distribution control strategies associated with the hash character strings in advance, so that the distribution control strategies of the data packets can be learned based on second hash character strings of the application layer data of the data packets.
Analyzing the data of the application layer of the data packet by the distribution learning module in the second processing module to acquire a distribution control strategy of the data packet, so as to guide the first processing module to transmit the data packet to the back-end server group based on the distribution control strategy of the data packet. Because the learning of the data packet distribution control strategy is independently executed by the distribution learning module, the transmission of the first processing module to other data packets is not influenced, and the overall execution speed of the data distribution equipment is improved.
In an alternative embodiment, as shown in fig. 4, the second processing module may further include a server monitoring module, where the server monitoring module is configured to obtain load information and/or health status information of the target server group. The distribution learning module is further configured to determine at least one server from the target backend server group based on the distribution control policy of the data packet and the load information and/or the health status information of the target backend server group.
The load information comprises load quantity, the health state information is used for indicating the state of the server, and the state of the server comprises available or unavailable. If the state of the server is unavailable, indicating that the server has a fault; if the state of the server is available, the server is indicated to be operating normally.
Specifically, after determining the target backend service group, the distribution learning module may determine one or more servers from the target backend service group through the following embodiments, so as to instruct the first processing module to transmit the data packet to the one or more servers:
in an alternative embodiment, the distribution learning module obtains the load amounts of the plurality of servers in the target backend server set from the server monitoring module, and selects one or more servers with load amounts smaller than the threshold value from the target backend server set.
In an alternative embodiment, the distribution learning module obtains the load amounts of a plurality of servers in the target backend server set from the server monitoring module, and selects a server with the smallest load amount from the target backend server set.
In an alternative embodiment, the distribution learning module obtains health status information of a plurality of servers in the target backend server set from the server monitoring module, and selects one or more servers in an available state from the target backend server set.
In an alternative embodiment, the distribution learning module obtains the load amounts and health state information of a plurality of servers in the target backend server set from the server monitoring module, and selects one or more servers which have load amounts smaller than a threshold value and are in an available state from the target backend server set.
In an alternative embodiment, the distribution learning module obtains the load capacity and the health status information of a plurality of servers in the target backend server set from the server monitoring module, and selects a server with the smallest load capacity from the servers in the available status of the target backend server set.
The server monitoring module is configured in the second processing module to monitor the load condition and the availability of each server in the back-end server group, so that a proper server is selected to transmit the data packet, the load balance of the back-end server is realized, the occurrence of the failure condition of data packet distribution is reduced, and the accuracy of data distribution is improved.
On the basis of the above embodiments, the first processing module is further configured to obtain a distribution control policy of the plurality of data packets that have been transmitted within a preset period of time; counting the use frequency of a distribution control strategy of a plurality of data packets; adjusting the priority of the distribution control strategy based on the frequency of use; and transmitting a plurality of new data packets to be distributed through the priority of the adjusted distribution control strategy from high to low. The priority of the distribution control policy of the initial data packet is typically preconfigured.
Taking a live broadcast data packet as an example, the distribution control policy of the initial live broadcast data packet includes a distribution control policy 1, a distribution control policy 2 and a distribution control policy 3 with priorities from high to low, where the distribution control policy 1 includes transmitting live broadcast data packets to the live broadcast server 1 and the live broadcast server 2, the distribution control policy 2 includes transmitting live broadcast data packets to the live broadcast server 2, and the distribution control policy 3 includes transmitting live broadcast data packets to the live broadcast server 3.
Based on the above example, the first processing module may transmit the data packet to the target backend server group based on the distribution control policy associated with the live data packet by obtaining the metadata of the live data packet if the metadata matches the distribution control policy associated with the live data packet. In some examples, there are a plurality of distribution control policies associated with the live data packet, such as the distribution control policy 1, the distribution control policy 2, and the distribution control policy 3, and the first processing module may select the distribution control policy 1 with the highest priority based on the distribution control policy associated with the live data packet and the priority of the distribution control policy of the live data packet, and transmit the live data packet to the live server 1 and the live server 2.
Based on the above example, if the first processing module counts the frequency of use of the distribution control policy for transmitting the live data packet in the preset period as follows: the first processing module can raise the priority of the distribution control strategy 3 and lower the priority of the distribution control strategy 2 based on the use frequency of the distribution control strategy of the live broadcast data packet, and the priority of the distribution control strategy of the adjusted live broadcast data packet can be the distribution control strategy 1, the distribution control strategy 3 and the distribution control strategy 2 from high to low according to the distribution control strategy 1 (20 times), the distribution control strategy 2 (1 time) and the distribution control strategy 3 (8 times).
Specifically, the first processing module may adjust the priority of the distribution control policy of the data packet according to the following embodiments:
in an alternative embodiment, if a certain distribution control policy of the data packet is not used within a first preset period, the priority of the distribution control policy may be reduced.
In an alternative embodiment, if a certain distribution control policy of the data packet is not used within the second preset period, the distribution control policy may be removed. The duration of the second preset time period is longer than the duration of the first preset time period.
In an alternative embodiment, if a certain distribution control policy of the data packet is used for a first preset period of time more than a preset number of times, the priority of the distribution control policy may be raised.
The priority of the distribution control strategy of the data package is managed by the first processing module, so that the use priority of the distribution control strategy of the data package is dynamically adjusted to adapt to the change of network environment, the data package is transmitted to a reliable back-end server, and the data distribution efficiency and accuracy of the data distribution equipment are improved.
The data distribution device shown in the above embodiments may further include a configuration control module, a database control module, and a packet control module, in addition to the first processing module and the second processing module.
Fig. 5 is a schematic diagram of a data distribution device according to an embodiment of the present application. As shown in fig. 5, the data distribution apparatus of the present embodiment includes: the system comprises a configuration control module, a database control module, a data packet control module, a first processing module and a second processing module.
The functions of the first processing module and the second processing module are referred to in the foregoing embodiments, and are not described herein.
The configuration control module mainly realizes two types of functions: firstly, providing a plurality of user access means such as a graphical user interface (graphical user interface, GUI), a Command Line Interface (CLI), a RESTful API and the like to the outside through a northbound interface of the data distribution equipment; and secondly, the configuration of various functional modules, such as the position deployment, data packet monitoring, data packet management, data packet distribution and the like of the data distribution equipment is realized through the southbound interface pair of the data distribution equipment.
The database control module is used for managing database resources with three purposes: the first is an OSS for storing a system log of the data distribution device; the second is a mongo db for storing various configurations of the data distribution device; the third is a dis for storing data packets in the data distribution device.
The data packet control module is used for managing data packets of STCP data packets, TCP data packets, UDP data packets and the like, and is also used for managing uplink data packets from the client device to the back-end server and downlink data packets from the back-end server to the client device.
It should be noted that the configuration control module, the database control module, and the packet control module described above may be regarded as related control modules for assisting the operation of the first processing module and the second processing module.
It should be further noted that, the configuration control module may deploy the data distribution device to at least two available areas in a distributed manner, for example, deploy the data distribution device to three available areas shown in fig. 5, such as available area a, available area B and available area C, where the power and network of the different available areas are independent, so that the problem that the single point deployment causes the failure of the data distribution service in the area can be avoided.
Fig. 6 is a schematic flow chart of a data processing method according to an embodiment of the present application. The data processing method of the present embodiment is applicable to the data distribution apparatus shown in fig. 1, as shown in fig. 6, and includes the steps of:
s601, a data packet to be distributed from the client device is received.
S602, acquiring metadata of the data packet through a preset packet detection algorithm.
If the metadata matches the distribution control policy associated with the data packet, S603 is performed.
If the distribution control policy associated with the data packet is not matched based on the metadata, S604 is performed.
S603, transmitting the data packet to the target back-end server group based on the distribution control strategy.
S604, determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet.
After S604, S603 is performed.
In this embodiment, the data distribution apparatus includes a first processing module and a second processing module.
In an alternative embodiment, the first processing module receives a data packet to be distributed from the client device, obtains metadata of the data packet through a preset packet detection algorithm, and if the metadata matches a distribution control policy associated with the data packet, the first processing module transmits the data packet to the target back-end server group based on the distribution control policy.
In an alternative embodiment, the first processing module sends the data packet and the metadata of the data packet to the second processing module if the metadata does not match the distribution control policy associated with the data packet. The second processing module determines a distribution control strategy of the data packet by analyzing application layer data of the data packet.
In an alternative embodiment, after determining the distribution control policy of the data packet, the second processing module sends the distribution control policy of the data packet and the metadata of the data packet to the first processing module, so that the first processing module stores the distribution control policy of the data packet and the metadata of the data packet locally in the first processing module, and after acquiring the same metadata, the first processing module can directly transmit the data packet based on the locally stored distribution control policy of the data packet associated with the metadata, thereby improving the data distribution efficiency.
In the data processing method shown in the above embodiment, for a data packet entering the second processing module, the second processing module analyzes the data packet to determine a distribution control policy associated with the data packet, so as to instruct the first processing module to transmit the data packet based on the distribution control policy determined by the second processing module. The processing process of the second processing module on the data packet can be regarded as the learning process of the data packet distribution control strategy, and the processing process is mutually independent from the processing process of the first processing module, so that the rapid transmission of the first processing module on other data packets is not influenced, and the data distribution efficiency can be improved.
The above data processing method will be described in detail with reference to the embodiment of fig. 6 by way of a specific embodiment. Fig. 7 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 7, the data processing method provided in this embodiment includes:
s701, the client device sends a data packet to be distributed to the data distribution device.
S702, a first processing module of the data distribution device acquires a first hash character string of the data packet by using a hash algorithm.
S703, the first processing module queries a hash stream table based on the first hash character string.
In one possible implementation, if the first hash string is not matched in the Ha Xiliu table, the following steps are performed:
s704, the first processing module sends the data packet and the first hash character string to a second processing module of the data distribution device.
S705, the second processing module determines a distribution control strategy of the data packet.
S706, the second processing module returns the data packet and the distribution control strategy of the data packet to the first processing module.
And S707, the first processing module transmits the data packet to the target back-end server group based on the distribution control strategy of the data packet determined by the second processing module.
In a possible implementation manner, if the first hash string is matched in the Ha Xiliu table, performing:
s708, the first processing module acquires a distribution control strategy associated with the first hash character string.
S709, the first processing module transmits the data packet to the target backend server group based on the distribution control policy associated with the first hash string.
The specific implementation principle and technical effect of each processing module of the data distribution device in this embodiment can be referred to the foregoing embodiments, and will not be described herein.
Fig. 8 is a schematic flow chart of a data processing method according to an embodiment of the present application. The data processing method of the present embodiment is applicable to the data distribution apparatus shown in fig. 1, as shown in fig. 8, and includes the steps of:
s801, receiving a live broadcast data packet to be distributed, wherein the live broadcast data packet comprises live broadcast service data uploaded by a main broadcasting terminal device or a spectator terminal device.
S802, acquiring metadata of the live broadcast data packet through a preset packet detection algorithm.
If the metadata matches the distribution control policy associated with the live data packet, S803 is performed.
If the metadata does not match the distribution control policy associated with the live data packet, S804 is performed.
S803, the live broadcast data packet is transmitted to the target live broadcast server group based on the distribution control strategy.
S804, determining a distribution control strategy of the live data packet by analyzing application layer data of the live data packet.
After S804, S803 is performed.
In this embodiment, the live broadcast data packet includes live broadcast service data with multiple functions, for example, live broadcast service data includes but is not limited to live broadcast pictures, praise, comments, and the like. Different distribution control policies are associated with different metadata for different functional live service data. For example, the distribution control policy corresponding to the data packet of the live view includes transmitting the data packet to the live server 1 and the live server 2, and the distribution control policy corresponding to the complimentary data packet includes transmitting the data packet to the live server 2, and so on.
It will be appreciated that in addition to the above examples, in the live broadcast process, the live broadcast service may further include other functions, for example, a system notification, a streaming control instruction, etc., where the system notification may be a live broadcast start time set by a user, and the streaming control instruction may be a resolution of an audio/video stream set according to a current network state, etc. In addition, more functions and corresponding distribution control strategies can be expanded according to live service requirements.
According to the data processing method shown in the embodiment, for the live broadcast data packet entering the second processing module, the distribution control strategy associated with the live broadcast data packet is determined through analysis of the live broadcast data packet by the second processing module, so that the first processing module is guided to transmit the live broadcast data packet based on the distribution control strategy determined by the second processing module. The processing process of the second processing module on the live broadcast data packet can be regarded as the learning process of the data packet distribution control strategy, and the processing process is mutually independent from the processing process of the first processing module, so that the quick transmission of the first processing module on other live broadcast data packets can not be influenced, and the data distribution efficiency can be improved.
Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 9, the electronic device includes: memory 901 and at least one processor (e.g., processor 902 of fig. 9). Memory 901 is used to store computer programs and may be configured to store various other data to support operations on the electronic device. The processor 902 is coupled to the memory 901, and is configured to execute the computer program in the memory 901, so as to implement the technical solution of the foregoing method embodiment, and its implementation principle and technical effect are similar, which is not described herein again.
In some embodiments, as shown in fig. 9, the electronic device further comprises: firewall 903, load balancer 904, communications component 905, power component 906, and other components. Only some of the components are schematically shown in fig. 9, which does not mean that the electronic device only comprises the components shown in fig. 9.
The embodiments of the present application provide a computer readable storage medium storing a computer program, which when executed by a processor, causes the processor to implement the technical solution of the foregoing method embodiments.
Embodiments of the present application provide a computer program product comprising a computer program/instruction which, when executed by a processor, causes the processor to implement the technical solution as in the previous method embodiments.
The embodiment of the application provides a chip, which comprises: the processing module and the communication interface, the processing module can execute the technical scheme in the embodiment of the method. In some embodiments, the chip further includes a storage module (e.g., a memory), where the storage module is configured to store instructions, and the processing module is configured to execute the instructions stored in the storage module, and execution of the instructions stored in the storage module causes the processing module to execute the technical solution in the foregoing method embodiment.
The memory may be an object store (Object Storage Service, OSS).
The memory may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device in which the communication component is located may access a wireless network based on a communication standard, such as a WiFi,2G, 3G, 4G/LTE, 5G, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
The power supply component provides power for various components of equipment where the power supply component is located. The power components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the devices in which the power components are located.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, compact disk-read only memory (CD-ROM), optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, user dialogue information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data is required to comply with the related laws and regulations and standards of the related country and region, and is provided with a corresponding operation portal for the user to select authorization or rejection.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (14)

1. The data processing system is characterized by comprising client equipment, data distribution equipment and a plurality of back-end server groups, wherein the client equipment is deployed at a cloud end, the data distribution equipment comprises a first processing module and a second processing module, and the back-end server groups comprise a plurality of servers for processing the same service;
The client device is configured to: transmitting a data packet to be distributed to the data distribution equipment;
the data distribution device is used for: matching, by the first processing module, a distribution control policy associated with the data packet, and transmitting the data packet to a target backend server group based on the distribution control policy; and if the first processing module is not matched with the distribution control strategy associated with the data packet, learning the distribution control strategy of the data packet through the second processing module so as to guide the first processing module to transmit the data packet.
2. The system of claim 1, wherein the first processing module configures a preset packet detection algorithm; the first processing module is specifically configured to:
acquiring metadata of the data packet through the preset packet detection algorithm;
transmitting the data packet to a target back-end server group based on a distribution control policy associated with the data packet if the metadata matches the distribution control policy;
if the distribution control policy associated with the data packet is not matched based on the metadata, the data packet and the metadata are sent to the second processing module to cause the second processing module to learn the distribution control policy of the data packet.
3. The system of claim 2, wherein the preset packet detection algorithm configured in the first processing module includes a hash algorithm, and a hash flow table is further configured in the first processing module, and the hash flow table includes an association relationship between a hash string and a distribution control policy;
the first processing module is specifically configured to: acquiring a first hash character string of the data packet through the hash algorithm, wherein the first hash character string is metadata of the data packet;
if the first hash character string is matched in the Ha Xiliu table, acquiring a distribution control strategy associated with the first hash character string, and transmitting the data packet to a target back-end server group based on the distribution control strategy;
and if the first hash character string is not matched in the Ha Xiliu table, the first processing module sends the data packet and the first hash character string to the second processing module so that the second processing module learns a distribution control strategy of the data packet.
4. The system according to claim 1, wherein the second processing module is specifically configured to:
and determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet.
5. The system of claim 4, wherein the second processing module comprises a packet listening module and a distribution learning module;
the data packet monitoring module is used for: the validity of the data packet is checked by acquiring quintuple information of the data packet, and if the data packet is determined to be legal, the data packet is sent to the distribution learning module;
the distribution learning module is used for: and determining a distribution control strategy of the data packet by analyzing the application layer data of the data packet.
6. The system of claim 5, wherein the distribution learning module is specifically configured to:
acquiring application layer data of the data packet, wherein the application layer data comprises services of the data packet;
obtaining a second hash character string of the application layer data of the data packet by using a hash algorithm;
and taking the distribution control strategy associated with the second hash character string as the distribution control strategy of the data packet.
7. The system of claim 5 or 6, wherein the system comprises a plurality of sensors,
the second processing module further comprises a server monitoring module, wherein the server monitoring module is used for: load information and/or health state information of the target server group are obtained;
The distribution learning module is further configured to: and determining at least one server from the target back-end server group based on the distribution control strategy of the data packet and the load information and/or the health state information of the target back-end server group.
8. The system of claim 1, wherein the first processing module is further configured to:
acquiring a distribution control strategy of a plurality of data packets transmitted in a preset period, wherein the plurality of data packets are data packets of the same service;
counting the use frequency of the distribution control strategy of the data packets;
adjusting the priority of the distribution control strategy based on the frequency of use;
and transmitting a plurality of new data packets to be distributed through the priority of the adjusted distribution control strategy from high to low.
9. The system of claim 1, wherein the system further comprises a controller configured to control the controller,
the distribution control policy includes at least one of: transmitting the transmission path information, protocol information, flow control information and flow metering information of the data packet.
10. A data processing method, characterized by being applied to a data distribution apparatus, the method comprising:
receiving a data packet to be distributed from a client device;
Acquiring metadata of the data packet through a preset packet detection algorithm;
transmitting the data packet to a target back-end server group based on a distribution control policy associated with the data packet if the data packet matches the distribution control policy based on the metadata;
if the distribution control policy associated with the data packet is not matched based on the metadata, determining the distribution control policy of the data packet by parsing the application layer data of the data packet.
11. A data processing method, characterized by being applied to a data distribution apparatus, the method comprising:
receiving a live broadcast data packet to be distributed, wherein the live broadcast data packet comprises live broadcast service data uploaded by a main broadcasting terminal device or a spectator terminal device;
acquiring metadata of the live broadcast data packet through a preset packet detection algorithm;
transmitting the live data packet to a target live server group based on a distribution control policy associated with the live data packet if the metadata matches the distribution control policy based on the metadata;
if the metadata does not match the distribution control policy associated with the live data packet, determining the distribution control policy of the live data packet by parsing the application layer data of the live data packet.
12. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor to cause the electronic device to perform the method of claim 10 or 11.
13. A computer readable storage medium having stored therein computing instructions which, when executed by a processor, implement the method of claim 10 or 11.
14. A computer program product comprising a computer program which, when executed by a processor, implements the method according to claim 10 or 11.
CN202310323795.9A 2023-03-30 2023-03-30 Data processing system, method and equipment Active CN116032929B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310323795.9A CN116032929B (en) 2023-03-30 2023-03-30 Data processing system, method and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310323795.9A CN116032929B (en) 2023-03-30 2023-03-30 Data processing system, method and equipment

Publications (2)

Publication Number Publication Date
CN116032929A true CN116032929A (en) 2023-04-28
CN116032929B CN116032929B (en) 2023-06-23

Family

ID=86089705

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310323795.9A Active CN116032929B (en) 2023-03-30 2023-03-30 Data processing system, method and equipment

Country Status (1)

Country Link
CN (1) CN116032929B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108282412A (en) * 2018-01-19 2018-07-13 世纪龙信息网络有限责任公司 Network shunt method, apparatus, system and computer equipment
CN110061921A (en) * 2019-04-17 2019-07-26 北京云杉世纪网络科技有限公司 A kind of cloud platform packet delivery method and system
CN110289999A (en) * 2019-06-21 2019-09-27 深圳前海微众银行股份有限公司 A kind of data processing method, system and device
CN111399749A (en) * 2019-01-02 2020-07-10 中国移动通信有限公司研究院 Data processing system and method
CN114500398A (en) * 2022-01-26 2022-05-13 中国农业银行股份有限公司 Processor cooperative acceleration method, device, equipment and medium
WO2022148363A1 (en) * 2021-01-06 2022-07-14 上海哔哩哔哩科技有限公司 Data transmission method and data transmission server

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108282412A (en) * 2018-01-19 2018-07-13 世纪龙信息网络有限责任公司 Network shunt method, apparatus, system and computer equipment
CN111399749A (en) * 2019-01-02 2020-07-10 中国移动通信有限公司研究院 Data processing system and method
CN110061921A (en) * 2019-04-17 2019-07-26 北京云杉世纪网络科技有限公司 A kind of cloud platform packet delivery method and system
CN110289999A (en) * 2019-06-21 2019-09-27 深圳前海微众银行股份有限公司 A kind of data processing method, system and device
WO2020253634A1 (en) * 2019-06-21 2020-12-24 深圳前海微众银行股份有限公司 Data processing method, system and device
WO2022148363A1 (en) * 2021-01-06 2022-07-14 上海哔哩哔哩科技有限公司 Data transmission method and data transmission server
CN114500398A (en) * 2022-01-26 2022-05-13 中国农业银行股份有限公司 Processor cooperative acceleration method, device, equipment and medium

Also Published As

Publication number Publication date
CN116032929B (en) 2023-06-23

Similar Documents

Publication Publication Date Title
US11089515B2 (en) Adaptable radio access network
US20170155695A1 (en) Method, device and system for uploading live video
US20170155928A1 (en) Method, Device and System for Playing Live Video
US20150333930A1 (en) Dynamic service function chaining
US11818100B2 (en) Automatic provisioning of streaming policies for video streaming control in CDN
US9219691B2 (en) Source-driven switch probing with feedback request
US9549010B2 (en) Method and apparatus for media session identification, tracking, and analysis
US20130103791A1 (en) Optimizing content delivery over a protocol that enables request multiplexing and flow control
CN108063714B (en) Network request processing method and device
US11711306B2 (en) Determining quality information for a route
US9807639B2 (en) Network traffic event management at the client terminal level
KR20140088172A (en) Methods to combine stateless and stateful server load balancing
US20150237027A1 (en) Apparatus, method and system for context-aware security control in cloud environment
US9331915B1 (en) Dynamic network traffic mirroring
EP3613183B1 (en) Content based byte-range caching using a dynamically adjusted chunk size
US10021221B2 (en) Methods and systems for detection and classification of multimedia content in secured transactions using pattern matching
CN114189885B (en) Network element information processing method, device and storage medium
US20180091631A1 (en) Systems and methods for writing prioritized http/2 data to a socket buffer
US11265254B1 (en) Systems and methods for determining a policy that allocates traffic associated with a network protocol type to a network slice
CN112751897A (en) Load balancing method, device, medium and equipment
US10244550B2 (en) Technique for scheduling transmission of content in an access network
CN116032929B (en) Data processing system, method and equipment
CN110798691B (en) Live content uploading and acquiring method and device
CN112995311B (en) Service providing method, device and storage medium
US20150229734A1 (en) Transparent internet cache and method for providing transparent internet cache

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant