CN113672410A - Data processing method and electronic device - Google Patents
Data processing method and electronic device Download PDFInfo
- Publication number
- CN113672410A CN113672410A CN202110982892.XA CN202110982892A CN113672410A CN 113672410 A CN113672410 A CN 113672410A CN 202110982892 A CN202110982892 A CN 202110982892A CN 113672410 A CN113672410 A CN 113672410A
- Authority
- CN
- China
- Prior art keywords
- service processing
- processes
- channel
- security service
- establishing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5016—Session
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The embodiment of the invention provides a data processing method and electronic equipment, wherein the method comprises the following steps: establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the working processes are respectively compatible with a processor core; establishing a security service processing process group which comprises a plurality of security service processing processes, wherein the plurality of security service processing processes are respectively compatible with one processor core and are used for completing security service processing of the message; establishing a first channel between the plurality of security service processing processes and the main process so that the plurality of security service processing processes complete registration with the main process through the first channel; and establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data processing method and an electronic device.
Background
With the rapid development of network technology, network security draws high attention of the whole society, network security communication equipment is deployed in a large amount in an actual network environment, and the performance and stability of the network security equipment are required to be higher by the complex topological environment and huge network traffic of the existing network. The excellent software design can make full use of hardware resources and improve the overall performance and stability of the network security equipment.
The multi-core network security device usually runs on a linux operating system and accelerates the network communication processing speed based on the DPDK technology. Typical implementation: starting DPDK data plane processes and a cpu core in one-to-one affinity, binding an interface receiving queue for each data plane process, selecting one receiving queue to receive messages by a DPDK user state driving interface through an rss hash algorithm, polling the bound interface queue by each data plane process, and realizing parallel message processing of a plurality of data plane processes by the whole machine. A basic forwarding engine in the data plane process is responsible for analyzing a two-layer Ethernet header, a three-layer IP header and a four-layer TCP header of a message, searching a route, maintaining session connection and receiving and sending an interface queue; and a security service engine in the data plane process is responsible for analyzing the application layer and carrying out deep security detection and processing on the message.
Currently employed secure data processing techniques include:
the first prior art is as follows:
the basic forwarding engine and the security service engine run in a data plane process to execute in series.
The second prior art is:
the basic forwarding engine runs on a DPDK data plane process, the security service engine runs on a linux management plane process, and the basic forwarding engine and the security service engine run independently.
However, the drawbacks of the two schemes include:
the first technology is as follows:
1) the security service cannot implement bypass. Because the safety service and the interface message receiving and sending logic are executed in series, the slow service processing can affect the interface packet receiving flow, so that the network card loses packet, and the direct forwarding can not be realized by skipping the service processing.
2) Independent restart cannot be implemented. Because the logics such as DPDK interface and memory initialization are uniformly set during starting, the logics cannot be set by a single core, shared memories are used among engines, the memory damage problem occurs to the security service of one core, and the complete machine can only be restarted to recover the security service.
3) When the device exceeds 64 cores, the number of the interface queues is limited, and the device cannot be bound with the cpu core one to one, so that the performance of the whole machine is reduced.
The second technology is as follows:
1) the data plane process needs to send the message to the linux kernel through special path mechanisms such as kni, the security service receives the message from the linux protocol stack for processing, then sends the message to the data plane process through the path and forwards the message through the interface, and the message is copied for multiple times in the whole process, so that the new establishment and data throughput performance of the whole machine are greatly reduced.
2) Each security service process needs to perform message analysis and session connection maintenance again, and is asynchronous with the session maintained by the data plane process, so that the code logic complexity and the development workload are greatly increased, and the subsequent maintenance cost is increased.
Disclosure of Invention
The invention provides a data processing method which can be applied to multi-core equipment and can improve the basic forwarding and safety service processing efficiency, and electronic equipment applying the method.
In order to solve the above technical problem, an embodiment of the present invention provides a data processing method, which is applied to a multi-core device, and the method includes:
establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the working processes are respectively compatible with a processor core;
establishing a security service processing process group which comprises a plurality of security service processing processes, wherein the plurality of security service processing processes are respectively compatible with one processor core and are used for completing security service processing of the message;
establishing a first channel between the plurality of security service processing processes and the main process so that the plurality of security service processing processes complete registration with the main process through the first channel;
and establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
Optionally, the session data created by the multiple work processes are shared, and the master process is shared with the global data variable among the multiple work processes;
the multiple security service processing processes have independent memories and support independent restarting, and the multiple security service processing processes only read and access data in the shared memory area created by the main process.
Optionally, the establishing a first channel between the multiple security service processing processes and the host process includes:
when each safety service processing process is started, the first channel is established between each safety service processing process and the main process, and the first channel comprises a data packet channel;
the establishing of the second channel between the plurality of work processes and the plurality of security service processing processes comprises:
and establishing the second channel between each safety service processing process and each work process, wherein the second channel comprises a DPDK lock-free queue.
Optionally, the service data recorded when the security service processing process processes a session message is private data corresponding to the session message and is associated with the corresponding session message;
the method further comprises the following steps:
the security service processing process determines a data management mode based on the memory occupation condition;
and the security service processing process independently maintains the private data and the session message associated with the private data based on the determined data management mode.
Optionally, the data management manner includes:
applying for a memory pool when each safety service processing process is initialized;
determining an ID address of the session message and a private data memory pool first address;
storing private data corresponding to the session message to the corresponding memory pool, and establishing a fast index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
Optionally, the data management manner includes:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a hash chain table for the target memory;
determining the ID address and the session age of the session message;
and calculating a hash index value at least based on the ID address and the session age of the session message.
Optionally, the method further comprises:
and the security service processing process acquires notification information sent by the working process after the session message is deleted, and releases the corresponding private data based on the notification information.
Optionally, the method further comprises:
the working process establishes a scheduling table for recording age locally, and the age of the scheduling table is consistent with the age of a state table of the safety processing service process recorded in the main process;
the working process carries out security service processing strategy matching on the newly established session message according to a strategy issued by a command line of the main process;
the working process determines a target safe service processing process based on the matching result and the scheduling table;
the work process records the ID address of the target security service processing process in the new session message;
when the target security service processing process finishes the processing of the newly established session message, the processed newly established session message is transferred back to the working process;
and the work process executes routing forwarding on the received new session message.
Optionally, the method further comprises:
and the working process determines whether the current operation scene is a target scene, if so, the message is forwarded to the safety service processing process, and the routing forwarding of the message is directly executed, wherein the target scene comprises the restarting of the safety service processing process, the total quantity of the target messages reaching a configured threshold value, and the second channel reaching full load.
Another embodiment of the present invention also provides an electronic device, including:
the first establishing module is used for establishing a basic forwarding process group and comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on the processor and is used for managing each process and resource, wherein the working processes are respectively compatible with a processor core;
the second establishing module is used for establishing a safety service processing process group which comprises a plurality of safety service processing processes, and the plurality of safety service processing processes are respectively compatible with the processor core and are used for completing the safety service processing of the message;
a third establishing module, configured to establish a first channel between the multiple security service processing processes and the host process, so that the multiple security service processing processes complete registration with the host process through the first channel;
and the fourth establishing module is used for establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention.
Fig. 2 is a diagram of a practical application process of the data processing method in another embodiment of the present invention.
Fig. 3 is a partial diagram of a practical application process of the data processing method according to another embodiment of the present invention.
Fig. 4 is a partial diagram of a practical application process of the data processing method according to another embodiment of the present invention.
Fig. 5 is a partial diagram of a practical application process of the data processing method according to another embodiment of the present invention.
Fig. 6 is a partial diagram of a practical application process of the data processing method according to another embodiment of the present invention.
Fig. 7 is a block diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The following detailed description of specific embodiments of the present invention is provided in connection with the accompanying drawings, which are not intended to limit the invention.
It will be understood that various modifications may be made to the embodiments disclosed herein. The following description is, therefore, not to be taken in a limiting sense, but is made merely as an exemplification of embodiments. Other modifications will occur to those skilled in the art within the scope and spirit of the disclosure.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and, together with a general description of the disclosure given above, and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.
These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment, given as a non-limiting example, with reference to the accompanying drawings.
It should also be understood that, although the invention has been described with reference to some specific examples, a person of skill in the art shall certainly be able to achieve many other equivalent forms of the invention, having the characteristics as set forth in the claims and hence all coming within the field of protection defined thereby.
The above and other aspects, features and advantages of the present disclosure will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.
Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely examples of the disclosure that may be embodied in various forms. Well-known and/or repeated functions and structures have not been described in detail so as not to obscure the present disclosure with unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure.
The specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may each refer to one or more of the same or different embodiments in accordance with the disclosure.
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
As shown in fig. 1 and fig. 2, an embodiment of the present invention provides a data processing method, which is applied to a multi-core device, and the method includes:
establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the plurality of working processes are respectively compatible with a processor core;
establishing a security service processing process group which comprises a plurality of security service processing processes, wherein the plurality of security service processing processes are respectively compatible with a processor core and are used for completing the security service processing of the message;
establishing a first channel between the plurality of security service processing processes and the main process so that the plurality of security service processing processes complete registration with the main process through the first channel;
and establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
The method in the embodiment can be applied to multi-core equipment, such as more than 64 cores, and all processor cores, such as CPU cores, can be fully utilized based on the method in the embodiment, so that the performance of the whole machine is effectively improved. Moreover, in the embodiment, data transmission is realized through different first channels and second channels for the safety service processing process and the work process in the basic forwarding process group respectively, and no influence is generated among the processes, and even if a certain process fails, the operation of other processes is not influenced, so that the basic forwarding process group (engine) and the safety service processing process group (engine) can operate in a high-efficiency and stable cooperation manner, and code coupling does not occur.
Specifically, the set of security service processing processes in this embodiment may be a security service processing engine, which includes a plurality of SE processes, each SE process is associated with one CPU core, and may be in one-to-one correspondence or many-to-one correspondence. The basic forwarding process group may be referred to as a basic forwarding engine, which includes a master process and a plurality of work processes, which may also be referred to as worker processes. The master process runs on a 0-core CPU and is responsible for resource initialization such as DPDK large-page memory, interface drive and other processes, including management of a work process and a security service processing process (SE process), and the worker process and the SE process need to be registered with the master after being started. Each worker process is compatible with one CPU core, and can be in one-to-one correspondence or many-to-one correspondence. And each worker process is simultaneously bound with a network card transceiving queue for receiving messages from a network card driver and then carrying out routing forwarding. In application, both the basic forwarding process group and the security service processing process group in this embodiment may be implemented based on the DPDK secondary process.
Further, in this embodiment, session data created by a plurality of work processes are shared, and global data variables between the main process and the plurality of work processes are shared;
the multiple security service processing processes have independent memories and support independent restarting, and the multiple security service processing processes only read and access data in the shared memory area created by the main process.
For example, in specific implementation, the process types running on the CPU cores in the multi-core device may be set through the configuration file, and then the DPDK running parameters may be set through the script. The master process and the worker process can be compiled into a binary program, the master process is started to read the number of workers and the bound CPU core information during running, and then all the worker processes are pulled up by the master. The session data structures created among the worker processes are shared, and certainly, other data structures can be shared. And sharing global data variables between the master process and the worker process. However, the worker process and the master process in this embodiment do not support separate restart.
Furthermore, the SE process is independently compiled into a binary program, the master process reads the configuration file through the script to obtain the starting quantity of the SE process and the binding information of the CPU core, then all the SE processes are pulled up, the resources among the SE processes are independent, the memory is independent, and the SE process supports independent restarting, so that when other SE processes are damaged, only the damaged SE process needs to be restarted, and the whole safety service processing engine does not need to be restarted. Meanwhile, each SE process can also perform independent memory management based on a dlmalloc library. Each SE process accesses the shared memory area by way of the shared memory area created by the read-only mapping master process, that is, only data such as session (message) can be read and accessed, so as to prevent writing bad shared memory from affecting the operation of other processes.
Further, in this embodiment, establishing the first channel between the multiple security service processing processes and the host process includes:
each safety business processing process establishes a first channel with a main process when being started, and the first channel comprises a data packet channel;
establishing a second channel between the plurality of work processes and the plurality of security service processing processes, comprising:
and a second channel is established between each safety service processing process and each work process, and the second channel comprises a DPDK lock-free queue.
For example, a channel established between each SE process and the master is a local socket channel, and is used for enabling the SE process to transmit registration information and heartbeat keep-alive information to the master process. The first number of channels may be M, for example, and is the same as the number of SE processes. The second channel established between each SE process and the worker process may optionally use the DPDK lock-free queue rte _ ring for transmitting messages and messages. Namely, each SE process can also receive a message forwarded by the worker process through the DPDK lock-free queue to perform security service processing. The number of the second channels may be N + M, where N is the number of worker processes started and M is the number of SE processes started.
Specifically, in this embodiment, all SE processes are uniformly managed by the master process, a local socket channel is established with the master process after each SE process is started, information such as a process type is registered to the master process, a heartbeat is sent to the master process at regular time, and the master process maintains SE state data. And for the worker process, the state information of the SE process on the master process is acquired through the shared memory to generate a local scheduling table for subsequently determining the SE process of the message processed by the user. As described above, the memory of each SE process is independent, data is not shared among the processes, the SE processes can be restarted independently after problems occur, and a master, a worker and other SE processes are not affected. In the restarting process, the worker process updates the local scheduling table, the old session flow of the damaged SE process is directly bypass, and the new session flow is scheduled to other SE processes of the same type; after restarting, the SE process registers to the master process again, so that the worker scheduling table is added to participate in flow processing.
Further, the session is composed of a five-tuple represented by a source IP, a destination IP, a source port, a destination port, and a four-layer protocol number, and represents a packet flow, which is used for caching a routing query result and a security service processing result. The service data recorded when each security service processing process processes the same session message is called the private data of the session. The session is created by the worker process, the private data is generated by the SE process, and the private data needs to be associated with the session in a one-to-one manner.
The method in this embodiment further includes:
the security service processing process determines a data management mode based on the memory occupation condition;
and the security service processing process independently maintains the private data and the session message associated with the private data based on the determined data management mode.
Specifically, the SE process performs read-only mapping on the session memory, independently maintains private data information and a corresponding relationship with the session, and ensures validity of the private data through the session age. The SE process provides two session private data management modes of a private data memory pool and a hash chain table, specifically selects which mode can be selected according to the actual memory occupation condition, if the memory occupation condition is serious, the hash chain table can be preferentially selected, and otherwise, the memory pool can be selected.
For example, the first embodiment:
the data management mode comprises the following steps:
applying for a memory pool when each safety business processing process is initialized;
determining an ID address of a session message and a first address of a private data memory pool;
storing private data corresponding to the session message to a corresponding memory pool, and establishing a fast index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
For example, as shown in fig. 3, when each SE process is initialized, a block of memory pool (total number of sessions × size of private data management header) is applied, and according to a session ID, a private data management header address can be quickly indexed and corresponding session private data information can be maintained. The private data management mode has the advantages of high indexing speed, no expense of memory application release, independence of each SE process, no mutual exclusion among processes and no risk of memory boundary crossing access. The dse _ private _ global in the graph is the private data memory pool first address, and each private data management structure is after the session _ id is offset. And Age is the session Age of the session message and is used for verifying the validity of the private data, and the 8-byte private _ data array stores the private data address.
Example two:
the data management mode comprises the following steps:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a Hash chain table for a target memory;
determining the ID address and the session age of the session message;
and calculating a hash index value at least based on the ID address and the session age of the session message.
For example, as shown in fig. 4, a hash chain table (hash chain table) may be used to manage private data, a hash value calculated according to a session _ id, a session _ age, and a hash _ lines is indexed to a corresponding hash chain table header, a session private data management structure memory needs to be applied when the private data is added, a corresponding chain table is inserted, and a corresponding chain table needs to be deleted and a private data management structure memory is released when the private data is deleted. And the dse _ private _ hash _ global in the graph is the initialized hash chain table head memory pool first address, and the chain table head is obtained by shifting according to the hash value calculated by the session _ id and the session _ age. The private data management structure body needs to apply for release, the age ensures the validity of the private data, the session _ id is used for hash collision detection, and the 8-byte private _ data is used for storing the private data address for the array.
Further, the method in this embodiment further includes:
and the safety service processing process acquires notification information sent after the session message is deleted by the working process, and releases corresponding private data based on the notification information.
For example, as shown in fig. 5, the release of the session private data in the SE process is performed by notifying the SE process through an msg message after the worker process deletes the session packet. Considering the influence of frequently issuing msg messages on performance under a newly-built flow model, when a worker process deletes a certain number (for example, 32) of session messages cumulatively, one msg message is encapsulated and sent to an SE process, and a session id is used for deleting private data msg messages: and encapsulating and sending the session age. After receiving the msg message, the SE process processes the session id in batch: and the session age indexes to the corresponding private data structure memory, and the validity of the private data is judged according to the age. Releasing effective private data according to destructor functions set by different modules; and for the condition that no private data release callback exists, the private data address is put into the se _ private _ free _ global resource recycling circular queue, the timer is used for timing, and the queue automatically releases the private data after the timeout.
Further, the method in this embodiment further includes:
the working process establishes a scheduling table for recording ages locally, and the age of the scheduling table is consistent with the age of a state table of the safety processing service process recorded in the main process;
the working process carries out security service processing strategy matching on the newly established session message according to a strategy issued by a command line of the main process;
the working process determines a target safety service processing process based on the matching result and the scheduling table;
the work process records the ID address of the target security service processing process in a newly-established session message;
when the target security service processing process finishes the processing of the newly established session message, the processed newly established session message is transferred back to the working process;
and the work process executes routing forwarding on the received new session message.
For example, when a worker process is subjected to message scheduling, in order to avoid performance influence caused by locking, as described above, each worker process locally generates a scheduling table, and records the age of the state table for comparison with the master process, and when the age of the state table of the SE process monitored by the master process changes, the local scheduling table of the worker process can be quickly updated. And when the worker process operates, performing security engine strategy matching on the newly-established session message according to a strategy in a command line issued by the master process, selecting a target SE process according to a strategy matching result and a scheduling table, recording the id of the target SE in the session, after one SE process finishes processing the message, returning the message to the worker process, continuously matching the next strategy by the worker process, recording the id of the matched and determined target SE in sequence until all strategies are processed, and then continuously executing routing forwarding. And the subsequent message of the session is directly forwarded according to the stored target SE id information, the survival state of the target SE process and the message channel state, namely the second channel state, are obtained before forwarding, and if the state is abnormal, the flow bypass is directly executed.
For example, as shown in FIG. 6, the WAF is run on the SE1 process and the SE3 process, and the IPS is run on the SE2 process and the SE4 process. After matching with WAF and IPS strategies, the Worker process selects an SE1 process and an SE2 process according to a local scheduling table, records corresponding SE id, sequentially forwards messages to an SE1 process, returns the messages and carried processing results to the Worker after the SE1 process finishes processing, dispatches the messages to an SE2 process for processing, and finally forwards the messages to the next level from the interface of the equipment according to a message output interface obtained by routing check. In the figure, the parts numbered 1 and 2 represent the first scheduling of the worker process and the SE process, and the parts numbered 3 and 4 represent the second scheduling.
Further, the method in this embodiment further includes:
and the working process determines whether the current operation scene is a target scene, if so, the message is forwarded to the safety service processing process, the routing forwarding of the message is directly executed, and the target scene comprises the restarting of the safety service processing process, the total quantity of the target messages reaching a configured threshold value and the full load of the second channel.
For example, under the following scenario, a worker process executes traffic bypass, that is, does not perform security service processing, and directly routes and forwards traffic, where the scenario includes:
1) the cross core queue rte _ ring is full;
2) the total number of the cross-core messages reaches a configured threshold value;
3) the SE process restarts.
The data processing method of the embodiment can fully utilize all CPU cores on multi-core safety equipment, such as equipment with more than 64 cores, and improve the performance of the whole machine; the method can realize the single-core restart of the security service engine and the traffic bypass on the basic forwarding engine on the premise of ensuring the performance, and improve the running stability of the equipment, thereby achieving the aims of improving the cost performance of the equipment and reducing the later maintenance cost. In addition, in this embodiment, a basic forwarding process and a security service process can be implemented based on a DPDK secondary process, the basic forwarding process is bound to a limited interface queue for message transceiving, the security service processing process only performs service processing, the two processes are independently compiled and run, and zero-copy transmission of a message between the two processes is implemented through a lock-free queue, thereby further improving data processing efficiency.
As shown in fig. 7, another embodiment of the present application also provides an electronic device, including:
the first establishing module is used for establishing a basic forwarding process group and comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on the processor and is used for managing each process and resource, wherein the working processes are respectively compatible with a processor core;
the second establishing module is used for establishing a safety service processing process group which comprises a plurality of safety service processing processes, and the plurality of safety service processing processes are respectively compatible with a processor core and are used for completing the safety service processing of the message;
the third establishing module is used for establishing a first channel between the plurality of security service processing processes and the main process so as to enable the plurality of security service processing processes to complete the registration with the main process through the first channel;
and the fourth establishing module is used for establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
Optionally, session data created by a plurality of working processes are shared, and global data variables between the main process and the plurality of working processes are shared;
the multiple security service processing processes have independent memories and support independent restarting, and the multiple security service processing processes only read and access data in the shared memory area created by the main process.
Optionally, establishing a first channel between the multiple security service processing processes and the host process includes:
each safety business processing process establishes a first channel with a main process when being started, and the first channel comprises a data packet channel;
establishing a second channel between the plurality of work processes and the plurality of security service processing processes, comprising:
and a second channel is established between each safety service processing process and each work process, and the second channel comprises a DPDK lock-free queue.
Optionally, when the security service processing process processes a session message, the recorded service data is private data of the corresponding session message and is associated with the corresponding session message;
the electronic device further includes:
and the security service processing module is used for determining a data management mode based on the memory occupation condition and independently maintaining the private data and the session message associated with the private data based on the determined data management mode.
Optionally, the data management manner includes:
applying for a memory pool when each safety business processing process is initialized;
determining an ID address of a session message and a first address of a private data memory pool;
storing private data corresponding to the session message to a corresponding memory pool, and establishing a fast index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
Optionally, the data management manner includes:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a Hash chain table for a target memory;
determining the ID address and the session age of the session message;
and calculating a hash index value at least based on the ID address and the session age of the session message.
Optionally, the secure transaction processing module is further configured to:
and obtaining notification information sent after the session message is deleted by the working process, and releasing corresponding private data based on the notification information.
Optionally, the electronic device further comprises a working module:
the system comprises a scheduling table used for establishing a scheduling table for recording age locally, wherein the age of the scheduling table is consistent with the age of a state table of a security processing service process recorded in a main process, and security service processing strategy matching is carried out on a newly-established session message according to a strategy issued by a command line of the main process; determining a target security service processing process based on the matching result and the scheduling table; recording the ID address of the target security service processing process in a newly-established session message; when the target security service processing process finishes the processing of the newly established session message, the processed newly established session message is transferred back to the working process; and executing routing forwarding on the received new session message.
Optionally, the work module is further configured to:
and determining whether the current operation scene is a target scene, if so, forwarding the message to a security service processing process, and directly executing routing forwarding of the message, wherein the target scene comprises restarting the security service processing process, enabling the total quantity of the target messages to reach a configured threshold value, and enabling the second channel to reach full load.
Another embodiment of the present application further provides an electronic device, including:
one or more processors;
a memory configured to store one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the processing methods described above.
An embodiment of the present application also provides a storage medium, on which a computer program is stored, which when executed by a processor implements the processing method as described above. It should be understood that each solution in this embodiment has a corresponding technical effect in the foregoing method embodiments, and details are not described here.
Embodiments of the present application also provide a computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed, cause at least one processor to perform a processing method such as the embodiments described above. It should be understood that each solution in this embodiment has a corresponding technical effect in the foregoing method embodiments, and details are not described here.
It should be noted that the computer storage media of the present application can be computer readable signal media or computer readable storage media or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
It should be understood that although the present application has been described in terms of various embodiments, not every embodiment includes only a single embodiment, and such description is for clarity purposes only, and those skilled in the art will recognize that the embodiments described herein may be combined as suitable to form other embodiments, as will be appreciated by those skilled in the art.
The above embodiments are only exemplary embodiments of the present invention, and are not intended to limit the present invention, and the scope of the present invention is defined by the claims. Various modifications and equivalents may be made by those skilled in the art within the spirit and scope of the present invention, and such modifications and equivalents should also be considered as falling within the scope of the present invention.
Claims (10)
1. A data processing method is applied to multi-core equipment, and comprises the following steps:
establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the working processes are respectively compatible with a processor core;
establishing a security service processing process group which comprises a plurality of security service processing processes, wherein the plurality of security service processing processes are respectively compatible with one processor core and are used for completing security service processing of the message;
establishing a first channel between the plurality of security service processing processes and the main process so that the plurality of security service processing processes complete registration with the main process through the first channel;
and establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
2. The method of claim 1, wherein each session data created by the plurality of work processes is shared, and the master process is shared with global data variables among the plurality of work processes;
the multiple security service processing processes have independent memories and support independent restarting, and the multiple security service processing processes only read and access data in the shared memory area created by the main process.
3. The method of claim 1, wherein the establishing a first channel between the plurality of secure transaction processes and the host process comprises:
when each safety service processing process is started, the first channel is established between each safety service processing process and the main process, and the first channel comprises a data packet channel;
the establishing of the second channel between the plurality of work processes and the plurality of security service processing processes comprises:
and establishing the second channel between each safety service processing process and each work process, wherein the second channel comprises a DPDK lock-free queue.
4. The method according to claim 2, wherein the service data recorded when the security service processing process processes a session message is private data corresponding to the session message and is associated with the corresponding session message;
the method further comprises the following steps:
the security service processing process determines a data management mode based on the memory occupation condition;
and the security service processing process independently maintains the private data and the session message associated with the private data based on the determined data management mode.
5. The method of claim 4, wherein the data management manner comprises:
applying for a memory pool when each safety service processing process is initialized;
determining an ID address of the session message and a private data memory pool first address;
storing private data corresponding to the session message to the corresponding memory pool, and establishing a fast index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
6. The method of claim 4, wherein the data management manner comprises:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a hash chain table for the target memory;
determining the ID address and the session age of the session message;
and calculating a hash index value at least based on the ID address and the session age of the session message.
7. The method of claim 4, further comprising:
and the security service processing process acquires notification information sent by the working process after the session message is deleted, and releases the corresponding private data based on the notification information.
8. The method of claim 2, further comprising:
the working process establishes a scheduling table for recording age locally, and the age of the scheduling table is consistent with the age of a state table of the safety processing service process recorded in the main process;
the working process carries out security service processing strategy matching on the newly established session message according to a strategy issued by a command line of the main process;
the working process determines a target safe service processing process based on the matching result and the scheduling table;
the work process records the ID address of the target security service processing process in the new session message;
when the target security service processing process finishes the processing of the newly established session message, the processed newly established session message is transferred back to the working process;
and the work process executes routing forwarding on the received new session message.
9. The method of claim 1, further comprising:
and the working process determines whether the current operation scene is a target scene, if so, the message is forwarded to the safety service processing process, and the routing forwarding of the message is directly executed, wherein the target scene comprises the restarting of the safety service processing process, the total quantity of the target messages reaching a configured threshold value, and the second channel reaching full load.
10. An electronic device, comprising:
the first establishing module is used for establishing a basic forwarding process group and comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on the processor and is used for managing each process and resource, wherein the working processes are respectively compatible with a processor core;
the second establishing module is used for establishing a safety service processing process group which comprises a plurality of safety service processing processes, and the plurality of safety service processing processes are respectively compatible with the processor core and are used for completing the safety service processing of the message;
a third establishing module, configured to establish a first channel between the multiple security service processing processes and the host process, so that the multiple security service processing processes complete registration with the host process through the first channel;
and the fourth establishing module is used for establishing a second channel between the plurality of working processes and the plurality of safety service processing processes so that the plurality of safety service processing processes obtain the message through the second channel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110982892.XA CN113672410B (en) | 2021-08-25 | 2021-08-25 | Data processing method and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110982892.XA CN113672410B (en) | 2021-08-25 | 2021-08-25 | Data processing method and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113672410A true CN113672410A (en) | 2021-11-19 |
| CN113672410B CN113672410B (en) | 2023-08-25 |
Family
ID=78546232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110982892.XA Active CN113672410B (en) | 2021-08-25 | 2021-08-25 | Data processing method and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113672410B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115150464A (en) * | 2022-06-22 | 2022-10-04 | 北京天融信网络安全技术有限公司 | Application proxy method, device, equipment and medium |
| CN115208941A (en) * | 2022-07-13 | 2022-10-18 | 北京天融信网络安全技术有限公司 | Data processing method and system based on session connection |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108270676A (en) * | 2016-12-31 | 2018-07-10 | 普天信息技术有限公司 | A kind of network data processing method and device based on IntelDPDK |
| CN109831394A (en) * | 2017-11-23 | 2019-05-31 | 杭州华为数字技术有限公司 | Data processing method, terminal and computer storage medium |
| CN109828842A (en) * | 2019-01-29 | 2019-05-31 | 上海兴畅网络技术股份有限公司 | A kind of high-performance data acquisition engine method based on DPDK technological development |
| CN111385296A (en) * | 2020-03-04 | 2020-07-07 | 深信服科技股份有限公司 | Business process restarting method, device, storage medium and system |
| CN112104572A (en) * | 2020-09-11 | 2020-12-18 | 北京天融信网络安全技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112437023A (en) * | 2020-10-12 | 2021-03-02 | 北京天融信网络安全技术有限公司 | Virtualized security network element data processing method, system, medium and cloud platform |
| CN112615784A (en) * | 2020-12-17 | 2021-04-06 | 北京天融信网络安全技术有限公司 | Method, device, storage medium and electronic equipment for forwarding message |
-
2021
- 2021-08-25 CN CN202110982892.XA patent/CN113672410B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108270676A (en) * | 2016-12-31 | 2018-07-10 | 普天信息技术有限公司 | A kind of network data processing method and device based on IntelDPDK |
| CN109831394A (en) * | 2017-11-23 | 2019-05-31 | 杭州华为数字技术有限公司 | Data processing method, terminal and computer storage medium |
| CN109828842A (en) * | 2019-01-29 | 2019-05-31 | 上海兴畅网络技术股份有限公司 | A kind of high-performance data acquisition engine method based on DPDK technological development |
| CN111385296A (en) * | 2020-03-04 | 2020-07-07 | 深信服科技股份有限公司 | Business process restarting method, device, storage medium and system |
| CN112104572A (en) * | 2020-09-11 | 2020-12-18 | 北京天融信网络安全技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112437023A (en) * | 2020-10-12 | 2021-03-02 | 北京天融信网络安全技术有限公司 | Virtualized security network element data processing method, system, medium and cloud platform |
| CN112615784A (en) * | 2020-12-17 | 2021-04-06 | 北京天融信网络安全技术有限公司 | Method, device, storage medium and electronic equipment for forwarding message |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115150464A (en) * | 2022-06-22 | 2022-10-04 | 北京天融信网络安全技术有限公司 | Application proxy method, device, equipment and medium |
| CN115150464B (en) * | 2022-06-22 | 2024-03-15 | 北京天融信网络安全技术有限公司 | Application proxy method, device, equipment and medium |
| CN115208941A (en) * | 2022-07-13 | 2022-10-18 | 北京天融信网络安全技术有限公司 | Data processing method and system based on session connection |
| CN115208941B (en) * | 2022-07-13 | 2024-04-23 | 北京天融信网络安全技术有限公司 | Data processing method and system based on session connection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113672410B (en) | 2023-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111371779B (en) | Firewall based on DPDK virtualization management system and implementation method thereof | |
| CN107634984B (en) | File synchronization method based on one-way transmission channel | |
| WO2019184164A1 (en) | Method for automatically deploying kubernetes worker node, device, terminal apparatus, and readable storage medium | |
| CN113672410B (en) | Data processing method and electronic device | |
| CN109800160B (en) | Cluster server fault testing method and related device in machine learning system | |
| CN105991412B (en) | Information push method and device | |
| CN112558948A (en) | Method and device for identifying message under mass flow | |
| WO2017148297A1 (en) | Method and device for joining tables | |
| CN108924008A (en) | A kind of dual controller data communications method, device, equipment and readable storage medium storing program for executing | |
| JP6359111B2 (en) | Method and apparatus for processing data packets based on parallel protocol stack instances | |
| CN107688512A (en) | One kind optimization virtual-machine data backup method and system | |
| CN111309700B (en) | Control method and system for multi-sharing directory tree | |
| CN104866528A (en) | Multi-platform data acquisition method and system | |
| CN114363269B (en) | Message transmission method, system, equipment and medium | |
| CN103618778A (en) | System and method for achieving data high concurrency through Linux virtual host | |
| CN114371935A (en) | Gateway processing method, gateway, device and medium | |
| EP3097662B1 (en) | Methods, systems and computer readable media for testing network devices using simulated application traffic | |
| US20190044835A1 (en) | Technologies for filtering network packets on ingress | |
| CN101789901A (en) | Accelerating method and device of transmitting system positioned on multinuclear platform | |
| CN110795209B (en) | Control method and device | |
| CN112114983A (en) | Communication method, device and equipment based on shared memory | |
| CN109246121B (en) | Attack defense method and device, Internet of things equipment and computer readable storage medium | |
| CN110445580A (en) | Data transmission method for uplink and device, storage medium, electronic device | |
| CN112217689B (en) | Network message tracking method and system based on OpenStack | |
| CN113489775B (en) | Seven-layer load balancing server and load balancing method based on VPP |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |