CN111240845A - Data processing method, device and storage medium - Google Patents
Data processing method, device and storage medium Download PDFInfo
- Publication number
- CN111240845A CN111240845A CN202010033177.7A CN202010033177A CN111240845A CN 111240845 A CN111240845 A CN 111240845A CN 202010033177 A CN202010033177 A CN 202010033177A CN 111240845 A CN111240845 A CN 111240845A
- Authority
- CN
- China
- Prior art keywords
- data
- data processing
- submodule
- processed
- processing module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application discloses a data processing method, a data processing device and a storage medium, wherein for each data processing module, an input submodule in the data processing module can be triggered to acquire data to be processed; triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read data to be processed. The embodiment of the application can improve the data processing efficiency.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, and storage medium.
Background
In recent years, with the rapid development of integrated circuits, people have increasingly strong requirements for high computing performance and low power consumption, and the development of general purpose processors (CPUs) is facing to the ceiling, and then the development is facing to the direction of increasing the number of processing cores, at present, in order to improve the data processing performance of a computer device, tens of CPUs (even hundreds of CPUs) can be combined into one computer device by using Non-uniform memory Access (NUMA) technology, and the computer device has a plurality of data processing modules, each data processing module includes at least one CPU and an independent local memory, and the data processing modules can Access each other through an interconnection module (for example, referred to as Crossbar). However, the latency to access remote memory far exceeds local memory.
Computer devices usually have only one external device for inputting data to be processed, which can be provided in only one data processing module. Therefore, when data is processed, the current data processing method is inefficient because a remote memory is accessed or only one data processing module is applied.
Specifically, referring to fig. 6a, the external device may store the data to be processed in the local memory, and the CPU in the other data processing module needs to access the local memory (remote memory with respect to the CPU) of the external device to read the data to be processed.
Specifically, referring to fig. 6b, the external device stores the data to be processed into the local memory and the remote memory, and the CPU reads the data to be processed from the respective corresponding local memories.
Specifically, referring to fig. 6c, the external device stores the data to be processed in the local memory, and only the CPU in the corresponding data processing module is used to read the data to be processed in the local memory for processing, and the CPUs in the other data processing modules are all idle.
Disclosure of Invention
In view of this, embodiments of the present application provide a data processing method, an apparatus, and a storage medium, which can improve data processing efficiency.
In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a computer device, where the computer device includes a plurality of data processing modules, and each data processing module includes an input submodule, a storage submodule, and a processing submodule, and the data processing method includes:
for each data processing module, triggering an input submodule in the data processing module to acquire data to be processed;
storing the acquired data to be processed to a storage submodule in the data processing module based on an input submodule in the data processing module;
triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and triggering a processing submodule in the data processing module to process the read data to be processed.
In an embodiment, the input sub-module includes at least one input channel, the processing sub-module includes at least one data processing core, and before storing the acquired data to be processed into a storage sub-module in the data processing module based on the input sub-module in the data processing module, the method further includes:
acquiring a data reading core according to a data processing core in a processing submodule and a preset data processing program in the data processing module;
determining a target input channel used for communicating with the storage submodule in the input submodule according to the data reading core;
establishing a mapping between the target input channel and the data reading core;
and connecting the target input channel to the storage sub-module corresponding to the data reading core according to the mapping, wherein the storage sub-module corresponding to the data reading core is the storage sub-module in the data processing module where the data reading core is located.
In an embodiment, according to the mapping, the target input channel is connected to the storage sub-module corresponding to the data reading core, where the storage sub-module corresponding to the data reading core is the storage sub-module in the data processing module where the data reading core is located, and the method includes:
determining a storage submodule corresponding to the data reading core according to the mapping;
analyzing the data to be processed acquired by the target input channel to acquire communication identification information in the data to be processed;
acquiring index information of the data to be processed in the storage submodule according to the communication identification information;
connecting the target input channel to the storage submodule based on the index information.
In an embodiment, the storing the acquired data to be processed to a storage sub-module in the data processing module based on an input sub-module in the data processing module includes:
and storing the data to be processed acquired by the target input channel to a storage submodule in the data processing module according to the index information.
In an embodiment, the triggering a processing submodule in the data processing module to read to-be-processed data in a storage submodule of the data processing module includes:
acquiring index information of the data to be processed corresponding to the data reading core in the storage submodule based on the mapping between the data reading core and the target input channel;
and triggering the data reading core to read the data to be processed corresponding to the data reading core from a storage submodule in the data processing module through the index information.
In an embodiment, acquiring a data reading core according to a data processing core in a processing sub-module and a preset data processing program in the data processing module includes:
acquiring a data reading subprogram according to a preset data processing program corresponding to the data processing module;
and establishing mapping between a processing core in the processing submodule in the data processing module and the data reading subprogram, and taking the data processing core after establishing mapping as a data reading core.
In an embodiment, the data reading sub-program includes a plurality of processes, a mapping is established between a processing core in the processing sub-module in the data processing module and the data reading sub-program, and the data processing core after the mapping is established is used as a data reading core, including:
and establishing mapping between different data processing cores in the processing sub-module and each process in the corresponding data reading sub-program to obtain a plurality of data reading cores.
In an embodiment, the data reading sub-program includes a plurality of threads, a processing core in the processing sub-module in the data processing module is mapped with the data reading sub-program, and the data processing core after mapping is set up is used as a data reading core, and the method further includes:
and establishing mapping between different data processing cores in the processing sub-module and each thread in the corresponding data reading program to obtain a plurality of data reading cores.
In an embodiment, triggering a processing sub-module in the data processing module to read data to be processed in a storage sub-module of the data processing module includes:
determining the calling sequence of each process or thread according to the preset data processing program;
and calling the corresponding data reading core according to the calling sequence, and reading the data to be processed corresponding to the data reading core from the corresponding storage sub-module.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the acquisition unit is used for triggering an input submodule in each data processing module to acquire data to be processed;
the storage-to unit is used for triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module;
the reading unit is used for triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and the processing unit is used for triggering the processing submodule in the data processing module to process the read data to be processed.
In a third aspect, a storage medium is provided in the embodiments of the present application, on which a computer program is stored, and when the computer program runs on a computer, the computer program is caused to execute the data processing method provided in any of the embodiments of the present application.
According to the embodiment of the application, for each data processing module, an input submodule in the data processing module is triggered to acquire data to be processed; triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read data to be processed. According to the embodiment of the application, the data to be processed is input to the computer equipment by simultaneously using the plurality of input sub-modules, each input sub-module only accesses the local storage sub-module, and each processing sub-module only accesses the local storage sub-module, so that on one hand, time waste caused by accessing a remote storage module is avoided, on the other hand, a plurality of groups of data to be processed can be simultaneously processed, and therefore the data processing efficiency can be improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a data processing method according to an embodiment of the present application.
Fig. 2a is a schematic flowchart of a data processing method according to an embodiment of the present application.
Fig. 2b is another schematic flow chart of the data processing method according to the embodiment of the present application.
Fig. 3a is a schematic structural diagram of a first data processing method according to an embodiment of the present application.
Fig. 3b is a schematic structural diagram of a second data processing method according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a server according to an embodiment of the present application.
Fig. 5a is a schematic diagram of a data processing module in a server according to an embodiment of the present application.
Fig. 5b is a schematic diagram of a first traffic processing scenario of the server according to the embodiment of the present application.
Fig. 5c is a schematic diagram of a second traffic processing scenario of the server according to the embodiment of the present application.
Fig. 5d is a schematic diagram of a third traffic processing scenario of the server according to the embodiment of the present application.
Fig. 6a is a schematic structural diagram of a first data processing module provided in the prior art.
Fig. 6b is a schematic structural diagram of a second data processing module provided in the prior art.
Fig. 6c is a schematic structural diagram of a third data processing module provided in the prior art.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a data processing method, a data processing device and a storage medium.
The term "data" in the embodiments of the present invention is a generic term of all media that can input symbols to a computer and be processed by a computer program, and is an object of storage, forwarding, and processing by the computer.
The processing of the data in the embodiment of the invention comprises the forms of forwarding, analysis, calculation, processing and the like.
An embodiment of the present invention provides a data processing system, including any one of the data processing apparatuses provided in the embodiments of the present invention, where the data processing apparatus may be specifically integrated in a server. The server may be represented as a single server, or may be represented as a server cluster formed by a plurality of servers.
In addition, the data processing system may also include other computer devices, such as a terminal and other servers, and the terminal may be: a mobile phone, a tablet Computer, a notebook Computer, or a Personal Computer (PC).
For example, referring to fig. 1, taking the example that the data processing apparatus is integrated on a single server, the data processing system includes a terminal and a server, and the terminal and the server are linked via a network. The network includes network entities such as routers and gateways.
For example, a user may operate on a terminal, the terminal may transmit data to be processed to a server integrated with a data processing apparatus based on the user operation, the server includes a plurality of data processing modules, the data processing modules include an input submodule, a storage submodule, and a processing submodule, and the server may acquire the data to be processed through the input submodule.
The server can trigger an input submodule in each data processing module to acquire data to be processed for each data processing module; triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read data to be processed.
The data processing method can be applied to process the traffic data, and accordingly, the server is a server for processing the traffic data, such as a traffic analysis server, a web server, a gateway server, and the like.
The above example of fig. 1 is only an example of a system architecture for implementing the embodiment of the present invention, and the embodiment of the present invention is not limited to the system architecture shown in fig. 1, and various embodiments of the present invention are proposed based on the system architecture.
The following are detailed below. The numbers in the following examples are not intended to limit the order of preference of the examples.
The present embodiment will be described from the perspective of a data processing apparatus, which may be particularly integrated in a server.
As shown in fig. 2a, a data processing method is provided, which can be executed by a processor of a server, and will be described below by taking an example in which a data processing apparatus is integrated in a single server, and the specific flow of the data processing method is as follows:
101. and for each data processing module, triggering an input submodule in the data processing module to acquire data to be processed.
The data to be processed is data which needs to be forwarded, analyzed, processed, calculated and the like. Data is the object of processing by a computer device. Wherein the data to be processed may have different forms such as symbols, words, numbers, voice, images, video, etc.
The input submodule is an external device in the computer device and is an input device for inputting data to be processed to a host computer of the computer device. The input submodule can be represented by a camera, a scanner, a hard disk, a network card for communication and the like.
In an embodiment, multiple sets of video or picture data may be sent to a host of a server integrated with a data processing apparatus through multiple sets of cameras, respectively, to obtain multiple sets of data to be processed. For each data processing module, video or picture data can be acquired through a camera in the data processing module and used as data to be processed.
In one embodiment, a plurality of sets of traffic processing information may be transmitted to the host of the server integrated with the data processing apparatus through a plurality of network cards, and the traffic processing information may be represented as a set of character data as an object to be processed in the server. The character data may be referred to as traffic data. For each data processing module, the network card in the data processing module can be used for acquiring flow data as data to be processed.
102. And triggering an input submodule in the data processing module to store the acquired data to be processed to a storage submodule in the data processing module.
The storage submodule is a module for data exchange between the processing submodule and external equipment, and can temporarily store data and applied programs in the data processing process. The storage submodule may be represented as a memory (or may also be referred to as a main memory) of the computer device.
The storage submodule corresponding to the input submodule is a storage submodule in the data processing module where the input submodule is located.
The server comprises a plurality of data processing modules, and each data processing module comprises an input submodule, a storage submodule and a processing submodule.
For each data processing module, the input sub-module may store the acquired data to be processed to a storage sub-module in the data processing module.
The processing submodule is a module used for operation and control in computer equipment, and is an execution module for data processing and program operation. May be embodied as a Central Processing Unit (CPU).
In one embodiment, the process of processing data by using a plurality of data processing modules involves NUMA (non uniform Memory Access) technology, which is a computer Memory design for a multiprocessor, and the Memory Access time depends on the Memory location of the processor. Under NUMA architecture, a processor accesses its own local memory more quickly than non-local memory (where memory goes to a processor or memory shared between another processor).
NUMA technology may combine tens of processors (even hundreds of processors) in one computer device to form a plurality of data processing modules, each of which may include at least one processor and an independent local memory, and the data processing modules may access each other through an interconnection module (e.g., referred to as Crossbar Switch). NUMA techniques may improve the efficiency of data processing by providing separate memories for each processor, avoiding performance penalties that may occur when multiple processors access the same memory. However, the latency to access remote memory far exceeds local memory. The local memory is a memory in the data processing module, and the remote memory is a memory in other data processing modules.
According to the embodiment of the invention, the acquired data to be processed is stored in the corresponding storage sub-module by each input sub-module. Access to the remotely stored submodule by the input submodule may be avoided. Therefore, the data storage speed can be improved, and the data processing efficiency is further improved.
In an embodiment, the input submodule includes at least one input channel, the processing submodule includes at least one data processing core, and before the acquired data to be processed is stored in the storage submodule in the data processing module based on the input submodule in the data processing module, the method may further include the following steps:
acquiring a data reading core according to a data processing core in a processing submodule and a preset data processing program in the data processing module;
determining a target input channel used for communicating with the storage submodule in the input submodule according to the data reading core;
establishing a mapping between the target input channel and the data reading core;
and connecting the target input channel to the storage sub-module corresponding to the data reading core according to the mapping, wherein the storage sub-module corresponding to the data reading core is the storage sub-module in the data processing module where the data reading core is located.
The data processing core is a complete computing unit which can complete reading and executing data processing instructions in the processing submodule. The processing submodule including a plurality of data processing cores may be represented as a multi-core processor, and the multi-core processor may be considered as a substrate on which a plurality of single-core processors are integrated, and each single-core processor is equivalent to one data processing core. The data processing efficiency can be improved by using the multi-core processor for data processing.
In order to simultaneously use a plurality of data processing modules for data processing, a data processing program may be set for each data processing module in advance, and the processing sub-module may read the data processing program, decode the data processing program to obtain a data processing instruction, and execute the data processing instruction.
In one embodiment, the setting of the data processing program for each data processing module in advance may be implemented as follows: the preset data processing program can be stored in a memory of the server, and the server system can enable the data processing module to obtain the data processing program through a preset interface with CPU affinity.
In one embodiment, a data reading subprogram is obtained according to a preset data processing program corresponding to the data processing module;
and establishing mapping between a processing core in the processing submodule in the data processing module and the data reading subprogram, and taking the data processing core after establishing mapping as a data reading core.
The processing process includes a plurality of steps, and generally the first step is to read the required data to be processed from the storage submodule.
The data reading subprogram is a part of the preset data processing program containing a data reading instruction.
The data reading subprogram can establish mapping with the data processing core through a CPU affinity interface provided by a system of the server to obtain the data reading core. The purpose of the mapping is to cause the data read core to execute the instruction corresponding to the data read subroutine.
When data reading is needed, the data reading core can read the data reading subprogram through the interface, carry out decoding to obtain a data reading instruction, and then execute the data reading instruction to finish the step of reading data from the storage submodule.
The step of reading data from the storage submodule may be further divided into a plurality of sub-steps, each sub-step may be referred to as a process, the step of reading data from the storage submodule may include a plurality of sub-steps performed in parallel, and each sub-step performed in parallel may be referred to as a thread. The data reading subroutine includes different threads and processes.
In an embodiment, the data reading sub-program includes a plurality of processes, a mapping is established between a processing core in the processing sub-module in the data processing module and the data reading sub-program, and the data processing core after the mapping is established is used as a data reading core, which may include the following steps:
and establishing mapping between different data processing cores in the processing sub-module and each process in the corresponding data reading sub-program to obtain a plurality of data reading cores.
In an embodiment, the data reading sub-program includes a plurality of threads, a processing core in the processing sub-module in the data processing module is mapped with the data reading sub-program, and the data processing core after mapping is set up is used as a data reading core, and the method further includes:
and establishing mapping between different data processing cores in the processing sub-module and each thread in the corresponding data reading program to obtain a plurality of data reading cores.
The input channel is an interface for transmitting data to the storage submodule, for example, when the input submodule is a network card, the input channel is an I/O bus in the network card. The target input channel is the I/O bus of the actual application.
In one embodiment, the number of target input channels is equal to or greater than the number of data processing cores in the same data processing module.
Wherein, the network card is provided with a processor and a memory (comprising a RAM and a ROM). The communication between the network card and the local area network is performed in a serial transmission manner through a cable or a twisted pair. The communication between the network card and the computer is carried out in a parallel transmission mode through an I/O bus on a computer mainboard. Therefore, one important function of the network card is to perform serial/parallel conversion. Since the data rate on the network is not the same as the data rate on the computer bus, a memory chip for buffering data must be installed in the network card. A driver for communicating with the server host may be stored in the ROM of the network card.
Data is transferred in the form of DMA queues on the target input channel (I/O bus). The DMA queue is a channel for data transmission by using a DMA (Direct Memory Access) technology. The DMA can directly store data from an external device into the storage submodule without the control of a CPU.
In one embodiment, the input submodule has a driver, where the driver is a special program for enabling the server and the external device to communicate with each other, and can be used for the input submodule to communicate with a preset data processing program in the server. Specifically, corresponding communication needs to be implemented between the driver and the preset data processing program, and the data processing program sends a corresponding instruction, for example, an instruction for applying for an input channel, to the input submodule through an interface provided by the driver.
In an embodiment, the processing submodule may send, according to a data processing program, an instruction for applying for an input channel to the input submodule through the driver, so as to initialize the DMA controller in the network card, the input submodule sends a DMA request, and the DMA controller determines priority and shielding of the DMA request, and provides a bus request to the bus arbitration logic. When the processing submodule executes the current bus cycle, the bus control right can be released. At this time, the bus arbitration logic outputs a bus response indicating that the DMA has responded, applies for the DMA queue, and notifies the input sub-module to start DMA transfer through the DMA controller. After the DMA controller obtains the bus control right, the processing sub-module (CPU) immediately hangs up or only executes internal operation, and the DMA controller outputs a read-write command and directly controls the input channel to carry out DMA transmission. Under the control of the DMA controller, data is directly transmitted between the storage submodule and the external equipment, and the participation of a central processing unit is not needed in the transmission process. The start position and data length of the data to be transmitted are initially provided.
In an embodiment, according to the mapping, connecting the target input channel to the storage sub-module corresponding to the data reading core, where the storage sub-module corresponding to the data reading core is the storage sub-module in the data processing module where the data reading core is located, may include the following steps:
determining a storage submodule corresponding to the data reading core according to the mapping;
analyzing the data to be processed acquired by the target input channel to acquire communication identification information in the data to be processed;
acquiring index information of the data to be processed in the storage submodule according to the communication identification information;
connecting the target input channel to the storage submodule based on the index information.
The communication identification information refers to a source IP address, a source port, a destination IP address, a destination port, and a transport layer protocol of data, and may also be referred to as a five-tuple, where each kind of information is referred to as an element, and correspondingly, a four-tuple is four kinds of the above five kinds of information.
In an embodiment, the input submodule parses the received to-be-processed data to obtain a source IP address, a source port, a destination IP address, a destination port, and transport layer protocol information, and the input submodule calculates a HASH value according to five tuple information by using a configured HASH function, or calculates according to two, three, or four tuples. The lower bits of the HASH value (each network card may be different) are taken as the index of RETA (redirection table) in the storage submodule. And distributing the data stored in the RETA to the corresponding processing sub-modules.
In one embodiment, the input submodule may store in a different RETA (reset table) of the storage submodule via the target input channel.
103. And triggering a processing submodule in the data processing module to read the data to be processed in a storage submodule of the data processing module.
In an embodiment, triggering a processing sub-module in the data processing module to read data to be processed in a storage sub-module of the data processing module may include the following steps:
acquiring index information of the data to be processed corresponding to the data reading core in the storage submodule based on the mapping between the data reading core and the target input channel;
and triggering the data reading core to read the data to be processed corresponding to the data reading core from a storage submodule in the data processing module through the index information.
In an embodiment, the data reading sub-program includes a plurality of processes, or includes a plurality of parallel threads, and triggers a processing sub-module in the data processing module to read data to be processed in a storage sub-module of the data processing module, and may further include the following steps:
determining the calling sequence of each process or thread according to the preset data processing program;
and calling the corresponding data reading core according to the calling sequence, and reading the data to be processed corresponding to the data reading core from the corresponding storage sub-module.
104. And triggering a processing submodule in the data processing module to process the read data to be processed.
According to a preset data processing program, the read data to be processed can be processed differently. Specifically, the processing sub-module may read a preset data processing program, decode the preset data processing program, and obtain different processing instructions, so as to perform different processing.
As can be seen from the above, in the embodiment of the present application, for each data processing module, an input sub-module in the data processing module may be triggered to acquire data to be processed; triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read data to be processed. According to the embodiment of the application, the data to be processed is input to the computer equipment by simultaneously using the plurality of input sub-modules, each input sub-module only accesses the local storage sub-module, and each processing sub-module only accesses the local storage sub-module, so that on one hand, time waste caused by accessing a remote storage module is avoided, on the other hand, a plurality of groups of data to be processed can be simultaneously processed, and therefore the data processing efficiency can be improved.
The method described in the above embodiments is further described in detail below.
In this embodiment, a method according to an embodiment of the present invention will be described in detail by taking an example in which a data processing apparatus is integrated in a single server to perform traffic data processing.
Referring to fig. 2b, the data processing method of the present application includes the following specific processes:
201. and for each data processing module, the server triggers a network communication submodule in the data processing module to acquire flow data to be processed.
The network communication submodule is hardware used for enabling the server to communicate in a network. May be embodied as a network card. The communication between the network card and the network is performed in a serial transmission mode through a cable or a twisted pair. And the communication between the network card and the server is carried out in a parallel transmission mode through an I/O bus on a mainboard of the server. Therefore, one important function of the network card is to perform serial/parallel conversion.
In one embodiment, a network card using a PCI-e slot may be deployed in each data processing module of the server.
The server can acquire a plurality of groups of flow data to be processed through a plurality of network cards.
Referring to FIG. 5a, when a server uses a NUMA architecture, the data processing modules may be referred to as NUMA nodes, the storage sub-modules may be referred to as memory, and the processing sub-modules may be CPUs. In the following of this embodiment, a NUMA node will be described as an example.
202. And the server stores the acquired traffic data to be processed to a storage submodule in the data processing module based on a network communication submodule in the data processing module.
In an embodiment, the network card stores the received network traffic data in the memory of the NUMA node in a DMA manner. The DMA method refers to the above embodiments, and is not described in detail.
The network card stores the received network traffic data into the memory of the NUMA node in a DMA mode, and the network card is realized by matching the network card drive with a preset traffic processing program. The specific implementation scheme is as follows:
in the preset flow processing program, there is a "flow reading subprogram" which is responsible for cooperating with the network card driver to obtain the flow data stored by the network card from the local NUMA memory. Local means that the NUMA nodes belong to the same NUMA node, and remote means that the NUMA nodes do not belong to the same NUMA node.
In one embodiment, the traffic reading subroutine has multiple processes or threads. A process or thread of the traffic reading subprogram may be deployed on the CPU of each NUMA node where the network card is deployed, to form a plurality of traffic data reading cores.
The number of data processing COREs (CORE) of the CPU occupied by the traffic reading subprogram on each NUMA node is determined by the size of the traffic data to be processed that the NUMA node needs to access. The method can be deployed on only one CORE of the NUMA node CPU, or can be deployed on a plurality of COREs of the NUMA node CPU. And the flow reading subprogram process or thread is bound to the CORE of the specified CPU through an interface provided by the server system and used for setting the CPU affinity.
In one embodiment, the server system may provide the following interfaces:
(1) setting an interface of process affinity: scheduled _ timing
(2) Interface to set thread affinity: pthread _ detail _ np
The flow reading subprogram initializes the DMA queue of the network card through the interface provided by the network card driver (the interface provided by different network card drivers is different, and a DPDK development kit can also be used in a unified way). The content that needs to be initialized is as follows:
the number of DMA queues needing to be initialized by each network card is more than or equal to the number of COREs binding the flow reading subprogram to the NUMA node corresponding to the network card.
Each DMA queue needs to be bound to a CORE to which the traffic receiver is bound (binding is done by the method of creating the mapping, see the above embodiment).
When initializing the DMA queue, a memory needs to be specified for the DMA queue, where the memory is a memory of a NUMA node where a CORE bound by the DMA queue is located (refer to the above embodiment). And ensuring that the network card stores the received flow into the memory of the NUMA node.
203. And the server triggers a processing submodule in the data processing module to read the to-be-processed flow data in a storage submodule of the data processing module.
And the flow reading subprogram receives the to-be-processed flow data of the DMA queue bound by the CORE of the flow reading subprogram through the driving interface.
For the specific process, refer to the above embodiments, and are not described again.
204. And the server triggers a processing submodule in the data processing module to process the read flow data to be processed.
The data processing means may be integrated in different servers of the network connection system, thereby increasing the speed of different traffic handling steps. Accordingly, the preset data processing program may be expressed as a program that performs different processing on the traffic data.
For example, referring to fig. 5b, the data processing system in this embodiment may be integrated in a gateway server, the preset data processing program may be represented as a traffic forwarding program, and a network card and the traffic forwarding program are deployed on each NUMA node of the gateway server. And when the traffic data is forwarded, the network card of the NUMA node is selected to send the traffic data. The gateway server may send the read traffic data to be processed to other servers through a network link, for example, the read traffic data to be processed is forwarded to a service server in the system.
For example, referring to fig. 5c, the data processing system in this embodiment may be integrated in a traffic analysis server, the preset data processing program may be represented as a traffic analysis program, and a network card and the traffic analysis program are deployed on each NUMA node of the traffic analysis server. The traffic analysis server may obtain traffic data copied by a port mirroring function of the optical splitter or the switch/router, as traffic data to be processed, and perform analysis calculation on the traffic data to be processed, where the optical splitter is configured to converge mirror traffic data copied by optical splitting or port mirroring, and forward the converged mirror traffic data to another server (which may be a traffic processing server). The mirror image traffic data is the traffic data which is obtained by copying the service traffic data in a light splitting or port mirror image mode and is forwarded to the network traffic analysis server through the light splitting switch. The service traffic data is traffic data generated by service interaction between a user (client) and a service server.
For example, referring to fig. 5d, the data processing system in this embodiment may be integrated in a web service server, the preset data processing program may be represented as a traffic analysis program, and a network card and the traffic analysis program are deployed on each NUMA node of the web service server. The web service server can generate the traffic data for responding the information by calculating and analyzing the information contained in the traffic data to be processed, and return the traffic data to the terminal.
As can be seen from the above, in the embodiment of the present application, for each data processing module, a network communication sub-module in the data processing module may be triggered to obtain flow data to be processed; triggering a network communication sub-module in the data processing module to store the acquired traffic data to be processed to a storage sub-module in the data processing module; triggering a processing submodule in the data processing module to read the flow data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read flow data to be processed. According to the embodiment of the application, the to-be-processed flow data are input into the computer equipment by simultaneously using the plurality of network communication sub-modules, each network communication sub-module only accesses the local storage sub-module, and each processing sub-module only accesses the local storage sub-module, so that on one hand, time waste caused by accessing the remote storage module is avoided, on the other hand, a plurality of groups of to-be-processed flow data can be simultaneously processed, and therefore the data processing efficiency can be improved.
In an embodiment, the data processing apparatus may be integrated into a server cluster to implement the data processing method of the present application, where the server cluster is composed of a plurality of servers, where each server has at least one data processing module integrated therein, and the data processing modules may access each other through an interconnection module (for example, referred to as Crossbar Switch).
In order to better implement the method, an embodiment of the present invention further provides a data processing apparatus, where the data processing apparatus may be specifically integrated in an electronic device, and the electronic device may be a terminal, a server, a personal computer, or the like.
For example, in the present embodiment, the method according to the embodiment of the present invention will be described in detail by taking an example in which a data processing apparatus is integrated in a server.
For example, as shown in fig. 3a, the data processing apparatus may include an acquisition unit 301, a storage unit 302, a reading unit 303, and a processing unit 304. The following were used:
the first acquiring unit 301 is configured to, for each data processing module, trigger an input submodule in the data processing module to acquire data to be processed.
And (ii) storing the data to be processed in the unit 302, which is used for triggering an input sub-module in the data processing module to store the acquired data to be processed in a storage sub-module in the data processing module.
In some embodiments, referring to fig. 3b, before executing the store-to-unit 302, the binding unit 305 is further executed, and the binding unit 305 may specifically include a core obtaining subunit, a channel core binding subunit, and a channel memory binding subunit, which are specifically as follows:
the core acquisition subunit is used for acquiring a data reading core according to a data processing core in the processing submodule and a preset data processing program in the data processing module;
the channel acquisition subunit is used for determining a target input channel used for communicating with the storage submodule in the input submodule according to the data reading core;
a channel core binding subunit, configured to establish mapping between the target input channel and the data read core;
and the channel memory binding subunit is used for connecting the target input channel to the storage submodule corresponding to the data reading core according to the mapping, wherein the storage submodule corresponding to the data reading core is the storage submodule in the data processing module where the data reading core is located.
In some embodiments, the channel memory binding subunit may be specifically configured to:
determining a storage submodule corresponding to the data reading core according to the mapping;
analyzing the data to be processed acquired by the target input channel to acquire communication identification information in the data to be processed;
acquiring index information of the data to be processed in the storage submodule according to the communication identification information;
connecting the target input channel to the storage submodule based on the index information.
In some embodiments, the core acquisition subunit may be specifically configured to:
acquiring a data reading subprogram according to a preset data processing program corresponding to the data processing module;
and establishing mapping between a processing core in the processing submodule in the data processing module and the data reading subprogram, and taking the data processing core after establishing mapping as a data reading core.
In an embodiment, the storing to unit 302 may be specifically configured to:
and storing the data to be processed acquired by the target input channel to a storage submodule in the data processing module according to the index information.
And a reading unit 303, configured to trigger a processing sub-module in the data processing module to read data to be processed in a storage sub-module of the data processing module.
In an embodiment, the reading unit 303 may be specifically configured to:
determining the calling sequence of each process or thread according to the preset data processing program;
and calling the corresponding data reading core according to the calling sequence, and reading the data to be processed corresponding to the data reading core from the corresponding storage sub-module.
In an embodiment, the reading unit 303 may be further configured to:
acquiring index information of the data to be processed corresponding to the data reading core in the storage submodule based on the mapping between the data reading core and the target input channel;
and triggering the data reading core to read the data to be processed corresponding to the data reading core from a storage submodule in the data processing module through the index information.
A processing unit 304, configured to allocate the sub-content to a corresponding data processing branch in the target data processing path for processing, so as to obtain a data processing result corresponding to each sub-content.
In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.
As can be seen from the above, the obtaining unit in the embodiment of the present application is configured to, for each data processing module, trigger an input submodule in the data processing module to obtain to-be-processed data; the storage-to unit is used for triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; the reading unit is used for triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and the processing unit is used for triggering the processing submodule in the data processing module to process the read data to be processed.
According to the embodiment of the application, the data to be processed is input to the computer equipment by simultaneously using the plurality of input sub-modules, each input sub-module only accesses the local storage sub-module, and each processing sub-module only accesses the local storage sub-module, so that on one hand, time waste caused by accessing a remote storage module is avoided, on the other hand, a plurality of groups of data to be processed can be simultaneously processed, and therefore the data processing efficiency can be improved.
The embodiment of the application also provides a server, and the server can be a mobile phone, a tablet computer, a micro processing box, an unmanned aerial vehicle or an image acquisition device and the like. As shown in fig. 4, it shows a schematic structural diagram of a server according to an embodiment of the present application, specifically:
the server may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that the server architecture shown in FIG. 4 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the processor 401 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the server. In some embodiments, processor 401 may include one or more processing cores; in some embodiments, processor 401 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.
The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.
The server also includes a power supply 403 for supplying power to the various components, and in some embodiments, the power supply 403 may be logically connected to the processor 401 via a power management system, so that the functions of managing charging, discharging, and power consumption are implemented via the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
The server may also include an input module 404, the input module 404 operable to receive entered numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
The server may also include a communication module 405, and in some embodiments, the communication module 405 may include a wireless sub-module through which the server may wirelessly transmit over short distances to provide wireless broadband internet access. For example, the communication module 405 may be used to assist a user in sending and receiving e-mails, browsing web pages, accessing streaming media, and the like.
Although not shown, the server may further include a display unit and the like, which will not be described in detail herein. Specifically, in this embodiment, the processor 401 in the server loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:
for each data processing module, triggering an input submodule in the data processing module to acquire data to be processed;
triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module;
triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and triggering a processing submodule in the data processing module to process the read data to be processed.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
As can be seen from the above, in the embodiment of the present application, for each data processing module, an input sub-module in the data processing module may be triggered to acquire data to be processed; triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module; triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module; and triggering a processing submodule in the data processing module to process the read data to be processed. According to the embodiment of the application, the data to be processed is input to the computer equipment by simultaneously using the plurality of input sub-modules, each input sub-module only accesses the local storage sub-module, and each processing sub-module only accesses the local storage sub-module, so that on one hand, time waste caused by accessing a remote storage module is avoided, on the other hand, a plurality of groups of data to be processed can be simultaneously processed, and therefore the data processing efficiency can be improved.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, the present application provides a storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the data processing methods provided by the present application. For example, the instructions may perform the steps of:
respectively acquiring a plurality of groups of data to be processed through an input submodule in the data processing module;
triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module;
triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and triggering a processing submodule in the data processing module to process the read data to be processed.
Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the storage medium can execute the steps in any data processing method provided in the embodiments of the present application, beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.
The foregoing detailed description is directed to a data processing method, a data processing apparatus, and a storage medium provided in the embodiments of the present application, and specific examples are applied in the present application to explain the principles and implementations of the present application, and the descriptions of the foregoing embodiments are only used to help understand the methods and core ideas of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (11)
1. A data processing method is applicable to computer equipment, the computer equipment comprises a plurality of data processing modules, the data processing modules comprise an input submodule, a storage submodule and a processing submodule, and the data processing method comprises the following steps:
for each data processing module, triggering an input submodule in the data processing module to acquire data to be processed;
storing the acquired data to be processed to a storage submodule in the data processing module based on an input submodule in the data processing module;
triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and triggering a processing submodule in the data processing module to process the read data to be processed.
2. The data processing method according to claim 1, wherein the input submodule includes at least one input channel, the processing submodule includes at least one data processing core, and before the triggering the input submodule in the data processing module to store the acquired data to be processed in the storage submodule in the data processing module, the method further includes:
for each processing submodule, acquiring a data reading core according to the data processing core in the processing submodule and a preset data processing program in the data processing module;
determining a target input channel used for communicating with the storage submodule in the input submodule according to the data reading core;
establishing a mapping between the target input channel and the data reading core;
and connecting the target input channel to the storage sub-module corresponding to the data reading core according to the mapping, wherein the storage sub-module corresponding to the data reading core is the storage sub-module in the data processing module where the data reading core is located.
3. The data processing method of claim 2, wherein said connecting the target input channel to the storage submodule corresponding to the data read core according to the mapping comprises:
determining a storage submodule corresponding to the data reading core according to the mapping;
analyzing the data to be processed acquired by the target input channel to acquire communication identification information in the data to be processed;
acquiring index information of the data to be processed in the storage submodule according to the communication identification information;
connecting the target input channel to the storage submodule based on the index information.
4. The data processing method according to claim 3, wherein the storing the acquired data to be processed to a storage submodule in the data processing module based on an input submodule in the data processing module comprises:
acquiring index information of the data to be processed corresponding to the data reading core in the storage submodule based on the mapping between the data reading core and the target input channel;
and storing the data to be processed acquired by the target input channel to a storage submodule in the data processing module according to the index information.
5. The data processing method of claim 3, wherein the triggering of the processing submodule in the data processing module to read the data to be processed in the storage submodule of the data processing module comprises:
and triggering the data reading core to read the data to be processed corresponding to the data reading core from a storage submodule in the data processing module through the index information.
6. The data processing method of claim 2, wherein the obtaining of the data reading core according to the data processing core in the processing sub-module and a preset data processing program in the data processing module comprises:
acquiring a data reading subprogram according to a preset data processing program corresponding to the data processing module;
and establishing mapping between a processing core in the processing submodule in the data processing module and the data reading subprogram, and taking the data processing core after establishing mapping as a data reading core.
7. The data processing method according to claim 6, wherein the data reading subprogram includes a plurality of processes, the mapping between the processing core in the processing submodule in the data processing module and the data reading subprogram, and the mapping being established for the data processing core as the data reading core includes:
and establishing mapping between different data processing cores in the processing sub-module and each process in the corresponding data reading sub-program to obtain a plurality of data reading cores.
8. The data processing method according to claim 6, wherein the data reading subroutine includes a plurality of threads, the mapping between the processing core in the processing submodule in the data processing module and the data reading subroutine, and the mapping being performed using the data processing core as a data reading core, further comprising:
and establishing mapping between different data processing cores in the processing sub-module and each thread in the corresponding data reading program to obtain a plurality of data reading cores.
9. The data processing method according to any one of claims 7 or 8, wherein the triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module comprises:
determining the calling sequence of each process or thread according to the preset data processing program;
and calling the corresponding data reading core according to the calling sequence, and reading the data to be processed corresponding to the data reading core from the corresponding storage sub-module.
10. A data processing apparatus, comprising:
the acquisition unit is used for triggering an input submodule in each data processing module to acquire data to be processed;
the storage-to unit is used for triggering an input submodule in the data processing module to store the acquired data to be processed into a storage submodule in the data processing module;
the reading unit is used for triggering a processing submodule in the data processing module to read data to be processed in a storage submodule of the data processing module;
and the processing unit is used for triggering the processing submodule in the data processing module to process the read data to be processed.
11. A storage medium having stored thereon a computer program, characterized in that, when the computer program runs on a computer, it causes the computer to execute a data processing method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033177.7A CN111240845B (en) | 2020-01-13 | 2020-01-13 | Data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033177.7A CN111240845B (en) | 2020-01-13 | 2020-01-13 | Data processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111240845A true CN111240845A (en) | 2020-06-05 |
| CN111240845B CN111240845B (en) | 2023-10-03 |
Family
ID=70874519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010033177.7A Active CN111240845B (en) | 2020-01-13 | 2020-01-13 | Data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111240845B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112350851A (en) * | 2020-10-14 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Data processing and checking method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102520915A (en) * | 2011-11-25 | 2012-06-27 | 华为技术有限公司 | Method and device for threading serial program in nonuniform memory access system |
| CN102541803A (en) * | 2011-12-31 | 2012-07-04 | 曙光信息产业股份有限公司 | Data sending method and computer |
| CN102571580A (en) * | 2011-12-31 | 2012-07-11 | 曙光信息产业股份有限公司 | Data receiving method and computer |
| CN103744799A (en) * | 2013-12-26 | 2014-04-23 | 华为技术有限公司 | Memory data access method, device and system |
| CN104050091A (en) * | 2012-12-28 | 2014-09-17 | 华耀(中国)科技有限公司 | Network equipment based on NUMA (Non Uniform Memory Access) system and setting method thereof |
| CN104182334A (en) * | 2013-05-24 | 2014-12-03 | 鸿富锦精密工业（深圳）有限公司 | Memory testing method and system for NUMA (non-uniform memory access) system |
| CN107346267A (en) * | 2017-07-13 | 2017-11-14 | 郑州云海信息技术有限公司 | A kind of cpu performance optimization method and device based on NUMA architecture |
| WO2018077292A1 (en) * | 2016-10-28 | 2018-05-03 | 北京市商汤科技开发有限公司 | Data processing method and system, electronic device |
-
2020
- 2020-01-13 CN CN202010033177.7A patent/CN111240845B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102520915A (en) * | 2011-11-25 | 2012-06-27 | 华为技术有限公司 | Method and device for threading serial program in nonuniform memory access system |
| CN102541803A (en) * | 2011-12-31 | 2012-07-04 | 曙光信息产业股份有限公司 | Data sending method and computer |
| CN102571580A (en) * | 2011-12-31 | 2012-07-11 | 曙光信息产业股份有限公司 | Data receiving method and computer |
| CN104050091A (en) * | 2012-12-28 | 2014-09-17 | 华耀(中国)科技有限公司 | Network equipment based on NUMA (Non Uniform Memory Access) system and setting method thereof |
| CN104182334A (en) * | 2013-05-24 | 2014-12-03 | 鸿富锦精密工业（深圳）有限公司 | Memory testing method and system for NUMA (non-uniform memory access) system |
| CN103744799A (en) * | 2013-12-26 | 2014-04-23 | 华为技术有限公司 | Memory data access method, device and system |
| WO2018077292A1 (en) * | 2016-10-28 | 2018-05-03 | 北京市商汤科技开发有限公司 | Data processing method and system, electronic device |
| CN107346267A (en) * | 2017-07-13 | 2017-11-14 | 郑州云海信息技术有限公司 | A kind of cpu performance optimization method and device based on NUMA architecture |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112350851A (en) * | 2020-10-14 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Data processing and checking method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111240845B (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10095645B2 (en) | Presenting multiple endpoints from an enhanced PCI express endpoint device | |
| CN106688208B (en) | Network communication using pooled storage in a rack scale architecture | |
| US5991797A (en) | Method for directing I/O transactions between an I/O device and a memory | |
| CN107995129B (en) | NFV message forwarding method and device | |
| US7941569B2 (en) | Input/output tracing in a protocol offload system | |
| US7827024B2 (en) | Low latency, high bandwidth data communications between compute nodes in a parallel computer | |
| US10932202B2 (en) | Technologies for dynamic multi-core network packet processing distribution | |
| CN108984327B (en) | Message forwarding method, multi-core CPU and network equipment | |
| US11449456B2 (en) | System and method for scheduling sharable PCIe endpoint devices | |
| US9319313B2 (en) | System and method of forwarding IPMI message packets based on logical unit number (LUN) | |
| US10649921B2 (en) | Composable serial console and KVM with remote accessibility | |
| US11044210B2 (en) | Technologies for performing switch-based collective operations in distributed architectures | |
| CN115080479B (en) | Transmission method, server, device, bare metal instance and baseboard management controller | |
| EP3588310B1 (en) | Technologies for demoting cache lines to shared cache | |
| US20090132582A1 (en) | Processor-server hybrid system for processing data | |
| CN111240845B (en) | Data processing method, device and storage medium | |
| US10719475B2 (en) | Method or apparatus for flexible firmware image management in microserver | |
| CN117041147B (en) | Intelligent network card equipment, host equipment, method and system | |
| CN117312229B (en) | Data transmission device, data processing equipment, system, method and medium | |
| CN113726723B (en) | UDP-based data transmission method, device and equipment | |
| Mahabaleshwarkar et al. | TCP/IP protocol accelaration | |
| EP3962031A1 (en) | System, apparatus and method for advanced monitoring in an edge system | |
| CN117370046A (en) | Inter-process communication method, system, device and storage medium | |
| KR20200064416A (en) | Scalable Event Delivery System and Method Supporting Flexibility | |
| US20160275038A1 (en) | Method for communication using pci express dedicated communication module and network device including the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40024059 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |