CN110333899B - Data processing method, device and storage medium - Google Patents
Data processing method, device and storage medium Download PDFInfo
- Publication number
- CN110333899B CN110333899B CN201910570004.6A CN201910570004A CN110333899B CN 110333899 B CN110333899 B CN 110333899B CN 201910570004 A CN201910570004 A CN 201910570004A CN 110333899 B CN110333899 B CN 110333899B
- Authority
- CN
- China
- Prior art keywords
- processor
- isolated
- bound
- hardware
- running process
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/4401—Bootstrapping
- G06F9/4411—Configuring for operating with peripheral devices; Loading of device drivers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4812—Task transfer initiation or dispatching by interrupt, e.g. masked
- G06F9/4825—Interrupt from clock, e.g. time of day
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Multi Processors (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a data processing device and a storage medium; in the embodiment of the invention, when a data processing request is received, a corresponding uninterrupted running process is called, then an isolation processor is distributed to the uninterrupted running process, the uninterrupted running process is bound with the distributed isolation processor to obtain a bound post processor, then scheduling and interruption in the bound post processor are closed to obtain a set post processor, and the uninterrupted running process is run in the set post processor to process the data; the scheme can effectively improve the efficiency of data processing.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data processing method, an apparatus, and a storage medium.
Background
In a server application scenario, a large amount of network and storage traffic data need to be processed, which is collectively called Input/Output (IO) data, and a conventional IO processing manner is to process the data through a protocol stack or a storage stack provided by a hardware interrupt and an Operating System (OS), which is relatively long in flow, complex in processing logic, and relatively low in efficiency.
In order to solve these problems, new IO processing methods, such as a Data Plane Development Kit (DPDK) and a Storage Performance Development Kit (SPDK), have appeared, which process IO Data mainly by means of user-state dead-cycle polling (polling), so as to reduce the dependency on hardware interrupts to the greatest extent, and simultaneously skip complex OS protocol stacks and Storage stacks, thereby greatly improving the IO processing Performance. However, the DPDK/SPDK operation mode must be exclusive to a Central Processing Unit (CPU), cannot be shared with other tasks, and must be continuously run, so that running such processes has high requirements on running processors.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, an apparatus, and a storage medium, which can effectively improve data processing efficiency.
The embodiment of the invention provides a data processing method, which comprises the following steps:
when a data processing request is received, calling a corresponding uninterrupted running process;
distributing an isolation processor for the uninterrupted running process, and binding the uninterrupted running process with the distributed isolation processor to obtain a bound processor;
closing scheduling and interruption in the bound post-processor to obtain a set post-processor;
and running the uninterrupted running process in the set post processor to process the data.
Correspondingly, an embodiment of the present invention further provides a data processing apparatus, including:
the calling unit is used for calling a corresponding uninterrupted running process when a data processing request is received;
the distribution unit is used for distributing the isolation processor for the uninterrupted running process and binding the uninterrupted running process with the distributed isolation processor to obtain a bound processor;
a closing unit, configured to close scheduling and interruption in the bound post-processor to obtain a set post-processor;
and the running unit is used for running the uninterrupted running process in the set post-processor so as to process the data.
Optionally, in some embodiments, the shutdown unit may include a shutdown scheduling subunit and a shutdown clock subunit, as follows:
the closing scheduling subunit is configured to close an interrupt return, an exception return, a system call return, a wakeup task, and modify a scheduling opportunity when the kernel preemption count is performed on the bound processor; and
and the clock closing subunit is used for closing the clock interrupt of the bound processor.
Optionally, in some embodiments, the clock turning-off subunit is specifically configured to call hardware of the system to provide an interface of the bound processor, and turn off clock interrupt of the bound processor; and modifying a time updating interface of the system kernel, and providing time for the uninterrupted running process by acquiring the time of the non-isolated processor.
Optionally, in some embodiments, the data processing apparatus further includes an identification unit, a configuration unit, and a determination unit, as follows:
the identification unit is used for identifying all hardware processors of the system when the system is initialized;
the configuration unit is used for configuring the candidate hardware processors and analyzing the configuration to obtain a target processor set, wherein the target processor set comprises at least one isolation processor;
the determining unit is configured to determine, as a non-isolated processor, a processor of the hardware processor other than the target processor set;
the allocating unit may specifically allocate an isolation processor to the uninterrupted running process based on the target processor set.
Optionally, in some embodiments, the configuration unit may be specifically configured to perform parameter setting on a configuration file of the system when the system is started, and transmit the set parameter to a system kernel; or when the system runs, configuring the candidate hardware processor through a dynamic configuration interface provided by the system.
Optionally, in some embodiments, the data processing apparatus further includes a binding unit, as follows:
and the binding unit is used for binding the processes except the uninterrupted running process in the system on the non-isolated processor to run.
Optionally, in some embodiments, the binding unit may include a binding subunit and a designated subunit, as follows:
the binding subunit is configured to traverse all hardware interrupts in the system, and bind the hardware interrupts to the non-isolated processor;
the appointing subunit is used for modifying the system kernel parameters and appointing processes in the system except the uninterruptedly-operated processes to operate on the non-isolated processor.
Optionally, in some embodiments, the binding subunit may be specifically configured to invoke an interrupt interface provided by hardware of the system; setting the relevance of hardware interrupt and the non-isolated processor based on the interrupt interface, and binding all hardware interrupt to the non-isolated processor.
In addition, an embodiment of the present invention further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to perform steps in any data processing method provided in the embodiment of the present invention.
In addition, an embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a storage medium, and the processor is configured to implement each instruction; the storage medium is used for storing a plurality of instructions, and the instructions are used for being loaded by the processor and executing any one of the data processing methods provided by the embodiments of the invention.
When a processing request of data is received, a corresponding uninterrupted running process can be called, then an isolation processor is distributed to the uninterrupted running process, the uninterrupted running process is bound with the distributed isolation processor to obtain a bound post processor, then scheduling and interruption in the bound post processor are closed to obtain a set post processor, and the uninterrupted running process is run in the set post processor to process the data; the scheme can effectively improve the efficiency of data processing.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1a is a schematic view of a scenario of a data processing method according to an embodiment of the present invention;
FIG. 1b is a first flowchart of a data processing method provided by the embodiment of the invention;
FIG. 2a is a second flowchart of a data processing method provided by the embodiment of the invention;
FIG. 2b is a first block diagram of a system provided by an embodiment of the invention;
FIG. 2c is a second block diagram of a system provided by an embodiment of the invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a network device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a data processing method, a data processing device and a storage medium. The data processing may be integrated in a network device, and the network device may be a server or a terminal.
For example, referring to fig. 1a, first, when receiving a data processing request, the network device integrated with a data processing apparatus invokes a corresponding uninterrupted running process, then allocates an isolation processor to the uninterrupted running process, binds the uninterrupted running process with the allocated isolation processor to obtain a bound processor, then closes scheduling and interruption in the bound processor to obtain a set post processor, and runs the uninterrupted running process in the set post processor to process the data.
According to the scheme, the uninterrupted running process is bound with the distributed isolation processors, and scheduling and interruption in the processors after binding are closed, so that the uninterrupted process can completely monopolize the isolation processors without any other interference, and the data processing efficiency is effectively improved.
The following are detailed descriptions. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.
This embodiment will be described from the perspective of a data processing apparatus, which may be specifically integrated in a network device, where the network device may be a server or a terminal; the terminal may include a mobile phone, a tablet Computer, a notebook Computer, and a Personal Computer (PC).
A method of data processing, comprising: when a processing request of data is received, calling a corresponding uninterrupted running process, then allocating an isolation processor for the uninterrupted running process, binding the uninterrupted running process with the allocated isolation processor to obtain a bound post processor, then closing scheduling and interruption in the bound post processor to obtain a set post processor, and running the uninterrupted running process in the set post processor to process the data.
As shown in fig. 1b, the specific flow of the data processing method may be as follows:
101. and when a processing request of the data is received, calling a corresponding uninterrupted running process.
Herein, the continuously running process refers to a process continuously running on the processor, for example, the DPDK and SPDK processes process data packets in a polling manner, rather than an interrupt.
For example, the data may be collected by the data collection device and then sent to the data processing device, and when the data processing device receives a data processing request, the data processing device invokes an uninterrupted running process corresponding to the data processing request. The data processing means may comprise hardware such as a plurality of processors, temporary storage media (such as memory) and permanent storage media (such as hard disks, etc.).
102. And allocating an isolation processor for the uninterrupted running process, and binding the uninterrupted running process with the allocated isolation processor to obtain a bound processor.
For example, in order to reduce the dependence on hardware interruption to the greatest extent by running processes without interruption, and simultaneously skip complex OS protocol stacks and memory stacks, so that the IO processing performance can be improved to the greatest extent, and the processes running without interruption can monopolize processors, and can perform isolation at the system start stage for processor resources required by such processes, so that the isolated processors do not participate in system scheduling, that is, other processes are not allowed to run on the isolated processors. For example, when the operating system is initialized, all hardware processors may be identified, the processors that need to be isolated may be configured, the configuration may be analyzed, the processors that need to be isolated may be obtained, the processors that need to be isolated may be deducted from all hardware processor lists, and the obtained processor list may be placed in the global scheduling domain. Thus, the kernel scheduler only sees the non-isolated processor, the isolated processor is invisible to the scheduler, and no task is scheduled on the isolated processor naturally. That is, before the step "allocating an isolation processor to the uninterrupted running process", the following steps may be included:
when a system is initialized, all hardware processors of the system are identified, candidate hardware processors are configured, the configuration is analyzed, a target processor set is obtained, the target processor set comprises at least one isolated processor (namely an isolated processor), and processors except the target processor set in the hardware processors are determined to be non-isolated processors. Allocating an isolated processor to the continuously running process may include allocating an isolated processor to the continuously running process based on the set of target processors.
The configuration mode of the target processor set may be various, for example, a mode of transferring a kernel start parameter of the operating system may be used, and a dynamic configuration interface provided by the operating system may also be used. Namely, the step "configure candidate hardware processor", may include:
when the system is started, setting parameters of a configuration file of the system, and transmitting the set parameters to a system kernel; or when the system runs, the candidate hardware processor is configured through a dynamic configuration interface provided by the system.
103. And closing the scheduling and interruption in the bound post-processor to obtain the set post-processor.
For example, the interrupt return, exception return, system call return, wakeup task, and modify scheduling time when the kernel preemption count of the bound processor may be specifically closed; and closing the clock interrupt of the bound processor.
Wherein, the clock interrupt is closed by calling the interface provided by the hardware. Then, relevant interfaces of the operating system kernel with respect to time update are modified, and time is provided for the isolated processor by acquiring time of other processors. Namely, the step "turning off the clock interrupt of the bound processor" may include:
calling hardware of the system to provide an interface of the bound processor, and closing local clock interrupt of the bound processor; and modifying the time updating interface of the system kernel, and providing time for the uninterrupted running process by acquiring the time of the non-isolated processor.
104. And running the uninterrupted running process in the set post processor to process the data.
For example, in order to make the uninterrupted running process completely monopolize the isolated processor resource without any other interference, and to improve the execution performance of the processes to the greatest extent, processes except the uninterrupted running process in the system may be bound to the non-isolated processor to run before the uninterrupted running process is run in the post-configuration processor. For example, interrupts may be bound, system processes may be isolated, and so forth. Among other things, isolated system processes may designate target processors on which these processes may run as non-isolated cores (i.e., non-isolated processors) when creating related system processes by modifying the operating system kernel. Namely, the step of "binding the process in the system except the uninterruptedly running process to run on the non-isolated processor", may include:
traversing all hardware interrupts in the system, and binding the hardware interrupts to the non-isolated processor; and modifying the system kernel parameters, and designating processes except the uninterrupted running process in the system to run on the non-isolated processor.
The specific processing mode of binding the interrupt can be all hardware interrupts in the traversal environment, and the affinity of the interrupt is set by calling an interface provided by the hardware, so that all the interrupts are bound to the non-isolated core. The step of "binding the hardware interrupt to the non-sequestered processor" may include:
calling an interrupt interface provided by hardware of the system; and setting the relevance of the hardware interrupt and the non-isolated processor based on the interrupt interface, so that all hardware interrupts are bound to the non-isolated processor.
It should be noted that, the processor in the present invention may refer to a Central Processing Unit (CPU), other processors, and the like.
As can be seen from the above, in the embodiments of the present invention, when a data processing request is received, a corresponding uninterrupted running process may be called, then, an isolation processor is allocated to the uninterrupted running process, the uninterrupted running process is bound to the allocated isolation processor, a bound processor is obtained, then, scheduling and interruption in the bound processor are closed, a set post-processor is obtained, and the uninterrupted running process is run in the set post-processor, so as to process the data. According to the scheme, the uninterrupted running process is bound with the allocated isolation processor, and scheduling and interruption in the bound processor are closed, so that the specified processor resource can be isolated (other interference tasks do not run), the isolated processor is not scheduled at all, the corresponding process can really and completely monopolize the processor, no interruption or other interference exists, the uninterrupted running process runs on the set processor by hundreds, and the data processing efficiency is effectively improved.
The method described in the previous embodiment is further illustrated in detail by way of example.
In the present embodiment, the data processing apparatus is specifically integrated in a network device, and a Linux operating system is taken as an example for description.
As shown in fig. 2a, a specific flow of a data processing method may be as follows:
201. when the Linux operating system is initialized, the network equipment determines a target processor set and a non-isolated processor of the system.
For example, the network device may specifically identify all hardware processors when the operating system is initialized, and configure the processors that need to be isolated according to the uninterrupted running process, so that the isolated processors are invisible to the scheduler. Analyzing the configuration, obtaining a processor needing isolation (namely a target processor set, wherein the target processor set comprises at least one isolation processor), deducting the processor needing isolation from all the hardware processor lists, and putting the obtained processor (namely a non-isolation processor) list into a global scheduling domain. For example, as shown in fig. 2b, a Linux operating system needs to run a DPDK process and an SPDK process, so as to ensure the execution efficiency of the uninterrupted running process to the maximum extent, a dedicated processor may be isolated for the DPDK process and the SPDK process (uninterrupted running process), and other processes (including a host process and a virtual machine related process) run on the non-isolated processor, an isolated processor of the DPDK and the SPDK needs to be configured, the processor needing to be isolated is deducted from all processors of the system to obtain a remaining processor, and the remaining processor is used to run processes other than the DPDK and the SPDK.
202. And when the network equipment receives a data processing request, calling a corresponding uninterrupted running process.
For example, the data may be collected by the data collection device and then sent to the network device, and when receiving a data processing request, the network device invokes an uninterrupted running process corresponding to the data processing request. For example, when the request for processing data is a function library and a driver set that require fast packet processing, DPDK and SPDK processes may be invoked, and so on.
203. The network device allocates an isolation processor to the uninterrupted running process.
For example, the network device may specifically allocate an isolated processor of DPDK to a DPDK process and allocate an isolated processor of SPDK to an SPDK process from a configured target processor set, where as shown in fig. 2c, the isolated processor of DPDK may include one or more isolated processors, such as CPU1, CPU2 \ 8230, CPUX, and so on in the DPDK. The isolated processors of the SPDK may include one or more isolated processors, such as CPU1, CPU2, 8230, CPUX, etc. in the SPDK. .
204. And the network equipment binds the uninterrupted running process with the distributed isolation processor to obtain a bound processor.
For example, as shown in fig. 2c, the network device may specifically bind the DPDK processes with the allocated DPDK isolation processors one by one, for example, each DPDK process is bound with CPU1 and CPU2 in the DPDK, 8230, \ 8230, CPUX to obtain bound DPDK isolation processors, and bind the SPDK processes with the allocated SPDK isolation processors one by one, for example, each SPDK process is bound with CPU1 and CPU2 in the SPDK 8230, and \8230, CPUX to obtain bound SPDK isolation processors.
205. And the network equipment closes scheduling and interruption in the bound postprocessor to obtain the set postprocessor.
For example, the network device may specifically turn off scheduling completely, turn off all scheduling opportunities, and turn off clock interrupts to obtain the set post-processor. That is, the kernel scheduler is not run at all in the bound post-processor, there is no scheduling opportunity, no scheduling action is generated, and the bound processes (such as DPDK and SPDK processes) are exclusively owned and run 100%. For example, taking a Linux system as an example, the shutdown scheduling specifically may be: closing the scheduling opportunity when the interrupt returns, namely skipping the call related to schedule (); closing the scheduling opportunity when the exception is returned, namely skipping the call related to schedule (); close scheduling opportunities when system calls return, i.e.Skipping the schedule () related call; closing the scheduling opportunity when the task is awakened, namely skipping the call related to schedule (); and closing the scheduling opportunity when the kernel preemption count is modified, namely skipping the call related to schedule (). For example, turning off the clock interrupt may be accomplished by calling a relevant interface (e.g., a register) provided by hardware, turning off the local clock interrupt; modifying the operating system kernel's associated interface with respect to time updates, time may be provided by obtaining time for other processors (i.e., non-isolated processors) for the isolated processor (i.e., the target set of processors). By turning off the local clock interrupt, the isolated processor does not interrupt the task due to the clock interrupt, and can not schedule accordingly, thereby reducing the overhead of scheduling and interrupt。
206. The network device binds the processes in the system except the processes which run uninterruptedly to the non-isolated processor for running.
For example, the network device may specifically traverse all hardware interrupts in the system and bind the hardware interrupts to the non-isolated processor, for example, may traverse all hardware interrupts in the environment, and set the affinity of the interrupts by calling an interface provided by the hardware and/or operating system, so that all interrupts are bound to the non-isolated processor. The network device may specifically designate processes in the system, except for the DPDK and SPDK processes, to run on the non-isolated processor, for example, may modify an operating system kernel, and when creating a related system process, for example, when creating a work queue and a kernel thread of each processor (per processor), designate a processor that the processes may run as the non-isolated processor. That is, the relevant logic in the operating system kernel may be modified to isolate the isolated processor from the target processor on which the per processor kernel task is running, so that all of the per processor kernel tasks are not running on the isolated processor. For example, for an isolated processor, all per processor tasks are flushed, including some resident per processor threads in the operating system, some threads dedicated to handling the present processor transaction, such as watchdog (watchdog) and migration (migration) kernel threads in a Linux system.
207. The network equipment runs the uninterrupted running process in the set post processor so as to process the data.
For example, the network device may run the DPDK and SPDK processes continuously for 100% in the post-setting processor to process the data without any interference.
As can be seen from the above, when the network device receives a data processing request, the network device in the embodiment of the present invention may invoke a corresponding uninterrupted running process, then allocate an isolation processor to the uninterrupted running process, bind the uninterrupted running process with the allocated isolation processor, obtain a bound post-processor, then close scheduling and interruption in the bound post-processor, obtain a set post-processor, and run the uninterrupted running process in the set post-processor, so as to process the data. According to the scheme, the uninterrupted running process and the allocated isolation processor are bound, scheduling and interruption in the bound processor are closed, the specified processor resource can be isolated (other interference tasks are not run), the isolated processor is not scheduled at all, the corresponding process can really and completely monopolize the processor, no interruption and other interference exist, the uninterrupted running process runs on the set processor by hundreds of percent, and the efficiency of data processing is effectively improved. And the local clock interrupt is closed, so that the system cannot interrupt the task due to the clock interrupt, the scheduling cannot be carried out, and the expenses of scheduling and interrupt are reduced.
In order to better implement the above method, correspondingly, an embodiment of the present invention further provides a data processing apparatus, where the data processing apparatus may be specifically integrated in a network device, and the network device may be a server or a terminal.
For example, as shown in fig. 3, the data processing apparatus may include a calling unit 301, an assigning unit 302, a closing unit 303, and an executing unit 304, as follows:
(1) A calling unit 301;
the invoking unit 301 is configured to invoke a corresponding uninterrupted running process when a data processing request is received.
For example, the calling unit 301 may specifically acquire data by the data acquisition device and then send the data to the calling unit 301, and when receiving a data processing request, the calling unit 301 calls an uninterrupted running process corresponding to the data processing request.
(2) A distribution unit 302;
the allocating unit 302 is configured to allocate an isolation processor to the uninterrupted running process, and bind the uninterrupted running process and the allocated isolation processor to obtain a bound processor.
Optionally, in some embodiments, the data processing apparatus further includes an identification unit, a configuration unit, and a determination unit, as follows:
the system comprises an identification unit, a processing unit and a control unit, wherein the identification unit is used for identifying all hardware processors of the system when the system is initialized;
the configuration unit is used for configuring the candidate hardware processors and analyzing the configuration to obtain a target processor set, wherein the target processor set comprises at least one isolation processor;
a determining unit, configured to determine processors in the hardware processor except the target processor set as non-isolated processors;
the allocating unit may specifically include allocating an isolated processor to the uninterrupted running process based on the target processor set.
Optionally, in some embodiments, the configuration unit may be specifically configured to perform parameter setting on a configuration file of the system when the system is started, and transmit the set parameter to a system kernel; or the candidate hardware processor is configured through a dynamic configuration interface provided by the system when the system runs.
(3) A closing unit 303;
a closing unit 303, configured to close scheduling and interruption in the bound post-processor, so as to obtain a set post-processor.
Optionally, in some embodiments, the shutdown unit may include a shutdown scheduling subunit and a shutdown clock subunit, as follows:
a scheduling closing subunit, configured to close an interrupt return, an exception return, a system call return, a wakeup task, and modify a scheduling time when the kernel preemption count of the bound processor is performed; and
and the clock closing subunit is used for closing the clock interrupt of the bound processor.
Optionally, in some embodiments, the clock subunit is turned off, specifically, the clock subunit is configured to call hardware of the system to provide an interface of the bound processor, and turn off clock interrupt of the bound processor; and modifying the time updating interface of the system kernel, and providing time for the uninterrupted running process by acquiring the time of the non-isolated processor.
(4) An operation unit 304;
an execution unit 304, configured to execute the uninterrupted execution process in the post-setting processor to process the data.
Optionally, in some embodiments, the data processing apparatus further includes a binding unit, as follows:
and the binding unit is used for binding the processes except the uninterrupted running process in the system to the non-isolated processor to run.
Optionally, in some embodiments, the binding unit may include a binding subunit and a designated subunit, as follows:
a binding subunit, configured to traverse all hardware interrupts in the system, and bind the hardware interrupts to the non-isolated processor;
and the specified subunit is used for modifying the system kernel parameters and specifying the processes in the system except the processes which run uninterruptedly to run on the non-isolated processor.
Optionally, in some embodiments, the binding subunit may be specifically configured to invoke an interrupt interface provided by hardware of the system; and setting the relevance of the hardware interrupt and the non-isolated processor based on the interrupt interface, so that all hardware interrupts are bound to the non-isolated processor.
In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.
As can be seen from the above, in the embodiment of the present invention, when receiving a data processing request, the invoking unit 301 may invoke a corresponding uninterrupted running process, then the allocating unit 302 allocates an isolation processor for the uninterrupted running process, and binds the uninterrupted running process with the allocated isolation processor to obtain a bound processor, then the closing unit 303 closes scheduling and interruption in the bound processor to obtain a set post processor, and the running unit 304 runs the uninterrupted running process in the set post processor to process the data. According to the scheme, the uninterrupted running process and the allocated isolation processor are bound, scheduling and interruption in the bound processor are closed, the specified processor resource can be isolated (other interference tasks are not run), the isolated processor is not scheduled at all, the corresponding process can really and completely monopolize the processor, no interruption and other interference exist, the uninterrupted running process runs on the set processor by hundreds of percent, and the efficiency of data processing is effectively improved.
In addition, an embodiment of the present invention further provides a network device, as shown in fig. 4, which shows a schematic structural diagram of the network device according to the embodiment of the present invention, specifically:
the network device may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the network device architecture shown in fig. 4 does not constitute a limitation of network devices and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:
the processor 401 is a control center of the network device, connects various parts of the entire network device by using various interfaces and lines, and performs various functions of the network device and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the network device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.
The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the network device, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.
The network device further includes a power supply 403 for supplying power to each component, and preferably, the power supply 403 is logically connected to the processor 401 through a power management system, so that functions of managing charging, discharging, and power consumption are implemented through the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
The network device may also include an input unit 404, the input unit 404 being operable to receive input numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
Although not shown, the network device may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 401 in the network device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:
when a processing request of data is received, calling a corresponding uninterrupted running process, then allocating an isolation processor for the uninterrupted running process, binding the uninterrupted running process with the allocated isolation processor to obtain a bound post processor, then closing scheduling and interruption in the bound post processor to obtain a set post processor, and running the uninterrupted running process in the set post processor to process the data.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
As can be seen from the above, in the embodiments of the present invention, when a data processing request is received, a corresponding uninterrupted running process may be called, then, an isolation processor is allocated to the uninterrupted running process, the uninterrupted running process is bound to the allocated isolation processor, a bound processor is obtained, then, scheduling and interruption in the bound processor are closed, a set post-processor is obtained, and the uninterrupted running process is run in the set post-processor, so as to process the data. According to the scheme, the uninterrupted running process and the allocated isolation processor are bound, scheduling and interruption in the bound processor are closed, the specified processor resource can be isolated (other interference tasks are not run), the isolated processor is not scheduled at all, the corresponding process can really and completely monopolize the processor, no interruption and other interference exist, the uninterrupted running process runs on the set processor by hundreds of percent, and the efficiency of data processing is effectively improved.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.
To this end, embodiments of the present invention further provide a storage medium, in which a plurality of instructions are stored, where the instructions can be loaded by a processor to execute the steps in any one of the data processing methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:
when a processing request of data is received, calling a corresponding uninterrupted running process, then allocating an isolation processor for the uninterrupted running process, binding the uninterrupted running process with the allocated isolation processor to obtain a bound processor, then closing scheduling and interruption in the bound processor to obtain a set post processor, and running the uninterrupted running process in the set post processor to process the data.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the storage medium can execute the steps in any data processing method provided in the embodiment of the present invention, the beneficial effects that can be achieved by any data processing method provided in the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.
The foregoing detailed description has provided a data processing method, apparatus, and storage medium according to embodiments of the present invention, and the present invention has been described in detail using specific examples to explain the principles and implementations of the present invention, and the description of the foregoing embodiments is only used to help understand the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (7)
1. A data processing method, comprising:
identifying all hardware processors of the system at system initialization;
configuring candidate hardware processors, analyzing the configuration to obtain a target processor set, wherein the target processor set comprises at least one isolation processor, deducting the at least one isolation processor from a hardware processor list, and putting the processor list obtained by deducting into a global scheduling domain;
determining processors in the hardware processors except the target processor set as non-isolated processors;
when a data processing request is received, calling a corresponding uninterrupted running process;
distributing an isolation processor for each uninterrupted running process based on the target processor set, and binding each uninterrupted running process with the distributed isolation processor one by one to obtain a bound processor;
closing scheduling and interruption in the bound post-processor to obtain a set post-processor; wherein, the closing of scheduling and interruption in the bound processor comprises: closing interrupt return, exception return, system call return, task awakening and scheduling opportunity during kernel preemption count modification of the bound processor, and closing clock interrupt of the bound processor;
binding processes except the uninterrupted running process in the system on the non-isolated processor to run, and cleaning up tasks in all isolated processors aiming at the isolated processors, wherein the tasks comprise processor threads resident in an operating system and threads special for processing the transactions of the processor; wherein the binding of the processes in the system except the uninterruptedly running process to the non-isolated processor comprises: traversing all hardware interrupts in the system, binding the hardware interrupts to the non-isolated processor, modifying system kernel parameters, and designating processes in the system except for an uninterrupted running process to run on the non-isolated processor;
and running the uninterrupted running process in the set post processor to process the data.
2. The method of claim 1, wherein said shutting down clock interrupts of the bound processor comprises:
calling hardware of the system to provide an interface of the bound processor, and closing clock interruption of the bound processor;
and modifying a time updating interface of the system kernel, and providing time for the uninterrupted running process by acquiring the time of the non-isolated processor.
3. The method of claim 1, wherein configuring the candidate processors comprises:
when the system is started, setting parameters of a configuration file of the system, and transmitting the set parameters to a system kernel; or
And when the system runs, configuring the candidate hardware processor through a dynamic configuration interface provided by the system.
4. The method of claim 1, wherein binding the hardware interrupt to the non-sequestered processor comprises:
calling an interrupt interface provided by hardware of the system;
setting the relevance of hardware interrupt and the non-isolated processor based on the interrupt interface, and binding all hardware interrupt to the non-isolated processor.
5. A data processing apparatus, comprising:
the system comprises an identification unit, a processing unit and a control unit, wherein the identification unit is used for identifying all hardware processors of the system when the system is initialized;
a configuration unit, configured to configure a candidate hardware processor, and analyze the configuration to obtain a target processor set, where the target processor set includes at least one isolated processor, deduct the at least one isolated processor from a hardware processor list, and place the deducted processor list in a global scheduling domain;
a determining unit, configured to determine a processor in the hardware processor except the target processor set as a non-isolated processor calling unit, and configured to call a corresponding uninterrupted running process when a processing request of data is received;
the distribution unit is used for distributing an isolation processor for each uninterrupted operation process based on the target processor set, and binding each uninterrupted operation process with the distributed isolation processor one by one to obtain a bound processor;
a closing unit, configured to close scheduling and interruption in the bound postprocessor, to obtain a set postprocessor; the shutdown unit comprises a shutdown scheduling subunit and a shutdown clock subunit, wherein the shutdown scheduling subunit is used for shutting down interrupt return, exception return, system call return, awakening task of the bound processor and modifying scheduling opportunity during kernel preemption count, and the shutdown clock subunit is used for shutting down clock interrupt of the bound processor;
a binding unit, configured to bind processes in the system except for an uninterruptedly running process to the non-isolated processor for running, and clean up tasks in all isolated processors for the isolated processors, where the tasks include processor threads resident in an operating system and threads dedicated to processing transactions of the processor; the binding unit comprises a binding subunit and a designated subunit, wherein the binding subunit is used for traversing all hardware interrupts in the system and binding the hardware interrupts to the non-isolated processor; the appointing subunit is used for modifying the system kernel parameters and appointing processes in the system except for the uninterruptedly-operated processes to operate on the non-isolated processor;
and the running unit is used for running the uninterrupted running process in the set post-processor so as to process the data.
6. The apparatus according to claim 5, wherein the clock shutdown subunit is configured to invoke hardware of the system to provide an interface of the bound processor, and shut down a clock interrupt of the bound processor; and modifying a time updating interface of the system kernel, and providing time for the uninterrupted running process by acquiring the time of the non-isolated processor.
7. A storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the data processing method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910570004.6A CN110333899B (en) | 2019-06-27 | 2019-06-27 | Data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910570004.6A CN110333899B (en) | 2019-06-27 | 2019-06-27 | Data processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110333899A CN110333899A (en) | 2019-10-15 |
| CN110333899B true CN110333899B (en) | 2022-11-01 |
Family
ID=68143506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910570004.6A Active CN110333899B (en) | 2019-06-27 | 2019-06-27 | Data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110333899B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368178A (en) * | 2017-08-21 | 2017-11-21 | 湖南大学 | Power consumption optimization method based on DPDK processing data packets platforms |
| CN107786458A (en) * | 2017-11-02 | 2018-03-09 | 下代互联网重大应用技术（北京）工程研究中心有限公司 | The method that multiport access standard based on DPDK goes out |
| CN108490893A (en) * | 2018-02-13 | 2018-09-04 | 烽台科技（北京）有限公司 | A kind of industrial control method, device and equipment |
| CN109672575A (en) * | 2019-01-30 | 2019-04-23 | 新华三技术有限公司合肥分公司 | Data processing method and electronic equipment |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW200511111A (en) * | 2003-07-30 | 2005-03-16 | Koninkl Philips Electronics Nv | Microcontroller with an interrupt structure having programmable priority levels with each priority level associated with a different register set |
| CN102831015B (en) * | 2012-08-01 | 2015-09-23 | 华为技术有限公司 | The dispatching method of polycaryon processor and equipment |
| CN105718320B (en) * | 2016-01-18 | 2020-11-06 | 华为技术有限公司 | Clock task processing method, device and equipment |
| US20180165117A1 (en) * | 2016-12-08 | 2018-06-14 | NoFutzNetworks Inc. | Software Switch Hypervisor for Isolation of Cross-Port Network Traffic |
| KR102316971B1 (en) * | 2017-05-08 | 2021-10-25 | 삼성전자 주식회사 | Method and apparatus for allocating resource in software defined network |
| US10776012B2 (en) * | 2017-05-19 | 2020-09-15 | Exten Technologies, Inc. | Lock-free datapath design for efficient parallel processing storage array implementation |
| CN107483237A (en) * | 2017-08-02 | 2017-12-15 | 赛尔网络有限公司 | Performance optimization system and method based on DPDK forwardings |
-
2019
- 2019-06-27 CN CN201910570004.6A patent/CN110333899B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368178A (en) * | 2017-08-21 | 2017-11-21 | 湖南大学 | Power consumption optimization method based on DPDK processing data packets platforms |
| CN107786458A (en) * | 2017-11-02 | 2018-03-09 | 下代互联网重大应用技术（北京）工程研究中心有限公司 | The method that multiport access standard based on DPDK goes out |
| CN108490893A (en) * | 2018-02-13 | 2018-09-04 | 烽台科技（北京）有限公司 | A kind of industrial control method, device and equipment |
| CN109672575A (en) * | 2019-01-30 | 2019-04-23 | 新华三技术有限公司合肥分公司 | Data processing method and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 基于DPDK实现企业网络性能优化的研究与设计;张郁;《中国优秀硕士论文全文数据库》;20190115;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110333899A (en) | 2019-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3425502B1 (en) | Task scheduling method and device | |
| EP3039540B1 (en) | Virtual machine monitor configured to support latency sensitive virtual machines | |
| US9507631B2 (en) | Migrating a running, preempted workload in a grid computing system | |
| US6466962B2 (en) | System and method for supporting real-time computing within general purpose operating systems | |
| EP3073373B1 (en) | Method for interruption affinity binding of virtual network interface card, and computer device | |
| JP6294586B2 (en) | Execution management system combining instruction threads and management method | |
| US7698540B2 (en) | Dynamic hardware multithreading and partitioned hardware multithreading | |
| JP5005191B2 (en) | System and method for implementing an operating system in a virtual machine environment | |
| US9396028B2 (en) | Scheduling workloads and making provision decisions of computer resources in a computing environment | |
| US9063783B2 (en) | Coordinating parallel execution of processes using agents | |
| US20070204268A1 (en) | Methods and systems for scheduling processes in a multi-core processor environment | |
| US9411649B2 (en) | Resource allocation method | |
| US11579918B2 (en) | Optimizing host CPU usage based on virtual machine guest OS power and performance management | |
| US20110219373A1 (en) | Virtual machine management apparatus and virtualization method for virtualization-supporting terminal platform | |
| US20150254113A1 (en) | Lock Spin Wait Operation for Multi-Threaded Applications in a Multi-Core Computing Environment | |
| KR20070090649A (en) | Apparatus and method for providing cooperative scheduling on multi-core system | |
| US11216301B2 (en) | Process scheduling in a processing system having at least one processor and shared hardware resources | |
| CN111459622B (en) | Method, device, computer equipment and storage medium for scheduling virtual CPU | |
| US8977752B2 (en) | Event-based dynamic resource provisioning | |
| Li et al. | Multi-mode virtualization for soft real-time systems | |
| CN110333899B (en) | Data processing method, device and storage medium | |
| Monaco et al. | Shared resource orchestration extensions for kubernetes to support real-time cloud containers | |
| KR101334842B1 (en) | Virtual machine manager for platform of terminal having function of virtualization and method thereof | |
| JP2001282560A (en) | Virtual computer control method, its performing device and recording medium recording its processing program | |
| Walters et al. | Enabling interactive jobs in virtualized data centers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230920 Address after: 100086 Beijing Haidian District Zhichun Road 49 No. 3 West 309 Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd. Address before: 518057 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 floors Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. |