CN115599563A - Parallel I/O control method, device and equipment for serial I/O access - Google Patents
Parallel I/O control method, device and equipment for serial I/O access Download PDFInfo
- Publication number
- CN115599563A CN115599563A CN202211503147.3A CN202211503147A CN115599563A CN 115599563 A CN115599563 A CN 115599563A CN 202211503147 A CN202211503147 A CN 202211503147A CN 115599563 A CN115599563 A CN 115599563A
- Authority
- CN
- China
- Prior art keywords
- parallel
- special
- serial
- service
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The embodiment of the invention provides a parallel I/O control method, a device and equipment for serial I/O access, wherein the method comprises the following steps: responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; the parallel I/O system is run on the I/O specific process and the parallel I/O system is used to execute the I/O request corresponding to the I/O service command. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of I/O request processing, in particular to a parallel I/O control method, a parallel I/O control device and parallel I/O control equipment for serial I/O access.
Background
During the operation of the application program, data needs to be input from the storage medium based on a file reading mode, and a calculation result is output to the storage medium in a file writing mode. Data Input/Output (I/O) is the slowest link in the calculation process of an application program, and becomes a bottleneck restricting the performance of the application program. With the increasing demand of applications, the amount of calculation of applications and the amount of data to be processed are increasing. With the increasing processing power of computers, modern high-performance computers provide not only a plurality of processor cores capable of being paralleled for accelerating computation, but also a parallel storage function containing hardware and a parallel file system for accelerating I/O.
In order to make full use of the parallel storage function by an application program to minimize the impact of I/O overhead on the running speed of the application program, parallel I/O techniques have emerged. Parallel I/O techniques are implemented directly or indirectly using I/O Message Passing interfaces (MPI-IO). Therefore, a basic premise of adopting the parallel I/O technology to accelerate I/O is that the application program is a multi-process parallel program based on MPI parallelization implementation. However, there still exist a lot of serial programs with high I/O overhead or application programs using a serial file format, and since parallel I/O technology cannot be used, data can be input and output only in a serial manner, and the I/O overhead cannot be reduced by using a parallel storage function.
In view of the foregoing, a need exists for a method for reducing I/O overhead of a serial program or an application program using a serial file format to improve program performance.
Disclosure of Invention
The embodiment of the invention provides a parallel I/O control method, a parallel I/O control device and parallel I/O control equipment for serial I/O access, which are used for solving the problem of high I/O overhead of a serial program or an application program adopting a serial file format in the conventional method.
In a first aspect, an embodiment of the present invention provides a parallel I/O control method for serial I/O access, including:
responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format;
when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes;
the parallel I/O system is run on the I/O specific process and the parallel I/O system is used to execute the I/O request corresponding to the I/O service command.
In one embodiment, running a parallel I/O system on an I/O-specific process and using the parallel I/O system to execute an I/O request corresponding to an I/O service command comprises:
the asynchronous parallel I/O system is run on the I/O specific process and the I/O request corresponding to the I/O service command is executed using the asynchronous parallel I/O system.
In one embodiment, determining an I/O specific process comprises:
acquiring the I/O process number of an I/O special process, and when the I/O process number is larger than a preset I/O process number upper limit threshold, setting the I/O process number as the preset I/O process number upper limit threshold;
judging whether the process corresponding to the I/O process number is started or not;
if the process is started, multiplexing the process corresponding to the I/O process number as an I/O special process;
if not, the message transmission interface is adopted to run a command to start a plurality of processes of the I/O process as I/O special processes.
In one embodiment, executing an I/O request corresponding to an I/O service command using a parallel I/O system includes:
establishing a special communication channel between the process of the application program and the I/O special process;
sending information of the I/O request from the process of the application program to the I/O dedicated process by using a dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing the parallel I/O function of the parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
the processing result of the I/O request is transmitted from the I/O-specific process to the process of the application program using the dedicated communication channel.
In one embodiment, the dedicated communication channel is implemented based on an anonymous pipe, a named pipe, a message queue, or a virtual shared memory between processes.
In one embodiment, when the I/O service mode is to serve an I/O-specific process, the method further comprises: after all I/O requests are processed, the I/O special process is controlled to enter a dormant state which does not occupy processor resources.
In one embodiment, the method further comprises:
when the I/O service mode is I/O special thread service, the process of the application program starts the I/O special thread;
sending information of the I/O request from the process of the application program to the I/O special thread in a memory sharing mode;
executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
In one embodiment, the I/O request comprises: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O to complete.
In a second aspect, an embodiment of the present invention provides a parallel I/O control apparatus for serial I/O access, including:
the response module is used for responding to an I/O service command initiated by the application program in a serial I/O access mode and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format;
the processing module is used for determining the I/O special process when the I/O service mode is I/O special process service, and the I/O special process comprises a plurality of processes;
and the execution module is used for running the parallel I/O system on the I/O special process and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executes the memory-stored computer-executable instructions to cause the at least one processor to perform the method of parallel I/O control of serial I/O access according to any of the first aspects.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the parallel I/O control method for serial I/O access according to any one of the first aspect.
The parallel I/O control method, the device and the equipment for serial I/O access respond to an I/O service command initiated by an application program in a serial I/O access mode, and acquire the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; the parallel I/O system is run on the I/O specific process and the parallel I/O system is used to execute the I/O request corresponding to the I/O service command. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart of a parallel I/O control method for serial I/O access according to an embodiment of the present invention;
FIG. 2 is a flow chart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention;
FIG. 3 is a flow chart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention;
FIG. 4 is a flow chart of a parallel I/O control method for serial I/O access according to yet another embodiment of the present invention;
FIG. 5 is a block diagram of a parallel I/O control apparatus for serial I/O access according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
With the above figures, certain embodiments of the invention have been illustrated and described in more detail below. The drawings and the description are not intended to limit the scope of the inventive concept in any way, but rather to illustrate it by those skilled in the art with reference to specific embodiments.
Detailed Description
The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. Wherein like elements in different embodiments are numbered with like associated elements. In the following description, numerous details are set forth in order to provide a better understanding of the present application. However, those skilled in the art will readily recognize that some of the features may be omitted or replaced with other elements, materials, methods in different instances. In some instances, certain operations related to the present application have not been shown or described in detail in order to avoid obscuring the core of the present application from excessive description, and it is not necessary for those skilled in the art to describe these operations in detail, so that they may be fully understood from the description in the specification and the general knowledge in the art.
Furthermore, the features, operations, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. Also, the various steps or actions in the description of the methods may be transposed or transposed in order, as will be apparent to a person skilled in the art. Thus, the various sequences in the specification and drawings are for the purpose of clearly describing certain embodiments only and are not intended to imply a required sequence unless otherwise indicated where a certain sequence must be followed.
The numbering of the components as such, e.g., "first", "second", etc., is used herein only to distinguish the objects as described, and does not have any sequential or technical meaning. The term "connected" and "coupled" when used in this application, unless otherwise indicated, includes both direct and indirect connections (couplings).
Fig. 1 is a flowchart of a parallel I/O control method for serial I/O access according to an embodiment of the present invention. As shown in fig. 1, the parallel I/O control method for serial I/O access provided in this embodiment may include:
s101, responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format.
The serial program or the application program adopting the serial file format cannot use the parallel I/O technology, and can only initiate an I/O service command in a serial I/O access manner.
The I/O service command may include information on the I/O service mode and the I/O request. The I/O service mode is used to indicate the mode by which the I/O request is completed, and may include, for example, an I/O-specific thread service and an I/O-specific process service. The I/O-specific process service is used to indicate that an I/O request is completed by a set of dedicated processes, and the I/O-specific thread service is used to indicate that an I/O request is completed by a set of dedicated threads. The information of the I/O request is used to indicate the I/O operation that needs to be performed.
S102, when the I/O service mode is the I/O special process service, determining the I/O special process, wherein the I/O special process comprises a plurality of processes.
When the I/O service mode obtained from the I/O service command is used for servicing the I/O special process, determining that the plurality of processes are specially used for executing the I/O operation. A plurality of processes dedicated to performing I/O operations are determined to be I/O specific processes.
S103, running the parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
The I/O requests corresponding to the I/O service commands are executed by a parallel I/O system running on an I/O specific process. The parallel I/O technology can dispersedly store data of an application program to a plurality of processes, and the plurality of processes can cooperatively read and write file data to improve the I/O speed. In this embodiment, supporting multiple types of I/O requests may include, for example: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O to complete.
In the parallel I/O control method for serial I/O access provided by this embodiment, an I/O service command initiated by an application program in a serial I/O access manner is responded, and an I/O service manner is obtained from the I/O service command, where the application program is a serial program or the application program adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; the parallel I/O system is run on the I/O specific process and the parallel I/O system is used to execute the I/O request corresponding to the I/O service command. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
On the basis of the above embodiments, in order to further reduce the I/O overhead and increase the running speed of the application program, the method provided by this embodiment uses the asynchronous parallel I/O system to execute the I/O request corresponding to the I/O service command on the I/O dedicated process. The parallel I/O technology dispersedly stores the data of the application program to a plurality of processes, and the plurality of processes improve the I/O speed by cooperatively reading and writing the file data. Asynchronous I/O techniques employ I/O-specific processes/threads other than compute processes/threads to complete data I/O operations, allowing an application to continue performing other computations without waiting for the completion of the I/O operations, thereby allowing the computation and I/O processing of the application to be performed in an overlapping manner. In the embodiment, the asynchronous parallel I/O system giving consideration to both parallel I/O and asynchronous I/O is adopted, so that the influence of I/O overhead on the performance of a serial program or an application program adopting a serial file format can be reduced to the greatest extent. That is, in an alternative embodiment, running the parallel I/O system on an I/O specific process and using the parallel I/O system to execute an I/O request corresponding to the I/O service command may include: the asynchronous parallel I/O system is run on the I/O specific process and the I/O request corresponding to the I/O service command is executed using the asynchronous parallel I/O system.
How to determine the I/O specific process is described in detail below. Referring to fig. 2, fig. 2 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 2, on the basis of the embodiment shown in fig. 1, in the parallel I/O control method for serial I/O access provided in this embodiment, determining an I/O dedicated process may include:
and S1031, acquiring the number of I/O processes of the I/O special process, and when the number of the I/O processes is larger than a preset upper limit threshold of the number of the I/O processes, setting the number of the I/O processes as the preset upper limit threshold of the number of the I/O processes.
Typically, I/O-specific processes need to run on the computer on which the current process (i.e., the process of the application, and also the process of serial I/O access) is located, and the number of cores on the current computer is limited, so the number of I/O processes cannot exceed the resource limit of the current computer. Therefore, the number of I/O processes actually used by the I/O special process can be limited by presetting the upper limit threshold of the number of I/O processes so as not to exceed the resource limit. Specifically, when the number of I/O processes is greater than a preset upper limit I/O process number threshold, the number of I/O processes is set as the preset upper limit I/O process number threshold. In an alternative embodiment, the number of I/O processes of the I/O specific process may be preset by the user. It can be understood that when the number of I/O processes increases to a certain number, the I/O speed may not increase but decrease, that is, the scalability of the performance of the parallel I/O with the increase of the number of I/O processes has a limit, which also limits the upper limit of the number of I/O processes. In yet another alternative embodiment, the number of I/O processes that make the I/O speed the fastest under the above-mentioned limiting conditions may also be obtained based on a machine learning method as the number of I/O processes of the I/O-specific process. In another alternative embodiment, the optimal number of I/O processes may also be found by debugging the number of I/O processes and running the program multiple times.
S1032, whether the process corresponding to the number of the I/O processes is started or not is judged. If so, go to step S1033; if not, go to step S1034.
S1033, multiplexing the process corresponding to the I/O process number to be used as the I/O special process.
S1034, starting a plurality of processes of the I/O process as I/O special processes by adopting the message transmission interface operation command.
After determining the number of I/O processes of the I/O dedicated processes, in order to avoid waste of processor resources caused by repeatedly starting the processes, it is first necessary to determine whether a group of dedicated processes corresponding to the number of I/O processes has been started. If the process is started, the group of special processes is reused without restarting; if not, starting a group of special processes according to the number of the I/O processes, and starting a parallel I/O system or an asynchronous parallel I/O system on the group of processes. Typically, the above-described startup procedure involves starting a set of dedicated processes running either a parallel I/O system or an asynchronous parallel I/O system with MPI run commands.
In the parallel I/O control method for serial I/O access provided in this embodiment, on the basis of the above embodiment, by determining whether a process corresponding to the number of I/O processes has been started, when the process has been started, the process corresponding to the number of I/O processes is multiplexed as an I/O dedicated process; when the I/O process is not started, a message transmission interface operation command is adopted to start a plurality of processes of the I/O process to be used as I/O special processes. Not only can enough processes be ensured to run the parallel I/O system, but also the waste of processor resources caused by repeatedly starting the processes can be avoided.
In an optional embodiment, in order to further improve the utilization efficiency of the processor resources, when the I/O service mode is to serve an I/O dedicated process, after all I/O requests are processed, the I/O dedicated process may be further controlled to enter a sleep state that does not occupy the processor resources, so as to avoid additional overhead to the maximum extent. When responding to the service of executing the I/O request by the parallel I/O system or the asynchronous parallel I/O system, if the I/O special process is in the dormant state, the I/O special process only needs to be awakened, and the I/O special process is determined to be in the working state.
When the parallel I/O system is used for executing the I/O request corresponding to the I/O service command, the application program needs to send the information of the I/O request to the parallel I/O system, and the parallel I/O system utilizes the parallel I/O function of the parallel I/O system to execute the corresponding I/O operation and needs to return the result to the application program. However, since the application and the parallel I/O system run on a group of processes initiated by different commands, it is not possible to establish an MPI communication domain that contains both application processes and I/O-specific processes. How to use the parallel I/O system to execute the I/O request corresponding to the I/O service command is described in detail by specific embodiments below. Referring to fig. 3, fig. 3 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 3, on the basis of any of the foregoing embodiments, in the parallel I/O control method for serial I/O access provided by this embodiment, executing an I/O request corresponding to an I/O service command by using a parallel I/O system may include:
s1041, establishing a special communication channel between the process of the application program and the I/O special process.
The specific implementation method of the dedicated communication channel in this embodiment includes, but is not limited to, anonymous pipes based on processes, named pipes, message queues, virtual shared memory, and the like.
S1042, information of the I/O request is sent to the I/O special process from the process of the application program by utilizing the special communication channel.
After the dedicated communication channel is established, information of the I/O request may be sent from the process of the application to the I/O-specific process using the dedicated communication channel.
S1043, executing corresponding operation corresponding to the I/O request by using the parallel I/O function of the parallel I/O system running on the I/O special process, and obtaining the processing result of the I/O request.
S1044, transmitting the processing result of the I/O request from the I/O special process to the process of the application program by utilizing the special communication channel.
It should be noted that although the present embodiment has been described with respect to the case of using a parallel I/O system, the present embodiment is also applicable to the case of using an asynchronous parallel I/O system. The corresponding operation corresponding to the I/O request is executed by using the parallel I/O function, the asynchronous I/O function or the asynchronous parallel I/O function of the asynchronous parallel I/O system running on the I/O dedicated process, so as to obtain the processing result of the I/O request. For a specific implementation manner of executing the I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system, reference may be made to the above steps, which are not described herein again.
Based on any of the foregoing embodiments, the parallel I/O control method for serial I/O access further completes information transfer between the process of the application program and the I/O dedicated process by establishing a dedicated communication channel between the process of the application program and the I/O dedicated process and using the dedicated communication channel, so that the parallel I/O system running on the I/O dedicated process completes an I/O request corresponding to an I/O service command.
The above embodiments are described with respect to the case where the I/O service scheme is an I/O-dedicated process service, and the following description is given with respect to the case where the I/O service scheme is an I/O-dedicated thread service. Referring to fig. 4, fig. 4 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 4, on the basis of any of the foregoing embodiments, the parallel I/O control method for serial I/O access provided by this embodiment may further include:
and S104, when the I/O service mode is I/O special thread service, starting the I/O special thread by the process of the application program.
And S105, sending the information of the I/O request from the process of the application program to the I/O special thread in a memory sharing mode.
S106, executing corresponding operation corresponding to the I/O request by using the asynchronous I/O function of the I/O special thread to obtain the processing result of the I/O request.
And S107, sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
In the parallel I/O control method for serial I/O access provided by this embodiment, by starting the I/O dedicated thread and performing information transfer between the I/O dedicated thread and the application program in a memory sharing manner, and by using the asynchronous I/O function of the I/O dedicated thread to execute the corresponding operation corresponding to the I/O request, the serial program or the application program adopting the serial file format can input and output data in an asynchronous manner, thereby reducing the I/O overhead and contributing to improving the performance of the application program.
How to use the method provided by the present application is illustrated below by two specific application scenarios.
Application scenario 1: single process applications or parallel programs with only one process performing I/O processing. The target is as follows: asynchronous parallel I/O acceleration is implemented for applications. The operation steps are as follows: the method is used for replacing the original serial I/O access in the application program by using the program interface for initiating the asynchronous parallel I/O service command provided by the method, and setting an I/O service mode and an I/O request in each service command. When an I/O request is a service command of asynchronous data input/output, a corresponding service command of the I/O request waiting for the completion of asynchronous I/O needs to be inserted into an application program. And running the improved application program to confirm the correct use of the asynchronous parallel I/O service command and the I/O acceleration effect. When the I/O special process service mode is adopted, the optimal preset special I/O process number can be found by debugging the preset special I/O process number and running the program for multiple times.
Application scenario 2: serial NetCDF library and other serial I/O access application file formats. NetCDF is a common application file format in the fields of numerical weather forecast and climate simulation prediction. The target is as follows: the I/O efficiency of the serial I/O access mode of the application program in the NetCDF library is improved. The method comprises the following operation steps: in the application program, the configuration program interface of the asynchronous parallel I/O service command provided by the method is utilized to set the I/O service mode and set the I/O request mode, including synchronous input/output and asynchronous input/output. Finding out APIs (application programming interfaces) related to data input and output in a NetCDF library, modifying implementation codes of the APIs, and acquiring information of corresponding I/O (input/output) requests from output information of the APIs; then, an I/O service mode and an I/O request mode are obtained from the configuration information of the method, an asynchronous parallel I/O service command is initiated by combining the information of the I/O request, and the original data input/output operation in the API is stopped. And running the improved application program and the NetCDF library to confirm the correct use of the asynchronous parallel I/O service command and the I/O acceleration effect. When the I/O special process service mode is adopted, the optimal preset special I/O process number can be found by debugging the preset special I/O process number and running the program for multiple times.
Fig. 5 is a schematic structural diagram of a parallel I/O control apparatus for serial I/O access according to an embodiment of the present invention. As shown in fig. 5, the parallel I/O control apparatus 50 for serial I/O access provided in this embodiment may include: a response module 501, a processing module 502 and an execution module 503.
A response module 501, configured to respond to an I/O service command initiated by an application program in a serial I/O access manner, and obtain the I/O service manner from the I/O service command, where the application program is a serial program or the application program adopts a serial file format;
a processing module 502, configured to determine an I/O dedicated process when the I/O service mode is an I/O dedicated process service, where the I/O dedicated process includes multiple processes;
and an executing module 503, configured to run the parallel I/O system on the I/O dedicated process, and execute the I/O request corresponding to the I/O service command by using the parallel I/O system.
The apparatus of this embodiment may be configured to implement the technical solution of the method embodiment shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.
In an optional embodiment, the executing module 503 is configured to run the parallel I/O system on the I/O dedicated process, and to execute the I/O request corresponding to the I/O service command by using the parallel I/O system specifically includes: the asynchronous parallel I/O system is run on the I/O specific process and the I/O request corresponding to the I/O service command is executed using the asynchronous parallel I/O system.
In an optional implementation, the processing module 502 configured to determine the I/O specific process may specifically include:
acquiring the number of I/O processes of the I/O special process, and setting the number of the I/O processes as a preset upper limit threshold of the number of the I/O processes when the number of the I/O processes is larger than the preset upper limit threshold of the number of the I/O processes;
judging whether the process corresponding to the I/O process number is started or not;
if the process is started, multiplexing the process corresponding to the I/O process number as an I/O special process;
if not, the message transmission interface is adopted to run a command to start a plurality of processes of the I/O process as I/O special processes.
In an optional implementation manner, the executing module 503, configured to execute the I/O request corresponding to the I/O service command by using the parallel I/O system, may specifically include:
establishing a special communication channel between the process of the application program and the I/O special process;
sending information of the I/O request from the process of the application program to the I/O dedicated process by using a dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing the parallel I/O function of the parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
the processing result of the I/O request is transmitted from the I/O-specific process to the process of the application using the dedicated communication channel.
In an alternative embodiment, the dedicated communication channel is implemented based on an anonymous pipe between processes, a named pipe, a message queue, or a virtual shared memory.
In an alternative embodiment, when the I/O service mode is to serve an I/O dedicated process, the execution module 503 is further configured to control the I/O dedicated process to enter a sleep state that does not occupy processor resources after all I/O requests are processed.
In an optional embodiment, the execution module 503 is further configured to, when the I/O service mode is an I/O dedicated thread service, start the I/O dedicated thread by a process of the application; sending information of the I/O request from the process of the application program to the I/O special thread in a memory sharing mode; executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request; and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
In an alternative embodiment, the I/O request may include: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O to complete.
Fig. 6 is a schematic view showing an electronic device according to an embodiment of the present invention, which is only illustrated in fig. 6, and the embodiment of the present invention is not limited thereto. Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 6, the electronic device 60 provided in the present embodiment includes: memory 601, processor 602, and bus 603. The bus 603 is used for connecting the elements.
The memory 601 stores a computer program, and the computer program can implement the technical solution of any of the above method embodiments when executed by the processor 602.
Wherein, the memory 601 and the processor 602 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements may be electrically connected to each other via one or more communication buses or signal lines, such as bus 603. The memory 601 stores therein a computer program for implementing a parallel I/O control method for serial I/O access, including at least one software functional module that can be stored in the memory 601 in the form of software or firmware, and the processor 602 executes various functional applications and data processing by running the software program and module stored in the memory 601.
The Memory 601 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like. The memory 601 is used for storing programs, and the processor 602 executes the programs after receiving the execution instructions. Further, the software programs and modules within the memory 601 may also include an operating system, which may include various software components and/or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.) and may communicate with various hardware or software components to provide an operating environment for other software components.
The processor 602 may be an integrated circuit chip having signal processing capabilities. The Processor 602 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and so on. The various methods, steps, and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. It will be appreciated that the configuration of fig. 6 is merely illustrative and may include more or fewer components than shown in fig. 6 or have a different configuration than shown in fig. 6. The components shown in fig. 6 may be implemented in hardware and/or software.
The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement technical solutions of any of the above method embodiments.
The embodiments in the disclosure are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments.
The scope of the present disclosure is not limited to the above-described embodiments, and it is apparent that various modifications and variations can be made to the present disclosure by those skilled in the art without departing from the scope and spirit of the present disclosure. It is intended that the present disclosure also cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.
Claims (11)
1. A parallel I/O control method for serial I/O access, comprising:
responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format;
when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes;
and running a parallel I/O system on the I/O special process, and executing an I/O request corresponding to the I/O service command by using the parallel I/O system.
2. The method of claim 1, wherein the running a parallel I/O system on the I/O specific process and using the parallel I/O system to execute the I/O request corresponding to the I/O service command comprises:
and running an asynchronous parallel I/O system on the I/O special process, and executing an I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system.
3. The method of claim 1, wherein determining the I/O specific process comprises:
acquiring the number of I/O processes of the I/O special process, and setting the number of the I/O processes as a preset upper limit threshold of the number of the I/O processes when the number of the I/O processes is larger than the preset upper limit threshold of the number of the I/O processes;
judging whether the process corresponding to the I/O process number is started or not;
if the process is started, multiplexing the process corresponding to the I/O process number as an I/O special process;
if not, starting a plurality of processes of the I/O process as I/O special processes by adopting a message transmission interface operation command.
4. The method of claim 1, wherein the executing, using the parallel I/O system, the I/O request corresponding to the I/O service command comprises:
establishing a special communication channel between the process of the application program and the I/O special process;
sending information of the I/O request from a process of the application to the I/O-specific process using the dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing a parallel I/O function of a parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special process to the process of the application program by utilizing the special communication channel.
5. The method of claim 4, wherein the dedicated communication channel is implemented based on an anonymous pipe between processes, a named pipe, a message queue, or a virtual shared memory.
6. The method according to any of claims 1-5, wherein when the I/O service mode is an I/O-specific process service, the method further comprises: and after all the I/O requests are processed, controlling the I/O special process to enter a dormant state which does not occupy processor resources.
7. The method of claim 1, further comprising:
when the I/O service mode is I/O special thread service, starting the I/O special thread by the process of the application program;
sending the information of the I/O request to the I/O special thread from the process of the application program in a memory sharing mode;
executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
8. The method of any of claims 1-5, wherein the I/O request comprises: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O to complete.
9. A parallel I/O control apparatus for serial I/O access, comprising:
the response module is used for responding to an I/O service command initiated by an application program in a serial I/O access mode and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format;
the processing module is used for determining an I/O special process when the I/O service mode is I/O special process service, and the I/O special process comprises a plurality of processes;
and the execution module is used for running the parallel I/O system on the I/O special process and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
10. An electronic device, comprising: at least one processor and memory;
the memory stores computer execution instructions;
the at least one processor executing the memory-stored computer-executable instructions cause the at least one processor to perform the method of parallel I/O control of serial I/O access of any of claims 1-8.
11. A computer-readable storage medium having stored therein computer-executable instructions for implementing the parallel I/O control method for serial I/O access of any one of claims 1-8 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211503147.3A CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211503147.3A CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115599563A true CN115599563A (en) | 2023-01-13 |
| CN115599563B CN115599563B (en) | 2023-06-13 |
Family
ID=84852325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211503147.3A Active CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115599563B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130054808A1 (en) * | 2011-08-31 | 2013-02-28 | International Business Machines Corporation | Management of storage cluster performance with hybrid workloads |
| US20130166886A1 (en) * | 2008-11-24 | 2013-06-27 | Ruchira Sasanka | Systems, apparatuses, and methods for a hardware and software system to automatically decompose a program to multiple parallel threads |
| CN107391422A (en) * | 2017-07-19 | 2017-11-24 | 上海航天测控通信研究所 | multi-path asynchronous serial communication data access system and method |
| CN107748700A (en) * | 2017-10-09 | 2018-03-02 | 北京东土科技股份有限公司 | A kind of MPI based on embedded OS realizes system and method |
| CN108228330A (en) * | 2018-02-06 | 2018-06-29 | 北京安博通科技股份有限公司 | The multi-process method for scheduling task and device of a kind of serialization |
| CN111679859A (en) * | 2020-06-11 | 2020-09-18 | 山东省计算中心（国家超级计算济南中心） | I/O intensive high-performance application-oriented automatic parallel MPI-I/O acceleration method |
| CN112818336A (en) * | 2020-12-28 | 2021-05-18 | 深圳市金微蓝技术有限公司 | Data access method, data access device and computer readable storage medium |
| CN114003342A (en) * | 2021-11-01 | 2022-02-01 | 北京天融信网络安全技术有限公司 | Distributed storage method and device, electronic equipment and storage medium |
| CN115033529A (en) * | 2022-08-11 | 2022-09-09 | 北京卡普拉科技有限公司 | Asynchronous I/O adaptation method, system, storage medium and electronic equipment |
-
2022
- 2022-11-29 CN CN202211503147.3A patent/CN115599563B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130166886A1 (en) * | 2008-11-24 | 2013-06-27 | Ruchira Sasanka | Systems, apparatuses, and methods for a hardware and software system to automatically decompose a program to multiple parallel threads |
| US20130054808A1 (en) * | 2011-08-31 | 2013-02-28 | International Business Machines Corporation | Management of storage cluster performance with hybrid workloads |
| CN107391422A (en) * | 2017-07-19 | 2017-11-24 | 上海航天测控通信研究所 | multi-path asynchronous serial communication data access system and method |
| CN107748700A (en) * | 2017-10-09 | 2018-03-02 | 北京东土科技股份有限公司 | A kind of MPI based on embedded OS realizes system and method |
| CN108228330A (en) * | 2018-02-06 | 2018-06-29 | 北京安博通科技股份有限公司 | The multi-process method for scheduling task and device of a kind of serialization |
| CN111679859A (en) * | 2020-06-11 | 2020-09-18 | 山东省计算中心（国家超级计算济南中心） | I/O intensive high-performance application-oriented automatic parallel MPI-I/O acceleration method |
| CN112818336A (en) * | 2020-12-28 | 2021-05-18 | 深圳市金微蓝技术有限公司 | Data access method, data access device and computer readable storage medium |
| CN114003342A (en) * | 2021-11-01 | 2022-02-01 | 北京天融信网络安全技术有限公司 | Distributed storage method and device, electronic equipment and storage medium |
| CN115033529A (en) * | 2022-08-11 | 2022-09-09 | 北京卡普拉科技有限公司 | Asynchronous I/O adaptation method, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115599563B (en) | 2023-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108647104B (en) | Request processing method, server and computer readable storage medium | |
| CN103942178A (en) | Communication method between real-time operating system and non-real-time operating system on multi-core processor | |
| CN109358956B (en) | Service calling method | |
| US20110107344A1 (en) | Multi-core apparatus and load balancing method thereof | |
| US11822958B2 (en) | Method and a device for data transmission between an internal memory of a system-on-chip and an external memory | |
| CN108762905B (en) | Method and device for processing multitask events | |
| CN115361451B (en) | Network communication parallel processing method and system | |
| CN115061803A (en) | Multi-core processing system and task scheduling method, chip and storage medium thereof | |
| JPH0594317A (en) | Input output interrupt processing system of virtual machine | |
| CN105373563B (en) | Database switching method and device | |
| CN112068960A (en) | CPU resource allocation method, device, storage medium and equipment | |
| CN114816709A (en) | Task scheduling method, device, server and readable storage medium | |
| CN110955461B (en) | Processing method, device, system, server and storage medium for computing task | |
| CN116089049B (en) | Asynchronous parallel I/O request-based process synchronous scheduling method, device and equipment | |
| CN115599563B (en) | Parallel I/O control method, device and equipment for serial I/O access | |
| CN111381946B (en) | Task processing method and device and related products | |
| CN115878333A (en) | Method, device and equipment for judging consistency between process groups | |
| CN111443898A (en) | Method for designing flow program control software based on priority queue and finite-state machine | |
| CN110780999A (en) | System and method for scheduling multi-core CPU | |
| CN112181641A (en) | Thread processing method, device, equipment and storage medium | |
| US20230325512A1 (en) | Graphics processing unit invoking method, central processing unit and apparatus | |
| CN110908952B (en) | Data processing method and device, CPU and computer | |
| CN117891583A (en) | Process scheduling method, device and equipment for asynchronous parallel I/O request | |
| CN117389625B (en) | Process synchronization method, system, equipment and medium based on active interrupt instruction | |
| TWI823655B (en) | Task processing system and task processing method applicable to intelligent processing unit |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |