CN115599563B - Parallel I/O control method, device and equipment for serial I/O access - Google Patents
Parallel I/O control method, device and equipment for serial I/O access Download PDFInfo
- Publication number
- CN115599563B CN115599563B CN202211503147.3A CN202211503147A CN115599563B CN 115599563 B CN115599563 B CN 115599563B CN 202211503147 A CN202211503147 A CN 202211503147A CN 115599563 B CN115599563 B CN 115599563B
- Authority
- CN
- China
- Prior art keywords
- request
- parallel
- service
- special
- serial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The embodiment of the invention provides a parallel I/O control method, a device and equipment for serial I/O access, wherein the method comprises the following steps: responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; and running the parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of I/O request processing, in particular to a parallel I/O control method, device and equipment for serial I/O access.
Background
In the running process of the application program, data is required to be input from a storage medium based on a file reading mode, and a calculation result is output to the storage medium in a file writing mode. Data Input/Output (I/O) is the slowest link in the computing process of an application program, and becomes a bottleneck for limiting the performance of the application program. With the increasing demands of applications, the amount of computation of applications and the amount of data to be processed are increasing. With the increasing processing power of computers, modern high-performance computers not only provide numerous processor cores that can be parallel for acceleration of computing, but also provide parallel storage functions including hardware and parallel file systems for acceleration of I/O.
Parallel I/O technology has emerged to allow applications to fully utilize parallel memory functions to minimize the impact of I/O overhead on the application's operating speed. Parallel I/O technology is implemented directly or indirectly using I/O messaging interfaces (Message Passing Interface-I/O, MPI-IO). Therefore, a basic premise of accelerating I/O by adopting parallel I/O technology is that the application program is a multi-process parallel program realized based on MPI parallelization. However, there are still a lot of serial programs with high I/O overhead or application programs adopting serial file format, and because the parallel I/O technology cannot be adopted, data can only be input and output in a serial manner, and the parallel storage function cannot be utilized to reduce the I/O overhead.
In view of the foregoing, there is a need for a method to reduce the I/O overhead of a serial program or an application program in a serial file format to improve program performance.
Disclosure of Invention
The embodiment of the invention provides a parallel I/O control method, device and equipment for serial I/O access, which are used for solving the problem of high I/O overhead of a serial program or an application program adopting a serial file format in the existing method.
In a first aspect, an embodiment of the present invention provides a parallel I/O control method for serial I/O access, including:
responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format;
when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes;
and running the parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
In one embodiment, running a parallel I/O system on an I/O specific process and executing an I/O request corresponding to an I/O service command using the parallel I/O system, comprises:
and running an asynchronous parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system.
In one embodiment, determining an I/O specific process includes:
acquiring the number of I/O processes of an I/O special process, and setting the number of I/O processes as a preset I/O process number upper limit threshold when the number of I/O processes is larger than the preset I/O process number upper limit threshold;
judging whether a process corresponding to the number of the I/O processes is started or not;
if the process is started, multiplexing the process corresponding to the number of the I/O processes as an I/O special process;
if not, adopting the message passing interface operation command to start the I/O process to be used as the I/O special process.
In one embodiment, executing an I/O request corresponding to an I/O service command using a parallel I/O system includes:
establishing a special communication channel between a process of an application program and an I/O special process;
transmitting information of the I/O request from the process of the application program to the I/O dedicated process by using the dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing a parallel I/O function of a parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
the processing result of the I/O request is sent from the I/O-specific process to the process of the application program using the specific communication channel.
In one embodiment, the dedicated communication channel is implemented based on an inter-process anonymous pipe, named pipe, message queue, or virtual shared memory.
In one embodiment, when the I/O service is an I/O specific process service, the method further comprises: after all I/O requests are processed, the I/O dedicated process is controlled to enter a sleep state in which processor resources are not occupied.
In one embodiment, the method further comprises:
when the I/O service mode is I/O special thread service, the process of the application program starts the I/O special thread;
transmitting information of the I/O request from a process of the application program to the I/O special thread in a memory sharing mode;
executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
In one embodiment, an I/O request includes: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O completion.
In a second aspect, an embodiment of the present invention provides a parallel I/O control apparatus for serial I/O access, including:
the response module is used for responding to an I/O service command initiated by an application program in a serial I/O access mode, acquiring the I/O service mode from the I/O service command, and enabling the application program to be a serial program or adopting a serial file format;
the processing module is used for determining an I/O special process when the I/O service mode is I/O special process service, wherein the I/O special process comprises a plurality of processes;
and the execution module is used for running the parallel I/O system on the I/O special process and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
at least one processor and memory;
the memory stores computer-executable instructions;
at least one processor executes computer-executable instructions stored in a memory, causing the at least one processor to perform the parallel I/O control method of serial I/O access as claimed in any one of the first aspects.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having stored therein computer-executable instructions, which when executed by a processor, are configured to implement a parallel I/O control method for serial I/O access as in any one of the first aspects.
The parallel I/O control method, the parallel I/O control device and the parallel I/O control equipment for the serial I/O access, provided by the embodiment of the invention, respond to an I/O service command initiated by an application program in a serial I/O access mode, acquire the I/O service mode from the I/O service command, and the application program is a serial program or adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; and running the parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow chart of a parallel I/O control method for serial I/O access according to an embodiment of the present invention;
FIG. 2 is a flow chart of a parallel I/O control method for serial I/O access provided by yet another embodiment of the invention;
FIG. 3 is a flow chart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention;
FIG. 4 is a flow chart of a parallel I/O control method for serial I/O access provided by yet another embodiment of the invention;
FIG. 5 is a schematic diagram of a parallel I/O control device with serial I/O access according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Specific embodiments of the present invention have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
The invention will be described in further detail below with reference to the drawings by means of specific embodiments. Wherein like elements in different embodiments are numbered alike in association. In the following embodiments, numerous specific details are set forth in order to provide a better understanding of the present application. However, one skilled in the art will readily recognize that some of the features may be omitted, or replaced by other elements, materials, or methods in different situations. In some instances, some operations associated with the present application have not been shown or described in the specification to avoid obscuring the core portions of the present application, and may not be necessary for a person skilled in the art to describe in detail the relevant operations based on the description herein and the general knowledge of one skilled in the art.
Furthermore, the described features, operations, or characteristics of the description may be combined in any suitable manner in various embodiments. Also, various steps or acts in the method descriptions may be interchanged or modified in a manner apparent to those of ordinary skill in the art. Thus, the various orders in the description and drawings are for clarity of description of only certain embodiments, and are not meant to be required orders unless otherwise indicated.
The numbering of the components itself, e.g. "first", "second", etc., is used herein merely to distinguish between the described objects and does not have any sequential or technical meaning. The terms "coupled" and "connected," as used herein, are intended to encompass both direct and indirect coupling (coupling), unless otherwise indicated.
FIG. 1 is a flow chart of a parallel I/O control method for serial I/O access according to an embodiment of the present invention. As shown in fig. 1, the parallel I/O control method for serial I/O access provided in this embodiment may include:
s101, responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring the I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format.
The serial program or the application program adopting the serial file format cannot use the parallel I/O technology, and can only initiate the I/O service command in a serial I/O access mode.
I/O service commands may include information on I/O service patterns and I/O requests. The I/O service manner is used to indicate how to complete the I/O request, and may include, for example, I/O dedicated thread service and I/O dedicated process service. The I/O dedicated process service is to indicate completion of the I/O request by a set of dedicated processes and the I/O dedicated thread service is to indicate completion of the I/O request by a set of dedicated threads. The information of the I/O request is used to indicate I/O operations that need to be performed.
S102, when the I/O service mode is the I/O special process service mode, determining the I/O special process, wherein the I/O special process comprises a plurality of processes.
When the I/O service acquired from the I/O service command is an I/O-specific process service, it is determined that a plurality of processes are dedicated to performing I/O operations. A plurality of processes dedicated to performing I/O operations are determined as I/O-specific processes.
S103, running a parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system.
I/O requests corresponding to I/O service commands are performed by a parallel I/O system running on an I/O specific process. The parallel I/O technology can store the data of the application program into a plurality of processes in a scattered way, and the speed of I/O is improved by cooperatively reading and writing file data among the processes. Supporting multiple types of I/O requests in this embodiment may include, for example: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O completion.
In the parallel I/O control method for serial I/O access provided in this embodiment, an I/O service command initiated by an application program in a serial I/O access manner is responded, and an I/O service manner is obtained from the I/O service command, where the application program is a serial program or the application program adopts a serial file format; when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes; and running the parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the parallel I/O system. The serial program or the application program adopting the serial file format can input and output data in a parallel mode, so that the I/O overhead is reduced, and the performance of the application program is improved.
On the basis of the above embodiment, in order to further reduce the I/O overhead and increase the running speed of the application program, in the method provided by the embodiment, an I/O request corresponding to the I/O service command is executed on the I/O dedicated process by using an asynchronous parallel I/O system. The parallel I/O technology dispersedly stores the data of the application program to a plurality of processes, and the processes improve the I/O speed by cooperatively reading and writing the file data. Asynchronous I/O techniques employ I/O-specific processes/threads other than computing processes/threads to complete data I/O operations, so that an application can continue to perform other computations without waiting for completion of the I/O operations, thereby overlapping the computation and I/O processing of the application. In the embodiment, an asynchronous parallel I/O system which takes both parallel I/O and asynchronous I/O into consideration is adopted, so that the influence of I/O overhead on the performance of a serial program or an application program adopting a serial file format can be reduced to the greatest extent. That is, in an alternative embodiment, running a parallel I/O system on an I/O dedicated process and executing an I/O request corresponding to an I/O service command using the parallel I/O system may include: and running an asynchronous parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system.
How the I/O specific process is determined is described in detail below. Referring to fig. 2, fig. 2 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 2, in the parallel I/O control method for serial I/O access provided in this embodiment, based on the embodiment shown in fig. 1, determining an I/O dedicated process may include:
s1031, acquiring the number of I/O processes of the I/O special process, and setting the number of I/O processes as the preset upper limit threshold of the number of I/O processes when the number of I/O processes is larger than the preset upper limit threshold of the number of I/O processes.
Typically, I/O-specific processes need to run on the computer where the current process (i.e., the process of the application program, also the process of serial I/O access) is located, and the number of cores on the current computer is limited, so the number of I/O processes cannot exceed the resource limit of the current computer. The number of I/O processes actually used by the I/O dedicated process may be limited by presetting an upper threshold for the number of I/O processes so as not to exceed the resource limit. Specifically, when the I/O process number is greater than a preset I/O process number upper threshold, the I/O process number is set to the preset I/O process number upper threshold. In an alternative embodiment, the number of I/O processes for the I/O specific process may be preset by the user. It will be appreciated that increasing the number of I/O processes may result in a situation where the I/O speed does not increase but decreases, i.e., there is a limit to the scalability of the performance of parallel I/O with increasing the number of I/O processes, which also limits the upper limit of the number of I/O processes. In yet another alternative embodiment, the number of I/O processes that maximizes the I/O speed under the above-described constraints may also be obtained based on a machine learning approach as the number of I/O processes for I/O-specific processes. In another alternative embodiment, the optimal number of I/O processes may also be found by debugging the number of I/O processes and running the program multiple times.
S1032, judging whether the process corresponding to the number of I/O processes is started. If so, executing step S1033; if not, step S1034 is performed.
S1033, multiplexing the process corresponding to the I/O process number as an I/O special process.
S1034, starting a plurality of processes of the I/O process by adopting a message passing interface operation command as an I/O special process.
After determining the number of I/O processes of the I/O dedicated process, in order to avoid wasting processor resources caused by repeatedly starting the process, it is first required to determine whether a group of dedicated processes corresponding to the number of I/O processes has been started. If so, multiplexing the group of special processes without restarting; if not, a set of special processes are started according to the number of I/O processes, and a parallel I/O system or an asynchronous parallel I/O system is started on the set of processes. Typically, the above-described boot process involves booting a set of dedicated processes running a parallel I/O system or an asynchronous parallel I/O system with MPI run commands.
According to the parallel I/O control method for serial I/O access, on the basis of the embodiment, whether the process corresponding to the number of I/O processes is started or not is judged, and when the process is started, the process corresponding to the number of I/O processes is multiplexed to be used as an I/O special process; when not started, the message passing interface running command is adopted to start a plurality of processes of the I/O process as I/O special processes. The method can ensure enough processes to be used for running the parallel I/O system, and avoid the waste of processor resources caused by repeatedly starting the processes.
In an alternative implementation manner, in order to further improve the utilization efficiency of the processor resource, when the I/O service mode is to serve the I/O dedicated process, after all the I/O requests are processed, the I/O dedicated process may be controlled to enter a sleep state that does not occupy the processor resource, so as to avoid the additional overhead to the greatest extent. When responding to the parallel I/O system or the asynchronous parallel I/O system to execute the service of the I/O request, if the I/O special process is in a dormant state, only the I/O special process is required to be awakened, and the I/O special process is determined to be in a working state.
When the parallel I/O system is used for executing the I/O request corresponding to the I/O service command, the application program needs to send the information of the I/O request to the parallel I/O system, and the parallel I/O system uses the parallel I/O function to execute the corresponding I/O operation and needs to return the result to the application program. However, since the application program and the parallel I/O system run on a process group started by different commands, an MPI communication domain containing both the application program process and the I/O dedicated process cannot be established. In the following, a detailed description will be given of how to execute an I/O request corresponding to an I/O service command using a parallel I/O system by means of a specific embodiment. Referring to fig. 3, fig. 3 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 3, in the parallel I/O control method for serial I/O access according to any one of the embodiments, the executing, by using the parallel I/O system, the I/O request corresponding to the I/O service command may include:
s1041, establishing a special communication channel between the process of the application program and the I/O special process.
Specific implementation methods of the dedicated communication channel in this embodiment include, but are not limited to, an inter-process-based anonymous pipe, a named pipe, a message queue, a virtual shared memory, and the like.
S1042, the information of the I/O request is sent from the process of the application program to the I/O special process by using the special communication channel.
After the dedicated communication channel is established, the information of the I/O request may be sent from the process of the application to the I/O dedicated process using the dedicated communication channel.
S1043, executing corresponding operation corresponding to the I/O request by utilizing the parallel I/O function of the parallel I/O system running on the I/O special process to obtain the processing result of the I/O request.
S1044, sending the processing result of the I/O request from the I/O dedicated process to the process of the application program by using the dedicated communication channel.
The present embodiment is described with respect to the case of using the parallel I/O system, but is equally applicable to the case of using the asynchronous parallel I/O system. Only the parallel I/O function, the asynchronous I/O function or the asynchronous parallel I/O function of the asynchronous parallel I/O system running on the I/O special process is needed to execute the corresponding operation corresponding to the I/O request, so as to obtain the processing result of the I/O request. Specific implementation manner of executing the I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system may refer to the above steps, and will not be repeated herein.
The parallel I/O control method for serial I/O access provided in this embodiment further establishes a dedicated communication channel between a process of an application program and an I/O dedicated process, and completes information transfer between the process of the application program and the I/O dedicated process by using the dedicated communication channel, so that the parallel I/O system running on the I/O dedicated process completes an I/O request corresponding to an I/O service command.
The above embodiments are all described with respect to the case where the I/O service mode is the I/O dedicated process service, and the following description will be made with respect to the case where the I/O service mode is the I/O dedicated thread service. Referring to fig. 4, fig. 4 is a flowchart of a parallel I/O control method for serial I/O access according to another embodiment of the present invention. As shown in fig. 4, on the basis of any one of the above embodiments, the parallel I/O control method for serial I/O access provided in this embodiment may further include:
s104, when the I/O service mode is I/O special thread service, the process of the application program starts the I/O special thread.
S105, information of the I/O request is sent to the I/O special thread from the process of the application program in a mode of sharing the memory.
S106, executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request.
S107, sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
According to the parallel I/O control method for serial I/O access, the I/O special thread is started, information is transferred between the I/O special thread and the application program in a shared memory mode, and corresponding operation corresponding to the I/O request is executed by utilizing the asynchronous I/O function of the I/O special thread, so that the serial program or the application program adopting a serial file format can input and output data in an asynchronous mode, I/O expenditure is reduced, and the performance of the application program is improved.
How to use the method provided by the present application is described below by two specific application scenarios.
Application scenario 1: single-process applications or parallel programs where only one process performs I/O processing. The object is: asynchronous parallel I/O acceleration is implemented for applications. The operation steps are as follows: the asynchronous parallel I/O service command initiating program interface provided by the method replaces the original serial I/O access in the application program, and sets the I/O service mode and I/O request in each service command. When an I/O request is a service command for asynchronous data input/output, a corresponding service command for I/O request waiting for asynchronous I/O completion needs to be inserted in the application program. And running the improved application program to confirm that the asynchronous parallel I/O service command is used correctly and the I/O acceleration effect. When the I/O special process service mode is adopted, the optimal preset special I/O process number can be found by debugging the preset special I/O process number and running the program for a plurality of times.
Application scenario 2: application file formats for serial I/O access, such as serial NetCDF library. NetCDF is an application file format commonly used in the fields of numerical weather forecast and climate simulation prediction. The object is: the I/O efficiency of the serial I/O access mode of the application program in the NetCDF library is improved. The operation steps are as follows: in an application program, the configuration program interface of the asynchronous parallel I/O service command provided by the method is utilized to set an I/O service mode and an I/O request mode, wherein the mode comprises synchronous input/output and asynchronous input/output. Finding out APIs (application program interfaces) related to data input and output in a NetCDF library, modifying the realization codes of the APIs, and acquiring information of corresponding I/O requests from the output information of the APIs; and then acquiring an I/O service mode and an I/O request mode from the configuration information of the method, initiating an asynchronous parallel I/O service command by combining the information of the I/O request, and stopping the original data input/output operation in the API. And running the improved application program and the NetCDF library to confirm that the asynchronous parallel I/O service command is used correctly and the I/O acceleration effect. When the I/O special process service mode is adopted, the optimal preset special I/O process number can be found by debugging the preset special I/O process number and running the program for a plurality of times.
FIG. 5 is a schematic diagram of a parallel I/O control device with serial I/O access according to an embodiment of the present invention. As shown in fig. 5, the parallel I/O control device 50 for serial I/O access provided in this embodiment may include: a response module 501, a processing module 502 and an execution module 503.
A response module 501, configured to respond to an I/O service command initiated by an application program in a serial I/O access manner, and obtain an I/O service manner from the I/O service command, where the application program is a serial program or the application program adopts a serial file format;
a processing module 502, configured to determine an I/O specific process when the I/O service is an I/O specific process service, where the I/O specific process includes a plurality of processes;
an execution module 503, configured to run a parallel I/O system on an I/O specific process, and execute an I/O request corresponding to the I/O service command using the parallel I/O system.
The device of this embodiment may be used to implement the technical solution of the method embodiment shown in fig. 1, and its implementation principle and technical effects are similar, and are not described here again.
In an alternative embodiment, the executing module 503 is configured to run a parallel I/O system on an I/O specific process, and execute, using the parallel I/O system, an I/O request corresponding to an I/O service command may specifically include: and running an asynchronous parallel I/O system on the I/O special process, and executing the I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system.
In an alternative embodiment, the processing module 502 configured to determine the I/O specific process may specifically include:
acquiring the number of I/O processes of the I/O special process, and setting the number of I/O processes as a preset upper limit threshold of the number of I/O processes when the number of I/O processes is larger than the preset upper limit threshold of the number of I/O processes;
judging whether a process corresponding to the number of the I/O processes is started or not;
if the process is started, multiplexing the process corresponding to the number of the I/O processes as an I/O special process;
if not, adopting the message passing interface operation command to start the I/O process to be used as the I/O special process.
In an alternative embodiment, the executing module 503, configured to execute, using the parallel I/O system, an I/O request corresponding to an I/O service command may specifically include:
establishing a special communication channel between a process of an application program and an I/O special process;
transmitting information of the I/O request from the process of the application program to the I/O dedicated process by using the dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing a parallel I/O function of a parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
the processing result of the I/O request is sent from the I/O-specific process to the process of the application program using the specific communication channel.
In an alternative embodiment, the dedicated communication channel is implemented based on an inter-process anonymous pipe, named pipe, message queue, or virtual shared memory.
In an alternative embodiment, when the I/O service is an I/O dedicated process, the execution module 503 is further configured to control the I/O dedicated process to enter a sleep state that does not occupy the processor resource after all the I/O requests are processed.
In an alternative embodiment, the executing module 503 is further configured to, when the I/O service mode is I/O dedicated thread service, enable the process of the application program to start the I/O dedicated thread; transmitting information of the I/O request from a process of the application program to the I/O special thread in a memory sharing mode; executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request; and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
In an alternative embodiment, the I/O request may include: synchronous data input requests, synchronous data output requests, asynchronous data input requests, asynchronous data output requests, and requests waiting for asynchronous I/O completion.
The embodiment of the present invention further provides an electronic device, and please refer to fig. 6, in which the embodiment of the present invention is illustrated by way of example only in fig. 6, and the present invention is not limited thereto. Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the invention. As shown in fig. 6, the electronic device 60 provided in this embodiment includes: a memory 601, a processor 602, and a bus 603. Wherein a bus 603 is used to enable connections between the elements.
The memory 601 stores a computer program which, when executed by the processor 602, implements the technical solutions of any of the method embodiments described above.
Wherein the memory 601 and the processor 602 are electrically connected directly or indirectly to enable transmission or interaction of data. For example, the elements may be electrically coupled to each other via one or more communication buses or signal lines, such as bus 603. The memory 601 stores therein a computer program for implementing a parallel I/O control method of serial I/O access, including at least one software functional module which may be stored in the memory 601 in the form of software or firmware, and the processor 602 executes various functional applications and data processing by running the software program and the module stored in the memory 601.
The Memory 601 may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc. The memory 601 is used for storing a program, and the processor 602 executes the program after receiving an execution instruction. Further, the software programs and modules within the memory 601 may also include an operating system, which may include various software components and/or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.
The processor 602 may be an integrated circuit chip with signal processing capabilities. The processor 602 may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), and the like. The disclosed methods, steps, and logic blocks in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. It will be appreciated that the configuration of fig. 6 is illustrative only and may include more or fewer components than shown in fig. 6 or have a different configuration than shown in fig. 6. The components shown in fig. 6 may be implemented in hardware and/or software.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the technical solution of any of the method embodiments described above.
The various embodiments in this disclosure are described in a progressive manner, and identical and similar parts of the various embodiments are all referred to each other, and each embodiment is mainly described as different from other embodiments.
The scope of the present disclosure is not limited to the above-described embodiments, and it is apparent that various modifications and variations can be made to the present disclosure by those skilled in the art without departing from the scope and spirit of the disclosure. Such modifications and variations are intended to be included herein within the scope of the following claims and their equivalents.
Claims (9)
1. A parallel I/O control method for serial I/O access, comprising:
responding to an I/O service command initiated by an application program in a serial I/O access mode, and acquiring an I/O service mode from the I/O service command, wherein the application program is a serial program or adopts a serial file format, and the I/O service mode comprises I/O special process service and I/O special thread service;
when the I/O service mode is I/O special process service, determining an I/O special process, wherein the I/O special process comprises a plurality of processes started by a message passing interface operation command;
running a parallel I/O system on the I/O special process, and executing an I/O request corresponding to the I/O service command by using the parallel I/O system, wherein the I/O request comprises: synchronous data input request, synchronous data output request, asynchronous data input request, asynchronous data output request and request waiting for asynchronous I/O completion;
the executing the I/O request corresponding to the I/O service command using the parallel I/O system includes:
establishing a special communication channel between the process of the application program and the I/O special process;
transmitting information of the I/O request from the process of the application program to the I/O dedicated process by using the dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing a parallel I/O function of a parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special process to the process of the application program by utilizing the special communication channel.
2. The method of claim 1, wherein the running a parallel I/O system on the I/O-specific process and executing the I/O request corresponding to the I/O service command using the parallel I/O system comprises:
and operating an asynchronous parallel I/O system on the I/O special process, and executing an I/O request corresponding to the I/O service command by using the asynchronous parallel I/O system.
3. The method of claim 1, wherein the determining an I/O specific process comprises:
acquiring the number of I/O processes of the I/O special process, and setting the number of I/O processes as a preset upper limit threshold of the number of I/O processes when the number of I/O processes is larger than the preset upper limit threshold of the number of I/O processes;
judging whether the process corresponding to the I/O process number is started or not;
if the process is started, multiplexing the process corresponding to the I/O process number as an I/O special process;
if not, adopting the message passing interface operation command to start a plurality of processes of the I/O process as I/O special processes.
4. The method of claim 1, wherein the dedicated communication channel is implemented based on an inter-process anonymous pipe, a named pipe, a message queue, or a virtual shared memory.
5. The method of any of claims 1-4, wherein when the I/O service is an I/O dedicated process service, the method further comprises: after all I/O requests are processed, the I/O special process is controlled to enter a dormant state without occupying processor resources.
6. The method according to claim 1, wherein the method further comprises:
when the I/O service mode is I/O special thread service, the process of the application program starts the I/O special thread;
transmitting the information of the I/O request from the process of the application program to the I/O special thread in a memory sharing mode;
executing corresponding operation corresponding to the I/O request by utilizing the asynchronous I/O function of the I/O special thread to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special thread to the process of the application program in a memory sharing mode.
7. A parallel I/O control apparatus for serial I/O access, comprising:
the system comprises a response module, a processing module and a processing module, wherein the response module is used for responding to an I/O service command initiated by an application program in a serial I/O access mode and acquiring an I/O service mode from the I/O service command, the application program is a serial program or adopts a serial file format, and the I/O service mode comprises I/O special process service and I/O special thread service;
the processing module is used for determining an I/O special process when the I/O service mode is I/O special process service, wherein the I/O special process comprises a plurality of processes started by a message passing interface operation command;
an execution module, configured to run a parallel I/O system on the I/O dedicated process, and execute, using the parallel I/O system, an I/O request corresponding to the I/O service command, where the I/O request includes: synchronous data input request, synchronous data output request, asynchronous data input request, asynchronous data output request and request waiting for asynchronous I/O completion;
the executing module is configured to execute, by using the parallel I/O system, an I/O request corresponding to the I/O service command, where the executing module specifically includes:
establishing a special communication channel between the process of the application program and the I/O special process;
transmitting information of the I/O request from the process of the application program to the I/O dedicated process by using the dedicated communication channel;
executing corresponding operation corresponding to the I/O request by utilizing a parallel I/O function of a parallel I/O system running on the I/O special process to obtain a processing result of the I/O request;
and sending the processing result of the I/O request from the I/O special process to the process of the application program by utilizing the special communication channel.
8. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing computer-executable instructions stored in the memory, causing the at least one processor to perform the parallel I/O control method of serial I/O access as recited in any of claims 1-6.
9. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to implement a parallel I/O control method of serial I/O access as claimed in any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211503147.3A CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211503147.3A CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115599563A CN115599563A (en) | 2023-01-13 |
| CN115599563B true CN115599563B (en) | 2023-06-13 |
Family
ID=84852325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211503147.3A Active CN115599563B (en) | 2022-11-29 | 2022-11-29 | Parallel I/O control method, device and equipment for serial I/O access |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115599563B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107748700A (en) * | 2017-10-09 | 2018-03-02 | 北京东土科技股份有限公司 | A kind of MPI based on embedded OS realizes system and method |
| CN112818336A (en) * | 2020-12-28 | 2021-05-18 | 深圳市金微蓝技术有限公司 | Data access method, data access device and computer readable storage medium |
| CN114003342A (en) * | 2021-11-01 | 2022-02-01 | 北京天融信网络安全技术有限公司 | Distributed storage method and device, electronic equipment and storage medium |
| CN115033529A (en) * | 2022-08-11 | 2022-09-09 | 北京卡普拉科技有限公司 | Asynchronous I/O adaptation method, system, storage medium and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9189233B2 (en) * | 2008-11-24 | 2015-11-17 | Intel Corporation | Systems, apparatuses, and methods for a hardware and software system to automatically decompose a program to multiple parallel threads |
| US9658892B2 (en) * | 2011-08-31 | 2017-05-23 | International Business Machines Corporation | Management of storage cluster performance with hybrid workloads |
| CN107391422A (en) * | 2017-07-19 | 2017-11-24 | 上海航天测控通信研究所 | multi-path asynchronous serial communication data access system and method |
| CN108228330B (en) * | 2018-02-06 | 2022-03-04 | 北京安博通科技股份有限公司 | Serialized multiprocess task scheduling method and device |
| CN111679859B (en) * | 2020-06-11 | 2023-08-18 | 山东省计算中心（国家超级计算济南中心） | Automatic parallel MPI-I/O acceleration method for I/O intensive high-performance application |
-
2022
- 2022-11-29 CN CN202211503147.3A patent/CN115599563B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107748700A (en) * | 2017-10-09 | 2018-03-02 | 北京东土科技股份有限公司 | A kind of MPI based on embedded OS realizes system and method |
| CN112818336A (en) * | 2020-12-28 | 2021-05-18 | 深圳市金微蓝技术有限公司 | Data access method, data access device and computer readable storage medium |
| CN114003342A (en) * | 2021-11-01 | 2022-02-01 | 北京天融信网络安全技术有限公司 | Distributed storage method and device, electronic equipment and storage medium |
| CN115033529A (en) * | 2022-08-11 | 2022-09-09 | 北京卡普拉科技有限公司 | Asynchronous I/O adaptation method, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115599563A (en) | 2023-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108647104B (en) | Request processing method, server and computer readable storage medium | |
| EP3092567B1 (en) | System and method for isolating i/o execution via compiler and os support | |
| EP2908252A1 (en) | Inter-core communication apparatus and method | |
| US8972702B2 (en) | Systems and methods for power management in a high performance computing (HPC) cluster | |
| CN103942178A (en) | Communication method between real-time operating system and non-real-time operating system on multi-core processor | |
| CN102693162A (en) | Method for process communication among multiple virtual machines on multi-core platform based on shared memory and intercore interruption | |
| CN109358956B (en) | Service calling method | |
| CN115203142A (en) | Multi-core real-time communication system and method | |
| US10459771B2 (en) | Lightweight thread synchronization using shared memory state | |
| CN115361451B (en) | Network communication parallel processing method and system | |
| CN115061803A (en) | Multi-core processing system and task scheduling method, chip and storage medium thereof | |
| CN114153783B (en) | Method, system, computer device and storage medium for implementing multi-core communication mechanism | |
| JP2017162522A (en) | Method and apparatus for allocating interrupts in multi-core system | |
| CN114697194A (en) | Blocking type event notification method and device | |
| CN115599563B (en) | Parallel I/O control method, device and equipment for serial I/O access | |
| CN110362314B (en) | Information processing method and device, computer readable medium and electronic equipment | |
| CN109614242B (en) | Computing capacity sharing method, device, equipment and medium | |
| CN115878333A (en) | Method, device and equipment for judging consistency between process groups | |
| CN113391821B (en) | Asymmetric multiprocessor embedded operating system | |
| US20130247065A1 (en) | Apparatus and method for executing multi-operating systems | |
| CN117891583B (en) | Process scheduling method, device and equipment for asynchronous parallel I/O request | |
| CN112181641A (en) | Thread processing method, device, equipment and storage medium | |
| CN111459620A (en) | Information scheduling method from security container operating system to virtual machine monitor | |
| US10152341B2 (en) | Hyper-threading based host-guest communication | |
| US20230325512A1 (en) | Graphics processing unit invoking method, central processing unit and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |