CN112463360A - Parallel read-in method for billion-hundred-GB-level grid data file - Google Patents
Parallel read-in method for billion-hundred-GB-level grid data file Download PDFInfo
- Publication number
- CN112463360A CN112463360A CN202011183930.7A CN202011183930A CN112463360A CN 112463360 A CN112463360 A CN 112463360A CN 202011183930 A CN202011183930 A CN 202011183930A CN 112463360 A CN112463360 A CN 112463360A
- Authority
- CN
- China
- Prior art keywords
- file
- files
- processes
- data
- grid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 157
- 238000005192 partition Methods 0.000 claims abstract description 37
- 238000004891 communication Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 5
- 238000004088 simulation Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 2
- 238000002156 mixing Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000000853 adhesive Substances 0.000 description 1
- 230000001070 adhesive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 238000012821 model calculation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Abstract
The invention provides a parallel reading-in method of a billion-hundred GB magnitude grid data file, which is characterized in that grouping files are adopted to store ultra-large scale grid data files generated by a plurality of objects, each grouping comprises a plurality of files, and each file comprises a plurality of data partitions; and during reading, reading the file by adopting a plurality of file processes, sending the file to the corresponding non-file processes, and carrying out data load balancing and allocation. By adopting the method provided by the invention, the grid file IO efficiency can be greatly improved, and when the same super-large-scale grid data is read, the time consumed by the technical scheme of the invention is only 1/50-1/10 which is consumed by the prior art. The invention can greatly save the grid data reading time and improve the calculation efficiency and the economic benefit.
Description
Technical Field
The invention relates to the field of data processing, in particular to a parallel reading method of a billion-hundred-GB-level grid data file.
Background
The invention relates to the rapid development of computer technology and numerical method, and the Computational Fluid Dynamics (CFD) numerical simulation is widely applied in the fields of aerospace and the like.After decades of development, aerodynamic force/moment prediction in a conventional state based on a reynolds average NS (reynolds average-mean-stocks, RANS) equation has not been difficult, but when swirl, separation, transition, turbulent noise, turbulent combustion and other abnormal and nonlinear phenomena flow obviously, solving the RANS equation on a million-level grid cannot obtain a sufficiently accurate numerical solution, and then a larger-scale grid and a numerical method with higher fidelity, such as a large vortex simulation (LES) and a Direct Numerical Simulation (DNS), need to be adopted. The common feature of these methods is that the requirement for the amount of lattice is high, and the LES method is generally considered to meet the requirement for the amount of lattice in the adhesive bottom layer to Re1.8Magnitude, while DNS requires a grid size up to Re9/4. For practical aircraft profiles, the Reynolds number Re is typically 106Above the magnitude, the grid quantity at least reaches over one billion magnitude, and the resolution requirement of the algorithm on the multi-scale flow structure can be met.
When the grid scale is in the order of tens of millions, all grid and flow field data are stored in a single file, the file size is in the order of 1GB, at the moment, file storage and serial reading and writing do not cause too much pressure on a computer file system, and the storage and reading and writing efficiency is still at an acceptable level. However, when the grid quantity reaches the billion magnitude, the grid file and the flow field file reach several hundreds of GB magnitude, and if data is still stored in a single file and serial read-write is performed by a single process, the file read-write speed will inevitably drop sharply, resulting in unacceptable CFD computational efficiency.
Therefore, aiming at the actual requirements of high-efficiency storage and IO of the billion-level grid, a new file storage and parallel IO method needs to be developed, and a foundation is laid for high-resolution numerical simulation of the ultra-large-scale grid of the aircraft in the future.
Disclosure of Invention
Aiming at the problems in the prior art, a method for reading in billions and hundreds of GB magnitude grid data files in parallel is provided.
The technical scheme adopted by the invention is as follows: a parallel reading-in method of billion-hundred GB magnitude grid data files is used for storing ultra-large scale grid data files generated by a plurality of objects in groups, wherein each group comprises a plurality of files, and each file comprises a plurality of data partitions; and during reading, reading the file by adopting a plurality of file processes, sending the file to the corresponding non-file processes, and carrying out data load balancing and allocation.
Further, the specific reading method comprises:
if the process number is larger than the number of the files, the file process number is equal to the number of the files, each file process reads one file, and after reading, the grid data are sent to the non-file processes, so that load balancing is realized;
if the number of the processes is smaller than the number of the files, when the number of the files is integral multiple of the number of the processes, the number of the file processes is the total number of the processes, all the processes read the files, and the number of the files read by each process is the same; when the number of files is not integral multiple of the number of processes, the number of the file processes is not more than the total number of the processes on the premise that the number of the files read by each file process is the same;
further, the grid data generated by a plurality of objects are stored in groups, each object is used as a group, each group comprises a plurality of files, and each grid data file contains a plurality of grid partition data.
Further, the amount of data in each grid partition data is approximately the same.
Furthermore, if the number of partitions in each process is different, the partition data in each process is allocated in a balanced manner.
Further, the specific balanced blending method comprises the following steps: communication through MPI enables each process to process the same number of partitions.
Compared with the prior art, the beneficial effects of adopting the technical scheme are as follows: when reading the same ultra-large scale grid data, the time consumption of the technical scheme of the invention is only 1/50-1/10 which is time-consuming in the prior art. The invention can greatly save the grid data reading time and improve the calculation efficiency and the economic benefit.
Drawings
FIG. 1 is a geometric drawing of an airfoil pylon marking die in accordance with an embodiment of the present invention.
FIG. 2 is a schematic view of a very large scale grid for an external wing pylon according to an embodiment of the invention.
Fig. 3 is a diagram illustrating a CS-mode reading grid in the prior art.
FIG. 4 is a diagram of a P2P reading grid according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating reading when the number of partitions in each file is the same according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating reading when the number of partitions in each file is different according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating reading when there are redundant non-file processes in an embodiment of the invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
The invention provides a parallel reading-in method of a billion-hundred GB magnitude grid data file, which is used for storing ultra-large scale grid data files generated by a plurality of objects in groups, wherein each group comprises a plurality of files, and each file comprises a plurality of data partitions; and during reading, reading the file by adopting a plurality of file processes, sending the file to the corresponding non-file processes, and carrying out data load balancing and allocation.
In the CFD calculation, the main large files include a mesh file and a flow field file, which contain the topology and the flow field value of each discrete unit and need to occupy a large amount of storage space, for example, for an unstructured mesh with ten million units, the occupied storage is about 1.5GB under the framework of a second-order finite volume method, and the size of the flow field file is about 5 GB; but for a billion-order unstructured grid, the occupied memory is about 150GB, and the size of the flow field file is about 500 GB; therefore, for a billion or even billion-level grid in the future, files of the grid reach the ultra-large-scale level of hundreds of GB, and the actual requirements cannot be met by storing and IO by adopting a single process and a single file.
Therefore, the invention provides a new storage mode, wherein the grid of each object is divided into a plurality of files which are grouped into one group, namely, each object corresponds to one group, each group comprises a plurality of files, and each file comprises a plurality of partition data; in this embodiment, taking a multi-body separation numerical simulation problem of a super-large-scale wing pendant (as shown in fig. 1) as an example, the number of partitions in each wing grid data file is 16, and the number of partitions in each pendant grid data file is 8 or 16, specifically as follows:
the wings and the external stores have separate grids, and due to the super-large scale of the grids of the single object, the grids can be stored in a group of files, so that for the problem of a plurality of objects, the storage form of the super-large scale grid files is a plurality of groups of a plurality of files, as shown in table 1.
TABLE 1 very large-scale grid storage form of multiple objects
For example, for the wing add-on modeling example, the total number of units is 28.8 billion non-structural grids, wherein the number of wing grids is 19.8 billion, the number of add-on grids is 9 billion, as shown in fig. 2, the grids are divided into 2 groups respectively, and stored in 8192 files (4096+4096) or 6144 files (4096+2048), the total file size is 160GB, and after the grids are divided into a plurality of files and stored, the size of each file is about 10-20MB, as shown in table 2.
TABLE 2 example of very large scale grid storage for wing stores
For such data of multiple sets of multiple files, the prior art reading method is as follows: data is read only by the main process and sequentially sent by the main process to the corresponding sub-processes, which is called a Server mode, i.e. a CS mode, similarly to the manner of the Server and the Client, and the reading process is as shown in fig. 3. However, this method has a great limitation, the reading speed of the grid is very slow, and the time consumption is great, as shown in table 3, it takes about 90s to read the data file with the size of 2.5 GB.
Therefore, in this embodiment, it is considered that a plurality of processes are adopted to read a file (file process), and after reading, the grid partition data is sent to the corresponding processes respectively, which is referred to as a Point-to-Point method (P2P).
The method comprises the following specific steps: if the process number is larger than the number of the files, the file process number is equal to the number of the files, each file process reads one file, and after reading, the grid data are sent to the non-file processes, so that load balancing is realized; if the number of the processes is smaller than the number of the files, when the number of the files is integral multiple of the number of the processes, the number of the file processes is the total number of the processes, all the processes read the files, and the number of the files read by each process is the same; when the number of files is not integral multiple of the number of processes, the number of the file processes is not more than the total number of the processes on the premise that the number of the files read by each file process is the same;
and if the number of the partitions in each process is different, performing balanced allocation on the partition data in each process by adopting MPI communication. The specific balanced blending method comprises the following steps: communication through MPI allows each process to process the same amount of partitioned data.
The reading method is explained in detail in this embodiment:
two situations can occur when reading:
1. the number of processes is greater than the number of files
At this time, the number of file taking processes is equal to the number of files, that is, each file process only needs to read 1 file, and after the reading is completed, the mesh data is sent to the non-file process, as shown in fig. 4.
2. The number of processes is less than the number of files
For this case, the number of file processes is necessarily smaller than the number of files, and at this time, the case of class 2 is to be classified:
(1) the number of files F is an integer multiple of the number of processes P
At this time, the file process number M is taken as the total process number P, that is, M equals to P, all processes read the file, and each process reads K equals to F/M files. There are still 2 cases at this time: a) if the number of partitions in each file is the same, then the number of partitions in each process is also the same at this time, and MPI communication is not required, as shown in fig. 5. b) If the number of partitions in each file is different, the number of partitions in each process is different, and MPI communication is also required to perform balanced allocation on the partition data in each process, as shown in fig. 6.
For example, when the process number is P-4 and the number of files is F-12, the process number M-P-4 of the files, and K-F/M-3 of the files read by each file process. a) If each file contains 2 partitions and the total number of the partitions is 24, each process only has 6 partitions, and communication is not needed at this time. b) If the first 6 files each contain 1 partition, the last 6 files each have 3 partitions, and the total partition number is 6+18 to 24, then the process 2 and the process 3 need to send mesh data of 3 partitions to the process 0 and the process 1, respectively, so as to ensure that each process has 6 partitions.
(2) The number of files F is not an integer multiple of the number of processes P.
At the moment, a part of processes M serve as file processes to read files, and the other part of processes N serve as redundant non-file processes to receive data, (M and N meet the condition that M + N equals P), on the premise that the number K of the read files of each file process is the same (M x K equals F), the number of the file processes is not more than the total number of the processes, namely M < equalsP, and in order to reduce communication and improve efficiency, the larger the number M of the file processes is, the better the file processes are;
if the number of processes is P ═ 6, the number of files is F ═ 20, and on the premise that the number of file processes read files per file process is the same, the number of file processes is not greater than the total number of processes, then the number of file processes M and the number of file processes read files per file process K may be M ═ 2, K ═ 10 or M ═ 4, and K ═ 5, and in order to reduce traffic, the number of file processes M ═ 4 is taken, that is, K ═ 5 files are read per file process, and at this time, N ═ 2 processes are still left as non-file processes, and these redundant non-file processes need to acquire mesh data through MPI communication, as shown in fig. 7.
The method is adopted for testing, the wing plug-in object model calculation example is tested, and the time consumed by grid reading is inspected. The results show that: on a domestic cluster, the time consumption for reading 3.6 hundred million grids and 28.8 hundred million grids by adopting a single-process CS mode is 161.64s and 2142.71s, and in contrast, the time consumption for reading grids by adopting 1024 processes and 8192 processes respectively in a P2P mode is 12.30s and 37.28s respectively, the time consumption is only 1/50-1/10 of the CS mode, and the IO efficiency of grid files is obviously improved by adopting a P2P mode.
TABLE 3 wing hangings grid and read-in time
| Grid mesh | Number of cells in grid | Number of documents | File size | P2P model File Process count | P2P mode is time consuming | CS mode time consuming |
| original | 4500 million | 1024 | 2.5GB | 1024 | 7.12s | 94.21s |
| adapt1 | 3.6 hundred million | 1024 | 20GB | 1024 | 12.30s | 161.64s |
| adapt2 | 28.8 hundred million | 8192 | 160GB | 8192 | 37.28s | 2142.71s |
The invention is not limited to the foregoing embodiments. The invention extends to any novel feature or any novel combination of features disclosed in this specification and any novel method or process steps or any novel combination of features disclosed. Those skilled in the art to which the invention pertains will appreciate that insubstantial changes or modifications can be made without departing from the spirit of the invention as defined by the appended claims.
All of the features disclosed in this specification, or all of the steps in any method or process so disclosed, may be combined in any combination, except combinations of features and/or steps that are mutually exclusive.
Any feature disclosed in this specification may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
Claims (6)
1. A parallel read-in method of billion-hundred GB magnitude grid data file is characterized in that grouping files are adopted to store ultra-large scale grid data generated by a plurality of objects, each grouping comprises a plurality of files, and each file comprises a plurality of data partitions; and during reading, reading the file by adopting a plurality of file processes, sending the file to the corresponding non-file processes, and carrying out data load balancing and allocation.
2. The parallel read-in method of the billion hundred GB magnitude grid data file according to claim 1, characterized in that the specific read-in method comprises:
if the process number is larger than the number of the files, the file process number is equal to the number of the files, each file process reads one file, and after reading, the grid data are sent to the non-file processes, so that load balancing is realized;
if the number of the processes is smaller than the number of the files, when the number of the files is integral multiple of the number of the processes, the number of the file processes is the total number of the processes, all the processes read the files, and the number of the files read by each process is the same; and when the number of the files is not integral multiple of the number of the processes, the number of the file processes is not more than the total number of the processes on the premise that the number of the files read by each file process is the same.
3. The method for parallelly reading in the billion hundred GB magnitude grid data file according to claim 2, wherein the grid data generated by a plurality of objects are stored in groups, each object is used as a group, each group comprises a plurality of files, and each grid data file contains a plurality of grid partition data.
4. The method for reading in the billion hundred GB magnitude grid data file in parallel according to claim 3, wherein the data volume in each grid partition data is approximately the same.
5. The method for parallel reading-in of billion hundred GB magnitude grid data files according to claim 3 or 4, characterized in that if the number of partitions on each process is the same, partition data allocation is not needed, and if the number of partitions on each process is different, load balancing allocation is performed on the partition data in each process.
6. The parallel read-in method of billions of GB magnitude grid data files according to claim 5, characterized in that the specific balanced allocation method is as follows: communication through MPI enables each process to process the same amount of grid partition data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011183930.7A CN112463360A (en) | 2020-10-29 | 2020-10-29 | Parallel read-in method for billion-hundred-GB-level grid data file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011183930.7A CN112463360A (en) | 2020-10-29 | 2020-10-29 | Parallel read-in method for billion-hundred-GB-level grid data file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463360A true CN112463360A (en) | 2021-03-09 |
Family
ID=74835617
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011183930.7A Pending CN112463360A (en) | 2020-10-29 | 2020-10-29 | Parallel read-in method for billion-hundred-GB-level grid data file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463360A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116755636A (en) * | 2023-08-16 | 2023-09-15 | 中国空气动力研究与发展中心计算空气动力研究所 | Parallel reading method, device and equipment for grid files and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859286A (en) * | 2005-11-19 | 2006-11-08 | 华为技术有限公司 | Load sharing method |
| CN102447638A (en) * | 2012-01-12 | 2012-05-09 | 中兴通讯股份有限公司 | Load balancing method and forwarding apparatus |
| CN109920059A (en) * | 2019-03-14 | 2019-06-21 | 空气动力学国家重点实验室 | Ultra-large overlapping grid concurrent assembly method based on auxiliary grid |
-
2020
- 2020-10-29 CN CN202011183930.7A patent/CN112463360A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859286A (en) * | 2005-11-19 | 2006-11-08 | 华为技术有限公司 | Load sharing method |
| CN102447638A (en) * | 2012-01-12 | 2012-05-09 | 中兴通讯股份有限公司 | Load balancing method and forwarding apparatus |
| CN109920059A (en) * | 2019-03-14 | 2019-06-21 | 空气动力学国家重点实验室 | Ultra-large overlapping grid concurrent assembly method based on auxiliary grid |
Non-Patent Citations (2)
| Title |
|---|
| 常兴华;马戎;张来平;: "并行化非结构重叠网格隐式装配技术", 航空学报, no. 06 * |
| 王年华;常兴华;赵钟;张来平: "非结构CFD软件MPI+OpenMP混合并行及超大规模非定常并行计算的应用", 航空学报, vol. 41, no. 010, pages 1 - 4 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116755636A (en) * | 2023-08-16 | 2023-09-15 | 中国空气动力研究与发展中心计算空气动力研究所 | Parallel reading method, device and equipment for grid files and storage medium |
| CN116755636B (en) * | 2023-08-16 | 2023-10-27 | 中国空气动力研究与发展中心计算空气动力研究所 | Parallel reading method, device and equipment for grid files and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Dai et al. | ForeGraph: Exploring large-scale graph processing on multi-FPGA architecture | |
| Bode et al. | The tree particle-mesh N-body gravity solver | |
| Hall | Towards a practical parallelisation of the simplex method | |
| WO2017156968A1 (en) | Neural network computing method, system and device therefor | |
| CN105740424A (en) | Spark platform based high efficiency text classification method | |
| WO2010024027A1 (en) | Virtual server system and physical server selection method | |
| CN105824780A (en) | Stand-alone multi-FPGA (Field Programmable Gate Array)-based parallel development method | |
| CN107609141A (en) | It is a kind of that quick modelling method of probabilistic is carried out to extensive renewable energy source data | |
| CN112613243B (en) | Method, device and computer readable storage medium for hydrodynamic simulation | |
| CN111210879A (en) | Hierarchical storage optimization method for super-large-scale drug data | |
| CN101419600A (en) | Data copy mapping method and device based on object-oriented LANGUAGE | |
| CN112463360A (en) | Parallel read-in method for billion-hundred-GB-level grid data file | |
| CN107992358A (en) | A kind of asynchronous IO suitable for the outer figure processing system of core performs method and system | |
| Xiang et al. | GPU acceleration of CFD algorithm: HSMAC and SIMPLE | |
| CN103064991A (en) | Mass data clustering method | |
| CN102799750B (en) | Method for quickly generating common side and non-common sides of geometry surface triangle | |
| CN103324577B (en) | Based on the extensive itemize file allocation system minimizing IO access conflict and file itemize | |
| CN103473368A (en) | Virtual machine real-time migration method and system based on counting rank ordering | |
| CN110110158A (en) | A kind of the memory space division methods and system of three-dimensional mesh data | |
| CN116303219A (en) | Grid file acquisition method and device and electronic equipment | |
| CN103425787B (en) | The triangle gridding of a kind of asymptotic optimization repeats the quick minimizing technology in summit | |
| Kim et al. | MapReduce Based Experimental Frame for Parallel and Distributed Simulation Using Hadoop Platform. | |
| CN113900808A (en) | MPI parallel data structure based on arbitrary polyhedron unstructured grid | |
| CN105631067A (en) | Low-computation complexity construction method of pneumatic flow field grid undirected graph | |
| del Campo et al. | Task replication and control for highly parallel in-memory stores |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |