CN112463739A - Data processing method and system based on ocean mode ROMS - Google Patents
Data processing method and system based on ocean mode ROMS Download PDFInfo
- Publication number
- CN112463739A CN112463739A CN201910849225.7A CN201910849225A CN112463739A CN 112463739 A CN112463739 A CN 112463739A CN 201910849225 A CN201910849225 A CN 201910849225A CN 112463739 A CN112463739 A CN 112463739A
- Authority
- CN
- China
- Prior art keywords
- computing
- sub
- output
- calculation
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 89
- 230000008569 process Effects 0.000 claims abstract description 82
- 238000004364 calculation method Methods 0.000 claims abstract description 70
- 238000012545 processing Methods 0.000 claims abstract description 16
- 230000000903 blocking effect Effects 0.000 claims abstract description 12
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000005192 partition Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004088 simulation Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
- G06F17/12—Simultaneous equations, e.g. systems of linear equations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Abstract
The invention discloses a data processing method and a data processing system based on an ocean mode ROMS (remote management system). an input file is split into N sub-input files according to the number N of computing nodes, the computing process of each computing node reads the corresponding sub-input file, and each computing node outputs a sub-computing result through the output process of each computing node after computing is finished; and finally combining the N sub-calculation results and outputting a calculation result. The method comprises the steps of using distributed I/O to replace the original serial I/O technology of an ocean model ROMS, blocking an input file according to the number of computing nodes, independently reading blocked data by each computing process for computing, outputting a computed result of the process to an independent file, and enabling each independent file to contain blocking information of a computing area so as to be combined in subsequent processing.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data processing method and system based on an ocean mode ROMS.
Background
The ocean numerical model is a process of discretely solving an original ocean equation set. The ocean mode ROMS is an open-source three-dimensional regional ocean model, is researched and developed by the university of Rutger ocean and coast scientific research institute and UCLA together, and is widely applied to hydrodynamic force and water environment simulation in oceans and estuary areas. The mode involves a large calculation area and a complex solving process, so that the large calculation scale is realized, and the calculation capacity and the large data output capacity become the key of the overall performance improvement of the system along with the expansion of the calculation area and the improvement of the calculation precision.
The ocean mode of I/O is characterized in that intensive concurrent I/O requests occur in the calculation interval, and data is read and written at a determined stage. The mode reads in data to set an integral step length, completes a calculation task, and outputs a calculation result to a file at a certain time interval. When the analog data meet the requirements of certain precision and resolution, the representation form is complex, and the method has the main characteristics of large data scale, hundreds of TB magnitude in partial conditions, multiple data dimensions including three-dimensional or four-dimensional data, multiple data formats including NetCDF, GRIB, HDF and the like, and multiple data file types including analog data, restart data and the like.
At the present stage, the computation part in the ocean numerical mode ROMS has already realized multi-node parallel computation, but the I/O module still adopts a serial mode, which becomes a bottleneck for improving the overall performance of the ROMS.
Fig. 1 shows the serial I/O mode of the conventional ROMS mode: d represents the input data of the mode, the input data is received by a process P independently according to the sequence, and then the input data is distributed to N corresponding computing processes C according to the corresponding relation between the data and the processes, the computing processes C can generate N corresponding data after completing corresponding computation, and an output process O collects the data of the N computing processes to generate a data file as an output file to be output.
The existing I/O of the ROMS has the following problems:
1. reading data by a single process P, and distributing the data needing to be calculated to the computing nodes through MPI communication: 1) Because the data file is read by a single node and a single process, the mode data file is usually large, and the problem that the data file cannot be read due to insufficient memory often exists; 2) a large amount of MPI communication is generated in the data distribution process, so that excessive burden of a root process is caused, and the performance of the overall ROMS mode is influenced; 3) because of the single-process reading, the capability of super-computation parallel reading cannot be exerted.
2. After the computation is completed, the process of generating a file by computing data is similar to reading in, and the computing data needs to be interacted from a plurality of computing nodes to a single output process O through the MPI to write in the file: 1) in the data collection process, data transmission and synchronization need to be performed through MPI, which can cause waiting among processes; 2) the problem that large mode data cannot be written into a file is caused by small memory of a single node.
Disclosure of Invention
The invention aims to provide a data processing method and a data processing system based on an ocean mode ROMS, which improve the technical problems of the existing ROMS system by improving the input and output processes of the ROMS system.
In order to solve the technical problems, the invention adopts the following technical scheme:
a data processing method based on ocean mode ROMS is provided, which comprises the following steps: splitting an input file into N sub input files according to the number N of the computing nodes; respectively reading the corresponding sub input files by the computing process of each computing node; each computing node outputs a sub-computing result through an output process of the computing node after computing is finished; and combining the N sub-calculation results and outputting a calculation result.
Firstly, splitting an input file into N parts of sub input files according to the number N of computing nodes, specifically: splitting the input file into N = NtileI × NtileJ blocks according to the number N of the calculation nodes, wherein each block is a sub input file; and recording the block position and/or the core number of the core calculation area of each block of the sub input file.
Further, the calculation process of each calculation node reads the corresponding sub-input file, specifically: and the computing process of each computing node reads the corresponding sub input file based on the core computing area block position and/or the core number.
Further, when each computing node outputs a sub-computation result through its output process, the method further includes: adding block attribute information in the sub-calculation result; the blocking attribute information includes the core computation region blocking position and/or the core number.
Further, combining the N sub-calculation results to output a calculation result, specifically: generating a sub-output file for each sub-calculation result; and merging the sub-output files based on the block attribute information to obtain an output result.
A data processing system based on ocean mode ROMS is provided, which comprises: n computing nodes; each computing node comprises a computing process; the system further comprises: the splitting module is used for splitting the input file into N sub input files according to the number N of the computing nodes; an output process contained in each compute node; the output module is used for outputting a calculation result; the computing process of each computing node is used for respectively reading the corresponding sub input files; each computing node outputs a sub-computing result through an output process of the computing node after computing is finished; and the output module combines N sub-calculation results and outputs a calculation result.
Further, the splitting module comprises: the splitting unit is used for splitting the input file into N = NtileI NtileJ blocks according to the number N of the calculation nodes, and each block is a sub input file; and the recording unit is used for recording the block position and/or the core number of the core calculation area of each block of the sub input file.
Further, the computing process of each computing node reads the corresponding sub-input file based on the core computing area block position and/or the core number.
Furthermore, the output process of each computing node is also used for adding the block attribute information in the sub-computing result; the blocking attribute information includes the core computation region blocking position and/or the core number.
Further, the output module is specifically configured to: generating a sub-output file from each sub-calculation result; and combining the sub-output files based on the block attribute information to obtain an output result.
The data processing method and system based on the ocean mode ROMS provided by the invention use distributed I/O to replace the original serial I/O technology of the ocean mode ROMS, an input file is firstly blocked according to the number of computing nodes, each computing process independently reads the blocked data for computing, and the computed result of the process is output to an independent file, and each independent file contains the blocked information of the computing area, so that the blocked information can be combined in the subsequent processing, and a high-efficiency I/O technology is realized; compared with the prior art, the invention has the advantages and positive effects that:
1. lifting mode simulation speed: because the parallelization of the I/O module of the ocean mode ROMS is realized, the bottleneck of serial I/O is broken through, the overall simulation speed of the mode is improved, and powerful guarantee is provided for realizing business operation.
2. Reducing the output time: the single process writes an output file, so that the dependency relationship of writing a file between processes is reduced, the fragmentation of the output file is reduced, the output time is reduced, the speed is increased, and the error caused by processing the output file by the single process can be reduced.
3. And communication time is reduced, because each core outputs data of the self-computing part, data does not need to be sent any more, and the communication time is effectively reduced.
4. The reading time is reduced: the reading of the single-process file is the reading of the single process of the input file, the splitting is carried out, and then the split file is distributed to enter each process, but the fragmentized data file reduces the time for reading and splitting the data, and the overall speed of the I/O can be changed.
5. And (3) realizing load balance: if the output data is collected to the root process, a great burden is caused to the memory of the core of the root process. And the fragmentation output is realized, and the consumption of computing resources and memory is balanced because each process does not need to collect and distribute. And no excessive burden is caused to a certain process.
Other features and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention when taken in conjunction with the accompanying drawings.
Drawings
FIG. 1 is a schematic diagram of a serial I/O scheme of a conventional ocean mode ROMS;
FIG. 2 is a flow chart of a data processing method based on the ocean mode ROMS proposed by the present invention;
FIG. 3 is a schematic diagram of input file blocking in the data processing method based on the ocean mode ROMS proposed in the present invention;
FIG. 4 is a schematic diagram of a parallel I/O mode provided by the data processing method based on the ocean mode ROMS provided by the present invention;
fig. 5 is an architecture diagram of a data processing system based on marine mode ROMS according to the present invention.
Detailed Description
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings.
As shown in fig. 2, the data processing method based on the marine mode ROMS provided by the invention includes the following steps:
step S21: and splitting the input file into N sub input files according to the number N of the computing nodes.
The method comprises the steps that N computing nodes are used in a computing cluster, each node is responsible for computing a part of area on a map, namely an input file, the input file is divided into N = N = NtileI NtileJ blocks according to the number of the computing nodes, each block serves as a sub-input file, and each sub-input file is responsible for computing through one computing node.
As shown in fig. 3, taking a square grid with 4 nodes as an example for calculation, I-direction and J-direction are 2 directions, and the number of cores is calculated in the 2 directions by using NtileI and NtileJ, respectively, so that the whole calculation region is divided into nitilei × NtileJ sub-regions; wherein Imin and Imax are the maximum value and the minimum value of the position of the gridding calculation area on the I-direction; jmin, Jmax is the maximum value and the minimum value of the position of the gridding calculation area on the J-direction; itile, Jtile is the block number, and also computes the core number, as shown as (0, 0), (0, 1), (1, 0), (1, 1); istr and Iend are the minimum value and the maximum value of the grid position in the calculation sub-area corresponding to the current calculation thread [ Itile, splice ], and represent the core calculation area block position of the calculation sub-area in the I-direction; jstr, Jend is the minimum value and the maximum value of the grid position in the calculation sub-area corresponding to the current calculation thread [ Itile, splice ], and represents the core calculation area block position of the calculation sub-area in the J-direction; NtileI is the number of calculation cores in the I-direction, and NtileJ is the number of calculation cores in the J-direction.
After the input file is blocked, the core calculation area blocking position (Istr, Iend), (Jstr, Jend) and/or the core number (Itile, Jtile) of each block of the sub input file are recorded.
Step S22: and respectively reading the corresponding sub input files by the computing process of each computing node.
The computing nodes are bound with the computing processes, and the computing process of each computing node reads a corresponding sub input file d based on the partitioned positions (Istr, Iend), (Jstr, Jend) and/or the core number (Itile, Jtile) of the partitioned core computing area.
The original data reading process of the ROMS is to read input information of all areas from one master P process and then distribute the input information to each thread for calculation. In the embodiment of the invention, as shown in fig. 4, the process master P is removed, and the block data reading function corresponding to each computing node is directly added into the computing process C (i, j), so that the distribution process is omitted, and the reading efficiency is higher compared with the reading of a single process.
And step S23, each computing node outputs the sub-computing result through the output process of the computing node after computing is completed.
The original export process in the ROMS is to collect partition data on all computing processes and then export from the single process master O to a single file. In the embodiment of the invention, as shown in fig. 4, the collection process master O is removed, the output process O (i, j) is added to each computing node, and the sub-computation results computed by the computation process C (i, j) are directly output.
In the embodiment of the present invention, the partition attribute information, including but not limited to the partition position of the core computation region, the core number, etc., is added to the sub-computation results, and it is intended to be able to merge the sub-computation results according to the partition attribute information.
For example, after each computing node calculates a sub-computation result, in order to merge a single file output by distributed I/O into a file completely consistent with single-threaded I/O output in subsequent processing, attribute information of Attributes of the chunks needs to be added to the dimension information of the single file output by distributed I/O: i-direction points, namely domain _ decompensation = [ Imin Imax Istr Iend ], number of J-direction points, domain _ decompensation = [ Jmin Jmax Jstr Jend ]; for example, domain _ decomposition = [ 11013467 ] in the I direction and domain _ decomposition = [ 1613 ] in the J direction.
Step S24: and combining the N sub-calculation results and outputting a calculation result.
The sub-calculation results generated by calculation of each calculation process are directly output by the output process of the calculation process to form N data files corresponding to the processes one by one, the output file at the previous moment can also be used as input data at the next moment, and communication of data transmission to the root process is reduced.
After the N processes output N independent sub-output files, when the files need to be merged and output, the files are arranged and merged according to the respective block attribute information.
Based on the data processing method based on the ocean mode ROMS, the invention also provides a data processing system based on the ocean mode ROMS, as shown in FIG. 5, the data processing system comprises N computing nodes 51, and each computing node comprises a computing process C; the splitting module 50 splits the input file into N sub-input files according to the number N of the calculation nodes; each computing node comprises an output process O; the output module 52 is used for outputting the calculation result. Wherein, the computing process C of each computing node 51 is configured to read the corresponding sub-input file thereof; each computing node 51 outputs a sub-computation result through its output process o after the computation is completed; the output module 52 combines the N sub-calculation results and outputs the calculation result.
Specifically, the splitting module 50 includes: the splitting unit is used for splitting the input file into N = NtileI × NtileJ blocks according to the number N of the calculation nodes, and each block is a sub input file; the recording unit is used for recording the block position and/or the core number of the core calculation area of each block of the sub input file. The computing process C of each computing node 51 reads the corresponding child input file based on the core computing area partition location and/or the core number. The output process o of each compute node 51 adds the partition attribute information to the sub-compute results; the output module 52 generates a sub-output file from each sub-calculation result, and merges the sub-output files based on the block attribute information to obtain an output result; the partition attribute information here includes a core calculation region partition position and/or a core number.
It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the above examples, and those skilled in the art should also make changes, modifications, additions or substitutions within the spirit and scope of the present invention.
Claims (10)
1. A data processing method based on ocean mode ROMS is characterized by comprising the following steps:
splitting an input file into N sub input files according to the number N of the computing nodes;
respectively reading the corresponding sub input files by the computing process of each computing node;
each computing node outputs a sub-computing result through an output process of the computing node after computing is finished;
and combining the N sub-calculation results and outputting a calculation result.
2. The data processing method based on the ocean mode ROMS as claimed in claim 1, wherein the input file is split into N sub-input files according to the number N of the computing nodes, specifically:
splitting the input file into N = NtileI × NtileJ blocks according to the number N of the calculation nodes, wherein each block is a sub input file;
and recording the block position and/or the core number of the core calculation area of each block of the sub input file.
3. The data processing method based on the ocean mode ROMS as claimed in claim 2, wherein the computing process of each computing node reads the corresponding sub-input file respectively, specifically:
and the computing process of each computing node reads the corresponding sub input file based on the core computing area block position and/or the core number.
4. The data processing method of claim 2, wherein when each computing node outputs a sub-computation result through its output process, the method further comprises:
adding block attribute information in the sub-calculation result;
the blocking attribute information includes the core computation region blocking position and/or the core number.
5. The data processing method based on the ocean mode ROMS as claimed in claim 4, wherein the N sub-calculation results are combined to output a calculation result, specifically:
generating a sub-output file for each sub-calculation result;
and merging the sub-output files based on the block attribute information to obtain an output result.
6. A data processing system based on marine mode ROMS, comprising:
n computing nodes; each computing node comprises a computing process;
characterized in that the system further comprises:
the splitting module is used for splitting the input file into N sub input files according to the number N of the computing nodes;
an output process contained in each compute node;
the output module is used for outputting a calculation result;
the computing process of each computing node is used for respectively reading the corresponding sub input files; each computing node outputs a sub-computing result through an output process of the computing node after computing is finished; and the output module combines N sub-calculation results and outputs a calculation result.
7. The data processing system of claim 6, wherein the splitting module comprises:
the splitting unit is used for splitting the input file into N = NtileI NtileJ blocks according to the number N of the calculation nodes, and each block is a sub input file;
and the recording unit is used for recording the block position and/or the core number of the core calculation area of each block of the sub input file.
8. The data processing system according to claim 7, wherein the computing process of each computing node reads the corresponding child input file based on core computing area block location and/or core number.
9. The data processing system according to claim 7, wherein the output process of each compute node is further configured to add tile attribute information to the sub-computed result;
the blocking attribute information includes the core computation region blocking position and/or the core number.
10. The data processing system of claim 9, wherein the output module is specifically configured to:
generating a sub-output file from each sub-calculation result; and combining the sub-output files based on the block attribute information to obtain an output result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910849225.7A CN112463739A (en) | 2019-09-09 | 2019-09-09 | Data processing method and system based on ocean mode ROMS |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910849225.7A CN112463739A (en) | 2019-09-09 | 2019-09-09 | Data processing method and system based on ocean mode ROMS |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463739A true CN112463739A (en) | 2021-03-09 |
Family
ID=74807397
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910849225.7A Pending CN112463739A (en) | 2019-09-09 | 2019-09-09 | Data processing method and system based on ocean mode ROMS |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463739A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190511A (en) * | 2021-04-21 | 2021-07-30 | 中国海洋大学 | Big data concurrent scheduling and accelerated processing method based on many-core cluster |
| CN115391069A (en) * | 2022-10-27 | 2022-11-25 | 山东省计算中心（国家超级计算济南中心） | Parallel communication method and system based on ocean mode ROMS |
| CN116185662A (en) * | 2023-02-14 | 2023-05-30 | 国家海洋环境预报中心 | Asynchronous parallel I/O method based on NetCDF and non-blocking communication |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183368A (en) * | 2007-12-06 | 2008-05-21 | 华南理工大学 | Method and system for distributed calculating and enquiring magnanimity data in on-line analysis processing |
| CN103166785A (en) * | 2011-12-15 | 2013-06-19 | 同程网络科技股份有限公司 | Distributed type log analysis system based on Hadoop |
| CN103309676A (en) * | 2013-06-27 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Web service encapsulation method and system for ocean numerical modeling regional ocean modeling system (ROMS) |
| CN103905300A (en) * | 2012-12-25 | 2014-07-02 | 华为技术有限公司 | Data message sending method, device and system |
| CN104506935A (en) * | 2014-12-31 | 2015-04-08 | 成都东方盛行电子有限责任公司 | Audio-video file synthesis method based on fragmentation technique |
| CN104699670A (en) * | 2015-03-31 | 2015-06-10 | 中译语通科技（北京）有限公司 | File splitting method and device |
| CN105205174A (en) * | 2015-10-14 | 2015-12-30 | 北京百度网讯科技有限公司 | File processing method and device for distributed system |
| CN106022245A (en) * | 2016-05-16 | 2016-10-12 | 中国资源卫星应用中心 | Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification |
| CN107885549A (en) * | 2017-11-16 | 2018-04-06 | 信阳师范学院 | Remove the method and system that process is remained in TORQUE computing cluster calculate nodes |
-
2019
- 2019-09-09 CN CN201910849225.7A patent/CN112463739A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183368A (en) * | 2007-12-06 | 2008-05-21 | 华南理工大学 | Method and system for distributed calculating and enquiring magnanimity data in on-line analysis processing |
| CN103166785A (en) * | 2011-12-15 | 2013-06-19 | 同程网络科技股份有限公司 | Distributed type log analysis system based on Hadoop |
| CN103905300A (en) * | 2012-12-25 | 2014-07-02 | 华为技术有限公司 | Data message sending method, device and system |
| CN103309676A (en) * | 2013-06-27 | 2013-09-18 | 曙光信息产业(北京)有限公司 | Web service encapsulation method and system for ocean numerical modeling regional ocean modeling system (ROMS) |
| CN104506935A (en) * | 2014-12-31 | 2015-04-08 | 成都东方盛行电子有限责任公司 | Audio-video file synthesis method based on fragmentation technique |
| CN104699670A (en) * | 2015-03-31 | 2015-06-10 | 中译语通科技（北京）有限公司 | File splitting method and device |
| CN105205174A (en) * | 2015-10-14 | 2015-12-30 | 北京百度网讯科技有限公司 | File processing method and device for distributed system |
| CN106022245A (en) * | 2016-05-16 | 2016-10-12 | 中国资源卫星应用中心 | Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification |
| CN107885549A (en) * | 2017-11-16 | 2018-04-06 | 信阳师范学院 | Remove the method and system that process is remained in TORQUE computing cluster calculate nodes |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190511A (en) * | 2021-04-21 | 2021-07-30 | 中国海洋大学 | Big data concurrent scheduling and accelerated processing method based on many-core cluster |
| CN113190511B (en) * | 2021-04-21 | 2022-09-13 | 中国海洋大学 | Big data concurrent scheduling and accelerated processing method based on many-core cluster |
| CN115391069A (en) * | 2022-10-27 | 2022-11-25 | 山东省计算中心（国家超级计算济南中心） | Parallel communication method and system based on ocean mode ROMS |
| CN115391069B (en) * | 2022-10-27 | 2023-02-03 | 山东省计算中心（国家超级计算济南中心） | Parallel communication method and system based on ocean mode ROMS |
| CN116185662A (en) * | 2023-02-14 | 2023-05-30 | 国家海洋环境预报中心 | Asynchronous parallel I/O method based on NetCDF and non-blocking communication |
| CN116185662B (en) * | 2023-02-14 | 2023-11-17 | 国家海洋环境预报中心 | Asynchronous parallel I/O method based on NetCDF and non-blocking communication |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112463739A (en) | Data processing method and system based on ocean mode ROMS | |
| Ma et al. | Query processing of massive trajectory data based on mapreduce | |
| CN103178516B (en) | Node merge method based dynamic network topology analysis method | |
| CN108280522A (en) | A kind of plug-in type distributed machines study Computational frame and its data processing method | |
| CN109726174A (en) | Data archiving method, system, equipment and storage medium | |
| CN102662639A (en) | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method | |
| CN102158417A (en) | Method and device for optimizing multi-constraint quality of service (QoS) routing selection | |
| CN110209353B (en) | I/O parallel acceleration method, device and medium for ROMS mode in area coupling forecast system | |
| CN111400326A (en) | Smart city data management system and method thereof | |
| CN102609990A (en) | Massive-scene gradually-updating algorithm facing complex three dimensional CAD (Computer-Aided Design) model | |
| CN103353866A (en) | Three-dimensional model file format conversion method supporting XNA technology | |
| CN110852930B (en) | FPGA graph processing acceleration method and system based on OpenCL | |
| CN102902590A (en) | Parallel digital terrain analysis-oriented massive DEM (Digital Elevation Model) deploying and scheduling method | |
| CN104125293A (en) | Cloud server and application method thereof | |
| CN111125131A (en) | Two-stage consensus block chain system with state buffering capacity and deployment method | |
| CN114444274A (en) | Method, medium and device for reconstructing original structure grid from non-structure grid | |
| CN112947870B (en) | G-code parallel generation method of 3D printing model | |
| CN102685222B (en) | A kind of cloud SRM device for electric power system | |
| CN103150157B (en) | Based on the GPU kernel program restructuring optimization method of memory access difference | |
| CN103699627B (en) | A kind of super large file in parallel data block localization method based on Hadoop clusters | |
| CN108763562A (en) | A kind of construction method based on big data skill upgrading data exchange efficiency | |
| CN102592251A (en) | Universal modular parallel coupling system adopting earth system model | |
| CN108334532A (en) | A kind of Eclat parallel methods, system and device based on Spark | |
| CN103870342B (en) | Task core value calculating method based on node attribute function in cloud computing environment | |
| CN115879543A (en) | Model training method, device, equipment, medium and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210309 |
|
| RJ01 | Rejection of invention patent application after publication |