CN110543361B - Astronomical data parallel processing device and astronomical data parallel processing method - Google Patents
Astronomical data parallel processing device and astronomical data parallel processing method Download PDFInfo
- Publication number
- CN110543361B CN110543361B CN201910693839.0A CN201910693839A CN110543361B CN 110543361 B CN110543361 B CN 110543361B CN 201910693839 A CN201910693839 A CN 201910693839A CN 110543361 B CN110543361 B CN 110543361B
- Authority
- CN
- China
- Prior art keywords
- astronomical data
- data processing
- parallel
- astronomical
- processing program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Devices For Executing Special Programs (AREA)
- Multi Processors (AREA)
Abstract
The invention discloses an astronomical data parallel processing device and method, wherein the processing device calculates clusters, and the method comprises the following steps: the system comprises a management server, a storage server and a plurality of calculation servers, wherein the storage server is used for storing a plurality of astronomical data files, parameter files and astronomical data processing instructions, and the instructions are executed by the plurality of calculation servers at the same time. The method comprises the following steps: the management server runs a starting module; the starting module distributes the parallel modules to a plurality of computing servers for operation, and distributes a task number for the astronomical data processing program; the parallel module starts an astronomical data processing program, and extracts parameters from a parameter file in a storage server according to a task number and inputs the parameters to the astronomical data processing program; and the astronomical data processing program processes astronomical data files in the storage server according to the parameters and stores the results in the storage server.
Description
Technical Field
The invention relates to the field of astronomical data processing, in particular to a device and a method for astronomical data parallel processing.
Background
With the continuous construction and precision upgrading of astronomical observation equipment, the acquisition capability of astronomical data is greatly enhanced, astronomical research enters a big data era, an original astronomical data processing program cannot meet the time requirement of big data processing, and massive parallelism becomes a necessary means for accelerating astronomical data processing.
HPC (High Performance Computing System, high-performance computing system) is a main environment of parallel computing, and is characterized in that a computing cluster is formed by a plurality of computing servers with the same architecture through a high-speed network, and computing tasks are distributed on different computing servers by parallel processing software, so that parallel execution of the computing tasks is realized. Thus, implementing parallel computing tasks, the software of parallel computing is an important component, except for the necessary computing hardware, and development of parallel software in HPC environments is currently performed based on MPI (Message Passing Interface, messaging interface).
MPI is a standard protocol interface for parallel program development in HPC environment, designs a basic framework and a data interaction mode between sub-processes of parallel program development, and has a plurality of realization modes such as openMPI, intelMPI, MPICH at present, but the realization principle is consistent, however, in a large-scale data processing environment, different processes need to execute different calculation tasks or correspond to different input data, at this time, the common practice needs to carry out parallel design again according to the framework of MPI on the basis of the original program, and different processes execute different actions according to different process numbers, so that the processes can execute correctly in parallel in HPC environment. Otherwise, although the multi-process is started, the purpose of parallel processing of data cannot be achieved because the executed commands are the same.
The parallelization transformation of the original serial program is a very complex technical work, on one hand, for the use of the program, the transformation cannot be carried out under the condition of lacking source codes or being unfamiliar with the source codes, on the other hand, even under the condition of the active codes, after the parallelization transformation, the original program flow is changed, and the value shifting of the program under different computing environments is inconvenient.
Disclosure of Invention
First, the technical problem to be solved
The invention discloses a device and a method for processing astronomical data in parallel, which at least partially solve the defects of low speed and low efficiency of astronomical data serial processing in the existing method.
(II) technical scheme
According to an aspect of the present invention, there is provided an apparatus for astronomical data parallel processing, including: a computing cluster including a management server, a storage server and a plurality of computing servers; the storage server is used for storing a plurality of astronomical data files, parameter files and astronomical data processing instructions, and the instructions are executed by a plurality of computing servers at the same time, and the execution comprises the following steps: simultaneously extracting parameters from the parameter file; and initiating an astronomical data processing task, and distributing the astronomical data in parallel to a plurality of computing servers for operation.
In a further aspect, the management server, the storage server and the computing server are connected by an ethernet network and operate within the same network segment.
According to another aspect of the present invention, there is also provided a method for astronomical data parallel processing, including: a running starting module; distributing the parallel modules to a plurality of computing servers for operation, and distributing a task number for the astronomical data processing program; starting an astronomical data processing program, extracting parameters from a parameter file in a storage server according to a task number, and inputting the parameters into the astronomical data processing program; and processing the astronomical data file in the storage server according to the parameters, and storing the result.
In a further aspect, the astronomical data processing program is concurrently running in the plurality of computing servers.
In a further aspect, the parallel module is concurrently running in the plurality of the computing servers, extracts parameters from a parameter file and initiates an astronomical data processing procedure, the parallel module conforming to the MPI standard specification.
In a further scheme, the starting module is operated on the management server, initiates astronomical data processing tasks and distributes the parallel modules to a plurality of computing servers for operation.
In a further scheme, the parameter file is in a text file format, and each row comprises a task number and an astronomical data file distributed for an astronomical data processing program corresponding to the task number.
(III) beneficial effects
The invention redesigns the flow of the parallel architecture, increases the parallel module as the middleware of the MPI frame and the astronomical data processing program, inputs different parameters for the astronomical data processing program in different calculation servers, so that the astronomical data processing program has the capability of parallel calculation, and simultaneously, a user can realize large-scale parallel execution without modifying the serial codes of the original astronomical data processing program, thereby improving the data processing efficiency.
In addition, the problem of parallelization of the astronomical data processing program which is passive code or complex code and cannot be subjected to parallelization modification is solved.
Drawings
Fig. 1 is a schematic diagram of a computing cluster of an astronomical data parallel processing device according to an embodiment of the present invention.
Fig. 2 is a flowchart of an astronomical data parallel processing method according to an embodiment of the present invention.
[ reference numerals description ]
1. A management server; 2. a storage server; 3. computing server
Detailed Description
The present invention will be further described in detail below with reference to specific embodiments and with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent.
In the present invention, "disposed on" or "affixed to" is intended to include a direct contact relationship with a single or multiple components. Moreover, the use of ordinal numbers such as "first," "second," "first," or "second," etc., in the description and the claims, to modify a claim element does not by itself connote any preceding ordinal number for the element, nor does it connote an ordering of one element by another or a method of manufacture, the ordinal numbers being used merely to distinguish one element having a certain name from another element having a same name. In a large-scale data processing environment, different processes need to execute different computing tasks or correspond to different input data, at this time, a common practice needs to carry out parallel design again according to an MPI framework based on an original program, so that different processes execute different actions according to different process numbers, and therefore correct parallel execution is performed in an HPC environment, otherwise, although multiple processes are started, the purpose of parallel processing of data cannot be achieved because executed commands are the same.
The invention provides an astronomical data parallel processing device, fig. 1 is a calculation cluster structure diagram of the astronomical data parallel processing device in the embodiment of the invention, as shown in fig. 1, comprising a management server 1, a storage server 2 and a plurality of calculation servers 3.
The storage server 2 is configured to store a plurality of astronomical data files, parameter files, and astronomical data processing instructions, where the instructions are executed by a plurality of computing servers 3 at the same time, and the execution includes the following steps:
simultaneously extracting parameters from the parameter file; and initiating an astronomical data processing task, and distributing astronomical data in parallel to a plurality of computing servers 3 for operation.
In this embodiment, the management server 1, the storage server 2 and the computing server 3 are connected by an ethernet network and operate in the same network segment.
The invention also provides a method for parallel processing of astronomical data, and fig. 2 is a flow chart of a method for parallel processing of astronomical data according to an embodiment of the invention, as shown in fig. 2, including:
a running starting module;
the parallel modules are distributed to a plurality of computing servers 3 for operation, and a task number is distributed to astronomical data processing programs;
starting an astronomical data processing program, extracting parameters from a parameter file in the storage server 2 according to the task number, and inputting the parameters into the astronomical data processing program;
and processing the astronomical data file in the storage server according to the parameters, and maintaining the result.
In this embodiment, the astronomical data processing program runs in 3 of the plurality of computing servers at the same time; the parallel module also operates in the plurality of the computing servers 3 at the same time, extracts parameters from the parameter file and starts an astronomical data processing program, and at the same time, complies with the MPI standard specification; the starting module operates on the management server 1, and initiates astronomical data processing tasks and distributes the parallel modules to a plurality of computing servers 3 for operation.
In addition, the parameter file is in a text file format, and each row comprises a task number and an astronomical data file distributed for an astronomical data processing program corresponding to the task number.
In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention. It may be evident, however, that one or more embodiments may be practiced without these specific details. It will be appreciated by persons skilled in the art that the following specific details are not to be construed as limiting the invention.
In the exemplary embodiment of the present invention, the number of the plurality of computing servers 3 is 60, the storage server 2 includes 60 hard disks, each hard disk has a capacity of 10T, and the hard disks are mounted under the setting directory of the 60 computing servers 3 through a network file system.
The astronomical data files are spectral observation data, 1680 files are taken as the total, the astronomical data processing program is a spectral parameter extraction program, the parameter files comprise parameters required by execution of the spectral parameter extraction program and are stored in the storage server 2, the parallel module acquires astronomical data parameters, namely spectral parameters, from corresponding rows of the parameter files according to task numbers of the astronomical data processing program, and the astronomical data processing program starts to process the astronomical data files in the storage server 2 according to the parameters.
The parallel module is responsible for analyzing the parameter file and inputting different parameters to the same astronomical data processing program, and the astronomical data processing program runs in different calculation servers 3 so as to achieve the parallel effect.
In this embodiment, the command for running the astronomical data processing module is:
mpirun-np 1680-hosts cu01，cu02，cu03，cu04，...cu60-ppn 28/opt/software/scaleMpi execfile-f filelist
wherein, "mpirun" is a start module run command;
"cu01, cu02, cu03, cu04," cu60 "is a compute server 3 identifier;
"scaleMpi" is a parallel module;
"execfile" is an astronomical data processing program;
"filelist" is a parameter file;
"-np 1680" refers to 1680 astronomical data files to be processed.
In this embodiment, the parallel module inputs different parameters for the astronomical data processing program in different calculation servers 3, so that the serialized astronomical data processing program processes different astronomical data files simultaneously, thereby having the capability of parallel calculation, and the user can implement large-scale parallel execution without modifying the serial code of the original astronomical data processing program, and the data processing efficiency is improved.
In other embodiments of the present invention, a job management system, such as PBS (Portable Batch System, portable batch processing system), may be further configured in the management server 1, where the job management system may set a plurality of astronomical data processing tasks as a task queue, and sequentially execute or automatically execute the astronomical data processing tasks according to a set time.
According to the embodiment of the invention, under the condition that the astronomical data processing program is a passive code or the code is complex and can not be subjected to parallelization modification, different astronomical data files can be processed simultaneously by using the same astronomical data processing program, and the parallel computing capability is provided.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the invention thereto, but to limit the invention thereto, and any modifications, equivalents, improvements and equivalents thereof may be made without departing from the spirit and principles of the invention.
Claims (8)
1. An astronomical data parallel processing device, comprising:
the system comprises a computing cluster, a storage server and a plurality of computing servers, wherein an operation management system and a starting module are configured in the management server, and the operation management system is used for setting a plurality of astronomical data processing tasks into a task queue, and executing the astronomical data processing tasks sequentially or automatically according to set time; the starting module is used for initiating astronomical data processing tasks and distributing the parallel modules to a plurality of computing servers for operation;
the storage server is used for storing a plurality of astronomical data files, parameter files and astronomical data processing instructions, and the instructions are executed by a plurality of computing servers at the same time, and the execution comprises the following steps:
extracting parameters from the parameter file by using the parallel module at the same time, and inputting different parameters into the same astronomical data processing program, so that the serialized astronomical data processing program processes different astronomical data at the same time; and initiating an astronomical data processing task, and distributing the astronomical data in parallel to astronomical data processing programs of a plurality of computing servers for operation.
2. The device of claim 1, wherein the management server, storage server, and computing server are connected by an ethernet network and operate within the same network segment.
3. A method of applying the apparatus of claim 1 to parallel processing of astronomical data, comprising:
a running starting module;
distributing the parallel modules to a plurality of computing servers for operation, and distributing a task number for the astronomical data processing program;
starting an astronomical data processing program, extracting parameters from a parameter file in a storage server according to a task number, and inputting the parameters into the astronomical data processing program;
and processing the astronomical data file in the storage server according to the parameters, and maintaining the result.
4. A method according to claim 3, wherein the astronomical data processing program runs simultaneously in the plurality of computing servers.
5. The method according to claim 4, comprising:
and simultaneously operating in the plurality of computing servers through a parallel module, extracting parameters from the parameter file and starting an astronomical data processing program.
6. The method according to claim 5, comprising:
and the starting module is operated on the management server, initiates astronomical data processing tasks and distributes the parallel modules to a plurality of computing servers for operation.
7. The method of claim 6, wherein the parallel module complies with an MPI standard specification.
8. The method of claim 7, wherein the parameter file is in a text file format, and each row includes a task number and an astronomical data file assigned to an astronomical data processing program corresponding to the task number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910693839.0A CN110543361B (en) | 2019-07-29 | 2019-07-29 | Astronomical data parallel processing device and astronomical data parallel processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910693839.0A CN110543361B (en) | 2019-07-29 | 2019-07-29 | Astronomical data parallel processing device and astronomical data parallel processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110543361A CN110543361A (en) | 2019-12-06 |
| CN110543361B true CN110543361B (en) | 2023-06-13 |
Family
ID=68710387
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910693839.0A Active CN110543361B (en) | 2019-07-29 | 2019-07-29 | Astronomical data parallel processing device and astronomical data parallel processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110543361B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8260840B1 (en) * | 2010-06-28 | 2012-09-04 | Amazon Technologies, Inc. | Dynamic scaling of a cluster of computing nodes used for distributed execution of a program |
| CN102959517A (en) * | 2010-06-10 | 2013-03-06 | Otoy公司 | Allocation of gpu resources accross multiple clients |
| CN103034534A (en) * | 2011-09-29 | 2013-04-10 | 阿尔斯通电网公司 | Electric power system analysis parallel computing method and system based on grid computation |
| CN104537180A (en) * | 2015-01-04 | 2015-04-22 | 中国科学院国家天文台南京天文光学技术研究所 | Numerical simulation method of astronomical site selection atmospheric optical parameter measurement instrument |
| CN108805187A (en) * | 2018-05-29 | 2018-11-13 | 北京佳格天地科技有限公司 | Celestial spectrum sequence automatic classification system and method |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100584193B1 (en) * | 2004-01-29 | 2006-05-26 | 한국과학기술정보연구원 | A Grid MPI Job Allocation System and Method using File-based MPI initialization in Grid Computing System |
| US20090172353A1 (en) * | 2007-12-28 | 2009-07-02 | Optillel Solutions | System and method for architecture-adaptable automatic parallelization of computing code |
| WO2015145598A1 (en) * | 2014-03-26 | 2015-10-01 | 株式会社 日立製作所 | Data distribution device for parallel operation processing system, data distribution method, and data distribution program |
| US20160026553A1 (en) * | 2014-07-22 | 2016-01-28 | Cray Inc. | Computer workload manager |
| CN104615487B (en) * | 2015-01-12 | 2019-03-08 | 中国科学院计算机网络信息中心 | Parallel task optimization system and method |
| US20170351536A1 (en) * | 2016-06-02 | 2017-12-07 | Hewlett Packard Enterprise Development Lp | Provide hypervisor manager native api call from api gateway to hypervisor manager |
| CN106681807B (en) * | 2016-11-28 | 2020-04-14 | 中国人民解放军国防科学技术大学 | Imaging satellite task preprocessing parallelization method based on Spark |
-
2019
- 2019-07-29 CN CN201910693839.0A patent/CN110543361B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102959517A (en) * | 2010-06-10 | 2013-03-06 | Otoy公司 | Allocation of gpu resources accross multiple clients |
| US8260840B1 (en) * | 2010-06-28 | 2012-09-04 | Amazon Technologies, Inc. | Dynamic scaling of a cluster of computing nodes used for distributed execution of a program |
| CN103034534A (en) * | 2011-09-29 | 2013-04-10 | 阿尔斯通电网公司 | Electric power system analysis parallel computing method and system based on grid computation |
| CN104537180A (en) * | 2015-01-04 | 2015-04-22 | 中国科学院国家天文台南京天文光学技术研究所 | Numerical simulation method of astronomical site selection atmospheric optical parameter measurement instrument |
| CN108805187A (en) * | 2018-05-29 | 2018-11-13 | 北京佳格天地科技有限公司 | Celestial spectrum sequence automatic classification system and method |
Non-Patent Citations (3)
| Title |
|---|
| Parallel astronomical data processing with Python: Recipes for multicore machines;Navtej Singh等;《Astronomy and Computing 》;第2卷;第1-10页 * |
| 多GPU-CPU混合异构平台下的光谱计算优化;徐星宇;《中国优秀硕士学位论文全文数据库 基础科学辑》(第10期);A007-1 * |
| 用MapReduce框架构建虚拟天文台数据节点;宋烜等;《天文研究与技术》;第9卷(第2期);第150-156页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110543361A (en) | 2019-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Elser et al. | An evaluation study of bigdata frameworks for graph processing | |
| WO2022088804A1 (en) | Message pushing method and apparatus, and electronic device, and storage medium | |
| US9986018B2 (en) | Method and system for a scheduled map executor | |
| US8307322B2 (en) | Wiring design apparatus | |
| CN110362401A (en) | Data run the member host in batch method, apparatus, storage medium and cluster | |
| CN105912387A (en) | Method and device for dispatching data processing operation | |
| WO2017193737A1 (en) | Software testing method and system | |
| CN104683472A (en) | Data transmission method capable of supporting large data volume | |
| Chen et al. | GPU-accelerated high-throughput online stream data processing | |
| CN111327692A (en) | Model training method and device and cluster system | |
| CN111783952A (en) | Configuration method, device, system, electronic equipment and storage medium | |
| CN108228330A (en) | The multi-process method for scheduling task and device of a kind of serialization | |
| US20150172369A1 (en) | Method and system for iterative pipeline | |
| CN111324667A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN110543361B (en) | Astronomical data parallel processing device and astronomical data parallel processing method | |
| CN110231931A (en) | A kind of power supply real-time control module and system | |
| CN113687926A (en) | Control method, device and equipment for server asset scanning | |
| Phan et al. | On Understanding the energy impact of speculative execution in Hadoop | |
| CN110502337B (en) | Optimization system for shuffling stage in Hadoop MapReduce | |
| CN109766131A (en) | The system and method for the intelligent automatic upgrading of software is realized based on multithreading | |
| CN110750362A (en) | Method and apparatus for analyzing biological information, and storage medium | |
| CN114064142A (en) | Batch-flow integrated data processing system and processing method | |
| KR20050057834A (en) | Method for mass data processing through scheduler in multi processor system | |
| CN115344370A (en) | Task scheduling method, device, equipment and storage medium | |
| Wei et al. | Performance and portability studies with openacc accelerated version of gtc-p |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |