CN103810140B - A kind of radio astronomy data processing method based on general purpose PC - Google Patents
A kind of radio astronomy data processing method based on general purpose PC Download PDFInfo
- Publication number
- CN103810140B CN103810140B CN201410055134.3A CN201410055134A CN103810140B CN 103810140 B CN103810140 B CN 103810140B CN 201410055134 A CN201410055134 A CN 201410055134A CN 103810140 B CN103810140 B CN 103810140B
- Authority
- CN
- China
- Prior art keywords
- data
- radio astronomy
- antenna
- general purpose
- day
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Mobile Radio Communication Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of radio astronomy data processing method based on general purpose PC, wherein the method is performed by multiple stage general purpose PC to process radio astronomy data the most on the same day respectively, and the method includes: the radio astronomy data receiving antenna carry out pretreatment;According to the pretreated all baselines of radio astronomy data statistics, the standard deviation square value of all passages;According to obtained standard deviation square value, the optical path difference of baseline two-by-two every day is carried out calibration correction;The radio astronomy data of every day are modified by the optical path difference according to revised baseline two-by-two every day, and multiple stage general purpose PC is processed the revised radio astronomy data obtained are overlapped finally being processed data.The present invention is based on general purpose PC, and low cost, volume are little, maintenance is simple, be easy to extension, utilizes the data that local SATA hard disc processes N days simultaneously so that data processing speed is greatly promoted.
Description
Technical field
The invention belongs to radio astronomy processing technology field, penetrate based on general purpose PC particularly to one
Electricity chronometer data processing method.
Background technology
21CMA (21 Centimeter Array) is that the large-scale metric wave being operated in 70-200MHz frequency range is combined
Close aperture antenna battle array, it is intended in the epoch that ionize again in detection universe.21CMA is by 80 sub-antenna arrays
Composition, the effective area of each sub antenna battle array is about 200 square metres, is distributed in thing, north and south 3 public affairs
In on baseline, accepting after the amplified filtering of signal respectively through optical cable transmission controller of 80 arrays,
The computer cluster being made up of 80 servers produces after signal carries out digital collection and relevant treatment
The data needed are processed for follow-up data.The data produced are saved on hard disk the data carrying out next step
Handling process. 40 antenna arrays of thing of work produce the data of 1.2T every day at present, under needing to carry out
If a step data process. 80 groups of antennas run simultaneously, have every day the data of nearly 2.5T need into
Row data process, and traditional radio astronomy data processing method is that data are placed on high performance work station
Carry out computing, and HPC data-interface generally used now, such as IDE (integrated development
Environment), SATA (serial advanced technology attachment), SCSI (small
Computer system interface) and the transmission speed of optical fiber be all limited.
Fig. 1 shows the structural representation of SGI work station in prior art.As it is shown in figure 1, it is described
SGI work station is one has 32 Intel Itanium double-core CPU, 128GB internal memory and band a width of
The fibre disk array capacity of 4G is 12TB.It is configured with dual controller 4 passage 4GB optical fiber hard disk
Array, has two dish casees to constitute, each dish case respectively as two groups of RAID5, and totally 4 groups of RAID, two
Dish case is connected with RAID controller to improve concurrent capability by optical fiber respectively.Data pass through 4
The optical-fibre channel of 4GB is connected respectively on 4 PCI optical channel cards, and each optical channel card is respectively
It is inserted in different pci bus.With the data volume of 1.2T every day now, process one month simultaneously
As a example by data 1.2T*30=36TB, need to carry out 4 secondary data importings at SGI work station, then enter
Row data process, and finally carry out result preservation.Data are directed through the NFS association of raid5 disk array
View is carried out, and it is 30MB/s that NFS protocol imports the speed of data, therefore imports the data needs of 30 days
3000 hours, the data preserved after carrying out data process derived and are also required to 3000 hours.
Along with coming into operation of wide-aperture telescope and aerial array, the thing followed is mass data
Producing, having the data of PB magnitude every day, so needing at the data of a kind of general computer
Reason method solves.
Summary of the invention
(1) to solve the technical problem that
Along with the generation of mass data, traditional radio astronomy data processing method is by computer data
The impact that transmission bandwidth limits is more and more obvious, causes processing method to run into a bottleneck phase.Pile up such as
The data on mountain are comed one after another, and processing the most fast and effectively is a problem primarily solving of the present invention.
Using work station carry out chronometer data to process is a kind of traditional method, and the most swollen along with data
Swollen, the drawback of work station extension is more and more obvious, is first affected by its framework, has the limit extended
Value;Secondly affected by data transfer bandwidth bottleneck, it is impossible to ensure that data are quickly transmitted.
(2) technical scheme
The invention provides a kind of radio astronomy data processing method based on general purpose PC, wherein should
Method is performed by multiple stage general purpose PC to process radio astronomy data the most on the same day, the method bag respectively
Include:
Step 1: the radio astronomy data receiving antenna carry out pretreatment;
Step 2: according to the pretreated all baselines of radio astronomy data statistics, the mark of all passages
Quasi-variance yields;
Step 3: the optical path difference of baseline two-by-two every day is calibrated according to obtained standard deviation square value
Revise;
Step 4: according to radio astronomy data to every day of the optical path difference of revised baseline two-by-two every day
It is modified, and the revised radio astronomy data that the process of multiple stage general purpose PC obtains are folded
Add and finally processed data.
Ever-increasing chronometer data inspection process side is solved by a kind of general computer system
Method, first passes through distributed general purpose PC and carrys out the different data of carry and carry out data process, unit number
According to processing the bottleneck solving data transmission, data exchange can be carried out more efficiently and data process;
Next data accumulation have employed INFINIBAND technology based on 4.5Gb and improves transfer function, number
900MB/s is reached according to transmitted in both directions speed.
(3) beneficial effect
This solves tradition chronometer data processing method to be difficult to extension, extended the deadline by output transmission
System, it is impossible to carrying out data exchange fast and effectively and data process, the method data-handling efficiency obtains
The biggest raising.
Accompanying drawing explanation
Fig. 1 shows the structural representation of SGI work station in prior art;
Fig. 2 shows data radio astronomy data processing method stream based on general purpose PC in the present invention
Cheng Tu;
Fig. 3 shows the structural representation that Computer system of the present invention configures.
Detailed description of the invention
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with concrete real
Execute example, and referring to the drawings, the present invention is described in further detail.
The present invention proposes a kind of data radio astronomy data processing method based on general purpose PC.This
The said method that invention proposes is at large-scale metric wave aperture synthesis antenna array 21CMA (21Cenimeter
Array) above try out and achieve good effect.
21CMA is the metric wave aperture synthesis antenna array being operated in 70-200MHz, it is intended to detection universe
Ionize the epoch again, signal through antenna system, acquisition system is by fiber-optic transfer to data acquisition center and preserves
On hard disk, the data of every day have about 1.2T, and the data obtained enter data processing platform (DPP) and carry out
The radio astronomy flow chart of data processing in later stage.Traditional radio astronomy data processing method, by big data
The impact of form, it is necessary to carry out on high performance work station, this type of calculating carrier bulk is big, cost
High, safeguard complicated, be unfavorable for extension, and the present invention is based on general purpose PC, low cost, volume
Little, safeguard simple, be easy to extension, the data utilizing local SATA hard disc simultaneously to process N days, make
Obtain data processing speed to be greatly promoted
Fig. 2 shows at a kind of based on general purpose PC the data radio astronomy data that the present invention proposes
Reason method flow diagram.As in figure 2 it is shown, the method first passes through distributed multiple stage general purpose PC
The different antenna data of carry carries out data process, and unit data process the bottleneck solving data transmission,
Data exchange can be carried out more efficiently and data process;Secondly data accumulation have employed based on 4.5Gb
INFINIBAND technology improve transfer function, data double-way transmission speed reaches 900MB/s;
In the method, every PC can process the antenna data received the most on the same day, the processing procedure of every
Including:
Step 1, the radio astronomy data receiving antenna carry out pretreatment, such as deduction bad point data
Deng, specifically include: according to Antenna Operation state recording daily record deduction do not work antenna data and by
The antenna data of striped disturbance;The number of corresponding time period inner question antenna is deducted according to time log file
According to.All of problem antenna includes: the interference signal produced in periphery human activity process, such as intercommunication
The station broadcast etc. in a distant place for machine, aircraft reflection, all may suppress within a certain period of time and really want to measure
Signal, additionally, the unusual condition that antenna element itself occurs at work, such as power-off, self-excitation
Unavailable Deng the signal being all likely to result in some node.Above-mentioned in order to avoid during processing in data
The interference of factor, it is necessary to reject.
Step 2, calculating threshold value flag, specifically for adding up the r.m.s. centrifugal pump of all aerial signals,
And calculate each each frequency of foundation line according to the r.m.s. centrifugal pump of all aerial signals added up
The standard deviation square value of phase place between the antenna two-by-two of baseline is constituted in rate passage whole day all time, as
Described threshold value flag;This standard deviation square value is calculated as below:
The definition of standard deviation isWherein V is current point in time, on current channel can
Diopter function, N is data point number.
Step 3, correction optical path difference, carry out school specifically according to reference value to the optical path difference of antenna two-by-two
Quasi-correction, it is thick that so-called reference value refers to that the experience according to instrument design and long-play averagely obtain
Optical path difference numerical value slightly;
Step 4, data file superposition, specifically add up each base-line data revised
Preserve after calculating.
Before flow process that data process is discussed in detail, first illustrate the form of data storage.Radio is looked in the distance
The direct measurement data of mirror is the correlated results of the signal on the antenna node at baseline two ends, described relevant
Result includes auto-correlation and cross correlation results.If there being Np antenna node, Np (Np-1)/2 just can be constituted
Bar baseline, adds autocorrelative data, has Np (Np+1)/2 mutually/autocorrelation calculation result altogether.
From the point of view of storing for file, generally will gather data and store according to baseline.Including autocorrelation
According to, total Np (Np+1)/2 data file.These file internal are divided into again some segments,
Each segment contains the related data of all passages that a period of time inner product is got.Therefore, often
The size of one file is equal to W*2*Nc*Nt, and wherein W is shared by each individual data
Byte number, for single precision floating datum, W=4, Nc are number of active lanes, and Nt is record in a day
Time point number, the factor 2 is corresponding to real and imaginary part.Additionally, separately there is a file storage
The clock time of each time point.
Introduce each step in detail below:
In step 1, in order to the antenna node information input data processing routine rejected will be needed, need
By antenna node numbering one text of write to be deducted, wait program automatically reads when running.
In step 2, the data file that data processor will read on disk one by one, then for often
One single passage, calculates the phase standard side between whole 24 hours interior two antennas
Difference.By the step for, all baselines, the standard deviation square value of all passages will be deposited as flag threshold value
Store up into a flag file.The size of this document is 4*Nc*Np (Np+1)/2Byte.
In step 3, data processor will process, to calculate each baseline by file one by one
Optical path difference correction data.When calculating optical path difference, for each cross-correlation data file, first
Read in the record in time file one by one, and read in belong to from current cross-correlation data file successively and work as
All frequency channel data of front time point, then go out current local sidereal hour angle according to Time Calculation
θ, and current cross-correlation data are inserted in a lattice point.Lattice point coordinate calculates according to equation below:
Wherein, w is the dimension of picture length of side in units of pixel, and f is antenna frequencies, fmaxFor instrument
The maximum of operating frequency
It follows that first suppose initial light path difference L, and utilize its related data to collecting
It is modified:
Wherein, V ' is revised related data, and V (f θ) is visibility function, for spatial coherence letter
Number, for represent antenna relevant to width export mutually.Revised numerical value is just a cancellation instrument light path
The observation data of difference, the namely two-dimension fourier transform of sky image, f is antenna frequencies, and θ is this
Ground sidereal hour angle.
Again the relevant dot array data through revising is carried out Fourier transform.Interfere the most former according to radio
Reason, this will obtain sky image.Owing to optical path difference now exists error, so the sky empty graph obtained
As the contrast of data is poor.I.e. become big by constantly adjusting L or diminish and repeat the above steps, making
The brightness obtaining point source the brightest in figure reaches maximum, and the correction of optical path difference just completes.Revise
During, need to read in the flag file that above-mentioned steps obtains, and the modulus value of visibility function is more than
The data setting threshold value certain multiple in flag file abandon.
Above-mentioned steps is implemented for all of baseline (without autocorrelation evidence), just completes Yi Tianguan
Survey the optical path difference correction of data.
Step 4, utilizes the optical path difference correction value of every day that above-mentioned steps obtains, the number to every day
According to being modified, then the data of many days are included original observed data, flag file, bad point data,
Optical path difference file is aggregated in superposition calculation machine.In superposition calculation machine, program will read by sky successively
Original cross-correlation data, optical path difference file, and abandon the data more than the defined threshold value of flag file.
The data being then passed through revising are added according to by baseline mode.Data after addition are the most directly counted
Calculate meansigma methods, but record the natural law that a summation is added.So contribute to eliminating in additive process
Round-off error.In this step, data accumulation have employed INFINIBAND technology based on 4.5Gb and carries
High-transmission function, data double-way transmission speed reaches 900MB/s.
The hardware configuration of the described general purpose PC used in the present invention is as shown in following table one:
Table one:
Processor | Intel core i7-3770@3.4GHz |
Internal memory | 16G |
Mainboard | MAHOBAY |
Mainboard hard-disk interface | SATA2 |
Hard disk | WD2002FAEX (7200 turns of caching 64MB) |
The most general method for reading data is roughly divided into local disk and directly reads and network reading two
The mode of kind, and the most the most frequently used local disk interface is divided into IDE, SATA, SCSI, optical-fibre channel
With SAS five kinds, ide interface hard disk is used in household products, and scsi interface hard disk is mainly used in
Server workstations, and optical-fibre channel is used in high-end server.Network reading manner typically passes through nfs
Agreement conducts interviews, and is restricted by network speed, and reading speed is slower.SATA is the most the most frequently used
A kind of hard-disk interface type.Under the big classification of IDE and SCSI, can separate again multiple concrete
Interface type, the most each has different technical specifications, possesses different transmission speeds, such as
ATA100 and SATA;Ultra160SCSI and Ultra320SCSI represents a kind of concrete hard
Dish interface, respective speed difference is the biggest.Hard-disk interface herein have employed stablize most general
SATA3.0 technology, the link theoretical velocity of SATA3.0 is at 600MB/s, due to by other software and hardware
The restriction of system, surveying speed under this paper hardware system is 140MB/s.And add NCQ
Number of instructions, towards needing the audio frequency of massive band width, Video Applications, it is ensured that the transmission of data syn-chronization.
Fig. 3 shows the structural representation that Computer system of the present invention configures.As it is shown on figure 3, should
System includes: machine-readable the fetching data of multiple pc carries out data handling system respectively, the data warp processed
Cross INFINIBAND technology and carry out data summarization.
The present invention is by 30 general purpose PC modes, every PC this locality carry one blocks of data simultaneously
Hard disk and a block system hard disk, can simultaneously parallel processing 30 day data.
The present invention proposes to realize the system of said method can carry out computing, often to 40 groups of antennas of thing
It file produced is 780 files, runs said procedure and calculate the data of a day under described system
Needing 5 day time, if carrying out computing with 30 PC, 5 day time can run 30 day data,
And under SGI work station, calculating 1 day data needs 2 day time, 30 day data need 15 days fortune
Calculating, efficiency is far below the computing of general purpose PC.
Particular embodiments described above, is carried out the purpose of the present invention, technical scheme and beneficial effect
Further describe it should be understood that the foregoing is only the specific embodiment of the present invention,
Be not limited to the present invention, all within the spirit and principles in the present invention, any amendment of being made,
Equivalent, improvement etc., should be included within the scope of the present invention.
Claims (7)
1. a radio astronomy data processing method based on general purpose PC, wherein the method is by multiple stage
General purpose PC performs to process radio astronomy data the most on the same day respectively, and the method includes:
Step 1: the radio astronomy data receiving antenna carry out pretreatment;
Step 2: according to the pretreated all baselines of radio astronomy data statistics, the mark of all passages
Quasi-variance yields;
Step 3: the optical path difference of antenna two-by-two every day is calibrated according to obtained standard deviation square value
Revise;
Step 4: according to radio astronomy data to every day of the optical path difference of revised antenna two-by-two every day
It is modified, and the revised radio astronomy data that the process of multiple stage general purpose PC obtains are folded
Add and finally processed data;
The optical path difference of one baseline of correction specific as follows in step 3, this baseline is by described sky two-by-two
Line is constituted:
Step 31: for each cross-correlation data file, successively from current cross-correlation data file
Read in all frequency channel data belonging to current point in time, then go out current basis according to Time Calculation
Ground sidereal hour angle θ, and current cross-correlation data are inserted in a lattice point;
Step 32: assuming that an initial light path difference, and utilize it that related data is modified;
Step 33: revised related data is carried out Fourier transformation, obtains sky image;
Step 34: adjust described optical path difference, and go to step 32 so that the brightest in described sky image
Point source brightness reach maximum, complete the correction of optical path difference.
2. radio astronomy data processing method as claimed in claim 1, wherein, pre-in step 1
Process the bad point data specifically included in the radio astronomy data that deduction receives.
3. radio astronomy data processing method as claimed in claim 2, wherein, deducts bad point number
According to specifically including: deduct the data of the antenna that do not works and by bar according to Antenna Operation state recording daily record
The antenna data of stricture of vagina disturbance;The data of corresponding time period inner question antenna are deducted according to time log file.
4. radio astronomy data processing method as claimed in claim 1, wherein, will in step 3
The visibility function modulus value of antenna abandons more than the data of flag threshold value prearranged multiple, wherein said visually
Degree function is spatial coherence function, for represent antenna relevant to width export mutually.
5. radio astronomy data processing method as claimed in claim 4, wherein, described flag threshold
Value is the phase standard variance yields that in each passage, antenna is interior at 24 hours two-by-two.
6. radio astronomy data processing method as claimed in claim 4, wherein, described method by
30 general purpose PCs perform, every PC this locality carry one blocks of data hard disk and a block system simultaneously
Hard disk, parallel processing simultaneously 30 day data.
7. the radio astronomy data processing method as described in any one of claim 1-6, wherein, step
In 4, data investigation uses INFINIBAND technology based on 4.5Gb to realize.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410055134.3A CN103810140B (en) | 2014-02-18 | 2014-02-18 | A kind of radio astronomy data processing method based on general purpose PC |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410055134.3A CN103810140B (en) | 2014-02-18 | 2014-02-18 | A kind of radio astronomy data processing method based on general purpose PC |
Publications (2)
Publication Number | Publication Date |
---|---|
CN103810140A CN103810140A (en) | 2014-05-21 |
CN103810140B true CN103810140B (en) | 2017-01-04 |
Family
ID=50706930
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410055134.3A Expired - Fee Related CN103810140B (en) | 2014-02-18 | 2014-02-18 | A kind of radio astronomy data processing method based on general purpose PC |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN103810140B (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107291751A (en) * | 2016-04-01 | 2017-10-24 | 中兴通讯股份有限公司 | A kind of chronometer data information processing method and device |
CN107967407B (en) * | 2017-12-19 | 2021-05-04 | 中国科学院上海天文台 | Radio astronomical data processing method |
CN110175313B (en) * | 2019-05-24 | 2020-07-14 | 中国科学院国家天文台 | Astronomical sky-patrol data processing method, system and storage medium |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103324812A (en) * | 2013-07-10 | 2013-09-25 | 中国科学院国家天文台 | Method for simulating space astronomy cosmic ray observation image |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8024152B2 (en) * | 2008-09-23 | 2011-09-20 | Microsoft Corporation | Tensor linear laplacian discrimination for feature extraction |
-
2014
- 2014-02-18 CN CN201410055134.3A patent/CN103810140B/en not_active Expired - Fee Related
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103324812A (en) * | 2013-07-10 | 2013-09-25 | 中国科学院国家天文台 | Method for simulating space astronomy cosmic ray observation image |
Non-Patent Citations (2)
Title |
---|
Data processing software for radio astronomy;Huib Jan van Langevelde;《The 8th European VLBI Network Symposium》;20060926;第1-8页 * |
甚长基线干涉测量技术在深空导航中的应用;蒋栋荣 等;《科学》;20080125;第60卷(第1期);第10-14页 * |
Also Published As
Publication number | Publication date |
---|---|
CN103810140A (en) | 2014-05-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Allen et al. | The Einstein@ Home search for radio pulsars and PSR J2007+ 2722 discovery | |
US20180231636A1 (en) | Increasing performance of a receive pipeline of a radar with memory optimization | |
CN102279386B (en) | SAR (Synthetic Aperture Radar) imaging signal processing data transposing method based on FPGA (Field Programmable Gata Array) | |
CN103810140B (en) | A kind of radio astronomy data processing method based on general purpose PC | |
De et al. | A real-time coherent dedispersion pipeline for the giant metrewave radio telescope | |
US11454697B2 (en) | Increasing performance of a receive pipeline of a radar with memory optimization | |
Naidu et al. | PONDER-A Real time software backend for pulsar and IPS observations at the Ooty Radio Telescope | |
Wang et al. | Distributed data-processing pipeline for mingantu ultrawide spectral radioheliograph | |
CN110082822A (en) | The method for carrying out earthquake detection using convolutional neural networks | |
Sun et al. | A robust RFI identification for radio interferometry based on a convolutional neural network | |
Liu et al. | Assessment of the X-and C-band polarimetric SAR data for plastic-mulched farmland classification | |
van der Merwe et al. | Low-cost COTS GNSS interference monitoring, detection, and classification system | |
Jian et al. | Comparative analysis of different empirical mode decomposition-kind algorithms on sea-level inversion by GNSS-MR | |
CN102930151A (en) | Method for simulating infrared detection system effect in real time based on texture | |
CN103116173A (en) | Error test device for photoelectric tracking | |
Megrey et al. | Past, present and future trends in the use of computers in fisheries research | |
Verkhodanov et al. | Cosmological Evolution of Average Continuum Spectra of Radio Sources at Z> 2 Redshifts | |
Li et al. | Mountain top-based atmospheric radio occultation observations with open/closed loop tracking: experiment and validation | |
Xie et al. | Advancements in spaceborne synthetic aperture radar imaging with system-on-chip architecture and system fault-tolerant technology | |
CN104748703A (en) | Leaf area index (LAI) downscaling method and system | |
Thomasson et al. | 3C 459: a highly asymmetric radio galaxy with a starburst | |
CN109470269A (en) | Scaling method, calibration facility and the calibration system of extraterrestrial target measuring mechanism | |
You et al. | Research on Multilevel Filtering Algorithm Used for Denoising Strong and Weak Beams of Daytime Photon Cloud Data with High Background Noise | |
CN103903272B (en) | A kind of StaMPS Algorithm parallelization processing method based on Hadoop | |
Yang et al. | Improving Typical Urban Land-Use Classification with Active-Passive Remote Sensing and Multi-Attention Modules Hybrid Network: A Case Study of Qibin District, Henan, China |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170104 Termination date: 20180218 |
|
CF01 | Termination of patent right due to non-payment of annual fee |