CN112579696B - Multi-point synchronous copying method, device, server and storage medium for radio astronomical data - Google Patents
Multi-point synchronous copying method, device, server and storage medium for radio astronomical data Download PDFInfo
- Publication number
- CN112579696B CN112579696B CN202011344618.1A CN202011344618A CN112579696B CN 112579696 B CN112579696 B CN 112579696B CN 202011344618 A CN202011344618 A CN 202011344618A CN 112579696 B CN112579696 B CN 112579696B
- Authority
- CN
- China
- Prior art keywords
- data
- task
- storage
- replication
- synchronous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 103
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000010076 replication Effects 0.000 claims abstract description 93
- 230000005540 biological transmission Effects 0.000 claims abstract description 38
- 238000013507 mapping Methods 0.000 claims abstract description 13
- 238000013500 data storage Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000008447 perception Effects 0.000 claims description 4
- 230000005055 memory storage Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000006854 communication Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- VQLYBLABXAHUDN-UHFFFAOYSA-N bis(4-fluorophenyl)-methyl-(1,2,4-triazol-1-ylmethyl)silane;methyl n-(1h-benzimidazol-2-yl)carbamate Chemical compound C1=CC=C2NC(NC(=O)OC)=NC2=C1.C=1C=C(F)C=CC=1[Si](C=1C=CC(F)=CC=1)(C)CN1C=NC=N1 VQLYBLABXAHUDN-UHFFFAOYSA-N 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a method, a device, a server and a storage medium for multi-point synchronous replication of radio astronomical data, wherein the method comprises the following steps: radio astronomical data is collected through a receiver and is filled into a buffer area, and after a first preset percentage of the size of the buffer area is approximate, the data is sent to a storage client; in the data transmission process, starting a corresponding multi-task synchronous replication module according to a storage point which needs to be synchronously replicated; performing memory address space mapping between the acquisition server and the storage client; after the data of each task is copied, notifying each task through a consistency guarantee module so as to make each task contrast to a time synchronization window; calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not; and calculating the number of tasks which are successfully copied synchronously, and judging whether the multi-point synchronous copying task is successful or not. The invention can solve the problem of real-time copying of radio astronomical data from the acquisition server to the multipoint storage client side in a high-speed synchronous mode.
Description
Technical Field
The invention relates to a multi-point synchronous replication method, a device, a server and a storage medium for radio astronomical data, belonging to the technical fields of astronomical technology and computer application.
Background
Radio astronomy is an important research field of astronomy, and since the last 30 th century, the blackbody spectrum characteristics of pulsar, universe microwave background, pulsar double star and universe microwave background radiation are closely related to radio astronomy. The next generation radio telescope represented by SKA has extremely high sensitivity (50 times higher than JVA sensitivity and 10000 times higher searching speed), extremely high spatial resolution is obtained by a base line of thousands kilometers, a fine time structure is obtained by nanosecond sampling, and data exceeding the total amount of the global Internet is generated at a rate of 10 Pb/s. The SKA taking wide view field, multiple beams, high dynamic, high resolution and big data as core concepts overturns the traditional research means of radio astronomy, brings revolutionary brand new concepts to astronomy research, and also brings great challenges to the storage of massive astronomical radio telescope data, wherein the huge problems are brought by multi-point data replication from an observation end to a data storage end.
The synchronous replication of radio astronomical observation data is required to save one part of collected data on a data archiving server and a real-time computing server, so that a great amount of time cost can be saved, and the on-site data processing is possible. In this way, how to ensure that reliable mass radio astronomical observation data are synchronously copied at high speed at different storage ends is a problem and urgent need faced by each large radio telescope.
Synchronous replication software and systems in this field currently rely on traditional TCP/IP transmission methods, or use copying, remote data transmission. While real-time synchronous copying of data is not possible with conventional techniques and schemes. With the increase of the data volume, the delay of synchronous replication is higher and higher, and the requirement of rapid synchronous replication of real-time big data is not met.
Disclosure of Invention
In view of the above, the present invention provides a method, an apparatus, a server and a storage medium for multi-point synchronous replication of radio astronomical data, which can solve the problem of real-time replication of radio astronomical data from an acquisition server to a multi-point storage client in high-speed synchronous replication.
A first object of the present invention is to provide a method for multi-point synchronous replication of radioastronomical data.
A second object of the present invention is to provide a radio astronomical data multipoint synchronous copying device.
A third object of the present invention is to provide a system for multi-point synchronous replication of radioastronomical data.
A fourth object of the present invention is to provide an acquisition server.
A fifth object of the present invention is to provide a storage medium.
The first object of the present invention can be achieved by adopting the following technical scheme:
a method of multi-point synchronous replication of radioastronomical data, the method comprising:
radio astronomical data is collected through a receiver, and the radio astronomical data is filled into a buffer area of a memory or a nonvolatile memory according to configuration requirements;
when the data size is close to the first preset percentage of the buffer area size, sending the data to a storage client;
in the data transmission process, starting a corresponding multi-task synchronous replication module according to a storage point which needs to be synchronously replicated; each task is responsible for data synchronization of one storage client;
calling a high-speed data transmission system through a multi-task synchronous copying module, and mapping a memory address space between an acquisition server and a storage client;
after the data of each task is copied, notifying each task through a consistency guarantee module so as to make each task contrast to a time synchronization window;
calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not;
and calculating the number of tasks which are successfully copied synchronously, if the number of the tasks is larger than a second preset percentage of the number of the tasks which are copied synchronously, judging that the tasks which are copied synchronously at the multiple points are successful, otherwise, judging that the tasks which are copied synchronously at the multiple points are failed.
Further, the calculating the time for completing the copying data of each task, and judging whether the task synchronous copying is successful, specifically includes:
and calculating the completion time of the replication data of each task, if the difference between the reference value and the completion time of the replication data of the task is smaller than the tolerable time difference, judging that the synchronous replication of the task is successful, otherwise, judging that the synchronous replication of the task is failed.
Further, the reference value is set as: the value obtained by subtracting the task starting time from the notification time of the consistency assurance module.
Further, the acquiring radio astronomical data by the receiver, before filling the radio astronomical data into the buffer area of the memory or the nonvolatile memory according to the configuration requirement, further includes:
configuring a storage client network address and a specific synchronous folder of synchronous replication;
configuring the number of tasks to be synchronously copied;
configuring a mode of temporarily storing radio astronomical data locally; the local temporary storage mode comprises memory storage and nonvolatile storage;
the buffer size is preset.
Further, the acquiring radio astronomical data by the receiver, before filling the radio astronomical data into the buffer area of the memory or the nonvolatile memory according to the configuration requirement, further includes:
checking the time of the acquisition server and the storage client, and prompting and waiting for a user to synchronously calibrate the time of the acquisition server and the storage client if the time is larger than a preset time threshold.
Furthermore, each task registers task service to the consistency ensuring module, and simultaneously, each task also serves as a subscriber role, and perception of task progress is realized by using the consistency ensuring module.
The second object of the invention can be achieved by adopting the following technical scheme:
a radioastronomical data multipoint synchronous replication device, the device comprising:
the filling unit is used for acquiring radio astronomical data through the receiver and filling the radio astronomical data into a buffer area of the memory or the nonvolatile memory according to configuration requirements;
the sending unit is used for sending the data to the storage client after the data size is close to the preset percentage of the buffer area size;
the starting unit is used for starting the corresponding multi-task synchronous copying module according to the storage point which needs to be synchronously copied in the data transmission process; each task is responsible for data synchronization of one of the storage clients;
the calling unit is used for calling the high-speed data transmission subsystem through the multi-task synchronous copying module and mapping the memory address space between the acquisition server and the storage client;
the notification unit is used for notifying each task through the consistency guarantee module after the data replication of each task is completed, so that each task is compared with the time synchronization window;
the first calculation unit is used for calculating the completion time of the replication data of each task and judging whether the task synchronous replication is successful or not;
and the second calculation unit is used for calculating the number of tasks which are successfully synchronously copied, judging that the multi-point synchronous copying task is successful if the number of the tasks is larger than a second preset percentage of the number of the synchronous copying tasks, and judging that the multi-point synchronous copying task fails if the number of the tasks is not larger than the second preset percentage of the number of the synchronous copying tasks.
The third object of the present invention can be achieved by adopting the following technical scheme:
the system comprises a radio telescope observation system, a distributed storage system and a high-speed data transmission system, wherein the radio telescope observation system comprises a radio antenna array, a receiver and an acquisition server, the distributed storage system comprises a storage client and a data storage server, the radio antenna array, the receiver and the acquisition server are sequentially connected, the storage client is connected with the data storage server, and the acquisition server is connected with the storage client through the high-speed data transmission system;
the acquisition server is used for executing the radio astronomical data multipoint synchronous replication method.
The fourth object of the present invention can be achieved by adopting the following technical scheme:
the acquisition server comprises a processor and a memory for storing a program executable by the processor, wherein the processor realizes the radio astronomical data multipoint synchronous replication method when executing the program stored by the memory.
The fifth object of the present invention can be achieved by adopting the following technical scheme:
a storage medium storing a program which, when executed by a processor, implements the above-described radioastronomical data multipoint synchronous replication method.
Compared with the prior art, the invention has the following beneficial effects:
the invention realizes the high-speed synchronous replication of the acquired data based on the nonvolatile memory and the consistency guarantee module, is simple to realize, improves the real-time performance of the multipoint replication by utilizing the high speed and the reliability of the nonvolatile memory, realizes the multipoint task synchronization by the consistency guarantee module, is more reliable than the traditional synchronization independent of the consistency guarantee module, and is possible to guarantee the reliability of the data at the acquisition server and the storage client based on the RDMA distributed data access mode of the nonvolatile memory; in addition, the multipoint synchronous replication enables the same data to have backup among different storage clients, and the safety of data acquisition is further improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a block diagram of a radio astronomical data multipoint synchronous replication system according to embodiment 1 of the present invention.
Fig. 2 is a schematic diagram of memory address space mapping between an acquisition server and a storage client according to embodiment 1 of the present invention.
Fig. 3 is a flowchart of a method for multi-point synchronous replication of radio astronomical data in embodiment 1 of the present invention.
Fig. 4 is a block diagram of a multi-point synchronous replication device for radio astronomical data according to embodiment 2 of the present invention.
Fig. 5 is a block diagram of the configuration of the acquisition server according to embodiment 3 of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
Example 1:
as shown in fig. 1, this embodiment provides a multi-point synchronous replication system of radio astronomical data, which includes a radio telescope observation system 101, a distributed storage system 102 and a high-speed data transmission system, where the radio telescope observation system 101 includes a radio antenna array 1011, a receiver 1012 and an acquisition server 1013, the acquisition server 1013 is a server based on a Linux platform, the distributed storage system 102 is a Lustre distributed file system, which includes a storage client 1021 and a data storage server 1022, optionally, the distributed storage system 102 further includes a metadata server, the radio antenna array 1011, the receiver 1012 and the acquisition server 1013 are sequentially connected, the storage client 1021 is connected with the data storage server 1022, the acquisition server 1013 is connected with the storage client 1021 through the high-speed data transmission system, where the number of the acquisition servers 1013 is N, and the number of the storage clients is M.
Further, the high-speed data transmission system includes a sending module, a receiving module, and a high-speed data transmission protocol, where the sending module and the receiving module are both communication modules based on a nonvolatile memory, where the nonvolatile memory of the sending module is shown as 103 in fig. 1, the sending module is disposed on the collecting server 1013, the nonvolatile memory of the receiving module is shown as 104 in fig. 1, the receiving module is disposed on the storage client 1021, and the sending module and the receiving module communicate through the data transmission protocol, where the high-speed data transmission protocol can select RoCE, and the nonvolatile memory is currently marketed and has advantages that the conventional memory does not possess: 1) High performance; 2) And data is not lost after power failure.
The embodiment also provides a multi-point synchronous replication method of radio astronomical data, and before the method is realized, the following core software and hardware are installed:
1) And installing a nonvolatile memory bank on the acquisition server.
2) And installing a high-speed transmission network card on the acquisition server.
3) And installing radio astronomical data acquisition software on the acquisition server.
4) A nonvolatile memory bank is installed on a storage client.
5) And installing a high-speed transmission network card on the storage client.
6) The high-speed transmission cable is connected with the acquisition server and the storage client.
In this embodiment, the nonvolatile memory bank is an Intel Optane nonvolatile memory bank, the high-speed transmission network card is a 10GbE high-speed transmission network card, the nonvolatile memory bank and the high-speed transmission network card installed on the acquisition server form a sending module, and the nonvolatile memory bank and the high-speed transmission network card installed on the storage client form a receiving module.
The mapping of the non-volatile memory space between the acquisition server and the storage client is as shown in fig. 2: comprises the following core parts, 201: collecting an NVM address space of a server, wherein the NVM is available memory space; 202: collecting an address space of a server application program; 203: RDMA address space, communication module address space; 204: storing an address space of the client application; 205: the client NVM address space is stored.
The communication process is as follows: 1) The acquisition server associates the required NVM address space with the address space of the acquisition server application program through the NVM.PM.FILE, thereby forming the required Mapped File; 2) The storage client associates the NVM address space with the address space of the storage client application via the nvm.pm.file, thereby forming a Mapped File of the storage File. 3) The RDMA address space is used for associating the memory address space of the acquisition server and the memory address space of the memory client through the Mapped File, so that the remote address mapping of the acquisition server and the memory client is completed, and the method is a basic environment for realizing the multi-point synchronous copying method of radio astronomical data.
As shown in fig. 3, the method for multi-point synchronous replication of radio astronomical data according to the present embodiment is implemented based on the above-mentioned acquisition server, and includes the following steps:
s301, acquiring radio astronomical data through a receiver, and filling the radio astronomical data into a buffer area of a memory or a nonvolatile memory according to configuration requirements.
1) And starting the acquisition server and starting the distributed storage system.
2) Storage clients that mount distributed storage systems on collection servers are typically accessible based on POSIX or a network file system.
3) And installing a receiver driver on the acquisition server.
4) And starting acquisition software on an acquisition server.
5) Checking the time of the acquisition server and the storage client, and prompting and waiting for a user to synchronously calibrate the time of the acquisition server and the storage client if the time is larger than a preset time threshold.
6) Loading a consistency guarantee module in acquisition software, wherein the consistency guarantee module is realized based on distributed consistency frameworks such as ETCD/Zookeeper, the specific implementation can be independent of the frameworks, and as long as a subscription and release mode can be completed, a latch perception mechanism similar to the Zookeeper can be performed, and the consistency guarantee module in the embodiment is realized based on ETCD; loading a nonvolatile memory management module based on PMDK provided by Intel; the high-speed data transmission system is loaded, and the module is realized based on RoCe and simplifies RDMA deployment on the Ethernet.
7) The following configuration is carried out through a configuration module of the acquisition software: a) Configuring a network address of a storage client terminal of synchronous replication and a specific synchronous folder, b) the number s of tasks to be synchronously replicated, s=3 in the embodiment; c) The configuration data is temporarily stored locally: the memory is also a nonvolatile memory, if the memory is stored in the nonvolatile memory, the security requirement of the data is high, and if the memory is stored in the memory, the loss of the data can be tolerated; d) The buffer size T is preset, in this embodiment t=128 GB.
After the operation is finished, the acquisition software acquires antenna data, namely radio astronomical data, through the acquisition receiver, and fills the antenna data into a buffer area of the memory or the nonvolatile memory according to configuration requirements.
S302, after the size of the data quantity is close to the first preset percentage of the size of the buffer area, the data is sent to the storage client.
Specifically, the first preset percentage is 80%, after the data size is close to 80% of the buffer area 128GB, the data is sent to the storage client, and the collection software starts to call the sending module to push the data from the high-speed network interface of the collection server to the storage client through the high-speed transmission cable.
S303, starting a corresponding multi-task synchronous copying module according to a storage point which needs to be synchronously copied in the data transmission process.
The storage points needing synchronous replication indicate that a plurality of storage points need synchronous replication (multi-point synchronous replication), the number of tasks is 3 according to the above configuration, and each task is responsible for data synchronization of one storage client.
S304, calling a high-speed data transmission system through a multi-task synchronous replication module, and mapping a memory address space between the acquisition server and the storage client.
Specifically, the high-speed data transmission system is called through the multi-task synchronous replication module, and memory address space mapping is performed between the acquisition server and the storage client by utilizing the characteristic that data is not lost after the nonvolatile memory is powered down, as shown in fig. 2.
And S305, after the data replication of each task is completed, notifying each task through a consistency assurance module so as to make each task contrast to a time synchronization window.
Specifically, each task registers task service to the consistency security module, and at the same time, each task also serves as a subscriber, perception of task progress is achieved by using a subscription and release mechanism of the consistency security module, when data replication of each task is completed, a watch mechanism of the consistency security module triggers to notify each task, each task receiving notification is against a time synchronization window W, and then calculation in steps S306 and S307 is performed, wherein w=2 seconds in this embodiment.
S306, calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not.
The reference value B is set as: the value obtained by subtracting the task start time from the notification time of the consistency assurance module, i.e., b=notification time-start time.
Calculating the completion time t of the replication data of the task i i If the reference value B-t i <200 ms, 200 ms is the tolerable time difference of this embodiment, the task is synchronous with the reference task, and it is judged that the task i is successfully copied synchronously, otherwise, it is judged that the task i is failed to be copied synchronously, and the synchronous copying situation of other tasks can be obtained by repeating the method.
S307, calculating the number of tasks which are successfully copied synchronously, if the number of tasks is larger than a second preset percentage of the number of tasks which are copied synchronously, judging that the tasks which are copied synchronously at the multiple points are successful, otherwise, judging that the tasks which are copied synchronously at the multiple points are failed.
And if the number of the tasks which are successfully copied in the synchronization is more than 50 percent (half) of the number of the tasks which are successfully copied in the synchronization, judging that the multi-point synchronous copying task is successful, otherwise, judging that the multi-point synchronous copying task is failed.
After the multipoint synchronous replication task is successful, the storage client side is stated to successfully receive data, the data is firstly cached locally through the nonvolatile memory, then the data is stored on a storage node according to the data persistence module of the storage client side of each distributed storage system, and thus the nonvolatile memory and the consistency assurance module are utilized to finish the high-reliability storage of the radio astronomical data at one time; and repeating the steps S301 to S307, so that the multipoint synchronous replication of continuous high-speed data acquisition can be realized.
It should be noted that although the method operations of the above embodiments are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in that particular order or that all illustrated operations be performed in order to achieve desirable results. Rather, the depicted steps may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
Example 2:
as shown in fig. 4, the embodiment provides a multi-point synchronous replication device of radio astronomical data, which is applied to an acquisition server and comprises a filling unit 401, a sending unit 402, a starting unit 403, a calling unit 404, a notification unit 405, a first calculating unit 406 and a second calculating unit 407, wherein the specific functions of the units are as follows:
and the filling unit 401 is used for collecting radio astronomical data through the receiver and filling the radio astronomical data into a buffer area of the memory or the nonvolatile memory according to configuration requirements.
A sending unit 402, configured to send data to the storage client after the size of the data volume approaches a preset percentage of the size of the buffer.
A starting unit 403, configured to start a corresponding multi-task synchronous replication module according to a storage point that needs to be synchronously replicated in a data sending process; wherein each task is responsible for data synchronization of one of the storage clients.
And the calling unit 404 is used for calling the high-speed data transmission subsystem through the multi-task synchronous replication module and mapping the memory address space between the acquisition server and the storage client.
And a notification unit 405, configured to notify each task through the consistency assurance module after the data replication of each task is completed, so that each task is against the time synchronization window.
The first calculating unit 406 is configured to calculate a replication data completion time of each task, and determine whether the task synchronous replication is successful.
The second calculating unit 407 is configured to calculate the number of tasks that are successfully synchronously replicated, and if the number of tasks is greater than a second preset percentage of the number of tasks that are synchronously replicated, determine that the multi-point synchronous replication task is successful, and if not, determine that the multi-point synchronous replication task fails.
Specific implementation of each unit in this embodiment may be referred to embodiment 1, and will not be described in detail herein; it should be noted that, in the system provided in this embodiment, only the division of the above functional units is used as an example, in practical application, the above functional allocation may be performed by different functional units according to needs, that is, the internal structure is divided into different functional units, so as to perform all or part of the functions described above.
It will be understood that the terms first, second, etc. used in the above devices may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, a first computing unit may be referred to as a second computing unit, and similarly, a second computing unit may be referred to as a first computing unit, both of which are computing units, but not the same computing unit, without departing from the scope of the invention.
Example 3:
as shown in fig. 5, this embodiment provides an acquisition server, which includes a processor 502, a memory and a network interface 503 connected by a system bus 501, where the processor is used to provide computing and control capabilities, the memory includes a nonvolatile storage medium 504 and an internal memory 505, where the nonvolatile storage medium 504 stores a computer program and a database, and the internal memory 505 provides an environment for running the computer program in the nonvolatile storage medium, and when the processor 502 executes the computer program stored in the memory, the foregoing method for multi-point synchronous copying of radio astronomical data according to embodiment 1 is implemented as follows:
radio astronomical data is collected through a receiver, and the radio astronomical data is filled into a buffer area of a memory or a nonvolatile memory according to configuration requirements;
when the data size is close to the first preset percentage of the buffer area size, sending the data to a storage client;
in the data transmission process, starting a corresponding multi-task synchronous replication module according to a storage point which needs to be synchronously replicated; each task is responsible for data synchronization of one storage client;
calling a high-speed data transmission system through a multi-task synchronous copying module, and mapping a memory address space between an acquisition server and a storage client;
after the data of each task is copied, notifying each task through a consistency guarantee module so as to make each task contrast to a time synchronization window;
calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not;
and calculating the number of tasks which are successfully copied synchronously, if the number of the tasks is larger than a second preset percentage of the number of the tasks which are copied synchronously, judging that the tasks which are copied synchronously at the multiple points are successful, otherwise, judging that the tasks which are copied synchronously at the multiple points are failed.
Example 4:
the present embodiment provides a storage medium, which is a computer readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the radioastronomical data multipoint synchronous replication method of the above embodiment 1, as follows:
radio astronomical data is collected through a receiver, and the radio astronomical data is filled into a buffer area of a memory or a nonvolatile memory according to configuration requirements;
when the data size is close to the first preset percentage of the buffer area size, sending the data to a storage client;
in the data transmission process, starting a corresponding multi-task synchronous replication module according to a storage point which needs to be synchronously replicated; each task is responsible for data synchronization of one storage client;
calling a high-speed data transmission system through a multi-task synchronous copying module, and mapping a memory address space between an acquisition server and a storage client;
after the data of each task is copied, notifying each task through a consistency guarantee module so as to make each task contrast to a time synchronization window;
calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not;
and calculating the number of tasks which are successfully copied synchronously, if the number of the tasks is larger than a second preset percentage of the number of the tasks which are copied synchronously, judging that the tasks which are copied synchronously at the multiple points are successful, otherwise, judging that the tasks which are copied synchronously at the multiple points are failed.
The computer readable storage medium of the present embodiment may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In this embodiment, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present embodiment, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with a computer-readable program embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable storage medium may be written in one or more programming languages, including an object oriented programming language such as Java, python, C ++ and conventional procedural programming languages, such as the C-language or similar programming languages, or combinations thereof for performing the present embodiments. The program may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
In summary, the invention realizes the high-speed synchronous replication of the acquired data based on the nonvolatile memory and the consistency guarantee module, is simple to realize, improves the real-time performance of multipoint replication by utilizing the high speed and the reliability of the nonvolatile memory, realizes the multipoint task synchronization by the consistency guarantee module, is more reliable than the traditional synchronization independent of the consistency guarantee module, and is possible to guarantee the reliability of the data at both the acquisition server and the storage client based on the RDMA distributed data access mode of the nonvolatile memory; in addition, the multipoint synchronous replication enables the same data to have backup among different storage clients, and the safety of data acquisition is further improved.
The above-mentioned embodiments are only preferred embodiments of the present invention, but the protection scope of the present invention is not limited thereto, and any person skilled in the art can make equivalent substitutions or modifications according to the technical solution and the inventive concept of the present invention within the scope of the present invention disclosed in the present invention patent, and all those skilled in the art belong to the protection scope of the present invention.
Claims (8)
1. A method for multi-point synchronous replication of radio astronomical data, the method comprising:
radio astronomical data is collected through a receiver, and the radio astronomical data is filled into a buffer area of a memory or a nonvolatile memory according to configuration requirements;
when the data size reaches a first preset percentage of the buffer area size, sending the data to a storage client;
in the data transmission process, starting a corresponding multi-task synchronous replication module according to a storage point which needs to be synchronously replicated; each task is responsible for data synchronization of one storage client;
calling a high-speed data transmission system through a multi-task synchronous copying module, and mapping a memory address space between an acquisition server and a storage client;
after the data of each task is copied, notifying each task through a consistency guarantee module so as to make each task contrast to a time synchronization window;
calculating the completion time of the replication data of each task, and judging whether the task synchronous replication is successful or not;
calculating the number of tasks which are successfully copied synchronously, if the number of tasks is larger than a second preset percentage of the number of the tasks which are copied synchronously, judging that the tasks which are copied synchronously at multiple points are successful, otherwise, judging that the tasks which are copied synchronously at multiple points are failed;
the method comprises the steps of collecting radio astronomical data through a receiver, and filling the radio astronomical data into a buffer area of a memory or a nonvolatile memory according to configuration requirements, and further comprising:
checking the time of the acquisition server and the storage client, and prompting and waiting for a user to synchronously calibrate the time of the acquisition server and the storage client if the time is greater than a preset time threshold;
configuring a storage client network address and a specific synchronous folder of synchronous replication;
configuring the number of tasks to be synchronously copied;
configuring a mode of temporarily storing radio astronomical data locally; the local temporary storage mode comprises memory storage and nonvolatile storage;
the buffer size is preset.
2. The method for multipoint synchronous replication of radio astronomical data according to claim 1, wherein the calculating the replication data completion time of each task, judging whether the task synchronous replication is successful, specifically comprises:
and calculating the completion time of the replication data of each task, if the difference between the reference value and the completion time of the replication data of the task is less than 200 milliseconds, judging that the synchronous replication of the task is successful, otherwise, judging that the synchronous replication of the task is failed.
3. The radioastronomical data multipoint synchronous replication method according to claim 2, wherein the reference value is set as: the value obtained by subtracting the task starting time from the notification time of the consistency assurance module.
4. A method of multipoint synchronous replication of radio astronomical data according to any one of claims 1-3, characterized in that each task registers task services with a consistency assurance module, and each task also acts as a subscriber role, and the consistency assurance module is utilized to realize perception of task progress.
5. A radioastronomical data multipoint synchronous replication device, characterized in that it comprises:
the filling unit is used for acquiring radio astronomical data through the receiver and filling the radio astronomical data into a buffer area of the memory or the nonvolatile memory according to configuration requirements;
the sending unit is used for sending the data to the storage client after the data size reaches a first preset percentage of the buffer area size;
the starting unit is used for starting the corresponding multi-task synchronous copying module according to the storage point which needs to be synchronously copied in the data transmission process; each task is responsible for data synchronization of one storage client;
the calling unit is used for calling the high-speed data transmission system through the multi-task synchronous copying module and mapping the memory address space between the acquisition server and the storage client;
the notification unit is used for notifying each task through the consistency guarantee module after the data replication of each task is completed, so that each task is compared with the time synchronization window;
the first calculation unit is used for calculating the completion time of the replication data of each task and judging whether the task synchronous replication is successful or not;
the second calculation unit is used for calculating the number of tasks which are successfully synchronously copied, judging that the multi-point synchronous copying task is successful if the number of the tasks is larger than a second preset percentage of the number of the synchronous copying tasks, and judging that the multi-point synchronous copying task fails if the number of the tasks is not larger than the second preset percentage of the number of the synchronous copying tasks;
the method comprises the steps of collecting radio astronomical data through a receiver, and filling the radio astronomical data into a buffer area of a memory or a nonvolatile memory according to configuration requirements, and further comprising:
checking the time of the acquisition server and the storage client, and prompting and waiting for a user to synchronously calibrate the time of the acquisition server and the storage client if the time is greater than a preset time threshold;
configuring a storage client network address and a specific synchronous folder of synchronous replication;
configuring the number of tasks to be synchronously copied;
configuring a mode of temporarily storing radio astronomical data locally; the local temporary storage mode comprises memory storage and nonvolatile storage;
the buffer size is preset.
6. The system is characterized by comprising a radio telescope observation system, a distributed storage system and a high-speed data transmission system, wherein the radio telescope observation system comprises a radio antenna array, a receiver and an acquisition server, the distributed storage system comprises a storage client and a data storage server, the radio antenna array, the receiver and the acquisition server are sequentially connected, the storage client is connected with the data storage server, and the acquisition server is connected with the storage client through the high-speed data transmission system;
the acquisition server is used for executing the radio astronomical data multipoint synchronous replication method according to any one of claims 1-4.
7. An acquisition server comprising a processor and a memory for storing a program executable by the processor, wherein the processor, when executing the program stored in the memory, implements the radioastronomical data multipoint synchronous replication method according to any one of claims 1-4.
8. A computer-readable storage medium storing a program, wherein the program, when executed by a processor, implements the radioastronomical data multipoint synchronous replication method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011344618.1A CN112579696B (en) | 2020-11-26 | 2020-11-26 | Multi-point synchronous copying method, device, server and storage medium for radio astronomical data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011344618.1A CN112579696B (en) | 2020-11-26 | 2020-11-26 | Multi-point synchronous copying method, device, server and storage medium for radio astronomical data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112579696A CN112579696A (en) | 2021-03-30 |
| CN112579696B true CN112579696B (en) | 2023-12-12 |
Family
ID=75123609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011344618.1A Active CN112579696B (en) | 2020-11-26 | 2020-11-26 | Multi-point synchronous copying method, device, server and storage medium for radio astronomical data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579696B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114661637B (en) * | 2022-02-28 | 2023-03-24 | 中国科学院上海天文台 | Data processing system and method for radio astronomical data intensive scientific operation |
| CN116225660B (en) * | 2023-03-17 | 2024-02-27 | 中国华能集团有限公司北京招标分公司 | Loading engine using method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101419278A (en) * | 2008-12-05 | 2009-04-29 | 航天恒星科技有限公司 | Multichannel high speed remote sensing data acquiring and processing device |
| US10068002B1 (en) * | 2017-04-25 | 2018-09-04 | Palantir Technologies Inc. | Systems and methods for adaptive data replication |
| US10547679B1 (en) * | 2018-01-02 | 2020-01-28 | Architecture Technology Corporation | Cloud data synchronization based upon network sensing |
| CN110986921A (en) * | 2019-11-13 | 2020-04-10 | 中国电子科技集团公司第二十研究所 | Data communication and processing method for astronomical integrated navigation system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7860825B2 (en) * | 2006-05-08 | 2010-12-28 | Palm, Inc. | Method for synchronizing software application and user data for asynchronous client-server and peer to peer computer networks |
-
2020
- 2020-11-26 CN CN202011344618.1A patent/CN112579696B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101419278A (en) * | 2008-12-05 | 2009-04-29 | 航天恒星科技有限公司 | Multichannel high speed remote sensing data acquiring and processing device |
| US10068002B1 (en) * | 2017-04-25 | 2018-09-04 | Palantir Technologies Inc. | Systems and methods for adaptive data replication |
| US10547679B1 (en) * | 2018-01-02 | 2020-01-28 | Architecture Technology Corporation | Cloud data synchronization based upon network sensing |
| CN110986921A (en) * | 2019-11-13 | 2020-04-10 | 中国电子科技集团公司第二十研究所 | Data communication and processing method for astronomical integrated navigation system |
Non-Patent Citations (2)
| Title |
|---|
| 基于Cassandra的海量MUSER数据分布式存储与检索研究;石玥 等;天文研究与技术;第15卷(第03期);第361-368页 * |
| 射电天文数据实时计算的关键技术研究;戴伟;《基础科学辑》(第03期);第A007-29页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112579696A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112579696B (en) | Multi-point synchronous copying method, device, server and storage medium for radio astronomical data | |
| CN109873863B (en) | Asynchronous calling method and device of service | |
| CN107577578B (en) | Method, device and equipment for counting delay of NFS operation word and storage medium | |
| CN107302582B (en) | Data acquisition and weak push method for million-level Internet of things scene | |
| US20210334791A1 (en) | Method and device for blockchain-based data traffic calculation | |
| US9513661B2 (en) | Calibrated timeout interval on a configuration value, shared timer value, and shared calibration factor | |
| CN111800511B (en) | Synchronous login state processing method, system, equipment and readable storage medium | |
| CN111274047A (en) | Information processing method, terminal, system, computer device and storage medium | |
| CN111161072A (en) | Block chain-based random number generation method, equipment and storage medium | |
| CN114138499B (en) | GPU resource utilization rate monitoring method and device, computer equipment and medium | |
| CN110347380A (en) | Front and back end development approach and equipment | |
| CN107491551B (en) | Method and device for realizing client access of distributed file system | |
| CN117331576A (en) | Application pre-download method, device, computer and storage medium | |
| CN110750424B (en) | Resource inspection method and device | |
| CN112637268B (en) | Radio astronomical data storage consistency guarantee method, device, system and storage medium | |
| CN112486696A (en) | Method and equipment for acquiring distributed lock | |
| CN113472850A (en) | Link data acquisition method, device, system, electronic equipment and storage medium | |
| CN112308532A (en) | Remote online video survey system and method for non-vehicle insurance public estimation | |
| EP3080706B1 (en) | Method of backup of data stored in a terminal | |
| CN110932927B (en) | Service processing method, device, equipment and readable storage medium | |
| CN112434050B (en) | Data synchronization method and device of power grid business processing system and business processing system | |
| CN105487040A (en) | Standard electric energy meter and data processing method thereof | |
| CN117479172A (en) | Privacy number binding method and device, electronic equipment and storage medium | |
| CN117729150A (en) | Flow control method and device, electronic equipment and storage medium | |
| US11271838B2 (en) | Timing synchronization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |