CN115408478B - Data storage and management system and method for sharing experimental instrument - Google Patents
Data storage and management system and method for sharing experimental instrument Download PDFInfo
- Publication number
- CN115408478B CN115408478B CN202211074151.2A CN202211074151A CN115408478B CN 115408478 B CN115408478 B CN 115408478B CN 202211074151 A CN202211074151 A CN 202211074151A CN 115408478 B CN115408478 B CN 115408478B
- Authority
- CN
- China
- Prior art keywords
- data
- instrument
- experimental
- folder
- shared
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data storage and management system and method of a shared experimental instrument. And the instrument end responds to an instrument user registration request of the user and creates a synchronous folder of the user under the shared folder of the shared experimental instrument. The database stores information items defining registered users of respective shared laboratory instruments under respective platforms. The data warehouse comprises a data server and saves the experimental data uploaded from the synchronous folder in a distributed storage mode, so that the experimental data of each experiment are stored as nodes in a lossless and distortion-free mode. And the processing server determines the uploading address of the experimental data of the synchronous folder in the data server according to the storage condition in the data server and transmits the uploading address to the instrument terminal. Thus, unified and efficient management can be performed, data security is improved, and centralized data management is realized.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a data storage and management system and method for sharing experimental instruments.
Background
Large-scale experimental equipment generally exists in institutions such as universities, scientific research institutes, laboratories and the like, the experimental equipment can generate experimental data, laboratory personnel need to store part or all of the experimental data for subsequent use, and one of the most powerful countermeasures in question is to provide original experimental data in academia. Unfortunately, not all raw experimental data is well preserved. On the one hand, the experimental data are often recorded on paper and are easily damaged or lost over time. On the other hand, in the case of poor management, the experimental data may be intentionally and unintentionally modified by the experimental students, and there is no way to give the most primitive data.
The USB flash disk/mobile hard disk is adopted in a traditional mode to store and transmit data, a USB port needs to be opened by an experimental instrument, the USB flash disk brought by a user can increase data security risks, and great influence can be caused on subsequent use of the experimental instrument if safety accidents such as poisoning/data leakage/data locking occur. When the experimental data volume is too large, the traditional U disk/mobile hard disk can not bear huge data volume, and if the experimental data volume is not well stored, data loss can also be caused.
At present, most experimental equipment has limited storage capacity, so a centralized storage management system and a server are needed, the existing nas storage server can solve the problems, but the requirement on the network state by using nas server storage mapping is high, and when the network is unstable or the IO data volume is large, the condition of failure of output to a network disk occurs, so that phenomena such as network driver disconnection and output software crash are caused. If the data are public large-scale instruments and equipment, the data cannot be effectively and automatically distributed to a specific experimental group or hands of experimenters, and the experimenters need to manually upload the data. When experimenters of public experimental equipment upload data, problems that login and logout are not timely, the data are uploaded to wrong user accounts, and the like easily occur among the experimenters. When the data volume is large, uploading and downloading both need longer time, and the efficiency is low under the condition of no automatic transmission. When the experimenter is more, for example, mechanisms such as colleges and universities laboratories mostly adopt modes such as network disk uploading and U disk/mobile hard disk storage, management is disordered, and specific personnel cannot be traced easily when data are retrieved.
Disclosure of Invention
The present application is provided to solve the above-mentioned problems occurring in the prior art. There is a need for a data storage and management system and method for sharing experimental instruments, which can improve the convenience of data storage management and the rationality of storage resource allocation, and can increase the security of data, so as to realize centralized data management and facilitate data operation and maintenance.
According to a first aspect of the present application, there is provided a data storage and management system for a shared laboratory instrument, comprising an administrator terminal, an instrument terminal, a data warehouse, a database and at least one processing server, wherein the instrument terminal and the shared laboratory instrument are located in the same communication area, so that the communication speed between the two is higher than a threshold value, the database is disposed in or in communication with the at least one processing server, and the administrator terminal is configured to: responding to an instrument registration request of an administrator, wherein the instrument registration request comprises the name of a shared experimental instrument to be registered, the platform to which the shared experimental instrument belongs and a local storage path of experimental data at the instrument end, and enabling the local storage path at the instrument end to automatically construct a shared folder of the shared experimental instrument. The instrument end is configured to: in response to an instrument user registration request of a user, creating a synchronization folder of the user under a shared folder of the shared laboratory instrument; storing experimental data generated by each experiment of the user using the shared experimental instrument in the synchronous folder of the user; and uploading the experimental data of each node in the synchronous folder to an uploading address in the data warehouse in a lossless and distortion-free mode through periodic communication with the at least one processing server. The database is configured to: information items defining registered users of respective shared laboratory instruments under respective platforms are stored. The data warehouse comprises at least one data server and saves the experimental data uploaded from the synchronous folder in a distributed storage mode, so that the experimental data of each experiment are stored as nodes in a lossless and distortion-free mode. The at least one processing server is configured to: and according to the storage condition in the at least one data server, determining the uploading address of the experimental data of the synchronous folder in the at least one data server and transmitting the experimental data to the instrument terminal.
According to a second aspect of the present application, an apparatus registration request including a name of a shared experimental apparatus to be registered, a platform to which the shared experimental apparatus belongs, and a local storage path of experimental data at an apparatus end is responded by an administrator end, so that the local storage path at the apparatus end automatically constructs a shared folder of the shared experimental apparatus. Responding to an instrument user registration request of a user through an instrument end, and creating a synchronous folder of the user under a shared folder of the shared experimental instrument; and storing experimental data generated by each experiment of the user using the shared experimental instrument in the synchronous folder of the user. And uploading the experimental data of each node in the synchronous folder to an uploading address in the data warehouse in a lossless and distortion-free mode through periodic communication with the at least one processing server, wherein the instrument end and the shared experimental instrument are positioned in the same communication area, so that the communication speed between the instrument end and the shared experimental instrument is higher than a threshold value. Storing information items defining registered users of the shared experimental instruments under each platform through a database, wherein the database is arranged in or communicated with at least one processing server; and saving the experimental data uploaded from the synchronous folder through at least one data server of the data warehouse in a distributed storage mode, so that the experimental data of each experiment are stored as nodes in a lossless and distortion-free mode. And determining an uploading address of the experimental data of the synchronous folder in the at least one data server through at least one processing server according to the storage condition in the at least one data server, and transmitting the uploading address to the instrument terminal.
Compared with the prior art, the beneficial effects of the embodiment of the application lie in that:
the data storage and management system that this application embodiment provided, through the administrator end, the instrument end, data warehouse, the laboratory technician automatic upload experimental data can be helped in the cooperation of database and at least one processing server, and simultaneously, can be to different users through the administrator end, experiment relevant data such as laboratory glassware and experimental data carry out unified efficient management, improve the flexibility of experimental data operation, promote the rationality of storage resource distribution, increase the security of data, effectively practice thrift the operating time of laboratory technician to data, realize centralized data management, make things convenient for data fortune dimension. The instrument end is in periodic communication with the at least one processing server, and uploads the experimental data of each node in the synchronous folder to the uploading address in the data warehouse in a lossless and distortion-free mode, so that the original experimental data can be stored, the data can be controlled by each node, and the resource utilization efficiency is improved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
In the drawings, which are not necessarily drawn to scale, like reference numerals may describe similar components in different views. The drawings illustrate various embodiments generally by way of example, and not by way of limitation, and together with the description and claims serve to explain the disclosed embodiments. The same reference numbers will be used throughout the drawings to refer to the same or like parts, where appropriate. Such embodiments are illustrative, and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.
FIG. 1 shows a block diagram of a data storage and management system according to an embodiment of the present application.
FIG. 2 illustrates a data storage and management system instrument registration interface diagram according to an embodiment of the present application;
FIG. 3 illustrates an interface diagram of a storage state of a data processor of a data storage and management system according to an embodiment of the present application.
Fig. 4 shows a storage table of experimental data stored in a database of the data storage and management system according to the embodiment of the present application.
FIG. 5 is a flow chart of a method for sharing data storage and management of a laboratory instrument according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood, the present application is described in detail below with reference to the accompanying drawings and the detailed description. The embodiments of the present application will be described in further detail below with reference to the drawings and specific embodiments, but the present application is not limited thereto. The order in which the various steps described herein are described as examples should not be construed as a limitation if there is no requirement for a context relationship between each other, and one skilled in the art would know that sequential adjustments may be made without destroying the logical relationship between each other, rendering the overall process impractical.
As used in this application, the terms "first," "second," and the like do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The use of the word "comprising" or "comprises" and the like in this application is intended to mean that the elements listed before this word cover the elements listed after this word and not to exclude the possibility that other elements may also be covered. In the present application, arrows shown in the figures of the respective steps are only used as examples of execution sequences, and are not limited, and the technical solution of the present application is not limited to the execution sequences described in the embodiments, and the respective steps in the execution sequences may be executed in a combined manner, may be executed in a split manner, and may be in an order-changed manner as long as the logical relationship of the execution content is not affected.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless specifically defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with a general description of the application given above and the detailed description of the embodiments given below, serve to explain the principles of the application.
Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely examples of the application, which can be embodied in various forms. Well-known and/or repeated functions and constructions are not described in detail to avoid obscuring the application of unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present application in virtually any appropriately detailed structure.
FIG. 1 shows a block diagram of a data storage and management system according to an embodiment of the present application. The data storage and management system 100 includes an administrator side 101, an instrument side 102, a data warehouse 103, a database 104, and at least one processing server 105. The administrator side 101, the instrument side 102, the data warehouse 103, and the database 104 may be communicatively connected through a local area network or a wide area network. The local area network or the wide area network may include a plurality of experiment platforms (e.g., medical experiment platforms), each experiment platform includes a plurality of experiment (or research) groups, each experiment group includes a plurality of experimenters, and each experiment group may adopt a plurality of experimental instruments to perform experiments.
Wherein the instrument end 102 and the shared laboratory instrument are located in the same communication area such that the communication speed between the two is higher than a threshold value. When experimenters utilize shared experimental instruments to carry out experiments, such as operating microscopes and the like, corresponding experimental data can be generated, the experimental data generated by the shared experimental instruments are directly generated on corresponding hosts, and the acquired experimental data are not directly stored in the data warehouse 103. Thus, the instrument end 102 and the shared experimental instrument are located in the same communication area, for example, in a local area network, so that the communication speed between the instrument end 102 and the shared experimental instrument is higher than a threshold value, and in this case, the experimental data generated by the shared experimental instrument can be timely and rapidly transmitted to the instrument end 102, so that the transmission of the experimental data can be continuously performed under the condition that a wide area network such as the internet is disconnected, the experimental data generated by the shared experimental instrument is saved, and the experimental data is prevented from being lost. In some embodiments, the instrument end 102 may also be integrated directly into a host that shares the laboratory instrument.
Compared with the method of directly transmitting the experimental data generated by the shared experimental instrument to the data warehouse 103, the communication speed between the instrument end 12 and the shared experimental instrument is higher than the threshold value under the same switch or in the same area, and the stability of the transmission speed of the experimental data is maintained, so that the experimental data is completely stored. The threshold may be a value set by a user or a default value of the data storage and management system 100, which is not limited to this.
The database 104 is disposed on or in communication with the at least one processing server 105, and the database 104 may be disposed on one or more processing servers 105, or may be disposed independently and in communication with the processing server 105, so that the processing server 105 can obtain data in the database 104, or edit and rewrite the data.
The administrator side 101 is configured to respond to an instrument registration request of an administrator, which includes a name of a shared experimental instrument to be registered, a platform to which the shared experimental instrument belongs, and a local storage path of experimental data at the instrument side 102, so that the local storage path at the instrument side 102 automatically constructs a shared folder of the shared experimental instrument. For example, as shown in FIG. 2, the administrator registers the instrument user, including the MACHINE NAME, enters the NAME of the MACHINE, e.g., 8 \ u MPMS3; PATH experimental data is stored locally at the instrument end 102, PATH is clicked, and a shared folder of the shared experimental instrument is selected or newly established. PLATFROM: and selecting a platform on which the shared experimental instrument is positioned. The administrator 101 automatically constructs a shared folder of the shared experimental instrument in the local storage path of the instrument 102, and the shared folder constructed based on the local storage path can completely store experimental data and upload the experimental data to the networked data warehouse 103 after the experimental data is completely stored. If the wide area network is disconnected suddenly, the instrument terminal 102 does not stop continuously generating and uploading the experimental data to the local shared folder, and the experimental data stored in the local shared folder is not lost due to the network disconnection.
The instrument terminal 102 is configured to respond to an instrument user registration request of a user, create a synchronization folder of the user under a shared folder of the shared experimental instrument, and store experimental data generated by each experiment of the shared experimental instrument used by the user in the synchronization folder of the user. Specifically, after each shared laboratory instrument accesses the data storage and management system 100 (for example, accesses a corresponding laboratory platform or laboratory group), the user registers on the shared laboratory instrument, and after the registration is completed, the instrument terminal 102 creates a synchronization folder of the user under the shared folder. In the process that an experimental user performs an experiment by using the shared experimental instrument, the shared experimental instrument continuously generates experimental data and stores the generated experimental data in a local synchronous folder, so that the experimental original data generated by the shared experimental instrument can be completely stored.
The instrument terminal 102 uploads the experimental data of each node in the synchronization folder to the upload address in the data warehouse 103 in a lossless and distortion-free manner through periodic communication with the at least one processing server 105. For example, the instrument terminal 102 sends a data upload request to the data warehouse 103, after receiving the data upload request, the data warehouse 103 forwards the data upload request to the processing server 105, and the processing server 105 determines an upload address of the instrument terminal 102 to the data warehouse 103 according to information such as an IP address of the shared experimental instrument and storage capacity of the data warehouse 103, and sends the upload address to the instrument terminal 102. The instrument terminal 102 obtains various experimental related information including the upload address through the periodic communication with the processing server 105. Based on the upload address, the instrument end 102 uploads the experimental data of each node generated by each shared experimental instrument to the data warehouse 103 in a lossless and distortion-free manner, so as to synchronize the experimental data between the local storage of the instrument end 102 and the data warehouse 103.
The database 104 is configured to store information items defining registered users of the shared laboratory instruments under the respective platforms, wherein the information items include, but are not limited to, instrument data of the shared laboratory instruments and laboratory user data related to the shared laboratory instruments. The shared experimental instrument data comprises but is not limited to the name of the instrument, the platform to which the instrument belongs, the mac address, the IP where the instrument is located, the user, the group, the file address information and the like; the shared experiment user data may include experiment user data such as user names of experiment users and experiment groups where the users are located.
The data repository 103 includes at least one data server 106 and maintains the experimental data uploaded from the synchronized folders in a distributed storage manner such that the experimental data for each experiment is stored as nodes without loss and distortion. For example, the data server a, the data server B, and the data server C are used together to construct the data warehouse 103, and at this time, the data warehouse 103 obtains experimental data and stores the experimental data in at least one of the data server a, the data server B, and the data server C in a distributed storage manner. Specifically, the experimental data may be allocated based on the remaining storage space of data server a, data server B, and data server C, such that the dynamic balance of the storage space of data server a, data server B, and data server C is maintained.
On same sharing laboratory glassware, the experimental data size that each time experiment obtained is the same almost, consequently to the experimental data of experiment at every turn is as the node to make the memory space of every node controllable, so, be favorable to realizing dynamic balance on a plurality of data server 106's storage, be favorable to each data server 106 of make full use of, prevent that each data server 106 accident from exploding the storehouse. When each user stores each experiment raw data, the management storage and management system 100 records a file path corresponding to each experiment raw data. The experimental data of each experiment is stored as nodes in a lossless and distortion-free mode, the nodes are not encrypted, fragmentation is not performed, compression is not performed, the original experimental data can be stored in a lossless mode, and the database 104 can be conveniently reconstructed through reverse pushing when the database 104 is damaged.
The at least one processing server 105 is configured to determine an uploading address of the experiment data of the synchronization folder in the at least one data server 106 according to the storage condition in the at least one data server 106, and transmit the uploading address to the instrument terminal 102. When a user uses a shared experimental instrument, a blank synchronous folder can be automatically created according to an experimental node, then generated experimental original data is stored in the synchronous folder, a data uploading request is sent to the data warehouse 103, the data uploading request carries relevant information of the synchronous folder, the processing server 105 determines the position (uploading address) of the data server 106 for storing the experimental original data according to the relevant information (including the size of the experimental original data) of the synchronous folder received by the data warehouse 103 and the storage condition of each data server 106 in the data warehouse 103, the uploading address is returned to the instrument end 102, the instrument end 102 uploads the experimental original data corresponding to the experimental node to the data warehouse 103 according to the returned uploading address, and therefore network storage of the experimental original data is achieved.
In some embodiments of the present application, the at least one data repository 103 includes at least two data servers 106 (e.g., the number of data servers 106 may also be increased dynamically according to actual needs, such as adding data servers 106 by dynamic expansion), and is communicatively connected to the processing server 105 to satisfy the storage of a large amount of experiment-related data. The at least one processing server 105 is further configured to obtain the storage residual capacity in each data server 106, and determine the storage address in the data server 106 with the largest storage residual capacity. As shown in fig. 3, the data warehouse 103 includes two data servers 106, namely a data server 1 and a data server 2, where the occupied storage capacity of the data server 1 is only 8.42%, and the occupied storage capacity of the data server 2 is 13.08%, at this time, the storage residual capacity of the data server 1 is the maximum, and the experiment data in the synchronization folder stored locally will be distributed and stored in the data server 1. By analogy, the experimental data uploaded to the data warehouse 104 is reasonably distributed and stored according to the remaining storage capacity of the data servers 106, so that the dynamic balance of the storage of each data server 106 in the data warehouse 104 can be maintained, and thus each data server 106 is fully utilized.
The at least one processing server 105 determines a storage address in the data server 106 having the largest storage remaining capacity and determines an upload address corresponding to the storage address. The upload address may be understood as a network address for uploading the experiment data in the locally stored synchronization folder to the data warehouse 104, and includes, but is not limited to, a transmission port, a transmission address, a transmission ID, and the like of the data server 106. And the storage address may be understood as which field of which unit of which data server 106 in the data warehouse 104 the experimental data acquired by the data warehouse 104 is stored.
Specifically, for example, the processing server 105 may be at least 1 linux system computing server, the system employs a CentOS7, and configures an IP address of the processing server 105, and the server is configured as follows:
AMD EPYC 7502 processor 2, 128G (64G 2) DDR4 memory, 2 blocks 480GIntel 4610SSD, 4G cache super capacitor module, 2 ports ten gigabit optical port network card (including optical module) and 4 ports gigabit electric port network card.
After configuring the processing server 105, the data server 106 can be configured, and the data server 106 can be at least 2 linux system storage servers, the system adopts a CentOS7, all the servers of the data server 106 are connected to the same switch and connected to the same network with the processing server 105, so that reliable connection between the processing server 105 and the data server 106 is ensured.
The IP address of the data server 106 is configured, and the server is configured as follows: the processor adopts Intel Xeon4210 (2.2 GHz/10 cores) × 2, 128G (64G × 2) DDR4 ECC memory, 2 blocks × 480G Intel4610SSD, 12 blocks × 12TB SATA 7.2K LFF hard disk, 12Gb 2 port SAS RAID card, 4G cache super capacitor module, 2 port gigabit optical port network card (including optical module) and 4 port gigabit electrical port network card.
In some embodiments of the present application, the at least one data server 106 is further configured to add, after determining the uploading address, a corresponding information item in the database 104, where the information item includes a local storage path of the synchronization folder at the instrument end 102, a storage address corresponding to the experimental data of each node, a creation time and an update time of the information item. After the locally stored experimental data of the instrument terminal 102 is uploaded to the data warehouse 103, a corresponding information item is left in the database 104, which at least records the position of the locally stored experimental data stored in the data server 106. As shown in fig. 4, in the table, path represents an experiment node, dir represents a type of data storage, if 1, the data storage is stored in a directory, if 0, the data storage is directly stored in a corresponding folder in a file form, and server represents a storage server model of the data storage. When the user checks the data, the user can directly check the stored data according to the experimental nodes and access the data, so that the experimental data can be conveniently stored and managed. The information items include, but are not limited to, a local storage Path (Path) of the synchronization folder at the instrument end 102, storage addresses (storage 1 or storage 2) corresponding to experimental data of each node, creation time (createtime) and update time (updatetime) of the information items. After each determination of an upload address, a corresponding information item is added to the database 104 to update the database 104. Thus, if there is an information item of the experimental data of a certain node in the database 104, it can be known that the experimental data of the node has been uploaded to the data warehouse 103.
In some embodiments of the present application, the at least one processing server 105 is further configured to transmit an upload stop instruction to the instrument terminal 102 if the remaining storage capacity in any one of the data servers 106 is lower than a preset capacity, where the preset capacity corresponds to a capacity increment required for experimental data modification of a node. Specifically, for example, the storage remaining capacity of the data server 1 or the data server 2 in fig. 3 is less than 5%, at this time, the processing server 105 transmits an upload stop instruction to the instrument terminal 102, and the instrument terminal 102 stops uploading the experimental data to the data server 1 and the data server 2 after receiving the upload stop instruction. The preset capacity is set, so that the requirement of a user for modifying the experimental data of the node is met. Assuming that the preset capacity is 5%, when the storage remaining capacity of the data server 1 or the data server 2 shown in fig. 3 is less than 5%, the instrument terminal 102 does not upload the experimental data to the data server 1 or the data server 2 any more, and if the user modifies the experimental data of the node 10 days ago at this time, the modified experimental data of the node is synchronized to the data server 1 or the data server 2, while the preset capacity of the original data server 1 or the data server 2 of 5% can store the capacity increased by the modified experimental data of the node, and the increased capacity is the capacity increment.
In some embodiments of the present application, the at least one data repository 103 includes at least two data servers 106, and the at least one processing server 105 is further configured to determine whether there are data servers 106 in the same communication area as the instrument end 102 in the at least two data servers 106, and if so, determine a storage address in the data server 106 in the same communication area and determine an upload address corresponding to the storage address, where the same communication area includes areas within the same local area or communicating through the same switch. The data warehouse 103 includes at least two data servers 106, and in the process of determining the storage addresses of the data servers 106, the storage address in the data server 106 with the largest storage remaining capacity may be determined first, and if the storage remaining capacities of two data servers 106 are the largest and the storage remaining capacities are the same, it is determined whether the two data servers 106 and the instrument terminal 102 are in the same communication area, and the storage address in the data server 106 in the same communication area as the instrument terminal 102 is selected preferentially. In addition, when determining the storage address of the data server 106, the storage address of the data server 106 in the same communication area as the instrument end 102 may be determined first, and if there are a plurality of data servers 106 in the same communication area as the instrument end 102, the data server 106 with the largest storage remaining capacity may be selected to store the experimental data uploaded to the data warehouse 103. Specifically, for example, the student completes the experiment at the instrument terminal 102 in the school zone and uploads the experiment data, and at this time, the data server 106 in the school zone is preferentially selected to store the experiment data, so that the data transmission speed is faster and more stable than the data stored in the cross-school zone.
In some embodiments of the present application, the database 104 is further configured to store information items, where the information items include a local storage path of the synchronization folder at the instrument end 102, storage addresses corresponding to experimental data of each node, and creation time and update time of the information items, which are specifically described above and are not described herein again. The instrument side 102 is further configured to request the at least one processing server 105 for the information of the registered user of the shared experimental instrument, that is, the instrument side 102 sends a request to the processing server 105 to obtain the information of the registered user of the shared experimental instrument, for example, the information of the registered user includes, but is not limited to, the name of the registered user and the registration time.
The at least one processing server 105 further checks the synchronization folders under the shared folder under the local storage path of the instrument end 102 for synchronization folders without missing registered users, and if yes, adds the synchronization folders of the registered users. Database 104 holds information items that may record registered users of the shared laboratory instrument in the form of a list. The processing server 105 feeds back the list of information items acquired from the database 104 to the instrument terminal 102, and the instrument terminal 102 checks the synchronization folder under the shared folder under the local storage path. If the information items in the database 104 record the storage information items of the synchronization folder, it indicates that the synchronization folder under the shared folder under the local storage path of the instrument 102 is missing, and the missing synchronization folder of the registered user needs to be added to the local storage path of the instrument 102.
In some embodiments of the present application, the at least one processing server 105 is further configured to determine, based on the information items saved by the database 104, a synchronization folder in which experimental data of nodes contained in the information items is missing in the instrument side 102. Determining an occurrence time difference of the missing compared to the creation time based on the creation time of the information item corresponding to the missing synchronization folder, deleting the corresponding information item in the database 104 when the occurrence time difference is within a range of a first predetermined time, but retaining the experimental data of the node of the corresponding storage address in the data warehouse 104 within a second predetermined time, wherein the first predetermined time enables a single experiment of the shared experimental instrument to be completed. Specifically, the instrument side 102 requests the database 104 for an information item, and checks the registered user information in the local storage path of the instrument side 102 based on the information item to determine the missing synchronization folder in the instrument side 102. Assuming that, if the creation time (createtime shown in fig. 4) of the corresponding information item of the missing synchronization folder in the database 104 is 2022-4-27-09. In the first predetermined time, the user intentionally deletes the synchronization folder at the instrument side 102, and at this time, the information item corresponding to the deleted synchronization folder in the database 104 may be directly deleted. The check may be based on a local storage path (path) of the synchronization folder in the information item at the instrument side and a user folder name, or other information, which is not particularly limited as long as it can be determined that the synchronization folder corresponding to the information item already recorded in the database 104 is missing at the instrument side 102.
In addition, the instrument side 102 generates an information item corresponding to the experimental data of the node in the database 104 after uploading the experimental data of the node in the locally stored synchronization folder to the data warehouse 103. That is, the synchronization between the instrument side 102 and the data warehouse 103 is completed, and then the information items in the instrument side 102 and the database 104 are checked. After the check, if the local missing synchronization folder of the instrument terminal 102 is found within the first predetermined time, at this time, the information item of the corresponding missing synchronization folder in the database 104 is directly deleted. However, the experimental data of the node corresponding to the storage address in the data warehouse 103 is reserved in the second predetermined time (note that the second predetermined time is greater than the first predetermined time) to prevent the local synchronization folder from being deleted by mistake. For example, the first predetermined time is 24 hours, the second predetermined time is 7 days, and if the deletion occurs by mistake in the instrument terminal 102 within 24 hours, at this time, although the information item corresponding to the deleted synchronization folder in the database 104 is deleted, the experimental data of the deleted synchronization folder node still remains in the data warehouse 103 for 7 days, and after 7 days, the experimental data of the deleted synchronization folder node is scanned and deleted.
The first preset time and the second preset time can be set by a user, or can be default time of the system, and the user can modify the first preset time and the second preset time based on actual needs. The specific values of the first predetermined time and the second predetermined time are not limited.
In some embodiments of the present application, a first file and a second file are stored under the local storage path of the instrument end 102, the first file stores the creation time of a directory or a folder under the local storage path, and the second file stores the latest modification time of the directory or the folder under the local storage path. Specifically, the first file is num creation. The instrument side 102 is further configured to check whether the latest modification time of the directory or folder is between the current communication time and the previous communication time by reading at least a second file in case of a periodical communication with the at least one processing server 105, and if so, upload the experimental data of the node, otherwise, not upload. Specifically, for example, the instrument side 102 communicates with the processing server 105 periodically at intervals, and checks whether the information items existing in the database 104 correspond to the registered user information stored locally in the instrument side 102, so as to synchronize information between the database 104 and the instrument side 102. Assume that the instrument side 102 communicates with the processing server 105 at a first time and, after a period of time, at a second time. The instrument terminal 102 can check the latest modification time of the second file locally, and if the latest modification time of the second file changes between the first time and the second time, the directory or the folder under the local storage path is modified, and at this time, the modified directory or folder contains the latest experimental data and uploads the experimental data of the modified directory or folder node. If the modification time of the directory or the folder of the second file is not changed between the first time and the second time, the experimental data does not need to be updated, and the experimental data of the directory or the folder node between the first time and the second time does not need to be uploaded.
When checking whether the directory or the folder is modified, only the modification time of the second file needs to be checked locally, and the instrument terminal 102 does not need to communicate with the processing server 105, so that the communication traffic between the instrument terminal 102 and the processing server 105 is greatly reduced, and the communication efficiency is improved.
In some embodiments of the present application, the at least one processing server 105 is further configured to obtain a creation time of a directory or a folder in the first file and a data deletion condition therein, if data deletion therein occurs within a first predetermined time from the creation time of the directory or the folder, cause the corresponding storage address in the data warehouse 103 to also delete corresponding data, and if data deletion therein occurs after the first predetermined time expires from the creation time of the directory or the folder, retain the corresponding data of the corresponding storage address in the data warehouse 103 within a second predetermined time (greater than the first predetermined time). Specifically, the first predetermined time is 24 hours, and the second predetermined time is 7 days. If data deletion occurs within 24 hours from the creation time of the directory or folder in the first file, it indicates that the deleted data is incorrect and is invalid data, and at this time, the corresponding storage address in the data warehouse 103 also deletes the corresponding data accordingly. If the data in the first file is deleted after 24 hours from the creation time of the directory or folder in the first file, the system can recognize that the data deletion behavior after more than 24 hours is to clean up the local storage space, at which time the data warehouse 103 still retains the corresponding data of the corresponding storage address for 7 days to prevent the user from repentance.
FIG. 5 is a flow chart of a method for sharing data storage and management of a laboratory instrument according to an embodiment of the present application. In step S501, an administrator end responds to an instrument registration request of the administrator, where the request includes a name of a shared experimental instrument to be registered, a platform to which the shared experimental instrument belongs, and a local storage path of experimental data at the instrument end, so that the local storage path at the instrument end automatically constructs a shared folder of the shared experimental instrument. In step S502, responding to an instrument user registration request of a user through an instrument end, creating a synchronization folder of the user under a shared folder of the shared experimental instrument; storing experimental data generated by each experiment of the user using the shared experimental instrument in the synchronous folder of the user; and uploading the experimental data of each node in the synchronous folder to an uploading address in the data warehouse in a lossless and distortion-free mode through periodic communication with the at least one processing server, wherein the instrument end and the shared experimental instrument are positioned in the same communication area, so that the communication speed between the instrument end and the shared experimental instrument is higher than a threshold value. At step 503, information items defining registered users of respective shared laboratory instruments under respective platforms are maintained via a database, wherein the database is disposed on or in communication with at least one processing server. In step S504, the experimental data uploaded from the synchronization folder is saved by at least one data server of the data warehouse in a distributed storage manner, so that the experimental data of each experiment is stored as a node without loss and distortion. In step S505, according to the storage status in the at least one data server, the at least one processing server determines an upload address of the experimental data of the synchronization folder in the at least one data server, and transmits the address to the instrument. So, cooperation through administrator's end, the instrument end, data warehouse, database and at least one processing server can assist the experimenter to upload experimental data automatically, and simultaneously, can be to different users through the administrator end, the relevant data of experiments such as laboratory glassware and experimental data carry out unified efficient management, improve the flexibility of experimental data operation, promote the rationality of storage resource distribution, increase the security of data, effectively practice thrift the operating time of experimenter to data, realize centralized data management, make things convenient for data fortune dimension.
In various embodiments of the present application, steps of the data storage and management method described in conjunction with the data storage and management system 100 (including the administrator side 101, the instrument side 102, the data warehouse 103, the database 104, and the at least one processing server 105) may be combined herein to form corresponding embodiments of the data storage and management method, which are not described herein again.
Moreover, although exemplary embodiments have been described herein, the scope thereof includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., of various embodiments across), adaptations or alterations. The elements of the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application, which examples are to be construed as non-exclusive. It is intended, therefore, that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.
The above description is intended to be illustrative and not restrictive. For example, the above-described examples (or one or more versions thereof) may be used in combination with each other. For example, other embodiments may be used by those of ordinary skill in the art upon reading the above description. In addition, in the above detailed description, various features may be grouped together to streamline the application. This should not be interpreted as an intention that a disclosed feature not claimed is essential to any claim. Rather, subject matter of the present application can lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that these embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
The above embodiments are only exemplary embodiments of the present application, and are not intended to limit the present invention, the scope of which is defined by the claims. Various modifications and equivalents may be made by those skilled in the art within the spirit and scope of the present application and such modifications and equivalents should also be considered as falling within the scope of the present invention.
Claims (10)
1. A data storage and management system of a shared experimental instrument is characterized by comprising an administrator side, an instrument side, a data warehouse, a database and at least one processing server, wherein the instrument side and the shared experimental instrument are located in the same communication area, the communication speed between the instrument side and the shared experimental instrument is higher than a threshold value, the database is arranged in or communicated with the at least one processing server,
the administrator side is configured to: responding to an instrument registration request of an administrator, wherein the instrument registration request comprises the name of a shared experimental instrument to be registered, a platform to which the shared experimental instrument belongs and a local storage path of experimental data at the instrument end, and enabling the local storage path at the instrument end to automatically construct a shared folder of the shared experimental instrument;
the instrument end is configured to: responding to an instrument user registration request of a user, and creating a synchronous folder of the user under a shared folder of the shared experimental instrument; storing experimental data generated by each experiment of the user using the shared experimental instrument in the synchronous folder of the user; uploading experimental data of each node in a synchronous folder to an uploading address in the data warehouse in a lossless and distortion-free mode through periodic communication with the at least one processing server;
the database is configured to: storing information items defining registered users of the shared experimental instruments under the platforms;
the data warehouse comprises at least one data server and stores the experimental data uploaded from the synchronous folder in a distributed storage mode, so that the experimental data of each experiment are stored as nodes in a lossless and distortion-free mode;
the at least one processing server is configured to: and according to the storage condition in the at least one data server, determining the uploading address of the experimental data of the synchronous folder in the at least one data server and transmitting the experimental data to the instrument terminal.
2. The data storage and management system of claim 1, wherein the at least one data repository comprises at least two data servers, the at least one processing server further configured to: acquiring the storage residual capacity in each data server; and determining a storage address in the data server with the largest storage residual capacity, and determining an uploading address corresponding to the storage address.
3. The data storage and management system of claim 2, wherein the at least one data server is further configured to: after the uploading address is determined, adding a corresponding information item in the database, wherein the information item comprises a local storage path of the synchronous folder at the instrument end, a storage address corresponding to experimental data of each node, and creation time and updating time of the information item.
4. The data storage and management system of claim 2, wherein the at least one processing server is further configured to: and transmitting an uploading stopping instruction to the instrument terminal under the condition that the storage residual capacity in any data server is lower than a preset capacity, wherein the preset capacity corresponds to a capacity increment required by experimental data modification of the node.
5. The data storage and management system of claim 1, wherein the at least one data repository comprises at least two data servers, the at least one processing server further configured to: and determining whether a data server in the same communication area with the instrument end exists in the at least two data servers, if so, determining a storage address in the data server in the same communication area and determining an uploading address corresponding to the storage address, wherein the same communication area comprises an area in the same local area network or communicated through the same switch.
6. The data storage and management system according to claim 1 or 2, wherein the database is further configured to store information items, the information items including a local storage path of the synchronization folder at the instrument end, storage addresses corresponding to experimental data of each node, creation time and update time of the information items,
the instrument end is further configured to: requesting, from the at least one processing server, registered user information of the shared laboratory instrument; and checking that the synchronous folders of the registered users are not missed if the synchronous folders of the registered users exist in the synchronous folders of the shared folders in the local storage path of the instrument end, and if the synchronous folders of the registered users exist in the synchronous folders of the shared folders, adding the synchronous folders of the registered users.
7. The data storage and management system of claim 6, wherein the at least one processing server is further configured to: determining a synchronous folder in which experimental data of nodes contained in the information item is missing in the instrument terminal based on the information item stored in the database; determining the occurrence time difference of the missing compared with the creation time based on the creation time of the information item corresponding to the missing synchronization folder; and when the difference is within a range of first preset time, deleting the corresponding information item in the database, and reserving the experimental data of the node of the corresponding storage address in the data warehouse within second preset time, wherein the first preset time enables the completion of the single experiment of the shared experimental instrument.
8. The data storage and management system of claim 3,
a first file and a second file are stored under the local storage path of the instrument end, the first file stores the creation time of a directory or a folder under the local storage path, the second file stores the latest modification time of the directory or the folder under the local storage path,
the instrument end is further configured to: and in the case of regular communication with the at least one processing server, checking whether the latest modification time of the directory or the folder is between the current communication time and the previous communication time, if so, uploading the experimental data of the node, and otherwise, not uploading the experimental data.
9. The data storage and management system of claim 3,
a first file and a second file are stored under the local storage path of the instrument end, the first file stores the creation time of a directory or a folder under the local storage path, the second file stores the latest modification time of the directory or the folder under the local storage path,
the at least one processing server is further configured to: acquiring the creation time of a directory or a folder in the first file and the data deletion condition in the directory or the folder; if the data in the directory or the folder is deleted within a first preset time from the creation time of the directory or the folder, the corresponding storage address in the data warehouse is also deleted corresponding data; if data deletion occurs after the first preset time expires from the creation time of the directory or the folder, the corresponding data of the corresponding storage address in the data warehouse is reserved in a second preset time.
10. A data storage and management method for sharing experimental instruments is characterized in that,
responding to an instrument registration request of an administrator, wherein the instrument registration request comprises the name of a shared experimental instrument to be registered, a platform to which the shared experimental instrument belongs and a local storage path of experimental data at an instrument end, and enabling the local storage path at the instrument end to automatically construct a shared folder of the shared experimental instrument;
responding to an instrument user registration request of a user through an instrument end, and creating a synchronous folder of the user under a shared folder of the shared experimental instrument; storing experimental data generated by each experiment of the user using the shared experimental instrument in the synchronous folder of the user; the method comprises the steps that experiment data of each node in a synchronous folder are uploaded to an uploading address in a data warehouse in a lossless and distortion-free mode through periodic communication with at least one processing server, and the instrument end and the shared experiment instrument are located in the same communication area, so that the communication speed between the instrument end and the shared experiment instrument is higher than a threshold value;
storing information items defining registered users of the shared experimental instruments under each platform through a database, wherein the database is arranged in or communicated with at least one processing server;
saving the experimental data uploaded from the synchronous folder in a distributed storage mode through at least one data server of a data warehouse, so that the experimental data of each experiment can be stored as nodes in a lossless and distortion-free mode;
and determining an uploading address of the experimental data of the synchronous folder in at least one data server and transmitting the uploading address to the instrument terminal through at least one processing server according to the storage condition in at least one data server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211074151.2A CN115408478B (en) | 2022-09-02 | 2022-09-02 | Data storage and management system and method for sharing experimental instrument |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211074151.2A CN115408478B (en) | 2022-09-02 | 2022-09-02 | Data storage and management system and method for sharing experimental instrument |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115408478A CN115408478A (en) | 2022-11-29 |
| CN115408478B true CN115408478B (en) | 2023-03-21 |
Family
ID=84164667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211074151.2A Active CN115408478B (en) | 2022-09-02 | 2022-09-02 | Data storage and management system and method for sharing experimental instrument |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115408478B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096685A (en) * | 2009-12-11 | 2011-06-15 | 阿里巴巴集团控股有限公司 | Method and device for synchronizing distributive data into data warehouse |
| CN102497353A (en) * | 2011-10-28 | 2012-06-13 | 深圳第七大道科技有限公司 | Processing method, server and system for multi-server distributed data |
| CN107743124A (en) * | 2017-10-18 | 2018-02-27 | 优来博（重庆）科技有限公司 | Lab resources are shared and management system and the monitoring method of laboratory apparatus or equipment |
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
| CN108694195A (en) * | 2017-04-10 | 2018-10-23 | 腾讯科技（深圳）有限公司 | A kind of management method and system of Distributed Data Warehouse |
| WO2019183547A1 (en) * | 2018-03-22 | 2019-09-26 | Datomia Research Labs Ou | Distributed storage system data management and security |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9864789B2 (en) * | 2013-04-08 | 2018-01-09 | Oracle International Corporation | Method and system for implementing an on-demand data warehouse |
| US9734230B2 (en) * | 2013-09-12 | 2017-08-15 | Sap Se | Cross system analytics for in memory data warehouse |
-
2022
- 2022-09-02 CN CN202211074151.2A patent/CN115408478B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096685A (en) * | 2009-12-11 | 2011-06-15 | 阿里巴巴集团控股有限公司 | Method and device for synchronizing distributive data into data warehouse |
| CN102497353A (en) * | 2011-10-28 | 2012-06-13 | 深圳第七大道科技有限公司 | Processing method, server and system for multi-server distributed data |
| CN108694195A (en) * | 2017-04-10 | 2018-10-23 | 腾讯科技（深圳）有限公司 | A kind of management method and system of Distributed Data Warehouse |
| CN107743124A (en) * | 2017-10-18 | 2018-02-27 | 优来博（重庆）科技有限公司 | Lab resources are shared and management system and the monitoring method of laboratory apparatus or equipment |
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
| WO2019183547A1 (en) * | 2018-03-22 | 2019-09-26 | Datomia Research Labs Ou | Distributed storage system data management and security |
Non-Patent Citations (3)
| Title |
|---|
| 吴惠娜 ; .数据仓库在铁路统计分析中的应用.2007,(03),全文. * |
| 张翅,唐谦.高校实验室云桌面数据存储方案的探索与实践.2022,全文. * |
| 禹禄君 ; .基于大数据的信息系统关键技术研究.2015,(03),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115408478A (en) | 2022-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI476610B (en) | Peer-to-peer redundant file server system and methods | |
| CN102662992B (en) | Method and device for storing and accessing massive small files | |
| US9104560B2 (en) | Two level addressing in storage clusters | |
| US9535630B1 (en) | Leveraging array operations at virtualized storage processor level | |
| WO2017167171A1 (en) | Data operation method, server, and storage system | |
| US20150269203A1 (en) | Accelerated access to objects in an object store implemented utilizing a file storage system | |
| CN103078898B (en) | File system, interface service device and data storage service supplying method | |
| WO2014183708A1 (en) | Method and system for realizing block storage of distributed file system | |
| CA2843886A1 (en) | Cloud-based distributed persistence and cache data model | |
| CN111400312B (en) | Edge storage database based on improved LSM tree | |
| CN108319623A (en) | A kind of fast resampling method, apparatus and data-base cluster | |
| JP2014529814A (en) | Data maintenance methods | |
| CN115878046B (en) | Data processing method, system, device, storage medium and electronic equipment | |
| CN109542861A (en) | File management method, device and system | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| US12032847B2 (en) | Cross-platform replication of logical units | |
| CN103631820A (en) | Metadata management method and device of distributed file system | |
| CN106547911A (en) | A kind of access method and system of mass small documents | |
| CN107291876A (en) | A kind of DDM method | |
| CN111813346A (en) | Method, system, device and medium for establishing Ceph distributed storage based on cloud platform | |
| CN104636441B (en) | NFS implementation method and device | |
| CN110362590A (en) | Data managing method, device, system, electronic equipment and computer-readable medium | |
| CN111435286A (en) | Data storage method, device and system | |
| CN115408478B (en) | Data storage and management system and method for sharing experimental instrument | |
| WO2023125630A1 (en) | Data management method and related apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |