CN115017163A - Pathological digital section data dynamic management system and method based on object storage - Google Patents
Pathological digital section data dynamic management system and method based on object storage Download PDFInfo
- Publication number
- CN115017163A CN115017163A CN202210768824.8A CN202210768824A CN115017163A CN 115017163 A CN115017163 A CN 115017163A CN 202210768824 A CN202210768824 A CN 202210768824A CN 115017163 A CN115017163 A CN 115017163A
- Authority
- CN
- China
- Prior art keywords
- object storage
- pathological
- file
- storage server
- pathological section
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
- G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data management, and discloses a pathological digital section data dynamic management system and method based on object storage, which comprises a block storage server and an object storage server; the block storage server uploads the pathological section file meeting the synchronization time to the object storage server; the object storage server is provided with a database, a standard storage area and a low-frequency storage area, wherein the database is used for recording the object storage address and the browsing times of the uploaded pathological digital slice file; receiving the uploaded pathological section file and storing the pathological section file to a standard storage area; when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database; and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address. The invention is based on the dynamic management of pathological digital section data stored by the object, and ensures the data reading efficiency.
Description
Technical Field
The invention belongs to the technical field of data management, and particularly relates to a pathological digital section data dynamic management system and method based on object storage.
Background
The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
The pathological digital section library is a network on-line management method for digital sections by using a computer and a network system. Compared with the traditional solid slice, the method is free from the time and space limitations. Meanwhile, various film reading meetings can be held on line, so that pathologists can conveniently communicate pathologists such as pathology film reading and remote pathology teaching. The slice library is uniformly deployed on one server, the storage space occupied by each digital slice is about 1GB or even more, the number of slices of a slice reading conference in a conventional scale is about 20 on average, the demand of the slice reading conference is more and more along with the improvement of technical development and people understanding, and the network slice reading conference is held more and more frequently. As storage requirements have increased over time, the block storage capacity expansion of the conventional server is expensive, and the Windows 10 system only supports capacity expansion to 16TB, and data storage space and management become a significant problem.
To address this problem, there is a need for a way to make digital slices simpler to store and manage, while supporting scalability, easy access to information, and lower cost. The common processing mode is to compress and upload the file to the server, but the method needs decompression when the client compresses for a long time and browses again, and the problem of insufficient space still exists after a long time; or the file is taken as an object by the object storage mode and stored into unstructured data. But object storage does not allow reading data in fragments (starting from a byte to the end of a byte), but only the entire object, which affects performance; meanwhile, the object storage cannot maintain the traditional database, and is more suitable for storing files with little or no change, and the traditional block storage is more suitable for the application of the database like a slice library which needs to be directly mapped with a storage bare disk.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a pathological digital section data dynamic management system and method based on object storage.
In order to achieve the above object, one or more embodiments of the present invention provide the following technical solutions:
a pathology digital slice data dynamic management system based on object storage comprises a block storage server and an object storage server; wherein the content of the first and second substances,
the block storage server is configured to acquire pathological section files meeting the synchronization time according to a set data synchronization period and upload the pathological section files to the object storage server;
the object storage server is provided with a database, a standard storage area and a low-frequency storage area, wherein the database is used for recording object storage addresses and browsing times of uploaded pathological digital slice files, and the standard storage area and the low-frequency storage area are used for storing folders corresponding to the pathological digital slice files; the object storage server is configured to:
receiving a pathological section file uploaded by a block storage server and storing the pathological section file in a standard storage area;
when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database;
and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address of the pathological section file in the database.
Further, each folder is used for storing basic information of a pathological section file, image information of each layer and an image information storage address file, wherein the image information storage address file is used for storing object storage addresses of images of each layer of the pathological section file; and the object storage address of the pathological digital section file adopts the object storage address of basic information of the pathological section file.
Further, the object storage address of the pathological section file is updated, and the image information storage address file is also updated at the same time.
Further, the uploading method of the pathology digital slice file comprises the following steps:
the block storage server acquires basic information of a pathological section file to be uploaded and uploads the basic information to the object storage server;
verifying the data consistency before and after uploading the basic information, and if the verification fails, generating an error log;
if the verification is successful, further uploading image information of each layer of the pathological section file, verifying data consistency before and after uploading the image information of each layer, and if the verification fails, generating an error log; if the verification is successful, generating an image information storage address file and uploading the image information storage address file to an object storage server;
and when the object storage server receives the basic information of the pathological section file, creating a folder for the pathological section file, wherein the folder is used for storing the basic information of the pathological section file, the image information of each layer and the image information storage address file.
Further, the checking the data consistency before and after the basic information is uploaded comprises:
after the object storage server receives the basic information of the pathological section file, calculating an MD5 value of the basic information and returning the MD5 value to the block storage server;
and the block storage server calculates the MD5 value based on the locally stored basic information, compares the MD5 value with the MD5 value acquired from the object storage server, and if the MD5 value is consistent with the MD5 value, the verification is successful.
Further, the checking the data consistency before and after uploading each layer of image information includes:
the object storage server calculates the MD5 value of each layer in turn according to the received image information of each layer and returns the value to the block storage server;
the block storage server calculates an MD5 value based on the layer of locally stored image information, compares the MD5 value with an MD5 value acquired from the object storage server, and generates an error log if one layer of image information fails to be verified;
if the layer of image information is consistent with the layer of image information, successfully verifying, acquiring an object storage address of the layer of image information in an object storage server, and storing the object storage address in an image information storage address file; and continuously performing the verification of the next layer of image information.
Further, the uploading method further comprises:
the object storage server acquires an object storage address corresponding to the basic information and stores the object storage address as the object storage address of the pathological section file; the browsing times is set to 0; sending a last successful message of the pathological section file to a block storage server, and deleting a local path in the object storage server;
the block storage server deletes the local pathological section file.
Further, browsing the designated pathological section file in the object storage server comprises:
the method comprises the steps that a block storage server obtains a pathological section file browsing request and sends the pathological section file browsing request to an object storage server; the browsing request comprises identification information of the pathological section file;
and the object storage server receives the browsing request, inquires a corresponding object storage address, determines a corresponding folder according to the object storage address, acquires images layer by layer according to the corresponding object storage address of each layer of image information, transmits the images to the block storage server, and adds 1 to the browsing times of the pathological section file after transmission is finished.
Further, the method for synchronizing the pathological section file uploaded to the object storage server comprises the following steps:
the object storage server acquires a pathological section file meeting the synchronization time according to the set data synchronization period, and acquires a corresponding object storage address;
acquiring corresponding basic information, calculating an MD5 value, and sending the value to a block storage server;
the block storage server searches whether the pathological section file exists locally, and if not, the case section file is not updated; if the pathological section file exists, calculating the MD5 value of the basic information, comparing the MD5 value with the MD5 value received from the object storage server, judging whether the pathological section file is consistent with the MD5 value, if the pathological section file is consistent with the MD 3578 value, representing that the pathological section file is not updated, if the pathological section file is not consistent with the MD5 value, updating and uploading the pathological section file to the object storage server based on the uploading method, and resetting the browsing frequency to be 0.
One or more embodiments provide a pathological digital section data dynamic management method based on object storage, which is applied to an object storage server, wherein the object storage server is provided with a database, a standard storage area and a low-frequency storage area, the database is used for recording object storage addresses and browsing times of uploaded pathological digital section files, and the standard storage area and the low-frequency storage area are used for storing corresponding folders of the pathological digital section files; the method comprises the following steps:
receiving the pathological section file uploaded by the block storage server and storing the pathological section file in a standard storage area;
when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database;
and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address of the pathological section file in the database.
The above one or more technical solutions have the following beneficial effects:
the method comprehensively utilizes the advantages of block storage and object storage, expands an object storage mode on the basis of adopting the block storage for pathological digital section data, uploads the data meeting the uploading time requirement to an object storage server, and in the object storage server, regularly moves pathological section files which are not browsed or rarely browsed for a long time in the standard storage area to the low-frequency storage area by dividing a storage area into the standard storage area and the low-frequency storage area, so that newer data are still managed in the block storage mode, the data reading efficiency is ensured, the older data adopt the object storage mode, and the low-frequency storage area is adopted for the data with few browsing times, on one hand, the data aiming at the object storage mode is ensured, the access is mainly aiming at the standard storage area, and the data reading efficiency is ensured; on the other hand, the low-frequency storage area is adopted, so that the cost is saved.
The method comprises the steps of respectively storing basic information of pathological section files and images on each layer, creating a folder for each pathological section file, and storing object storage addresses of the images on each layer by adopting an image information storage address file, so that when the pathological section files are read, the folders can be searched on the basis of the whole object storage addresses of the files, then the images on each layer of the pathological section files can be quickly acquired on the basis of the image information storage address files in the folders, and the reading efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.
FIG. 1 is a schematic overall view of the dynamic management of a pathological section file according to one or more embodiments of the present invention;
FIG. 2 is a flowchart of an overall process of uploading a pathological section file to a standard storage area according to one or more embodiments of the present invention;
fig. 3 is a flowchart of a dynamic management method for pathological digital section data according to one or more embodiments of the present invention;
FIG. 4 is a flowchart illustrating an implementation of a specific algorithm for uploading a pathological section file to a standard storage area according to one or more embodiments of the present invention;
FIG. 5 is a flow diagram illustrating an implementation of a pathological section file browsing algorithm in accordance with one or more embodiments of the invention;
fig. 6 is a flowchart of an algorithm implementation of moving a pathological section file from a standard storage area to a low-frequency storage area according to one or more embodiments of the present invention.
Detailed Description
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The embodiments and features of the embodiments of the present invention may be combined with each other without conflict.
Example one
In order to solve the pressure brought by storage space and management due to the increasing storage demand, on the basis that a server manages pathological digital section files based on block storage, the embodiment adds an object storage server, wherein the object storage server is provided with a database, a standard storage area and a low-frequency storage area, the database is used for recording the synchronization time, the object storage address and the browsing times of uploaded pathological digital section files, and the standard storage area and the low-frequency storage area are used for storing folders corresponding to the pathological digital section files; each folder is used for storing basic information of a pathological section file, each layer of image information array and an object storage address file corresponding to each layer of image information; the object storage address recorded in the database of the pathological digital section file adopts the object storage address of the basic information of the pathological section file.
Wherein the basic information includes, but is not limited to, original sample map, navigation map, label map, and scanning information. Each layer of image information array is an array formed by each layer of slice images of the pathological section file.
The standard storage area and the low frequency storage area are provided by the OSS service provider. The standard storage is calculated according to the actual size of the object, the lowest storage time requirement is not required, real-time access is supported, the uploading and downloading speed is fastest, and the price is higher than that of low-frequency storage; files with the minimum metering size of 64KB and less than 64KB are charged according to 64KB, the minimum storage time is required to be 30 days, and the uploading and downloading speed is relatively slow but the price is lower.
The embodiment provides a pathological digital section data dynamic management system based on object storage, which, as shown in fig. 1, comprises a block storage server and an object storage server, wherein,
the block storage server is used for acquiring pathological section files meeting the synchronization time in the block storage region according to the set data synchronization period and uploading the pathological section files to the target storage server;
the object storage server is used for receiving the pathological section files uploaded by the block storage server and storing the pathological section files in the standard storage area; when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database; and
and determining a corresponding folder according to the corresponding object storage address, acquiring an image according to the corresponding object storage address of each layer of image information, moving the image to a low-frequency storage area, and updating the object storage address of the pathological section file in the database.
Wherein, uploading the pathology digital slice file to the object storage server specifically comprises:
step A1: according to a set data synchronization period, a block storage server acquires basic information of a pathological section file to be uploaded and uploads the basic information to an object storage server;
step B1: the object storage server creates a folder for the pathological section file, stores the basic information, calculates the MD5 value of the basic information and returns the MD5 value to the block storage server;
step C1: the block storage server calculates an MD5 value based on locally stored basic information, compares the MD5 value with an MD5 value acquired from a target storage server, passes verification if the MD5 value is consistent with the MD5 value, and further uploads image information of each layer of the pathological section file; if the verification fails, generating an error log;
step D1: the object storage server sequentially calculates the MD5 value of each layer according to each layer of image information of the pathological section file and returns the value to the block storage server;
step E1: the block storage server sequentially calculates MD5 values based on locally stored image information of each layer, the MD5 values are compared with the MD5 values of the corresponding layer obtained from the object storage server, and if the MD5 values are all consistent, the check is passed; each layer of image information is verified, an object storage address of the layer of image information in an object storage server is obtained and stored in an image information storage address file; if a layer of image information fails to be checked, generating an error log;
step F1: uploading the image information storage address file to a corresponding folder of the object storage server;
step G1: acquiring a target storage address corresponding to the basic information, and storing the target storage address as a target storage address of the pathological section file; the browsing times is set to 0; sending a last successful message of the pathological section file to a block storage server, and deleting a local path in an object storage server;
step H1: the block storage server deletes the local pathological section file.
The method for synchronizing the pathological section files uploaded to the object storage server specifically comprises the following steps:
step A2: the object storage server acquires pathological section files meeting the synchronization time according to the set data synchronization period, and acquires corresponding object storage addresses;
step B2: acquiring corresponding basic information, calculating an MD5 value, and sending the value to a block storage server;
step C2: the block storage server searches whether the pathological section file exists locally, and if not, the case section file is not updated; if the pathological section file exists, calculating the MD5 value of the basic information, comparing the MD5 value with the MD5 value received from the object storage server, judging whether the pathological section file is consistent with the MD5 value, if the pathological section file is consistent with the MD 3578 value, representing that the pathological section file is not updated, if the pathological section file is not consistent with the MD5 value, updating and uploading the pathological section file to the object storage server based on the uploading method, and resetting the browsing frequency to be 0.
Specifically, browsing the designated pathological section file in the object storage server includes:
step A3: the method comprises the steps that a block storage server obtains a pathological section file browsing request and sends the pathological section file browsing request to an object storage server; the browsing request comprises identification information of the pathological section file; such as the filename of the pathological section file;
step B3: and the object storage server receives the browsing request, inquires a corresponding object storage address, determines a corresponding folder according to the object storage address, acquires images layer by layer according to the corresponding object storage address of each layer of image information, transmits the images to the block storage server, and adds 1 to the browsing times of the pathological section file after transmission is finished.
The algorithm implementation steps of the dynamic management method specifically comprise:
(1) setting a timing synchronization time A, e.g., beginning at 0:00 per day;
(2) setting the time period B of the pathological section files to be synchronized;
(3) time of arrival A;
(4) acquiring all files with pathological section files added earlier than the files under the condition B in the block storage server, and placing the files in a list C; if the list C is empty, setting a to a ═ a × 2 (set once every two or more days), and going to step (1); otherwise, turning to the step (5);
(5) c is traversed through a for loop, and i is set to be 0;
(5.1) z ═ ci ], and it is determined whether the file exists in the object storage database. If not, going to the step (5.2), and if so, going to the step (5.6);
(5.2) acquiring contents of Z local pathological section files of a block storage server, acquiring basic information BI (an original sample map, a navigation map, a label map and scanning information) of the pathological section files in the contents and image information arrays TI of all levels of the images, and creating JSON files TX with Z names in Z local paths;
(5.3) generating the MD5 value of z.BI. Creating a folder in the object storage according to the current file name, uploading, acquiring the MD5 value of the BI in the object storage address after uploading is finished, comparing whether the MD5 value is consistent with the MD5 value in the z object storage address, consistently transferring to the step (5.4) to upload the pathological section image information TI, and inconsistently writing into an error log and transferring to the step (5.3) to upload again;
(5.4) traversing the array z.ti through a for loop, setting u to 0;
(5.4.1) generate MD5 values for local z.TI [ u ]. Uploading, acquiring the MD5 value of z.TI [ u ] in the object storage address after uploading is finished, comparing whether the z.TI [ u ] local MD5 value is consistent with the MD5 value in the object storage address, and if not, uniformly turning to the step (5.4.2) and not, turning to the step (5.4.3);
(5.4.2) additionally write the address of z.ti [ u ] in the object store and u as a key value pair to the TX file in the z-local path, and go to step (5.4.4).
And (5.4.3) writing an error log and transferring to the step (5.4.1) to upload again.
(5.4.4) u + +, if u is less than or equal to the length of z.TI, continuing to circulate in the step (5.4.1), and if the u is greater than the length of z.TI, uploading the JSON file TX to the object storage under the corresponding z name folder, ending the circulation and then turning to the step (5.5);
and (5.5) writing the address of the z.BI in the object storage as z whole in the object storage into the address field of the object storage of the database, and deleting the local z pathological section file and the local path of the z in the database. Setting the browsing times in the database to be 0, and turning to the step (5.8);
(5.6) acquiring the MD5 value of the pathological section file of z in the object storage, judging whether the MD5 value of basic information BI of the z local pathological section file is consistent with the MD5 value of the pathological section file in the object storage, if the value is consistent, indicating that the file is not updated, setting a local address F to be null, and turning to the step (5.8); if the inconsistent file represents that the file is changed, the slice file is required to be uploaded and the object storage address is required to be updated, and if the step (5.2) to the step (5.5) is carried out, the step (5.7) is carried out;
(5.7) uploading the z update to a subject for storage, deleting the local z pathological section file and the local path of z in the database, setting the browsing times 0 in the database, and turning to the step (5.8);
(5.8) i + +, if i is less than or equal to the length of C, continuing to circulate in the step (5.1), and if i is greater than C, ending the circulation and turning to the step (6);
(6) uploading the pathological section file to the end of high-frequency object storage; if C is more than 10000, A is A/2; go to step (1).
And (3) browsing the slices, which comprises the following steps:
(7) browsing the pathological section file FS, judging whether the object storage path of the FS in the database is empty, and if the object storage path of the FS in the database is empty, the pathological section file is not uploaded to the object storage, and turning to the step (7.1); if not, it represents that the pathological section file has been uploaded to the subject storage or the local existence update file, and goes to step (7.2).
(7.1) acquiring the path address of the local pathological section of the FS, opening the FS for browsing, and enabling the browsing frequency of the FS in the database to be + 1;
(7.2) judging whether the FS is empty locally, if not, idling to the step (7.1), and if so, idling to the step (7.3);
(7.3) acquiring an object storage address (possibly a high-frequency storage address and possibly a low-frequency storage address without distinguishing) of the FS in the database, creating a local file according to the generation of a slice folder where the address is located and a suffix name as a format, and downloading basic information of the pathological slice file to the local;
(7.4) acquiring a first layer address in a JSON file TX in the current object storage folder;
and (7.5) downloading the first-layer image file according to the address, additionally writing the first-layer image file into the BI file (the suffix name of the BI file is the original file suffix name), and browsing. And before downloading, locally generating a folder for storing pathological section files according to the current year and month, and when browsing the Nth layer, pre-acquiring addresses of the (N + 1) layers in the TX and downloading. And updating the local path address of the FS in the database, and enabling the browsing times of the FS in the database to be + 1.
Since the low frequency storage area takes longer to download, a progress bar may be displayed at the time of downloading for better user experience.
(8) And finishing pathological section browsing.
Moving from high frequency storage to low frequency storage, the steps are as follows:
(9) setting the moving time U of the timed high-low frequency file;
(10) setting a file time period R to be stored at low frequency;
(11) when the time reaches U, acquiring all pathological section files which meet the R condition and are empty in the local path F from the database, placing the pathological section files in a list G, if G is empty, setting U to U2 (setting to be once every two days or once every more days), and turning to the step 9; otherwise go to step 11.
(12) And f, cycling through G and setting j to be 0.
(12.1) judging whether the browsing times of the T are equal to 0, turning to the step (12.2) when the browsing times of the T are equal to 0, and turning to the step (12.3) when the browsing times are larger than 0;
and (12.2) according to the object storage address of the T in the database, the pathological section file is transferred from the standard storage area to the low-frequency storage area, and the low-frequency storage address of the T is stored in the database after the movement is finished. Go to step (12.3);
(12.3) j + +, judging whether j is smaller than the length of G, if so, going to the step (12.1), and if so, going to the step 13.
(13) Moving the pathological section file from the standard storage area to the low-frequency storage area to finish; if U is greater than 10000, U is equal to U/2, and the step (9) is carried out.
The pathological digital section data dynamic management method based on object storage is applied to an object storage server and specifically comprises the following steps:
the block storage server acquires pathological section files meeting the synchronization time in the block storage region according to the set data synchronization period, and uploads the pathological section files to the object storage server;
step 1: receiving a pathological section file uploaded by a block storage server and storing the pathological section file in a standard storage area;
step 2: when the set moving time is reached, searching a pathological section file with the uploading time earlier than the set time and the browsing times less than a set threshold value from the standard storage area;
and 3, step 3: and determining a corresponding folder according to the corresponding object storage address, acquiring an image according to the corresponding object storage address of each layer of image information, moving the image to a low-frequency storage area, and updating the object storage address of the pathological section file.
Example two
Based on the method provided by the first embodiment, the present embodiment provides an object storage server, which dynamically manages pathological digital section data based on object storage, and the object storage server is used for storing uploading time, object storage addresses, browsing times and corresponding folders of uploaded pathological digital section files, and is provided with a standard storage area and a low-frequency storage area.
When the set moving time is reached, searching a pathological section file with the uploading time earlier than the set time and the browsing times less than a set threshold value from the standard storage area;
and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address of the pathological section file.
Each folder is used for storing basic information of a pathological section file, image information of each layer and an image information storage address file, wherein the image information storage address file is used for storing object storage addresses of images of each layer of the pathological section file; and the object storage address of the pathological digital section file adopts the object storage address of basic information of the pathological section file.
And updating the object storage address of the pathological section file and updating the image information storage address file at the same time.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive changes in the technical solutions of the present invention.
Claims (10)
1. A pathology digital section data dynamic management system based on object storage is characterized by comprising a block storage server and an object storage server; wherein the content of the first and second substances,
the block storage server is configured to acquire pathological section files meeting the synchronization time according to a set data synchronization period and upload the pathological section files to the object storage server;
the object storage server is provided with a database, a standard storage area and a low-frequency storage area, wherein the database is used for recording object storage addresses and browsing times of uploaded pathological digital slice files, and the standard storage area and the low-frequency storage area are used for storing folders corresponding to the pathological digital slice files; the object storage server is configured to:
receiving a pathological section file uploaded by a block storage server and storing the pathological section file in a standard storage area;
when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database;
and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address of the pathological section file in the database.
2. The system for dynamically managing pathological digital section data based on object storage according to claim 1, wherein each folder is used for storing basic information of a pathological section file, image information of each layer and an image information storage address file, wherein the image information storage address file is used for storing object storage addresses of images of each layer of the pathological section file; and the object storage address recorded in the database of the pathological digital section file adopts the object storage address of the basic information of the pathological section file.
3. The system for dynamically managing pathological digital section data based on object storage according to claim 2, wherein said image information storage address file is updated simultaneously with the object storage address of said pathological section file.
4. The system for dynamically managing pathological digital section data based on object storage according to claim 1, wherein the uploading method of pathological digital section file comprises:
the block storage server acquires basic information of a pathological section file to be uploaded according to a set data synchronization period, and uploads the basic information to the object storage server;
verifying the data consistency before and after uploading the basic information, and if the verification fails, generating an error log;
if the verification is successful, further uploading each layer of image information of the pathological section file, verifying the data consistency before and after each layer of image information is uploaded, and if the verification fails, generating an error log; if the verification is successful, generating an image information storage address file and uploading the image information storage address file to an object storage server;
when the object storage server receives the basic information of the pathological section file, a folder is created for the pathological section file in the standard storage area, and the folder is used for storing the basic information of the pathological section file, the image information of each layer and the image information storage address file.
5. The subject storage based pathological digital section data dynamic management system according to claim 4, wherein the checking of the data consistency before and after uploading the basic information comprises:
after the object storage server receives the basic information of the pathological section file, calculating an MD5 value of the basic information and returning the MD5 value to the block storage server;
and the block storage server calculates the MD5 value based on the locally stored basic information, compares the MD5 value with the MD5 value acquired from the object storage server, and if the MD5 value is consistent with the MD5 value, the verification is successful.
6. The system of claim 4, wherein the checking of the consistency of the data before and after uploading the image information of each layer comprises:
the object storage server calculates the MD5 value of each layer in turn according to the received image information of each layer and returns the value to the block storage server;
the block storage server calculates an MD5 value based on the layer of locally stored image information, compares the MD5 value with an MD5 value acquired from the object storage server, and generates an error log if one layer of image information fails to be verified;
if the layer of image information is consistent with the layer of image information, the verification is successful, the object storage address of the layer of image information in the object storage server is obtained, and the layer of image information is stored in an image information storage address file; and continuously performing the verification of the next layer of image information.
7. The dynamic management system for pathological digital slice data stored on an object basis according to any one of claims 4 to 6, wherein the uploading method further comprises:
the object storage server acquires an object storage address corresponding to the basic information and stores the object storage address as the object storage address of the pathological section file; the browsing times is set to 0; sending a last successful message of the pathological section file to a block storage server, and deleting a path of the block storage server in the object storage server;
the block storage server deletes the local pathological section file.
8. The system for dynamic management of pathological digital section data based on object storage according to claim 4, wherein browsing designated pathological section files in the object storage server comprises:
the method comprises the steps that a block storage server obtains a pathological section file browsing request and sends the pathological section file browsing request to an object storage server; the browsing request comprises identification information of the pathological section file;
and the object storage server receives the browsing request, inquires a corresponding object storage address, determines a corresponding folder according to the object storage address, acquires images layer by layer according to the corresponding object storage address of each layer of image information, transmits the images to the block storage server, and adds 1 to the browsing times of the pathological section file after transmission is finished.
9. The system for dynamic management of pathological digital section data based on object storage according to claim 4, wherein the method for synchronizing pathological section files uploaded to the object storage server comprises:
the object storage server acquires pathological section files meeting the synchronization time according to the set data synchronization period, and acquires corresponding object storage addresses;
acquiring corresponding basic information, calculating an MD5 value, and sending the value to a block storage server;
the block storage server searches whether the pathological section file exists locally or not, and if the pathological section file does not exist, the case section file is not updated; if the pathological section file exists, calculating the MD5 value of the basic information, comparing the MD5 value with the MD5 value received from the object storage server, judging whether the pathological section file is consistent with the MD5 value, if the pathological section file is consistent with the MD 3578 value, representing that the pathological section file is not updated, if the pathological section file is not consistent with the MD5 value, updating and uploading the pathological section file to the object storage server based on the uploading method, and resetting the browsing frequency to be 0.
10. A pathology digital section data dynamic management method based on object storage is applied to an object storage server and is characterized in that a database, a standard storage area and a low-frequency storage area are arranged in the object storage server, the database is used for recording object storage addresses and browsing times of uploaded pathology digital section files, and the standard storage area and the low-frequency storage area are used for storing corresponding folders of the pathology digital section files; the method comprises the following steps:
receiving a pathological section file uploaded by a block storage server and storing the pathological section file in a standard storage area;
when the set moving time is reached, searching pathological section files with uploading time earlier than the set time and browsing times less than a set threshold value in a standard storage area according to a database;
and determining a corresponding folder according to the corresponding object storage address, moving to a low-frequency storage area, and updating the object storage address of the pathological section file in the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768824.8A CN115017163A (en) | 2022-07-01 | 2022-07-01 | Pathological digital section data dynamic management system and method based on object storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768824.8A CN115017163A (en) | 2022-07-01 | 2022-07-01 | Pathological digital section data dynamic management system and method based on object storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115017163A true CN115017163A (en) | 2022-09-06 |
Family
ID=83078173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210768824.8A Pending CN115017163A (en) | 2022-07-01 | 2022-07-01 | Pathological digital section data dynamic management system and method based on object storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115017163A (en) |
-
2022
- 2022-07-01 CN CN202210768824.8A patent/CN115017163A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3806424A1 (en) | File system data access method and file system | |
| US10452271B2 (en) | Reconstructing in-memory indices in a distributed data storage system | |
| US7860907B2 (en) | Data processing | |
| JP5661188B2 (en) | File system and data processing method | |
| US20160110109A1 (en) | Using scratch extents to facilitate copying operations in an append-only storage system | |
| US10296518B2 (en) | Managing distributed deletes in a replicated storage system | |
| US8095678B2 (en) | Data processing | |
| US9195666B2 (en) | Location independent files | |
| US9772783B2 (en) | Constructing an index to facilitate accessing a closed extent in an append-only storage system | |
| WO2012090549A1 (en) | Information management method, and computer for providing information | |
| CN108776682B (en) | Method and system for randomly reading and writing object based on object storage | |
| WO2021237467A1 (en) | File uploading method, file downloading method and file management apparatus | |
| US20150254320A1 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| US11422721B2 (en) | Data storage scheme switching in a distributed data storage system | |
| US20160139980A1 (en) | Erasure-coding extents in an append-only storage system | |
| CN112334891B (en) | Centralized storage for search servers | |
| US8176087B2 (en) | Data processing | |
| CN107220248B (en) | Method and device for storing data | |
| CN113688139A (en) | Object storage method, gateway, device and medium | |
| CN115017163A (en) | Pathological digital section data dynamic management system and method based on object storage | |
| CN105162891A (en) | Data storage method based on IP network | |
| CN112286448B (en) | Object access method and device, electronic equipment and machine-readable storage medium | |
| US8886656B2 (en) | Data processing | |
| CN112364026A (en) | Distributed data fragment storage and reading method | |
| CN111562936B (en) | Object history version management method and device based on Openstack-Swift |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |