CN113032348A - Spatial data management method, system and computer readable storage medium - Google Patents
Spatial data management method, system and computer readable storage medium Download PDFInfo
- Publication number
- CN113032348A CN113032348A CN202110568927.5A CN202110568927A CN113032348A CN 113032348 A CN113032348 A CN 113032348A CN 202110568927 A CN202110568927 A CN 202110568927A CN 113032348 A CN113032348 A CN 113032348A
- Authority
- CN
- China
- Prior art keywords
- file
- metadata information
- small
- information
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a spatial data management method, a system and a computer readable storage medium, wherein the method comprises the following steps: acquiring an object file generation request; generating an object file and an index file according to the object file generation request, wherein the object file is a continuous storage space of a file system; when the spatial data need to be written into a file system, acquiring metadata information and small file contents corresponding to small files of the spatial data; writing the metadata information and the small file content into an object file; and writing the metadata information into the index file, and generating positioning information, wherein the positioning information is used for positioning the metadata information in the index file, and the metadata information is used for positioning the small file content in the object file.
Description
Technical Field
The present application relates to the field of data storage and processing, and in particular, to a method and system for managing spatial data and a computer-readable storage medium.
Background
Currently, spatial data is composed of many types of small files, such as the gdb format or the shp format, which is commonly used in the vector data industry, or other types of formats, which mostly contain a plurality of small files, such as map tiles, composed of many small files with file sizes ranging from several KB to several tens of KB. The related art generally stores the spatial data composed of small files directly to the file system of the operating system, which has the advantage of simple operation, but causes the following three problems:
the file retrieval method includes the steps that firstly, excessive index nodes (inodes) are caused, the inodes are metadata information of files and include information of byte numbers, authority limits, file block numbers and the like of the files, each file stores one inode on a disk, under the same capacity, a large number of inodes are generated in small files compared with large files, the large number of small files reaches the upper limit of the inode of a file system, and therefore file retrieval performance is low, and the fact that the files cannot be written continuously under extreme conditions is caused.
And secondly, disk fragments are formed, a large number of small files are not stored in a continuous disk space, so that a large number of disk fragments are caused, the discontinuous disk storage causes that a magnetic head consumes a large amount of time for addressing and positioning when the small files are read, and the file reading and writing speed is greatly reduced.
Thirdly, the storage space is expanded, for example, in a Windows file system, the minimum unit of disk management is a cluster (block), for example, the minimum space of the cluster is 4KB, even a 1KB file occupies a 4KB hard disk space, a 5KB file occupies an 8KB hard disk space, and other operation principles such as Linux are similar, so that the storage of a small file can cause the expansion of the disk occupied space.
Based on the above 3 problems, the related art may cause the operating system to have limited ability to manage small files of massive spatial data, limited data capacity, and extremely low writing and reading efficiency.
Disclosure of Invention
In order to help improve the read-write efficiency and the management capability of small files of spatial data, the application provides a spatial data management method, a spatial data management system and a computer-readable storage medium.
In a first aspect, the present application provides a spatial data management method, which adopts the following technical solution:
a spatial data management method, comprising:
receiving a file writing request, and generating an object file for storing data;
writing spatial data into an object file and generating an index file according to the file writing request, wherein the object file is a continuous storage space of a file system;
when spatial data need to be written into the file system, acquiring metadata information and small file contents corresponding to small files of the spatial data;
writing the metadata information and the small file content into the object file;
and writing the metadata information into the index file, and generating positioning information, wherein the positioning information is used for positioning the metadata information in the index file, and the metadata information is used for positioning the small file content in the object file.
By adopting the technical scheme, before the space data is written, the object file and the index file are produced according to the object file production request, the object file is a contiguous storage space in the disk space of the file system, and when space data needs to be written to the file system, obtaining the metadata information and the small file content corresponding to the small file of the space data, writing the metadata information and the small file content into an object file, writing the metadata information into an index file, and generating positioning information, thereby establishing an index association relationship between the positioning information and the small file content, compared with the related art, because the small files are stored in the object files in the continuous storage space in a centralized manner, a large number of index nodes cannot be generated in the storage of the massive small files, disk fragments cannot be formed, the expansion of the occupied space of the disk cannot be caused, and the read-write efficiency and the management capability of the small files of the space data are improved.
Optionally, the obtaining metadata information and the small file content corresponding to the small file of the spatial data includes:
when space data need to be written into the file system, acquiring small file contents corresponding to small files of the space data, wherein the number of the small files is at least one;
determining the number of bytes of the content according to the content of the small file;
determining the number of written bytes according to the number of bytes of the content and the number of bytes of preset metadata;
judging whether the residual storage space of the object file meets the written byte number or not;
if yes, determining a starting byte bit;
generating metadata information according to the initial byte bit and the written byte number;
and if not, not writing the spatial data into the file system.
By adopting the technical scheme, the actual content of the small file is the content of the small file, the number of bytes of the content of the small file is determined, the metadata information corresponding to the small file also needs to be stored, the fixed number of bytes is provided, the number of bytes written in is obtained according to the number of bytes of the content and the number of bytes of preset metadata, the storage space needed by the small file is stored, and when the residual storage space of the object file is judged to be satisfied, the initial byte bit and the number of bytes written in are determined, and the metadata information is generated. The generation of the metadata information ensures that the content of the small file is accurately positioned, and the judgment of the residual storage space of the object file ensures that the small file can be written into the object file.
Optionally, after the writing the metadata information and the small file content into the object file, the method further includes:
when the metadata information and the small file content are written into the object file, recording whether the writing process of the metadata information and the small file content is abnormal or not;
if the writing process of the metadata information and the small file content is not abnormal, writing the metadata information into the index file and generating positioning information;
and if the writing process of the metadata information and the small file content is abnormal, reporting first abnormal information.
By adopting the technical scheme, in the process of writing the small file, whether the writing process is abnormal or not is recorded, the metadata information is written into the index file when no abnormality occurs, the positioning information is generated, and if the abnormality occurs, the first abnormal information is reported to prompt that the abnormality occurs in the writing process of the metadata information and the content of the small file and needs to be processed. And the exception check of the small file writing process is added, and the correctness of the writing process is ensured.
Optionally, the writing the metadata information into the index file and generating the positioning information include:
writing the metadata information to the index file;
recording whether the writing process of the metadata information is abnormal or not;
if the writing process of the metadata information is not abnormal, generating positioning information, wherein the positioning information is used for positioning the metadata information in the index file;
and if the writing process of the metadata information is abnormal, reporting second abnormal information.
By adopting the technical scheme, when the metadata information is written into the index file, whether the writing process is abnormal or not is recorded, the positioning information is generated, the second abnormal information is generated when the writing process is abnormal, and the processing is required when the writing process of the metadata information is abnormal. The method and the device increase the abnormal check in the process of writing the metadata information into the index file and ensure the correctness in the writing process.
Optionally, the method further includes:
when a target small file needs to be read from the object file, acquiring positioning information corresponding to the target small file;
reading corresponding metadata information from the index file according to the positioning information;
analyzing the metadata information to obtain a starting byte bit and a written byte number;
and reading the target small file from the object file according to the starting byte bit and the written byte number.
By adopting the technical scheme, when the target small file needs to be read from the target file, the corresponding metadata information is read from the index file according to the positioning information corresponding to the target small file, and the initial byte bit and the writing byte number of the target small file when the target small file is written into the target file are determined according to the metadata information, so that the target small file can be simply read without consuming time for addressing and positioning.
Optionally, the metadata information further includes an information processing item, the information processing item is used for indicating whether the small file content is deleted in the object file,
the method further comprises the following steps:
and marking the information processing item of the metadata information as deletion, and synchronously modifying the metadata information in the object file to realize soft deletion of the small file content.
By adopting the technical scheme, the metadata information comprises an information processing item besides the initial byte bit and the written byte number, the information processing item is used for indicating whether the content of the small file is deleted in the object file, the information processing item of the metadata information is marked to be deleted, and the metadata information in the object file is synchronously modified, so that the content of the small file is softly deleted.
Optionally, the method further includes:
and writing the modified file corresponding to the small file content into the object file, and deleting the small file content from the object file.
By adopting the technical scheme, if the small file is modified, the modified file corresponding to the content of the small file needs to be written into the object file firstly, and then the content of the small file is deleted from the object file, so that the modification of the small file in the object file is realized.
Optionally, the method further includes:
when the information processing item of the metadata information is marked to be deleted, the metadata information is read out from the index file regularly;
and deleting the corresponding small file content and the corresponding metadata information in the object file according to the metadata information, and deleting the metadata information in the index file.
By adopting the technical scheme, when the information processing item of the metadata information is marked as deleted, namely soft deletion is performed, the content of the small file is actually stored in the object file, in order to release the storage space, the soft deleted small file needs to be physically deleted, the metadata information is read from the index file, the corresponding small file content and the metadata information in the object file are positioned according to the metadata information, then deletion is performed, and then the metadata information in the index file is deleted, so that the physical deletion of the small file is realized, and the storage space after soft deletion is released.
In a second aspect, the present application provides a spatial data management system, which adopts the following technical solutions:
a spatial data management system, comprising:
the system comprises a central node, a file storage node and a file system;
the central node is used for acquiring an object file generation request;
the file storage node is used for generating an object file and an index file according to the object file generation request, wherein the object file is a continuous storage space of a file system;
the file storage node is further configured to, when spatial data needs to be written into the file system, obtain metadata information and small file content corresponding to a small file of the spatial data, write the metadata information and the small file content into the object file, write the metadata information into the index file, and generate positioning information, where the positioning information is used to position the metadata information in the index file, and the metadata information is used to position the small file content in the object file.
By adopting the technical scheme, before the spatial data is written into the file system, the central node acquires the object file generation request, the file storage node firstly generates the object file and the index file according to the object file production request, the object file is a continuous storage space in the disk space of the file system, when the spatial data needs to be written into the file system, the file storage node acquires the metadata information and the small file content corresponding to the small file of the spatial data, writes the metadata information and the small file content into the object file, writes the metadata information into the index file, and generates the positioning information, thereby establishing the index association relationship between the positioning information and the small file content, compared with the related technology, because the small files are intensively stored in the object file of the continuous storage space, even a large number of small files can not generate a large number of index nodes, and disk fragments can not be formed, the expansion of the occupied space of the disk can not be caused, and the read-write efficiency and the management capability of the small files of the space data are improved.
In a third aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executes the spatial data management method in the above first aspect.
In summary, the present application includes at least one of the following beneficial technical effects:
1. because the small files are stored in the object files in the continuous storage space in a centralized manner, a large number of index nodes cannot be generated in the storage of the massive small files, disk fragments cannot be formed, the expansion of the occupied space of the disk cannot be caused, and the read-write efficiency and the management capability of the small files of the space data are improved;
2. the metadata information comprises a start byte bit and a write byte number, so that the accurate positioning of the content of the small file in the object file is ensured, and the judgment of the residual storage space of the object file ensures that the small file can be successfully written into the object file.
Drawings
Fig. 1 is a flowchart illustrating a spatial data management method according to an embodiment of the present application.
Fig. 2 is a schematic flowchart of acquiring metadata information and small file content according to an embodiment of the present application.
FIG. 3 is a first flowchart of a write abort determination according to an embodiment of the present disclosure.
FIG. 4 is a second flowchart of a write abort determination according to an embodiment of the present disclosure.
FIG. 5 is a flowchart illustrating a process of reading a doclet according to an embodiment of the present application.
Fig. 6 is a schematic structural diagram of a spatial data management system according to an embodiment of the present application.
Description of reference numerals: 601. a central node; 602. a file storage node; 603. a file system.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to fig. 1-6 and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The embodiment of the application discloses a spatial data management method.
Example 1
Referring to fig. 1, the method is performed by steps including:
101, obtaining the object file generation request.
In the related art, the disk space of the file system of the operating system is discontinuous when storing a large amount of small files, and therefore, an object file having a continuous storage space needs to be generated to obtain an object file generation request.
And 102, generating an object file and an index file according to the object file generation request.
After the object file generation request is obtained, a continuous storage space, for example, a 64GB storage space, is applied to the operating system for storing the actual content of the small file of the space data, and then a 64GB file with a suffix name of dat (short for data), that is, the object file, is generated. An index file with the suffix name idx (shorthand for index) is also created.
103, when the space data needs to be written into the file system, the metadata information and the small file content corresponding to the small file of the space data are obtained.
The spatial data is in the field of geographic information, and is composed of many types of small files, such as a gdb format or a shp format or other types of formats, which are commonly adopted in the vector data industry, and most of the formats contain a plurality of small files, such as map tiles, which are composed of many small files with the file size ranging from a few KB to a few tens of KB. And when the spatial data need to be written into the file system, acquiring the corresponding metadata information and the small file content of each small file of the spatial data. The metadata information corresponds to an inode (inode) of the doclet for locating the doclet content in the object file.
And 104, writing the metadata information and the small file content into the target file.
The metadata information and the small file content form the common content of the small file, and need to be written into the object file together.
And 105, writing the metadata information into the index file and generating the positioning information.
The metadata information is used as an index of the content of the small files and needs to be stored in the index file, so that the actual content of all the small files is written into the object file, the metadata information of all the small files is written into the index file, and the positioning information of the metadata information is generated and is used for positioning the metadata information in the index file.
The implementation principle of the embodiment 1 is as follows: before writing in the space data, the object file and the index file are produced according to the object file production request, when the space data needs to be written in a file system, the metadata information and the small file content corresponding to the small file of the space data are obtained, the metadata information and the small file content are written in the object file, the metadata information is written in the index file, and the positioning information is generated, so that the index association relation is established between the positioning information and the small file content.
It should be noted that, under the condition that the index file is damaged, because the metadata information and the small file content are stored in the object file, the index file can be restored and constructed by acquiring the metadata information in the object file; according to the business needs, object files and index files can be simultaneously stored among multiple machine rooms, multiple racks, multiple servers and multiple disks to form multiple copies; after the small file is written, the file HASH value (HASH) can be compared, and HASH is calculated according to the file content, such as md5, so that successful writing can be ensured. The security performance of the storage is improved.
It should be noted that the index file can be cached in the memory, so that the efficiency of file retrieval is improved; meanwhile, a plurality of index files and object files are stored, the file contents are consistent, small files can be read from different positions through a load balancing mechanism, and the reading speed is increased by times along with the number of copies.
In addition to optimizing the storage of small files in the spatial data, the large files can still be cut into pieces for storage, and the advantage of the storage of multiple copies can be utilized to improve the reading efficiency.
In addition, because the metadata information and the small file content of the small file are stored in the pair consisting of the index file and the object file, the small file is not limited by the capacity of a single machine and can be distributed in a plurality of disks and a plurality of servers, and the positioning information comprises the information capable of directly positioning the index file and the object file, the metadata information is stored and maintained without a uniform central node, the small file has horizontal linear expansion capability, and the problem of performance reduction along with the increase of the object file and the index file pair is solved.
Example 2
Referring to fig. 2, the present embodiment is different from embodiment 1 in that, in step 101, metadata information and small file content corresponding to a small file of spatial data are acquired, and specific execution steps include:
201, when the space data needs to be written into the file system, acquiring the small file content corresponding to the small file of the space data.
When the space data needs to be written into the file system, the small file content corresponding to each small file of the space data is obtained, and the small file content is the actual content of the small file.
202, determining the number of bytes of the content according to the content of the small file.
The small files of the space data are in a data mode, so the content of the small files is also in a byte data mode, and the number of bytes of the content of the small files is determined.
And 203, determining the number of written bytes according to the number of bytes of the content and the number of bytes of the preset metadata.
According to a file storage rule, metadata information needs to be configured for each small file, the content of the metadata information has a fixed byte length and can be used as the number of bytes of preset metadata, and then the number of bytes of written-in bytes can be determined by the number of bytes of the content plus the number of bytes of the preset metadata.
204, determining whether the remaining storage space of the object file meets the number of written bytes, and if so, executing step 205; if not, go to step 207.
Wherein, the object file may already store the file, or the storage space is limited when applying for, then it needs to be determined whether the remaining storage space of the object file can satisfy the number of bytes written, if so, the object file can be written, and step 205 is executed; if not, go to step 207.
The start byte bit is determined 205.
When the remaining storage space of the object file meets the number of written bytes, the object file is a continuous storage space, and then the starting position of the remaining storage space is used as a starting byte bit.
And 206, generating metadata information according to the initial byte bit and the written byte number.
After the initial byte bit is determined, the storage position of the content corresponding to the small file in the object file can be obtained by combining the number of written bytes, the index function is realized, and the initial byte bit and the number of written bytes are put into the metadata information.
207, the spatial data is not written to the file system.
If the residual storage space of the object file meets the number of written bytes, the file system can not completely write the space data, and the writing is stopped.
The implementation principle of the embodiment 2 is as follows: the actual content of the small file is called as the content of the small file, the content byte number of the content of the small file is determined, the preset metadata byte number is the byte number of the metadata information which needs to be corresponding to the content of the small file, the sum of the content byte number and the preset metadata byte number is the actually written byte number which needs to be written, when the residual storage space of the object file is judged to meet the written byte number, the stored initial byte position and the written byte number are determined, and the metadata information is generated. The generation of the metadata information ensures that the small file is accurately positioned, and the judgment of the residual storage space of the object file ensures that the small file can be written into the object file of the file system.
Example 3
Referring to fig. 3, this embodiment is different from the above embodiments 1 and 2 in that an exception check is added in the writing process of the metadata information and the small file content, so as to ensure that the file in the writing process is correct, and after the metadata information and the small file content are written into the target file, the method includes:
301, when the metadata information and the small file content are written into the object file, recording whether the writing process of the metadata information and the small file content is abnormal or not, and if not, executing step 302; if there is an exception, go to step 303.
When the metadata information and the small file content are written into the object file, recording the abnormal condition in the writing process, if no abnormality occurs, indicating that the object file is normally written, executing step 302; if an exception occurs, indicating that a problem occurred with the write, possibly a write error, step 303 is performed.
302, writing the metadata information to the index file, and generating the positioning information.
When there is no abnormality, the metadata information and the small file content are written into the object file normally, and at this time, the metadata information needs to be written into the index file to generate the positioning information.
303, reporting the first anomaly information.
The first abnormal information is used for prompting that abnormality occurs in the writing process of the metadata information and the small file content, so that a user or an operating system can conveniently process the abnormality.
The implementation principle of the embodiment 3 is as follows: and in the process of writing the small file, recording whether the writing process is abnormal or not, writing the metadata information into the index file when no abnormality occurs, generating the positioning information, and reporting the first abnormal information when the abnormality occurs. And the exception check of the small file writing process is added, and the correctness of the writing process is ensured.
Example 4
Referring to fig. 4, the present embodiment is different from the above embodiments 1 to 3 in that an exception check is added for writing metadata information into an index file, correctness of the metadata information is ensured, the metadata information is written into the index file, and positioning information is generated, including:
401, the metadata information is written to the index file.
402, recording whether the writing process of the metadata information is abnormal or not, and if not, executing a step 403; if there is an exception, go to step 404.
Wherein, record whether the exception occurs in the writing process, if there is no exception, prove that the intact index file of writing into of metadata information, carry out step 403; if an exception occurs, indicating that the metadata information is not completely written to the index file, step 404 is performed.
And 403, generating positioning information.
The positioning information is used for positioning metadata information in the index file.
404, reporting second anomaly information.
The second abnormal information is used for prompting that an abnormality occurs in the writing process of the metadata information so as to facilitate processing of a user or an operating system.
The implementation principle of the embodiment 4 is as follows: and when the metadata information is written into the index file, recording whether the writing process is abnormal or not, generating positioning information when no abnormality occurs, and generating second abnormal information when abnormality occurs. The method and the device increase the abnormal check in the process of writing the metadata information into the index file and ensure the correctness in the writing process.
Example 5
Referring to fig. 5, the difference between this embodiment and the above embodiments 1-4 is that when the writing process is completed and the positioning information is obtained, and then the reading is performed, the specific steps include:
501, when the target small file needs to be read from the target file, the positioning information corresponding to the target small file is obtained.
If the target small file needs to be read from the object file, the positioning information corresponding to the target small file needs to be acquired first.
502, reading corresponding metadata information from the index file according to the positioning information.
The positioning information corresponding to the target small file can position the metadata information in the index file, so that the metadata information can be read from the index file according to the positioning information.
503, analyze the metadata information to obtain the start byte bit and the number of bytes written.
Here, it is explained in the above embodiments 1 to 4 that the metadata information includes the start byte bit and the number of written bytes, and then the parsing can be performed.
And 504, reading the target small file from the object file according to the initial byte bit and the written byte number.
When the starting byte bit and the number of written bytes are known, reading is started from the object file according to the starting byte bit, and the data size of the number of written bytes is read, so that the target small file is obtained.
The implementation principle of the embodiment 5 is as follows: reading corresponding metadata information from the index file according to the positioning information corresponding to the target small file, and determining the initial byte bit and the writing byte number when the target small file is written into the object file according to the metadata information, so that the target small file can be simply read without consuming time for addressing and positioning.
Preferably, in some embodiments of the present application, the metadata information further includes an information processing item, where the information processing item is used to indicate whether the small file content is deleted in the object file,
the spatial data management method further includes:
and marking the information processing item of the metadata information as deletion, and synchronously modifying the metadata information in the object file, so that the content of the small file is softly deleted.
The principle of this embodiment is: the metadata information may include a creation date, a modification date, a file HASH value (HASH), an information processing item, and the like, in addition to the start byte bit and the number of written bytes. The information processing item is specifically used for indicating whether the small file content is deleted in the object file, the information processing item of the metadata information is marked to be deleted, and the metadata information in the object file is synchronously modified, so that the small file content is softly deleted.
Preferably, in some embodiments of the present application, the spatial data management method further includes:
and writing the modified file corresponding to the small file content into the object file, and deleting the small file content from the object file.
The principle of this embodiment is: if the small file is modified, the modified file corresponding to the content of the small file needs to be written into the object file first, and then the content of the small file is deleted from the object file, so that the modification of the small file in the object file is realized.
Preferably, in some embodiments of the present application, the spatial data management method further includes:
when the information processing item of the metadata information is marked as deleted, the metadata information is read out from the index file regularly;
and deleting the corresponding small file content and the metadata information in the object file according to the metadata information, and deleting the metadata information in the index file.
The principle of this embodiment is: when the information processing item of the metadata information is marked as deleted, namely soft deletion is performed, the content of the small file is actually stored in the object file, in order to release the storage space, the soft deleted small file needs to be physically deleted regularly, the metadata information is read from the index file, the corresponding small file content and the metadata information in the object file are positioned according to the metadata information and then deleted, and then the metadata information in the index file is deleted, so that the physical deletion of the small file is realized, and the storage space after the soft deletion is released.
The embodiment of the application also discloses a spatial data management system.
Example 6
Referring to fig. 6, the system includes:
a central node 601, a file storage node 602, and a file system 603;
the central node 601 is used for acquiring an object file generation request;
the file storage node 602 is configured to generate an object file and an index file according to an object file generation request, where the object file is a continuous storage space of the file system 603;
the file storage node 602 is further configured to, when the spatial data needs to be written into the file system 603, obtain metadata information and small file content corresponding to a small file of the spatial data, write the metadata information and the small file content into an object file, write the metadata information into an index file, and generate positioning information, where the positioning information is used to position the metadata information in the index file, and the metadata information is used to position the small file content in the object file.
The implementation principle of the embodiment 6 is as follows: before the spatial data is written into the file system, the central node 601 obtains the object file generation request, the file storage node 602 first generates the object file and the index file according to the object file generation request, when the spatial data needs to be written into the file system 603, the file storage node 602 obtains the metadata information and the small file content corresponding to the small file of the spatial data, writes the metadata information and the small file content into the object file, writes the metadata information into the index file, and generates the positioning information, thereby establishing index association relation between the positioning information and the small file content, compared with the related technology, because the small files are intensively stored in the object files of the continuous storage space, even if a large number of small files cannot generate a large number of index nodes, disk fragments cannot be formed, the expansion of occupied space of a disk cannot be caused, and the read-write efficiency and the management capacity of the small files of the space data are improved.
It should be noted that the central node 601 and the file storage node 602 in embodiment 6 are also used for executing the steps in the above embodiments of the spatial data management method.
The present application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executes the above spatial data management method.
The computer-readable storage medium includes, for example: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The foregoing is a preferred embodiment of the present application and is not intended to limit the scope of the application in any way, and any features disclosed in this specification (including the abstract and drawings) may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
Claims (10)
1. A method for spatial data management, the method comprising:
acquiring an object file generation request;
generating an object file and an index file according to the object file generation request, wherein the object file is a continuous storage space of a file system;
when spatial data need to be written into the file system, acquiring metadata information and small file contents corresponding to small files of the spatial data;
writing the metadata information and the small file content into the object file;
and writing the metadata information into the index file, and generating positioning information, wherein the positioning information is used for positioning the metadata information in the index file, and the metadata information is used for positioning the small file content in the object file.
2. The method for managing spatial data according to claim 1, wherein the acquiring metadata information and small file content corresponding to the small file of the spatial data comprises:
when space data need to be written into the file system, acquiring small file contents corresponding to small files of the space data, wherein the number of the small files is at least one;
determining the number of bytes of the content according to the content of the small file;
determining the number of written bytes according to the number of bytes of the content and the number of bytes of preset metadata;
judging whether the residual storage space of the object file meets the written byte number or not;
if yes, determining a starting byte bit;
generating metadata information according to the initial byte bit and the written byte number;
and if not, not writing the spatial data into the file system.
3. The spatial data management method according to claim 2, wherein after the writing of the metadata information and the small file content into the object file, further comprising:
when the metadata information and the small file content are written into the object file, recording whether the writing process of the metadata information and the small file content is abnormal or not;
if the writing process of the metadata information and the small file content is not abnormal, writing the metadata information into the index file and generating positioning information;
and if the writing process of the metadata information and the small file content is abnormal, reporting first abnormal information.
4. The spatial data management method according to claim 3, wherein said writing the metadata information to the index file and generating the positioning information includes:
writing the metadata information to the index file;
recording whether the writing process of the metadata information is abnormal or not;
if the writing process of the metadata information is not abnormal, generating positioning information, wherein the positioning information is used for positioning the metadata information in the index file;
and if the writing process of the metadata information is abnormal, reporting second abnormal information.
5. The spatial data management method of claim 4, further comprising:
when a target small file needs to be read from the object file, acquiring positioning information corresponding to the target small file;
reading corresponding metadata information from the index file according to the positioning information;
analyzing the metadata information to obtain a starting byte bit and a written byte number;
and reading the target small file from the object file according to the starting byte bit and the written byte number.
6. The spatial data management method according to any one of claims 1 to 5, wherein said metadata information further includes an information processing item for indicating whether or not said small file content is deleted in said object file,
the method further comprises the following steps:
and marking the information processing item of the metadata information as deletion, and synchronously modifying the metadata information in the object file to realize soft deletion of the small file content.
7. The spatial data management method of claim 6, further comprising:
and writing the modified file corresponding to the small file content into the object file, and deleting the small file content from the object file.
8. The spatial data management method of claim 7, further comprising:
when the information processing item of the metadata information is marked to be deleted, the metadata information is read out from the index file regularly;
and deleting the corresponding small file content and the corresponding metadata information in the object file according to the metadata information, and deleting the metadata information in the index file.
9. A spatial data management system, the system comprising:
the system comprises a central node, a file storage node and a file system;
the central node is used for acquiring an object file generation request;
the file storage node is used for generating an object file and an index file according to the object file generation request, wherein the object file is a continuous storage space of a file system;
the file storage node is further configured to, when spatial data needs to be written into the file system, obtain metadata information and small file content corresponding to a small file of the spatial data, write the metadata information and the small file content into the object file, write the metadata information into the index file, and generate positioning information, where the positioning information is used to position the metadata information in the index file, and the metadata information is used to position the small file content in the object file.
10. A computer-readable storage medium, in which a computer program capable of being loaded by a processor and executing the spatial data management method according to any one of claims 1 to 8 is stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110568927.5A CN113032348A (en) | 2021-05-25 | 2021-05-25 | Spatial data management method, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110568927.5A CN113032348A (en) | 2021-05-25 | 2021-05-25 | Spatial data management method, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113032348A true CN113032348A (en) | 2021-06-25 |
Family
ID=76455716
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110568927.5A Pending CN113032348A (en) | 2021-05-25 | 2021-05-25 | Spatial data management method, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032348A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986838A (en) * | 2021-12-28 | 2022-01-28 | 成都云祺科技有限公司 | Mass small file processing method and system based on file system and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104484471A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Implementation method of high-performance data storage engine |
| CN104765876A (en) * | 2015-04-24 | 2015-07-08 | 中国人民解放军信息工程大学 | Massive GNSS small file cloud storage method |
| CN105630779A (en) * | 2014-10-27 | 2016-06-01 | 杭州海康威视系统技术有限公司 | Hadoop distributed file system based small file storage method and apparatus |
| CN109063192A (en) * | 2018-08-29 | 2018-12-21 | 广州洪荒智能科技有限公司 | A kind of high-performance mass file storage system working method |
-
2021
- 2021-05-25 CN CN202110568927.5A patent/CN113032348A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105630779A (en) * | 2014-10-27 | 2016-06-01 | 杭州海康威视系统技术有限公司 | Hadoop distributed file system based small file storage method and apparatus |
| CN104484471A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Implementation method of high-performance data storage engine |
| CN104765876A (en) * | 2015-04-24 | 2015-07-08 | 中国人民解放军信息工程大学 | Massive GNSS small file cloud storage method |
| CN109063192A (en) * | 2018-08-29 | 2018-12-21 | 广州洪荒智能科技有限公司 | A kind of high-performance mass file storage system working method |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986838A (en) * | 2021-12-28 | 2022-01-28 | 成都云祺科技有限公司 | Mass small file processing method and system based on file system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112463724B (en) | Data processing method and system for lightweight file system | |
| CN111309687A (en) | Object storage small file processing method, device, equipment and storage medium | |
| US10572335B2 (en) | Metadata recovery method and apparatus | |
| CN113568582B (en) | Data management method, device and storage equipment | |
| US9858287B2 (en) | Storage system | |
| CN112463026A (en) | Method and apparatus for deduplication of supplemental data in a distributed object storage system | |
| CN107506466B (en) | Small file storage method and system | |
| CN113032348A (en) | Spatial data management method, system and computer readable storage medium | |
| CN106874329A (en) | The implementation method and device of database table index | |
| CN113253932A (en) | Read-write control method and system for distributed storage system | |
| CN110287164B (en) | Data recovery method and device and computer equipment | |
| CN114780489B (en) | Method and device for realizing distributed block storage bottom layer GC | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| CN110795031A (en) | Data deduplication method, device and system based on full flash storage | |
| CN113703673B (en) | Single machine data storage method and related device | |
| CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
| CN114647658A (en) | Data retrieval method, device, equipment and machine-readable storage medium | |
| CN110471623B (en) | Hard disk file writing method, device, computer equipment and storage medium | |
| CN109241011B (en) | Virtual machine file processing method and device | |
| CN112416261A (en) | Data storage method, device, equipment and system and storage medium | |
| KR100925523B1 (en) | Method Of Generating Error In Storage Device And Method Of Selecting Error Generating Location In Storage Device | |
| CN112860486B (en) | Method and device for creating data copy | |
| CN110941508B (en) | Post-disaster recovery system, method and device for virtualization system and readable storage medium | |
| CN112597074B (en) | Data processing method and device | |
| CN112463025B (en) | Tile record disk data management method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |