CN110109884A - A kind of file reading, device, equipment and medium - Google Patents

A kind of file reading, device, equipment and medium Download PDF

Info

Publication number
CN110109884A
CN110109884A CN201910434286.7A CN201910434286A CN110109884A CN 110109884 A CN110109884 A CN 110109884A CN 201910434286 A CN201910434286 A CN 201910434286A CN 110109884 A CN110109884 A CN 110109884A
Authority
CN
China
Prior art keywords
file
preset threshold
mds
data
size
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910434286.7A
Other languages
Chinese (zh)
Inventor
张端
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Wave Intelligent Technology Co Ltd
Original Assignee
Suzhou Wave Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Wave Intelligent Technology Co Ltd filed Critical Suzhou Wave Intelligent Technology Co Ltd
Priority to CN201910434286.7A priority Critical patent/CN110109884A/en
Publication of CN110109884A publication Critical patent/CN110109884A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • G06F16/148File search processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of file reading, device, equipment and medium, the step of this method includes: to read demand according to file to read corresponding target metadata in MDS, and obtain the target data size for the file destination being recorded in target metadata;When target data size is in preset threshold or is equal to preset threshold, file destination is read in MDS;Wherein, size of data is pre-stored within MDS in preset threshold or equal to the file of preset threshold.File of the size of data in certain threshold range by being pre-stored in MDS by this method in advance, the occupancy to the OSD communication resource can be reduced relatively, the opposite OSD that avoids generates delay and obstruction in data communication process, and then ensures the stability of file reading process.In addition, the present invention also provides a kind of document reading apparatus, equipment and medium, beneficial effect is same as above.

Description

A kind of file reading, device, equipment and medium
Technical field
The present invention relates to data communication fields, more particularly to a kind of file reading, device, equipment and medium.
Background technique
With the development of the universal and Internet technology of computer, data file just in magnanimity and promptly increases, these Data file is usually stored in computer memory system so that user retrieves data file, analyzes.
In order to realize to file read global reliability, currently generally use distributed file system to data file into Row stores, and is memory space needed for providing file system jointly by more memory nodes in specific implementation, therefore in order to more Add the reading for accurately carrying out data file in distributed file system, currently generallys use metadata (Metadata) to depositing The relevant information for being stored in the data file of memory node is recorded, metadata, also known as broker data, relaying data, for description The data (data about data) of data, describe the information of data attribute (property), mainly for supporting such as to indicate The functions such as storage location, historical data, resource lookup, file record.In order to preferably be safeguarded to metadata, current In usage scenario, metadata is often stored separately in MDS (Metadata Server, meta data server), and data are literary Part is stored in OSD (Object Storage Device, object storage device), due to the sum of current data file It is often huger according to measuring, therefore be usually divided and be stored in multiple OSD in advance, and then the client of distributed file system When accessing some data file, need first to access metadata of the MDS to get the data file, and then further according to first number According to corresponding data file segment is obtained in multiple paths of multiple OSD or OSD, final group is combined into complete data file, It is related to the client of distributed file system and the frequent interaction of OSD in this course, needs to occupy more communication money Source, on this basis, when distributed file system client access data volume it is smaller and need not move through division can store to When data file in OSD, it is still desirable in accordance with above-mentioned operating process, eventually by distributed file system client with OSD is interacted to obtain the data file, this will be further exacerbated by occupancy to the communication resource of OSD, may cause number The abnormal conditions such as delay and obstruction are generated according to communication process, it is difficult to ensure the stability of file reading process.
It can be seen that a kind of file reading is provided, it is opposite to avoid with opposite reduction to the occupancy of the OSD communication resource Delay and obstruction are generated in data communication process, and then ensures the stability of file reading process, are those skilled in the art Problem to be solved.
Summary of the invention
The object of the present invention is to provide a kind of file reading, device, equipment and media, logical to OSD with opposite reduction Believe the occupancy of resource, it is opposite to avoid generating delay and obstruction in data communication process, and then ensure the steady of file reading process It is qualitative.
In order to solve the above technical problems, the present invention provides a kind of file reading, comprising:
Demand is read according to file and reads corresponding target metadata in MDS, and is obtained and be recorded in target metadata File destination target data size;
When target data size is in preset threshold or is equal to preset threshold, file destination is read in MDS;Wherein, Size of data is pre-stored within MDS in preset threshold or equal to the file of preset threshold.
Preferably, file of the size of data in preset threshold is pre-stored within MDS and includes:
File of the size of data in preset threshold is used as the extended attribute of metadata to be stored in MDS in advance.
Preferably, this method further comprises:
The target data size of file destination is caused to be greater than default threshold after carrying out additional data writing operation to file destination When value, file destination is read in the extended attribute of target metadata, and file destination is stored to OSD.
Preferably, this method further comprises:
When target data size is greater than preset threshold, file destination is read in OSD according to target metadata.
In addition, the present invention also provides a kind of document reading apparatus, comprising:
Metadata read module reads corresponding target metadata for reading demand according to file in MDS, and obtains It is recorded in the target data size of the file destination in target metadata;
Small documents read module is used for when target data size in preset threshold or when being equal to preset threshold, in MDS Read file destination;Wherein, file of the size of data in preset threshold is pre-stored within MDS.
Preferably, which further comprises:
Big file read module is used for when target data size is greater than preset threshold, according to target metadata in OSD Read file destination.
In addition, the present invention also provides a kind of files to read equipment, comprising:
Memory, for storing computer program;
Processor is realized when for executing computer program such as the step of above-mentioned file reading.
In addition, being stored with meter on computer readable storage medium the present invention also provides a kind of computer readable storage medium Calculation machine program is realized when computer program is executed by processor such as the step of above-mentioned file reading.
File reading provided by the present invention, the file by size of data in preset threshold or equal to preset threshold It is pre-stored within MDS, demand is read according to file first when executing File read operation and reads corresponding target element in MDS Data, and then the target data size for the file destination being recorded in target metadata is obtained, when target data size is default In threshold value or when being equal to preset threshold, file destination is directly read in MDS.As it can be seen that this method passes through in advance by size of data File in certain threshold range is pre-stored in MDS, and then when user obtains size of data in the threshold range When file destination, then without interacting with OSD, therefore the occupancy to the OSD communication resource can be reduced relatively, it is opposite to avoid OSD generates delay and obstruction in data communication process, and then ensures the stability of file reading process.In addition, of the invention It is same as above also to provide a kind of document reading apparatus, equipment and medium, beneficial effect.
Detailed description of the invention
In order to illustrate the embodiments of the present invention more clearly, attached drawing needed in the embodiment will be done simply below It introduces, it should be apparent that, drawings in the following description are only some embodiments of the invention, for ordinary skill people For member, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart of file reading provided in an embodiment of the present invention;
Fig. 2 is a kind of structure chart of document reading apparatus provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, rather than whole embodiments.Based on this Embodiment in invention, those of ordinary skill in the art are without making creative work, obtained every other Embodiment belongs to the scope of the present invention.
Core of the invention is to provide a kind of file reading, with opposite reduction to the occupancy of the OSD communication resource, relatively It avoids generating delay and obstruction in data communication process, and then ensures the stability of file reading process.Of the invention is another Core is to provide a kind of document reading apparatus, equipment and medium.
In order to enable those skilled in the art to better understand the solution of the present invention, with reference to the accompanying drawings and detailed description The present invention is described in further detail.
Embodiment one
Fig. 1 is a kind of flow chart of file reading provided in an embodiment of the present invention.Referring to FIG. 1, file reading side The specific steps of method include:
Step S10: demand is read according to file and reads corresponding target metadata in MDS, and obtains and is recorded in target The target data size of file destination in metadata.
It should be noted that the executing subject of this method can be point in distributed file system in addition to MDS and OSD The client of cloth file system, the client are interacted by user's control according to user demand is corresponding with MDS and OSD progress And obtain corresponding file.This step i.e. according to the file of user read demand is read in MDS used to store metadata and The corresponding target metadata of file needed for user, and then get in target metadata the association attributes letter of file needed for user Breath, i.e. the target data size of file destination, target data size herein refer to file destination number shared in OSD equipment According to space.
Step S11: when target data size is in preset threshold or is equal to preset threshold, target text is read in MDS Part.
Wherein, size of data is pre-stored within MDS in preset threshold or equal to the file of preset threshold.
In this step, illustrate have in preset threshold or when being equal to preset threshold when the target data size got There is the corresponding document of the target data size to meet preset target file type, and the file destination of the type is stored in advance In MDS.Since this step is when target data size is in preset threshold or is equal to preset threshold, target is obtained in MDS File, therefore being stored in advance in the file destination in MDS is the relatively small file of size of data in distributed file system, in advance First the relatively small file of size of data is stored in MDS, can directly be obtained in MDS when user accesses this document It arrives, client and the OSD equipment without distributed file system interact.Preset threshold in this step should be according to distribution Depending on the actual use scene of file system, such as the data capacity and the amount of metadata in MDS equipment etc. of MDS equipment, it answers Guarantee that MDS normally can provide metadata to client.
File reading provided by the present invention, the file by size of data in preset threshold or equal to preset threshold It is pre-stored within MDS, demand is read according to file first when executing File read operation and reads corresponding target element in MDS Data, and then the target data size for the file destination being recorded in target metadata is obtained, when target data size is default In threshold value or when being equal to preset threshold, file destination is directly read in MDS.As it can be seen that this method passes through in advance by size of data File in certain threshold range is pre-stored in MDS, and then when user obtains size of data in the threshold range When file destination, then without interacting with OSD, therefore the occupancy to the OSD communication resource can be reduced relatively, it is opposite to avoid OSD generates delay and obstruction in data communication process, and then ensures the stability of file reading process.
Embodiment two
On the basis of the above embodiments, the present invention also provides a series of preferred embodiments.
As a preferred embodiment, file of the size of data in preset threshold is pre-stored within MDS includes:
File of the size of data in preset threshold is used as the extended attribute of metadata to be stored in MDS in advance.
It should be noted that due to consideration that being previously provided in the data structure of metadata for increasing data item Extended attribute, therefore in the present embodiment, it is stored in advance in extended attribute in the metadata using by file, is realized with this File is stored in the purpose in MDS, since the data structure of metadata has the stability of data storage, opposite can be ensured Reliability when file stores in MDS.
On the basis of the above embodiment, as a preferred embodiment, this method further comprises:
The target data size of file destination is caused to be greater than default threshold after carrying out additional data writing operation to file destination When value, file destination is read in the extended attribute of target metadata, and file destination is stored to OSD.
It should be noted that present embodiment allows for the case where writing in the presence of the addition for carrying out data to file destination When, the size of data of file destination can be continuously increased, and then be more than default threshold there may be the target data size of file destination The case where value, in the case, in order to ensure the storage to file destination does not cause excessive occupancy, this reality to the space of MDS Apply mode causes the target data size of file destination to be greater than preset threshold after carrying out additional data writing operation to file destination When, file destination is read out in the extended attribute of target metadata, and then file destination is stored into OSD, with this It avoids and is excessively occupied caused by the space MDS because the data volume of file destination is continuously increased, guarantee that metadata can be normal It is stored in MDS, and then ensures that MDS normally can provide Metadata Service to client.
On the basis of a series of above-mentioned embodiments, as a preferred embodiment, this method further comprises:
When target data size is greater than preset threshold, file destination is read in OSD according to target metadata.
It should be noted that in the present embodiment, target data size to be greater than to the file destination of preset threshold in advance It directly stores into OSD, and then when target data size is greater than preset threshold, mesh is read in OSD according to target metadata File is marked, opposite avoid causes the space MDS due to storing the relatively large file destination of target data size into MDS Excessive occupancy, and then ensure that metadata can normal storage in MDS, it is ensured that MDS can normally to client provide member Data service.
Embodiment three
Hereinbefore the embodiment of file reading is described in detail, the present invention also provides one kind and is somebody's turn to do The corresponding document reading apparatus of method, since the embodiment of device part is corresponded to each other with the embodiment of method part, dress Set part embodiment refer to method part embodiment description, wouldn't repeat here.
Fig. 2 is a kind of structure chart of document reading apparatus provided in an embodiment of the present invention.Text provided in an embodiment of the present invention Part reading device, comprising:
Metadata read module 10 reads corresponding target metadata for reading demand according to file in MDS, and obtains Take the target data size for the file destination being recorded in target metadata.
Small documents read module 11 is used for when target data size in preset threshold or when being equal to preset threshold, in MDS Middle reading file destination;Wherein, file of the size of data in preset threshold is pre-stored within MDS.
Document reading apparatus provided by the present invention, the file by size of data in preset threshold or equal to preset threshold It is pre-stored within MDS, demand is read according to file first when executing File read operation and reads corresponding target element in MDS Data, and then the target data size for the file destination being recorded in target metadata is obtained, when target data size is default In threshold value or when being equal to preset threshold, file destination is directly read in MDS.As it can be seen that the present apparatus passes through in advance by size of data File in certain threshold range is pre-stored in MDS, and then when user obtains size of data in the threshold range When file destination, then without interacting with OSD, therefore the occupancy to the OSD communication resource can be reduced relatively, it is opposite to avoid OSD generates delay and obstruction in data communication process, and then ensures the stability of file reading process.
On the basis of embodiment three, the device further include:
Big file read module is used for when target data size is greater than preset threshold, according to target metadata in OSD Read file destination.
Example IV
In addition, the present invention also provides a kind of files to read equipment, comprising:
Memory, for storing computer program;
Processor is realized when for executing computer program such as the step of above-mentioned file reading.
File provided by the present invention reads equipment, the file by size of data in preset threshold or equal to preset threshold It is pre-stored within MDS, demand is read according to file first when executing File read operation and reads corresponding target element in MDS Data, and then the target data size for the file destination being recorded in target metadata is obtained, when target data size is default In threshold value or when being equal to preset threshold, file destination is directly read in MDS.As it can be seen that this equipment passes through in advance by size of data File in certain threshold range is pre-stored in MDS, and then when user obtains size of data in the threshold range When file destination, then without interacting with OSD, therefore the occupancy to the OSD communication resource can be reduced relatively, it is opposite to avoid OSD generates delay and obstruction in data communication process, and then ensures the stability of file reading process.
In addition, being stored with meter on computer readable storage medium the present invention also provides a kind of computer readable storage medium Calculation machine program is realized when computer program is executed by processor such as the step of above-mentioned file reading.
Computer readable storage medium provided by the present invention, by size of data in preset threshold or equal to preset threshold File be pre-stored within MDS, when executing File read operation first according to file read demand read in MDS accordingly Target metadata, and then the target data size for the file destination being recorded in target metadata is obtained, when target data size In preset threshold or when being equal to preset threshold, file destination is directly read in MDS.As it can be seen that this computer-readable storage medium Matter works as user and obtains data by the way that file of the size of data in certain threshold range is pre-stored in MDS in advance When file destination of the size in the threshold range, then without being interacted with OSD, therefore it opposite can reduce and OSD is communicated The occupancy of resource, the opposite OSD that avoids generates delay and obstruction in data communication process, and then ensures file reading process Stability.
A kind of file reading provided by the present invention, device, equipment and medium are described in detail above.It says Each embodiment is described in a progressive manner in bright book, and the highlights of each of the examples are the differences with other embodiments Place, the same or similar parts in each embodiment may refer to each other.For the device disclosed in the embodiment, due to its with Method disclosed in embodiment is corresponding, so being described relatively simple, reference may be made to the description of the method.It should refer to It out, for those skilled in the art, without departing from the principle of the present invention, can also be to the present invention Some improvement and modification can also be carried out, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged Except there is also other identical elements in the process, method, article or apparatus that includes the element.

Claims (8)

1. a kind of file reading characterized by comprising
Demand is read according to file and reads corresponding target metadata in MDS, and is obtained and be recorded in the target metadata File destination target data size;
When the target data size is in preset threshold or is equal to the preset threshold, the mesh is read in the MDS Mark file;Wherein, size of data is pre-stored within described in the preset threshold or equal to the file of the preset threshold MDS。
2. the method according to claim 1, wherein file of the size of data in the preset threshold is pre- Being first stored in the MDS includes:
File of the size of data in the preset threshold is used as the extended attribute of metadata to be stored in the MDS in advance.
3. according to the method described in claim 2, it is characterized in that, this method further comprises:
The target data size of the file destination is caused to be greater than institute after carrying out additional data writing operation to the file destination When stating preset threshold, the file destination is read in the extended attribute of the target metadata, and the file destination is deposited It stores up to OSD.
4. according to claim 1 to method described in 3, which is characterized in that this method further comprises:
When the target data size is greater than the preset threshold, the mesh is read in OSD according to the target metadata Mark file.
5. a kind of document reading apparatus characterized by comprising
Metadata read module reads corresponding target metadata for reading demand according to file in MDS, and obtains record The target data size of file destination in the target metadata;
Small documents read module is used for when the target data size in preset threshold or when being equal to the preset threshold, The file destination is read in the MDS;Wherein, file of the size of data in the preset threshold is pre-stored within described MDS。
6. device according to claim 5, which is characterized in that the device further comprises:
Big file read module is used for when the target data size is greater than the preset threshold, according to the target element number According to reading the file destination in OSD.
7. a kind of file reads equipment characterized by comprising
Memory, for storing computer program;
Processor realizes such as Claims 1-4 described in any item file readings when for executing the computer program The step of.
8. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium Program is realized when the computer program is executed by processor such as the described in any item file readings of Claims 1-4 Step.
CN201910434286.7A 2019-05-23 2019-05-23 A kind of file reading, device, equipment and medium Pending CN110109884A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910434286.7A CN110109884A (en) 2019-05-23 2019-05-23 A kind of file reading, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910434286.7A CN110109884A (en) 2019-05-23 2019-05-23 A kind of file reading, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN110109884A true CN110109884A (en) 2019-08-09

Family

ID=67491832

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910434286.7A Pending CN110109884A (en) 2019-05-23 2019-05-23 A kind of file reading, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN110109884A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112738270A (en) * 2021-01-07 2021-04-30 苏州浪潮智能科技有限公司 File transmission method, device, equipment and storage medium
CN113961968A (en) * 2021-12-22 2022-01-21 北京安华金和科技有限公司 Method and device for carrying out data desensitization interrupt post-processing on file

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105141685A (en) * 2015-08-18 2015-12-09 浪潮(北京)电子信息产业有限公司 File read-write system and meta data memory thereof as well as method and device for reading and writing files

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105141685A (en) * 2015-08-18 2015-12-09 浪潮(北京)电子信息产业有限公司 File read-write system and meta data memory thereof as well as method and device for reading and writing files

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李孟等: "《基于Hadoop的小文件存储优化方案》", 《电子科技大学学报》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112738270A (en) * 2021-01-07 2021-04-30 苏州浪潮智能科技有限公司 File transmission method, device, equipment and storage medium
CN112738270B (en) * 2021-01-07 2022-12-30 苏州浪潮智能科技有限公司 File transmission method, device, equipment and storage medium
US11792258B1 (en) 2021-01-07 2023-10-17 Inspur Suzhou Intelligent Technology Co., Ltd. File transmission method and apparatus, device, and storage medium
CN113961968A (en) * 2021-12-22 2022-01-21 北京安华金和科技有限公司 Method and device for carrying out data desensitization interrupt post-processing on file

Similar Documents

Publication Publication Date Title
US9658759B2 (en) User adjustable data sharing between multiple graph elements
US8793573B2 (en) Continuous content item view enhanced through smart loading
US8731973B2 (en) Overlaying images in automated insurance policy form generation
US8996981B2 (en) Managing forms in electronic documents
US20200210032A1 (en) Method for e-note management of electronic books, and electronic device
AU2012238127B2 (en) Recovery of tenant data across tenant moves
US9547668B2 (en) Event-based content item view
US11030259B2 (en) Document searching visualized within a document
US20090319910A1 (en) Automatic content and author emphasis for shared data
JP5677029B2 (en) Data migration system, data migration method, program
US9384575B2 (en) Space constrained small format visual analytic labeling
US8631316B2 (en) Systems and methods for generating sitelets using information assets
CN110442341A (en) Interface configuration method, device and the terminal device of application program
US20170010869A1 (en) Multistage customizing of web-based application in a browser independent of platform and operating system
US20150153938A1 (en) Displaying dynamic graphical content in graphical user interface (gui) controls
CN112651826B (en) Credit line management and control system, method and readable storage medium
CN110109884A (en) A kind of file reading, device, equipment and medium
CN102789458B (en) A kind of image file disposal route and system
KR102230729B1 (en) An online task system for an Excel file based on templates
Hahn et al. Hybrid-treemap layouting
US8091016B2 (en) Visually manipulating instance collections
CN105574164A (en) Excel document data analysis method and device
US20070028180A1 (en) Tab order management in a portal environment
WO2015182271A1 (en) Information processing device and information processing method
JP5682984B1 (en) Document processing system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20190809