CN109542361A - A kind of distributed memory system file reading, system and relevant apparatus - Google Patents
A kind of distributed memory system file reading, system and relevant apparatus Download PDFInfo
- Publication number
- CN109542361A CN109542361A CN201811475963.1A CN201811475963A CN109542361A CN 109542361 A CN109542361 A CN 109542361A CN 201811475963 A CN201811475963 A CN 201811475963A CN 109542361 A CN109542361 A CN 109542361A
- Authority
- CN
- China
- Prior art keywords
- request
- memory
- read
- reading
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of distributed memory system file reading, system and a kind of electronic equipment and computer readable storage mediums, this method comprises: pre-reading the upper limit according to the determination of the size of data of the I/O request when receiving the I/O request of client；Size is sent to object storage device every time and pre-reads the read request of the upper limit to be described, and returning the result for the read request is stored to memory, is completed until the corresponding data of the I/O request are all read；The corresponding data of the I/O request stored in memory are back to the client.It can be seen that file reading provided by the present application can reduce the interaction times of client and object storage device, IO number of disk, promotion sequence reading performance are reduced, to improve the performance of entire storage system, enhances product competitiveness.
Description
Technical field
This application involves field of computer technology, more specifically to a kind of distributed memory system file reading side
Method, system and a kind of electronic equipment and a kind of computer readable storage medium.
Background technique
As distributed block storage system is widely applied in each field, performance requirement of the client to storage system
Higher and higher, the performance for improving distributed block storage system, which seems, to become more and more important.(Chinese is complete using IP for distributed block storage system
Claim: the agreement interconnected between network, full name in English: Internet Protocol) high bandwidth of network, high IOPS (Input/
Output Operations Per Second, one is used for computer memory device) polymerizing power the features such as, system is externally provided
One block data access service, meanwhile, client passes through ISCSI (Internet Small Computer System
Interface) protocol access distributed block storage system.Since client carries out IO (Input/ by iscsi agreement
Output) upper limit read and write is 512k, if the I O size of client access storage system is 1M or is bigger, the IO of client
Request is just split into multiple 512k request, such client need repeatedly with the OSD (Chinese name: object in storage system
Store equipment, full name in English: Object Storage Devices) it interacts, IO number of disk is increased in this way, is reduced
The readwrite performance of entire storage system.
Therefore, how to improve the readwrite performance of storage system is those skilled in the art's problem to be solved.
Summary of the invention
The application's is designed to provide a kind of distributed memory system file reading, system and a kind of electronic equipment
With a kind of computer readable storage medium, the readwrite performance of storage system is improved.
To achieve the above object, this application provides a kind of distributed memory system file readings, comprising:
When receiving the I/O request of client, the upper limit is pre-read according to the determination of the size of data of the I/O request；
Sending size to object storage device every time is the read request for pre-reading the upper limit, and by the return of the read request
As a result it stores to memory, is completed until the corresponding data of the I/O request are all read；
The corresponding data of the I/O request stored in memory are back to the client.
Wherein, it is determined before pre-reading the upper limit according to the size of data of the I/O request, further includes:
Judge in the memory with the presence or absence of the corresponding data of the I/O request；
If so, executing the step that the corresponding data of the I/O request stored in memory are back to the client
Suddenly；
The step of pre-reading the upper limit, is determined according to the size of data of the I/O request if it is not, then executing.
Wherein, it is determined before pre-reading the upper limit according to the size of data of the I/O request, further includes:
Whether the type for judging the I/O request is that sequence is read；
The step of pre-reading the upper limit, is determined according to the size of data of the I/O request if so, executing.
Wherein, the upper limit is pre-read according to the determination of the size of data of the I/O request, comprising:
Judge whether the size of data of the I/O request is greater than 1M；
If so, by the integral multiple for pre-reading the upper limit and being determined as 512K；
If it is not, then by the integral multiple for pre-reading the upper limit and being determined as the size of data of the I/O request.
Wherein, returning the result for the read request is stored to memory, comprising:
Judge whether the size of data stored in memory is greater than the memory management upper limit；
If so, returning the result for the read request is covered the data that storage time is earliest in the memory；
If it is not, then returning the result for the read request is directly stored into the memory.
Wherein, the memory management upper limit is the integral multiple for pre-reading the upper limit.
To achieve the above object, this application provides a kind of distributed memory system files to read system, comprising:
Determining module, for being determined according to the size of data of the I/O request pre- when receiving the I/O request of client
Read the upper limit；
Read module is the read request for pre-reading the upper limit for sending size to object storage device every time, and by institute
It states returning the result for read request to store to memory, be completed until the corresponding data of the I/O request are all read；
Return module, for the corresponding data of the I/O request stored in memory to be back to the client.
Wherein, the read module is specially and sends size to object storage device every time to ask for the reading for pre-reading the upper limit
It asks, and judges whether the size of data stored in memory is greater than the memory management upper limit；If so, returning the read request
It returns result and covers the data that storage time is earliest in the memory；If it is not, then returning the result for the read request is stored to institute
It states in memory, until the corresponding data of the I/O request all read the module completed.
To achieve the above object, this application provides a kind of electronic equipment, comprising:
Memory, for storing computer program；
Processor is realized when for executing the computer program such as above-mentioned distributed memory system file reading
Step.
To achieve the above object, this application provides a kind of computer readable storage medium, the computer-readable storages
It is stored with computer program on medium, is realized when the computer program is executed by processor such as above-mentioned distributed memory system text
The step of part read method.
By above scheme it is found that a kind of distributed memory system file reading provided by the present application, comprising: when connecing
When receiving the I/O request of client, the upper limit is pre-read according to the determination of the size of data of the I/O request；Every time to object storage device
It sends size and pre-reads the read request of the upper limit to be described, and returning the result for the read request is stored to memory, until the IO
It requests corresponding data all to read to complete；The corresponding data of the I/O request stored in memory are back to the visitor
Family end.
File reading provided by the present application is splitted data into and is repeatedly pre-read into memory, reads pre-read the upper limit every time
Data, return value client after the completion of all reading, client once interacted with object storage device value.Thus may be used
See, file reading provided by the present application can reduce the interaction times of client and object storage device, reduce disk
IO number, promotion sequence reading performance enhances product competitiveness to improve the performance of entire storage system.The application is also
It discloses a kind of distributed memory system file and reads system and a kind of electronic equipment and a kind of computer readable storage medium, together
Sample is able to achieve above-mentioned technical effect.
It should be understood that the above general description and the following detailed description are merely exemplary, this can not be limited
Application.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of application for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.Attached drawing is and to constitute specification for providing further understanding of the disclosure
A part, be used to explain the disclosure together with following specific embodiment, but do not constitute the limitation to the disclosure.Attached
In figure:
Fig. 1 is a kind of flow chart of distributed memory system file reading shown according to an exemplary embodiment；
Fig. 2 is the process of another distributed memory system file reading shown according to an exemplary embodiment
Figure；
Fig. 3 is a kind of structure chart of distributed memory system file reading system shown according to an exemplary embodiment；
Fig. 4 is the structure chart according to a kind of electronic equipment shown in an exemplary embodiment.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of embodiments of the present application, instead of all the embodiments.It is based on
Embodiment in the application, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall in the protection scope of this application.
The embodiment of the present application discloses a kind of distributed memory system file reading, improves the read-write of storage system
Performance.
Referring to Fig. 1, a kind of process of distributed memory system file reading shown according to an exemplary embodiment
Figure, as shown in Figure 1, comprising:
S101: when receiving the I/O request of client, the upper limit is pre-read according to the determination of the size of data of the I/O request；
The executing subject of the present embodiment can receive for the pre- read apparatus in distributed memory system, the pre- read apparatus
When the I/O request of client, it is first determined pre-read the upper limit.This pre-reads the upper limit and can be carried out according to the corresponding size of data of I/O request
Setting, the preferably integral multiple of the size of data of I/O request.
This step may include judging whether the size of data of I/O request is greater than 1M as a preferred implementation manner,；If
It is the integral multiple that will then pre-read the upper limit and be determined as 512K；If it is not, then by pre-read the upper limit be determined as I/O request size of data it is whole
Several times.
For example, the corresponding size of data of I/O request is 2M, then it can will pre-read the upper limit and be set as 1M, the corresponding number of I/O request
It is 64K according to size, then can will pre-reads the upper limit and be set as 256K, which is generally less than 1M.
It is read it should be noted that file reading provided in this embodiment is directed to sequence, i.e., according to I/O request
Before size of data determination pre-reads the upper limit, further include the steps that judging whether the type of the I/O request is that sequence is read, if so, executing
The step of pre-reading the upper limit is determined according to the size of data of I/O request, if otherwise terminating process.
It preferably, further include judging whether deposit in memory before pre-reading the upper limit according to the determination of the size of data of I/O request
The I/O request corresponding data the step of, if step S103 is then directly executed, if otherwise executing big according to the data of I/O request
The step of small determination pre-reads the upper limit, i.e., when there are return value client direct when the returned data of I/O request, Ke Yiti in memory
High reading efficiency.
S102: size is sent as the read request for pre-reading the upper limit to object storage device, and returning the read request
Result is returned to store to memory；
In specific implementation, sending size to object storage device every time is that the reading for pre-reading the upper limit that above-mentioned steps determine is asked
It asks, and returning the result for the read request is stored to memory, when the reading I O access storage system of same type, directly from memory
It returns and reads result, improve the response efficiency of I/O request.
S103: judge that the corresponding data of the I/O request are all read and complete；If so, into S104；If it is not, then entering
S102；
In this step, judge that the corresponding data of I/O request are all read to complete；If so, into S104；If it is not, then into
Enter S102.It can be seen that repeatedly interacting for client in the prior art and object storage device is become point in the present embodiment
Equipment and the multiple interaction with object storage device are pre-read in cloth storage system, reduces IO number of disk, is promoted suitable
Sequence reading performance.
S104: the corresponding data of the I/O request stored in memory are back to the client.
When the corresponding data of I/O request, which are all read, to be completed, by response data return value client stored in memory,
So far the response process of the I/O request is completed.
File reading provided by the embodiments of the present application is splitted data into and is repeatedly pre-read into memory, is read every time pre-
The data of the upper limit, return value client after the completion of all reading are read, client is once interacted with object storage device value.By
This as it can be seen that file reading provided by the embodiments of the present application can reduce the interaction times of client and object storage device,
IO number of disk, promotion sequence reading performance are reduced, to improve the performance of entire storage system, enhances product competition
Power.
The embodiment of the present application discloses a kind of distributed memory system file reading, relative to a upper embodiment, originally
Embodiment has made further instruction and optimization to technical solution.It is specific:
Referring to fig. 2, the stream of another distributed memory system file reading shown according to an exemplary embodiment
Cheng Tu, as shown in Figure 2, comprising:
S201: when receiving the I/O request of client, judge corresponding with the presence or absence of the I/O request in the memory
Data；If so, into S207；If it is not, then entering S202；
In the present embodiment, it is first determined whether when receiving the I/O request of client in memory with the presence or absence of the I/O request pair
The data answered then are read directly from memory if it exists as a result, if it does not exist, then entering S202.
S202: the upper limit is pre-read according to the determination of the size of data of the I/O request；
S203: size is sent to object storage device and pre-reads the read request of the upper limit to be described, and judges to deposit in the memory
Whether the size of data of storage is greater than the memory management upper limit；If so, into S204；If it is not, then entering S205；
In the present embodiment, memory is managed the result of read request using eliminative mechanism, i.e., in this step first
Judge whether size of data stored in memory is greater than the memory management upper limit, memory management upper limit those skilled in the art can herein
To be configured and modify according to the actual situation.Preferably, which can be the integral multiple for pre-reading the upper limit, such as
32M。
S204: returning the result for the read request is covered into the data that storage time is earliest in the memory；
It in this step, will be new using eliminative mechanism when size of data stored in memory is greater than the memory management upper limit
The old data of data cover, i.e., read request in above-mentioned steps is returned the result into the number that storage time is earliest in covering memory
According to saving memory headroom improves performance of storage system.
S205；Returning the result for the read request is directly stored into the memory；
S206: judge that the corresponding data of the I/O request are all read and complete；If so, into S207；If it is not, then entering
S203；
S207: the corresponding data of the I/O request stored in memory are back to the client.
The Application Example of distributed memory system file reading provided by the present application is received below.Specifically, point
Pre- read apparatus is provided in cloth block storage system, mainly by IO identification module, pre- read through model, OSD communication module, memory pipe
The four modules such as reason composition.IO identification module passes to pre- read through model for judging I/O type and I O size；Pre- read through model according to
I/O type, I O size and the upper limit is pre-read, the read request for pre-reading the upper limit is sent to OSD communication module；OSD communication module is read request
The data of return are stored in memory；Memory management module is managed these data using eliminative mechanism.
Distributed block storage system pre-reads that detailed process is as follows:
Step 1: by the received IO of iscsi server-side, judging I/O type and size.If sequence is read, then sent to OSD
The read request of upper limit size is pre-read, wherein pre-reading upper limit value can be set and modify (such as 1M).If it is non-sequential reading then directly and
OSD in storage system is interacted.
Step 2: the read request for pre-reading the upper limit being returned the result and is put into memory；Memory is using eliminative mechanism to these results
It is managed, if it exceeds then new data can eliminate old data the memory management upper limit, wherein memory management upper limit value
It can be set and modify (such as 32M).
Step 3: finally when the reading I O access storage system of same type, being returned directly from memory and read result.According to
Above-mentioned rule is completed until client reads data.
System is read to a kind of distributed memory system file provided by the embodiments of the present application below to be introduced, and is hereafter retouched
A kind of distributed memory system file stated reads system and a kind of above-described distributed memory system file reading
It can be cross-referenced.
Referring to Fig. 3, a kind of structure of distributed memory system file reading system shown according to an exemplary embodiment
Figure, as shown in Figure 3, comprising:
Determining module 301, for being determined according to the size of data of the I/O request when receiving the I/O request of client
Pre-read the upper limit；
Read module 302 is the read request for pre-reading the upper limit for sending size to object storage device every time, and will
Returning the result for the read request is stored to memory, is completed until the corresponding data of the I/O request are all read；
Return module 303, for the corresponding data of the I/O request stored in memory to be back to the client
End.
File provided by the embodiments of the present application reads system, splits data into and repeatedly pre-reads into memory, reads every time pre-
The data of the upper limit, return value client after the completion of all reading are read, client is once interacted with object storage device value.By
This can reduce the interaction times of client and object storage device as it can be seen that file provided by the embodiments of the present application reads system,
IO number of disk, promotion sequence reading performance are reduced, to improve the performance of entire storage system, enhances product competition
Power.
On the basis of the above embodiments, as a preferred implementation manner, further include:
First judgment module, for judging in the memory with the presence or absence of the corresponding data of the I/O request；If so, opening
The workflow of dynamic return module 301；If it is not, then starting the workflow of determining module 301.
On the basis of the above embodiments, as a preferred implementation manner, further include:
Second judgment module, for judging whether the type of the I/O request is that sequence is read；If so, starting determining module
301 workflow.
On the basis of the above embodiments, the determining module 301 includes: as a preferred implementation manner,
Whether judging unit, the size of data for when receiving the I/O request of client, judging the I/O request are big
In 1M；
First determination unit, for when the size of data of the I/O request is greater than 1M, the upper limit that pre-reads to be determined as
The integral multiple of 512K；
Second determination unit, for pre-reading the upper limit for described when the size of data of the I/O request is less than or equal to 1M
It is determined as the integral multiple of the size of data of the I/O request.
On the basis of the above embodiments, as a preferred implementation manner, the read module 302 be specially every time to
Object storage device sends size and pre-reads the read request of the upper limit to be described, and whether judges the size of data stored in memory
Greater than the memory management upper limit；If so, returning the result for the read request is covered the number that storage time is earliest in the memory
According to；If it is not, then returning the result for the read request is stored into the memory, until the corresponding data of the I/O request are whole
Read the module completed.
About the system in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
Present invention also provides a kind of electronic equipment, referring to fig. 4, a kind of electronic equipment 400 provided by the embodiments of the present application
Structure chart, as shown in figure 4, may include processor 11 and memory 12.The electronic equipment 400 can also include multimedia group
Part 13, one or more of input/output (I/O) interface 14 and communication component 15.
Wherein, processor 11 is used to control the integrated operation of the electronic equipment 400, to complete above-mentioned distributed storage system
All or part of the steps in system file reading.Memory 12 is for storing various types of data to support in the electronics
The operation of equipment 400, these data for example may include any application program or side for operating on the electronic equipment 400
The instruction of method and the relevant data of application program, such as contact data, the message of transmitting-receiving, picture, audio, video etc..
The memory 12 can be realized by any kind of volatibility or non-volatile memory device or their combination, such as static state
Random access memory (Static Random Access Memory, abbreviation SRAM), electrically erasable programmable read-only memory
(Electrically Erasable Programmable Read-Only Memory, abbreviation EEPROM), erasable programmable
Read-only memory (Erasable Programmable Read-Only Memory, abbreviation EPROM), programmable read only memory
(Programmable Read-Only Memory, abbreviation PROM), and read-only memory (Read-Only Memory, referred to as
ROM), magnetic memory, flash memory, disk or CD.Multimedia component 13 may include screen and audio component.Wherein shield
Curtain for example can be touch screen, and audio component is used for output and/or input audio signal.For example, audio component may include one
A microphone, microphone is for receiving external audio signal.The received audio signal can be further stored in memory
It 12 or is sent by communication component 15.Audio component further includes at least one loudspeaker, is used for output audio signal.I/O interface
14 provide interface between processor 11 and other interface modules, other above-mentioned interface modules can be keyboard, mouse, button
Deng.These buttons can be virtual push button or entity button.Communication component 15 for the electronic equipment 400 and other equipment it
Between carry out wired or wireless communication.Wireless communication, such as Wi-Fi, bluetooth, near-field communication (Near Field
Communication, abbreviation NFC), 2G, 3G or 4G or they one or more of combination, therefore corresponding communication
Component 15 may include: Wi-Fi module, bluetooth module, NFC module.
In one exemplary embodiment, electronic equipment 400 can be by one or more application specific integrated circuit
(Application Specific Integrated Circuit, abbreviation ASIC), digital signal processor (Digital
Signal Processor, abbreviation DSP), digital signal processing appts (Digital Signal Processing Device,
Abbreviation DSPD), programmable logic device (Programmable Logic Device, abbreviation PLD), field programmable gate array
(Field Programmable Gate Array, abbreviation FPGA), controller, microcontroller, microprocessor or other electronics member
Part is realized, for executing above-mentioned distributed memory system file reading.
In a further exemplary embodiment, a kind of computer readable storage medium including program instruction is additionally provided, it should
The step of above-mentioned distributed memory system file reading is realized when program instruction is executed by processor.For example, the computer
Readable storage medium storing program for executing can be the above-mentioned memory 12 including program instruction, and above procedure instruction can be by electronic equipment 400
Device 11 is managed to execute to complete above-mentioned distributed memory system file reading.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For system disclosed in embodiment
Speech, since it is corresponded to the methods disclosed in the examples, so being described relatively simple, related place is referring to method part illustration
?.It should be pointed out that for those skilled in the art, under the premise of not departing from the application principle, also
Can to the application, some improvement and modification can also be carried out, these improvement and modification also fall into the protection scope of the claim of this application
It is interior.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or apparatus that includes the element.
Claims (10)
1. a kind of distributed memory system file reading characterized by comprising
When receiving the I/O request of client, the upper limit is pre-read according to the determination of the size of data of the I/O request；
Size is sent as the read request for pre-reading the upper limit to object storage device every time, and returning the result the read request
It stores to memory, is completed until the corresponding data of the I/O request are all read；
The corresponding data of the I/O request stored in memory are back to the client.
2. distributed memory system file reading according to claim 1, which is characterized in that according to the I/O request
Size of data determines before pre-reading the upper limit, further includes:
Judge in the memory with the presence or absence of the corresponding data of the I/O request；
If so, executing the step of corresponding data of the I/O request stored in memory are back to the client；
The step of pre-reading the upper limit, is determined according to the size of data of the I/O request if it is not, then executing.
3. distributed memory system file reading according to claim 1, which is characterized in that according to the I/O request
Size of data determines before pre-reading the upper limit, further includes:
Whether the type for judging the I/O request is that sequence is read；
The step of pre-reading the upper limit, is determined according to the size of data of the I/O request if so, executing.
4. distributed memory system file reading according to claim 1, which is characterized in that according to the I/O request
Size of data determination pre-reads the upper limit, comprising:
Judge whether the size of data of the I/O request is greater than 1M；
If so, by the integral multiple for pre-reading the upper limit and being determined as 512K；
If it is not, then by the integral multiple for pre-reading the upper limit and being determined as the size of data of the I/O request.
5. any one of -4 distributed memory system file reading according to claim 1, which is characterized in that by the reading
Returning the result for request is stored to memory, comprising:
Judge whether the size of data stored in memory is greater than the memory management upper limit；
If so, returning the result for the read request is covered the data that storage time is earliest in the memory；
If it is not, then returning the result for the read request is directly stored into the memory.
6. distributed memory system file reading according to claim 5, which is characterized in that the memory management upper limit
For the integral multiple for pre-reading the upper limit.
7. a kind of distributed memory system file reads system characterized by comprising
Determining module, for being pre-read according to the determination of the size of data of the I/O request when receiving the I/O request of client
Limit；
Read module is the read request for pre-reading the upper limit for sending size to object storage device every time, and by the reading
Returning the result for request is stored to memory, is completed until the corresponding data of the I/O request are all read；
Return module, for the corresponding data of the I/O request stored in memory to be back to the client.
8. stating distributed memory system file according to claim 7 reads system, which is characterized in that the read module is specially
Size is sent to object storage device every time and pre-reads the read request of the upper limit to be described, and judges that the data stored in memory are big
It is small whether to be greater than the memory management upper limit；If so, the read request is returned the result in the covering memory storage time most
Early data；If it is not, then returning the result for the read request is stored into the memory, until the corresponding number of the I/O request
According to the module for all reading completion.
9. a kind of electronic equipment characterized by comprising
Memory, for storing computer program；
Processor realizes the distributed memory system as described in any one of claim 1 to 6 when for executing the computer program
The step of file reading.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes the distributed memory system text as described in any one of claim 1 to 6 when the computer program is executed by processor
The step of part read method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811475963.1A CN109542361B (en) | 2018-12-04 | 2018-12-04 | Distributed storage system file reading method, system and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811475963.1A CN109542361B (en) | 2018-12-04 | 2018-12-04 | Distributed storage system file reading method, system and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109542361A true CN109542361A (en) | 2019-03-29 |
| CN109542361B CN109542361B (en) | 2022-06-07 |
Family
ID=65852633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811475963.1A Active CN109542361B (en) | 2018-12-04 | 2018-12-04 | Distributed storage system file reading method, system and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109542361B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110516817A (en) * | 2019-09-03 | 2019-11-29 | 北京华捷艾米科技有限公司 | A kind of model training data load method and device |
| CN111309257A (en) * | 2020-02-10 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Pre-reading method and device for reading file at constant speed and computer readable storage medium |
| CN112199618A (en) * | 2020-10-22 | 2021-01-08 | Oppo广东移动通信有限公司 | File page pre-reading method and device, terminal and storage medium |
| CN112328185A (en) * | 2020-12-28 | 2021-02-05 | 烽火通信科技股份有限公司 | Intelligent pre-reading method based on distributed storage |
| CN112558866A (en) * | 2020-12-03 | 2021-03-26 | Oppo（重庆）智能科技有限公司 | Data pre-reading method, mobile terminal and computer readable storage medium |
| CN113031873A (en) * | 2021-03-26 | 2021-06-25 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN113672176A (en) * | 2021-08-13 | 2021-11-19 | 济南浪潮数据技术有限公司 | Data reading method, system, equipment and computer readable storage medium |
| CN114168272A (en) * | 2022-02-14 | 2022-03-11 | 麒麟软件有限公司 | Random-reading kernel IO optimization method during file caching and reading |
| CN114327299A (en) * | 2022-03-01 | 2022-04-12 | 苏州浪潮智能科技有限公司 | Sequential reading and pre-reading method, device, equipment and medium |
| CN114489469A (en) * | 2021-07-20 | 2022-05-13 | 荣耀终端有限公司 | Data reading method, electronic equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174294A1 (en) * | 1999-03-30 | 2002-11-21 | Mitsubishi Denki Kabushiki Kaisha | Information processing system and recording medium recording a program to cause a computer to execute steps |
| CN101382955A (en) * | 2008-09-28 | 2009-03-11 | 中国科学院计算技术研究所 | File reading method in cluster file system and system |
| CN102904923A (en) * | 2012-06-21 | 2013-01-30 | 华数传媒网络有限公司 | Data reading method and data reading system capable of relieving disk reading bottleneck |
| CN103716413A (en) * | 2014-01-13 | 2014-04-09 | 浪潮(北京)电子信息产业有限公司 | Acceleration method for mass small document IO operation transmission in distribution type document system |
| CN104765575A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Information storage processing method |
| CN105549905A (en) * | 2015-12-09 | 2016-05-04 | 上海理工大学 | Method for multiple virtual machines to access distributed object storage system |
| CN105677754A (en) * | 2015-12-30 | 2016-06-15 | 华为技术有限公司 | Method, apparatus and system for acquiring subitem metadata in file system |
| CN107480150A (en) * | 2016-06-07 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of file loading method and device |
| CN107992271A (en) * | 2017-12-21 | 2018-05-04 | 郑州云海信息技术有限公司 | Data pre-head method, device, equipment and computer-readable recording medium |
| US20190181887A1 (en) * | 2017-12-11 | 2019-06-13 | Rubrik, Inc. | Techniques for fast io and low memory consumption while using erasure codes |
-
2018
- 2018-12-04 CN CN201811475963.1A patent/CN109542361B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174294A1 (en) * | 1999-03-30 | 2002-11-21 | Mitsubishi Denki Kabushiki Kaisha | Information processing system and recording medium recording a program to cause a computer to execute steps |
| CN101382955A (en) * | 2008-09-28 | 2009-03-11 | 中国科学院计算技术研究所 | File reading method in cluster file system and system |
| CN102904923A (en) * | 2012-06-21 | 2013-01-30 | 华数传媒网络有限公司 | Data reading method and data reading system capable of relieving disk reading bottleneck |
| CN103716413A (en) * | 2014-01-13 | 2014-04-09 | 浪潮(北京)电子信息产业有限公司 | Acceleration method for mass small document IO operation transmission in distribution type document system |
| CN104765575A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Information storage processing method |
| CN105549905A (en) * | 2015-12-09 | 2016-05-04 | 上海理工大学 | Method for multiple virtual machines to access distributed object storage system |
| CN105677754A (en) * | 2015-12-30 | 2016-06-15 | 华为技术有限公司 | Method, apparatus and system for acquiring subitem metadata in file system |
| CN107480150A (en) * | 2016-06-07 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of file loading method and device |
| US20190181887A1 (en) * | 2017-12-11 | 2019-06-13 | Rubrik, Inc. | Techniques for fast io and low memory consumption while using erasure codes |
| CN107992271A (en) * | 2017-12-21 | 2018-05-04 | 郑州云海信息技术有限公司 | Data pre-head method, device, equipment and computer-readable recording medium |
Non-Patent Citations (1)
| Title |
|---|
| 董晓明等: "分布式文件系统的写性能优化", 《微型电脑应用》 * |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110516817A (en) * | 2019-09-03 | 2019-11-29 | 北京华捷艾米科技有限公司 | A kind of model training data load method and device |
| CN111309257A (en) * | 2020-02-10 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Pre-reading method and device for reading file at constant speed and computer readable storage medium |
| CN112199618A (en) * | 2020-10-22 | 2021-01-08 | Oppo广东移动通信有限公司 | File page pre-reading method and device, terminal and storage medium |
| CN112558866A (en) * | 2020-12-03 | 2021-03-26 | Oppo（重庆）智能科技有限公司 | Data pre-reading method, mobile terminal and computer readable storage medium |
| CN112558866B (en) * | 2020-12-03 | 2022-12-09 | Oppo（重庆）智能科技有限公司 | Data pre-reading method, mobile terminal and computer readable storage medium |
| CN112328185B (en) * | 2020-12-28 | 2021-03-23 | 烽火通信科技股份有限公司 | Intelligent pre-reading method based on distributed storage |
| CN112328185A (en) * | 2020-12-28 | 2021-02-05 | 烽火通信科技股份有限公司 | Intelligent pre-reading method based on distributed storage |
| CN113031873A (en) * | 2021-03-26 | 2021-06-25 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN113031873B (en) * | 2021-03-26 | 2023-02-28 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN114489469A (en) * | 2021-07-20 | 2022-05-13 | 荣耀终端有限公司 | Data reading method, electronic equipment and storage medium |
| CN114489469B (en) * | 2021-07-20 | 2022-12-23 | 荣耀终端有限公司 | Data reading method, electronic equipment and storage medium |
| CN113672176A (en) * | 2021-08-13 | 2021-11-19 | 济南浪潮数据技术有限公司 | Data reading method, system, equipment and computer readable storage medium |
| CN113672176B (en) * | 2021-08-13 | 2023-12-29 | 济南浪潮数据技术有限公司 | Data reading method, system, equipment and computer readable storage medium |
| CN114168272A (en) * | 2022-02-14 | 2022-03-11 | 麒麟软件有限公司 | Random-reading kernel IO optimization method during file caching and reading |
| CN114168272B (en) * | 2022-02-14 | 2022-04-19 | 麒麟软件有限公司 | Random-reading kernel IO optimization method during file caching and reading |
| CN114327299A (en) * | 2022-03-01 | 2022-04-12 | 苏州浪潮智能科技有限公司 | Sequential reading and pre-reading method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109542361B (en) | 2022-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109542361A (en) | A kind of distributed memory system file reading, system and relevant apparatus | |
| CN111240837B (en) | Resource allocation method, device, terminal and storage medium | |
| CN107750466B (en) | Pairing nearby devices using synchronized alert signals | |
| CN107832100B (en) | APK plug-in loading method and terminal thereof | |
| CN109542614B (en) | Resource allocation method, device, terminal and storage medium | |
| WO2017005085A1 (en) | Data compression method, apparatus, and terminal | |
| WO2014205903A1 (en) | File display method and device | |
| CN111382123B (en) | File storage method, device, equipment and storage medium | |
| CN110019024B (en) | Directory query method, system, electronic device and storage medium | |
| CN111666497A (en) | Application program loading method and device, electronic equipment and readable storage medium | |
| WO2020020202A1 (en) | Account management method and apparatus of application program, and device and medium | |
| CN104899039A (en) | Method and device for providing screen shooting service in terminal device | |
| WO2019071618A1 (en) | Image processing method and device | |
| KR20180111981A (en) | Edit real-time content with limited interaction | |
| CN107835986A (en) | The data manipulation method and electronic equipment of a kind of electronic equipment | |
| CN103677517A (en) | Method and device for searching pictures | |
| CN109117153A (en) | Processing method, device, terminal and the storage medium of application program | |
| CN109302639B (en) | Bullet screen message distribution method, bullet screen message distribution device, terminal and storage medium | |
| WO2017219872A1 (en) | Set top box upgrading and channel changing methods and systems | |
| WO2020077681A1 (en) | Method and apparatus for lazy loading of js script | |
| CN108205559B (en) | Data management method and equipment thereof | |
| JP2008129678A (en) | System, method and program for automatically determining file compression | |
| US20150264113A1 (en) | Dynamic Batching | |
| CN110445874B (en) | Session processing method, device, equipment and storage medium | |
| US8498622B2 (en) | Data processing system with synchronization policy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |