CN111241312A - Efficient picture storage and reading system and method - Google Patents
Efficient picture storage and reading system and method Download PDFInfo
- Publication number
- CN111241312A CN111241312A CN201911426318.5A CN201911426318A CN111241312A CN 111241312 A CN111241312 A CN 111241312A CN 201911426318 A CN201911426318 A CN 201911426318A CN 111241312 A CN111241312 A CN 111241312A
- Authority
- CN
- China
- Prior art keywords
- picture
- image
- module
- storage
- storage module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0635—Configuration or reconfiguration of storage systems by changing the path, e.g. traffic rerouting, path reconfiguration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
Abstract
The invention relates to a high-efficiency picture storing and reading system and a method, comprising the following steps: the picture storage module is used for directly storing the picture and distributing a stored parameter to the picture; the picture parameter storage module is used for storing parameters of pictures, and the parameters correspond to storage positions in the picture storage module; the picture caching module caches pictures when the pictures are stored in the picture storage module; the image parameter storage module is used for storing the parameters of the image, and the image parameter storage module is used for storing the parameters of the image according to the request and calling the corresponding image from the image storage module according to the parameters.
Description
Technical Field
The invention relates to a high-efficiency picture storage and reading system and a high-efficiency picture storage and reading method, and belongs to the technical field of data storage.
Background
At present, most of picture storage systems directly store a single picture on a disk, one picture corresponds to one file, and then a server generates a unique ID for each picture file and stores the unique ID in different directories, which directly causes the problems that when more and more pictures are obtained, the number of files is increased, the number and depth of directories are increased, if a certain picture needs to be read, a traditional POSIX file system needs to read MetaData, namely MetaData, of the picture file first, and then reads the corresponding picture file according to the MetaData, wherein multiple times of file I/O is needed, and under a large number of reading requests, no matter the response speed or the cost is huge, and the waiting time of a user is longer.
With the development of servers, currently, pictures at a network server are stored, as shown in fig. 1, a user generally requests a picture from a WebServer, the WebServer returns a URL of the picture, and then the user downloads the picture according to the returned URL, the downloading process generally directly requests a CDN node, and if a corresponding picture is stored in a cache of the CDN, the CDN node directly returns the picture; if no corresponding picture exists, reading the picture from the picture storage service by the CDN again, wherein the file reading process of the picture storage server comprises the following steps: 1) reading metaData corresponding to the picture; 2) reading inode information; 3) reading picture information; generally, at least 3 times of file I/O is needed, the file I/O belongs to system call and is very slow, and when a file storage service processes a large number of requests, the excessive file I/O times obviously become a bottleneck.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: in order to overcome the problems, an efficient image storage and reading system and method for reducing the number of file IO times are provided.
The technical scheme adopted by the invention for solving the technical problems is as follows:
an efficient picture storage and reading system comprising:
the picture storage module is used for directly storing the picture and distributing a stored parameter to the picture;
the picture parameter storage module is used for storing parameters of pictures, and the parameters correspond to storage positions in the picture storage module;
the picture caching module caches pictures when the pictures are stored in the picture storage module;
and the calling service module is used for calling the parameters of the pictures from the picture parameter storage module according to the request and calling the corresponding pictures from the picture storage module according to the parameters.
In a preferred embodiment, the picture storage module divides a storage space into a plurality of physical volumes, the physical volumes directly store pictures, and the physical volumes are displayed in a read-only state after being fully stored.
In a preferred embodiment, the image storage module further includes a volume folder, and the parameters allocated to the image stored in the volume folder include an image ID, a physical tag, a logical tag, a correspondence between the logical tag and the physical tag, an image offset, and an image data size.
In a preferred embodiment, the picture parameter storage module stores part of parameters of a picture, including a picture ID, a physical tag, a logical tag, and a correspondence between the logical tag and the physical tag.
In a preferred embodiment, the system further comprises a node caching module, configured to cache recently accessed pictures in the network environment.
In a preferred embodiment, the picture caching module uses a distributed hash table, which caches recently stored pictures, and deletes a picture cache from a cached picture after a physical volume corresponding to the cached picture is full.
The picture storing and reading method of the picture storing and reading system comprises the following steps:
s1: the calling service module obtains a picture access request and calls the parameters of the picture from the picture parameter storage module;
s2: the calling service module firstly calls the picture to the picture caching module according to the parameters of the picture, and if the picture caching module has a cache, the picture is directly called and sent;
s3: and if the image cache module does not cache, calling the image from the image storage module and sending the image to finish reading.
In a preferred embodiment, in step S3, the invoking service module obtains the picture ID, the physical tag, the logical tag, and the corresponding relationship between the logical tag and the physical tag from the picture parameter storage module, selects one physical tag according to the logical tag, finds a corresponding physical volume in the picture storage module, finds the picture offset and the picture data size of the picture in the volume folder, finds the position of the picture in the physical volume according to the picture offset, and reads data of the picture data size backwards, that is, reads the corresponding picture.
In a preferred embodiment, the method further comprises step S4: after step S1, the calling service module searches for a picture in the node cache module through the picture ID, if the corresponding picture is found, the calling service module directly calls and sends the picture, and finishes reading, and if the corresponding picture is not found, the calling service module continues to execute steps S2 and S3.
The invention has the beneficial effects that:
the invention greatly reduces the number of times of file IO when accessing the picture, does not have an addressing process when reading the picture file, only needs to find the corresponding physical volume to carry out one-time sequential reading, and obviously improves the reading efficiency.
Drawings
The invention is further illustrated with reference to the following figures and examples.
FIG. 1 is a prior art flow diagram as described in the background;
fig. 2 is a schematic structural diagram of the image storage and reading system according to the present invention.
Detailed Description
The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic views illustrating only the basic structure of the present invention in a schematic manner, and thus show only the constitution related to the present invention.
Example 1
The invention shown in fig. 2 provides an efficient picture storage and reading system, which includes:
the picture storage module is used for directly storing the picture and distributing a stored parameter to the picture;
the picture storage module divides a storage space into a plurality of physical volumes, when the physical volumes are implemented specifically, the physical volumes are preferably corresponding to stored hard disks, one hard disk is divided into one physical volume, the physical volume directly stores pictures, directories or folders are not set in the physical volume, the physical volume is taken as a whole storage folder, the pictures are directly stored, the pictures are stored in the physical volume in a stacking mode until the physical volume is full, and the pictures are displayed in a read-only state after the physical volume space is full.
The picture storage module further comprises a volume folder, wherein parameters allocated to the pictures are stored in the volume folder, the parameters comprise a picture ID, a physical volume label, a logical volume label, a corresponding relation between the logical volume label and the physical volume label, a picture offset and a picture data size, the picture ID is a parameter generated when the pictures are generated, the physical volume label corresponds to a physical volume in which the pictures are stored in the picture storage module, the logical volume label is unique to the pictures, the physical volume labels may be multiple, because the pictures uploaded by a user are repeatedly or repeatedly stored, the situation that one logical volume label corresponds to multiple physical volume labels may occur, and the picture offset is a position where the pictures are stacked in the physical volume, for example:
for example, in a single physical volume of 100G, there are 10W pictures of 1MB, if the 5W picture is to be read;
offset (50000-1) × 1MB (49999 MB) 49.999G;
the file size is 1M;
the file pointer is shifted to a position of 49999MB and 1MB of picture data is sequentially read as the picture.
The picture parameter storage module is used for storing parameters of pictures, and the parameters correspond to storage positions in the picture storage module; the picture parameter storage module stores partial parameters of the picture, including picture ID, physical volume label, logical volume label and the corresponding relation between the logical volume label and the physical volume label; since the picture parameter storage module needs to process quickly, and when all picture parameters are stored, the processing speed of the picture parameters is influenced, so that only necessary parameters which are convenient to search are stored in the picture parameter storage module, and the processing efficiency is improved.
The picture caching module caches pictures when the pictures are stored in the picture storage module; the picture caching module adopts a Distributed Hash Table (Distributed Hash Table) DHT for short, and is similar to a network of a Tracker which returns seed information according to seed feature codes, so that the method is a Distributed storage method. Under the condition that a server is not needed, each client is responsible for a small-range route and is responsible for storing a small part of data, so that the addressing and the storage of the whole DHT network are realized, the picture caching module is used for caching at a storage end or a local end, the latest stored pictures are cached, when a physical volume corresponding to the cached pictures is stored fully, the cached pictures are deleted from the picture caching, the reading task can be prevented from being carried out while the physical volume is stored, the efficiency is reduced under the condition that the physical volume, particularly a hard disk, is stored and read simultaneously, therefore, the latest stored pictures are cached by the picture caching module, when a reading command is given, the reading is preferentially carried out from the picture caching module, the physical volume only executes the storage command, and when the physical volume is stored fully, the corresponding pictures in the picture caching module are also deleted from the cache, and when a reading command is sent, the reading command is directly read from the physical volume, and the physical volume only executes the reading command at the moment, so that the efficiency is higher, and the storage and reading efficiency is further increased by the arrangement of the picture caching module.
And the calling service module is used for calling the parameters of the pictures from the picture parameter storage module according to the request and calling the corresponding pictures from the picture storage module according to the parameters. During reading, a client initiates an access request at a user end, and the calling service module calls the corresponding picture according to the request.
The invention greatly reduces the number of times of file IO when accessing the picture, does not have an addressing process when reading the picture file, only needs to find the corresponding physical volume to carry out one-time sequential reading, and obviously improves the reading efficiency.
In a preferred embodiment, the system further includes a node cache module, configured to cache a recently accessed picture in a Network environment, where the node cache module is used in a local area Network, and more preferably, the node cache module may be established in the internet, and a CDN is preferably used as the node cache module, and is called a Content Delivery Network, that is, a Content Delivery Network, and caches the picture that is recently stored in the picture storage module, so as to further increase reading efficiency.
Example 2
A picture storing and reading method of the picture storing and reading system according to embodiment 1, comprising the steps of:
s1: the calling service module obtains a picture access request and calls the parameters of the picture from the picture parameter storage module;
s2: the calling service module firstly calls the picture to the picture caching module according to the parameters of the picture, and if the picture caching module has a cache, the picture is directly called and sent;
s3: and if the image cache module does not cache, calling the image from the image storage module and sending the image to finish reading.
In a preferred embodiment, in step S3, the invoking service module obtains the picture ID, the physical tag, the logical tag, and the corresponding relationship between the logical tag and the physical tag from the picture parameter storage module, selects one physical tag according to the logical tag, finds a corresponding physical volume in the picture storage module, finds the picture offset and the picture data size of the picture in the volume folder, finds the position of the picture in the physical volume according to the picture offset, and reads data of the picture data size backwards, that is, reads the corresponding picture.
In a preferred embodiment, the method further comprises step S4: after step S1, the calling service module searches for a picture in the node cache module through the picture ID, if the corresponding picture is found, the calling service module directly calls and sends the picture, and finishes reading, and if the corresponding picture is not found, the calling service module continues to execute steps S2 and S3.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
Claims (9)
1. An efficient picture storage and reading system, comprising:
the picture storage module is used for directly storing the picture and distributing a stored parameter to the picture;
the picture parameter storage module is used for storing parameters of pictures, and the parameters correspond to storage positions in the picture storage module;
the picture caching module caches pictures when the pictures are stored in the picture storage module;
and the calling service module is used for calling the parameters of the pictures from the picture parameter storage module according to the request and calling the corresponding pictures from the picture storage module according to the parameters.
2. The system as claimed in claim 1, wherein the image storage module divides a storage space into a plurality of physical volumes, the physical volumes directly store images, and the physical volumes are displayed in a read-only state after being fully stored.
3. The efficient image storage and reading system of claim 1, wherein the image storage module further comprises a volume folder, and the parameters allocated to the image stored in the volume folder comprise an image ID, a physical tag, a logical tag, a correspondence between the logical tag and the physical tag, an image offset, and an image data size.
4. The efficient image storage and reading system as claimed in claim 3, wherein the image parameter storage module stores partial parameters of the image, including an image ID, a physical tag, a logical tag, and a correspondence between the logical tag and the physical tag.
5. The efficient picture storage and reading system as claimed in any one of claims 1-4, further comprising a node caching module for caching recently accessed pictures in a network environment.
6. The system as claimed in claim 2, wherein the image caching module uses a distributed hash table, which caches recently stored images, and deletes cached images from the image cache when the physical volume corresponding to the cached images is full.
7. A picture storage and reading method of the picture storage and reading system according to claims 1-6, comprising the steps of:
s1: the calling service module obtains a picture access request and calls the parameters of the picture from the picture parameter storage module;
s2: the calling service module firstly calls the picture to the picture caching module according to the parameters of the picture, and if the picture caching module has a cache, the picture is directly called and sent;
s3: and if the image cache module does not cache, calling the image from the image storage module and sending the image to finish reading.
8. The method as claimed in claim 7, wherein in step S3, the invoking service module obtains the picture ID, the physical tag, the logical tag, and the correspondence between the logical tag and the physical tag from the picture parameter storage module, selects a physical tag according to the logical tag, finds the corresponding physical volume in the picture storage module, finds the picture offset and the picture data size of the picture in the volume folder, finds the position of the picture in the physical volume according to the picture offset, and reads the data of the picture data size backwards, that is, reads the corresponding picture.
9. The efficient picture storage and reading method according to claim 7, further comprising step S4: after step S1, the calling service module searches for a picture in the node cache module through the picture ID, if the corresponding picture is found, the calling service module directly calls and sends the picture, and finishes reading, and if the corresponding picture is not found, the calling service module continues to execute steps S2 and S3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911426318.5A CN111241312A (en) | 2019-12-31 | 2019-12-31 | Efficient picture storage and reading system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911426318.5A CN111241312A (en) | 2019-12-31 | 2019-12-31 | Efficient picture storage and reading system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111241312A true CN111241312A (en) | 2020-06-05 |
Family
ID=70870758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911426318.5A Withdrawn CN111241312A (en) | 2019-12-31 | 2019-12-31 | Efficient picture storage and reading system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111241312A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113848454A (en) * | 2021-09-09 | 2021-12-28 | 海光信息技术股份有限公司 | Chip testing method and chip testing machine |
-
2019
- 2019-12-31 CN CN201911426318.5A patent/CN111241312A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113848454A (en) * | 2021-09-09 | 2021-12-28 | 海光信息技术股份有限公司 | Chip testing method and chip testing machine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7444389B2 (en) | Methods and apparatus for generating a content address to indicate data units written to a storage system proximate in time | |
| US7162571B2 (en) | Methods and apparatus for parsing a content address to facilitate selection of a physical storage location in a data storage system | |
| US10353621B1 (en) | File block addressing for backups | |
| US8260833B2 (en) | Naming scheme for media storage | |
| CN106294352B (en) | A kind of document handling method, device and file system | |
| CN109739815B (en) | File processing method, system, device, equipment and storage medium | |
| CN106506587A (en) | A kind of Docker image download methods based on distributed storage | |
| CN109697016B (en) | Method and apparatus for improving storage performance of containers | |
| US20050125626A1 (en) | Methods and apparatus for facilitating access to content in a data storage system | |
| CN107562757B (en) | Query and access method, device and system based on distributed file system | |
| US8977662B1 (en) | Storing data objects from a flat namespace in a hierarchical directory structured file system | |
| US10754825B2 (en) | Path resolver for client access to distributed file systems | |
| CN107197359B (en) | Video file caching method and device | |
| US11586672B2 (en) | Remote virtualized asset delivery and local provisioning | |
| CN103096126B (en) | Towards the cooperative caching method and system of video-on-demand service in cooperative caching cluster | |
| US9128944B2 (en) | File system enabling fast purges and file access | |
| KR20170123336A (en) | File manipulation method and apparatus | |
| JP2005276198A (en) | System and method for balancing multiple memory buffer sizes | |
| CN106817388B (en) | Method and device for acquiring data by virtual machine and host machine and system for accessing data | |
| CN108021333B (en) | System, device and method for randomly reading and writing data | |
| US10515055B2 (en) | Mapping logical identifiers using multiple identifier spaces | |
| US20090327303A1 (en) | Intelligent allocation of file server resources | |
| CN110267077B (en) | Offline caching method, device, terminal and readable storage medium | |
| CN111241312A (en) | Efficient picture storage and reading system and method | |
| CN107181773A (en) | Data storage and data managing method, the equipment of distributed memory system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200605 |