CN105025306B - Video storage method and system based on virtual management distributed file system - Google Patents
Video storage method and system based on virtual management distributed file system Download PDFInfo
- Publication number
- CN105025306B CN105025306B CN201510372701.2A CN201510372701A CN105025306B CN 105025306 B CN105025306 B CN 105025306B CN 201510372701 A CN201510372701 A CN 201510372701A CN 105025306 B CN105025306 B CN 105025306B
- Authority
- CN
- China
- Prior art keywords
- video
- file system
- distributed file
- stored
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23103—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion using load balancing strategies, e.g. by placing or distributing content on different disks, different memories or different servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23109—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion by placing content in organized collections, e.g. EPG data repository
Abstract
The present invention relates to a kind of video storage method and system based on virtual management distributed file system, to solve the problems, such as how to improve storage efficiency and be effectively utilized memory space.This method includes：The type and size of video to be stored are obtained, and generate the storage request of the video to be stored；Corresponding distributed file system is selected according to the type and size of the video to be stored, and the storage request is sent to corresponding distributed file system；Corresponding distributed file system carries out storage operation according to the storage request to the video to be stored.The distributed file system that the present invention stores video to be stored is selected according to the type and size of the video to be stored, video to be stored is stored using suitable distributed file system, the storage efficiency of video is improved, while the memory space of distributed file system has been obtained effective utilization.
Description
Technical field
The present invention relates to technical field of video storage, especially a kind of video based on virtual management distributed file system
Storage method and system.
Background technology
In recent years, as computer network and the high speed development of multimedia technology, various multimedia messages swarm forward.Depending on
Frequency is the carrier of the information such as sound, image and word, becomes users and like as a kind of common multimedia form
Communication media, it is closely bound up with the daily life of the mankind.Due to people for the quality requirement of video also higher and higher, high definition
Chemical conversion is for a kind of trend, this causes the file size of video to increase in magnanimity level, and video is also concurrent at the same time there are more people
The characteristics of access, therefore how to distribute, store, manage and share these massive video resources and become urgent problem.
Research is found, video can be deposited as video management platform using traditional distributed database architecture at present
Storage and management.But its efficiency is not high, and its storage capacity is limited by the ability of relied on data base management system, logarithm
It is also weaker according to the management and issue support of resource.
With the rapid development of Internet technology and the rise of cloud computing technology, cloud storage is increasingly becoming modern storage service
Presentation mode, while internet scale explosive growth, requires webserver data disposal ability higher and higher, clothes
Business device Clustering relies on its good retractility and cheap cost, is increasingly becoming the one of webserver processing mass data
Kind effective solution.The appearance of cloud storage causes this above-mentioned puzzlement to have good solution, and cloud storage is mainly adopted at present
With distributed file system, the type of distributed file system is varied, including Network File System (Network File
System, abbreviation NFS), Google's file system (Google File System, abbreviation GFS), parallel network file system
(Parallel Network File System, abbreviation PNFS), distributed mode of increasing income file system (Free Distributed
File System, abbreviation FastDFS), Hadoop distributed file systems (Hadoop Distributed File System,
Abbreviation HDFS) etc..Different distributed file systems have the characteristics of different, for example, GFS, HDFS usually handle big file, if sharp
More memories can be expended on the contrary by handling small documents with it；Certainly, the Swift file system of OpenStack can be used for storing
The file of arbitrary size, and support the concurrently write-in of multi-user, but there is no structure category, do not support nesting, also do not support
The rewriting of file, can only create.
In fact, the type of video is varied, vary, according to single a certain distributed file system, no
Storage efficiency can be only restricted, memory space can not be effectively utilized.
The content of the invention
The technical problems to be solved by the invention are how to improve storage efficiency and be effectively utilized memory space.
In order to solve the above technical problems, the present invention proposes a kind of video based on virtual management distributed file system and deposits
Method for storing.The video storage method includes：
The type and size of video to be stored are obtained, and generate the storage request of the video to be stored；
According to the type and size of the video to be stored phase is selected from the distributed file system of at least two types
The distributed file system answered, and the storage request is sent to corresponding distributed file system；
Corresponding distributed file system carries out storage operation according to the storage request to the video to be stored.
Further, which further includes：Treated described in corresponding distributed file system Real-time Feedback
Store the storage condition of video.
Further, each distributed file system includes multiple back end and is stored with the position of each back end
The host node of confidence breath；
Corresponding distributed file system carries out storage operation to the video to be stored, specifically includes：
Video to be stored is subjected to piecemeal, obtains some data blocks；
Each data block is stored on different back end according to the positional information stored on host node.
Further, which further includes：Monitor each data section in each distributed file system
The working status of point.
Further, which further includes：Monitor the resource distribution situation of each distributed file system.
In order to solve the above technical problems, present invention also offers a kind of regarding based on virtual management distributed file system
Frequency storage system.The Video Storage System includes：Point of video storage tube platform, virtual management system and at least two types
Cloth file system；
Wherein, the video storage tube platform, for obtaining the type and size of video to be stored, generation is to be stored to be regarded
The storage request of frequency, and the type of video to be stored, size and storage request are sent to the virtual management system；
The virtual management system includes scheduling of resource unit, and the scheduling of resource unit is used to receive institute's video to be stored
Type, size and storage request, according to the type and size from the distributed file system of at least two type
Corresponding distributed file system is selected, and the storage request is sent to corresponding distributed file system；
Corresponding distributed file system, is treated for receiving the storage request, and according to the storage request
Storage video carries out storage operation.
Further, corresponding distributed file system is additionally operable in real time pass through the storage condition of video to be stored
Scheduling of resource unit in the virtual management system feeds back to the video storage tube platform.
Further, each distributed file system includes multiple back end and is stored with the position of each back end
The host node of confidence breath；
Corresponding distributed file system is specifically used for performing under the triggering of the storage request：
Video to be stored is subjected to piecemeal, obtains some data blocks；
Each data block is stored on different back end according to the positional information stored on host node.
Further, the host node is additionally operable to obtain the work of each back end in its affiliated distributed file system
Make status information, and the work state information is sent to the virtual management system；
The virtual management system further includes rm-cell；The rm-cell is used to receive each distribution
The work state information that the host node of file system is sent.
Further, the rm-cell is additionally operable to monitor the resource distribution situation of each distributed file system.
Video storage method provided by the invention and system, the distributed file system stored to video to be stored are
Selected according to the type and size of the video to be stored, so, the present invention can only profit relative to all videos in the prior art
For distributed file system with a certain fixation, the storage efficiency of video is improved, while also makes distributed file system
Memory space has obtained effective utilization.
Brief description of the drawings
The features and advantages of the present invention can be more clearly understood by reference to attached drawing, attached drawing is schematically without that should manage
Solve to carry out any restrictions to the present invention, in the accompanying drawings：
Fig. 1 shows the flow diagram of the video storage method of the invention based on virtual management distributed file system；
Fig. 2 shows the structure diagram of the Video Storage System of the invention based on virtual management distributed file system.
Embodiment
The embodiment of the present invention is described below in detail, the example of the embodiment is shown in the drawings, wherein from beginning to end
Same or similar label represents same or similar element or has the function of same or like element.Below with reference to attached
The embodiment of figure description is exemplary, and is only used for explaining the present invention, and is not construed as limiting the claims.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singulative " one " used herein, " one
It is a ", " described " and "the" may also comprise plural form.It is to be further understood that what is used in the specification of the present invention arranges
Diction " comprising " refer to there are the feature, integer, step, operation, element and/or component, but it is not excluded that in the presence of or addition
One or more other features, integer, step, operation, element, component and/or their groups.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (including technology art
Language and scientific terminology), there is the meaning identical with the general understanding of the those of ordinary skill in fields of the present invention.Should also
Understand, those terms such as defined in the general dictionary, it should be understood that have with the context of the prior art
The consistent meaning of meaning, and unless by specific definitions, otherwise will not be explained with the implication of idealization or overly formal.
Embodiment 1
The present invention provides a kind of video storage method based on virtual management distributed file system, as shown in Figure 1, the party
Method includes：
Step S1, the type and size of video to be stored are obtained, and generate the storage request of the video to be stored；
Step S2, selected according to the type and size of video to be stored from the distributed file system of at least two types
Corresponding distributed file system, and the storage request is sent to corresponding distributed file system；
Step S3, corresponding distributed file system carries out storage behaviour according to the storage request to video to be stored
Make.
The distributed file system that is stored to video to be stored of the present invention be according to the type of the video to be stored and
Size selection, the video of different type or size can select different distributed file systems to be stored.Treated for example, working as
When storage video is larger, it can select to be good at GFS, the HDFS for handling big file.So the present invention is relative to institute in the prior art
There is video to utilize for the distributed file system of a certain fixation, improve the storage efficiency of video, while also make distribution
The memory space of formula file system has obtained effective utilization.
In addition, in order to make storage request can be identified by corresponding distributed file system, can also will storage message into
Retransmited after row certain processing and encapsulation to corresponding distributed file system.
The first improvement to above-mentioned technical proposal, the video storage method further include：Corresponding distributed document
The storage condition of system Real-time Feedback video to be stored.
The storage condition can include storage progress, storage result (such as storing successfully or store failure) etc..Pass through reality
When feed back storage condition, allow users to understand current storage condition in real time.
Second of improvement to above-mentioned technical proposal, each distributed file system include multiple back end and are stored with
The host node of the positional information of each back end；
Corresponding distributed file system carries out storage operation to the video to be stored, specifically includes：
Video to be stored is subjected to piecemeal, obtains some data blocks；
Each data block is stored on different back end according to the positional information stored on host node.
This solution provides a kind of concrete mode of distributed file system storage video, which divides video file
Block, since the size of each data block is much smaller relative to whole video file, so each data block is stored at the same time corresponding
Back end on compared to whole video file is stored, greatly reduce the storage used time, improve storage efficiency.
The further of above-mentioned second of improvement project is improved, which further includes：Monitor each data section
The working status of point.
This programme learns that can back end carry out storage work, to sentence by the working status of monitoring data node
It is disconnected whether data block to be stored in the back end.
The third improvement to above-mentioned further improvement project, the video storage method further include：Monitor each distribution
The resource distribution situation of file system.
So-called resource distribution situation, refers to the capacity of each distributed file system and current institute's amount of storage etc..We
Case can estimate the storage energy of the current Video Storage System by monitoring the resource distribution situation of each distributed file system
The size of power.
Embodiment 2
The present invention provides a kind of Video Storage System based on virtual management distributed file system.As shown in Fig. 2, should
Video Storage System includes：The distributed file system of video storage tube platform, virtual management system and at least two types；
Wherein, the video storage tube platform, for obtaining the type and size of video to be stored, generation is to be stored to be regarded
The storage request of frequency, and the type of video to be stored, size and storage request are sent to the virtual management system；
The virtual management system includes scheduling of resource unit, and the scheduling of resource unit is used to receive institute's video to be stored
Type, size and storage request, according to the type and size from the distributed file system of at least two type
Corresponding distributed file system is selected, and the storage request is sent to corresponding distributed file system；
Corresponding distributed file system, is treated for receiving the storage request, and according to the storage request
Storage video carries out storage operation.
Distributed file system in such scheme is located at bottom, its physical memory resources managed is not necessarily directly connected to
On the local node, can be also connected by computer network with node.Since the Video Storage System has point of multiple types
Cloth file system, distributed file system can be located on different physical mediums, the distributed file system of each type
There is different, the video of suitable storage different type and size.And virtual management system provide one it is abstract logical
With access interface, the difference of the physical medium of the distributed file system of bottom is shielded, seamless use is multiple not
The distributed file system of same type.The video storage tube platform is positioned at the application layer with user mutual, video storage tube
The interface of user's upload or foradownloaded video is designed with platform, can also input or show on the platform certainly the letter of video
Breath, such as title, classification, uplink time etc..So the present invention is integrated with different types of distributed file system, using virtual
Management system is managed multiple distributed file systems, different distributed file systems is cooperated, also makes storage
It can realize in the resource of different distributed file systems shared, be deposited for the different size of video of efficient storage, quick obtaining
Take the video data in different distributions formula file system to provide one kind and refer to approach and scientific basis.
The present invention uses the distribution text that virtual management system is adapted to for the selection of the video to be stored of different type and size
Part system, so as to improve the storage efficiency of video, while also makes the memory space of distributed file system obtain effectively
Utilize.
The first improvement to above-mentioned technical proposal, corresponding distributed file system is additionally operable in real time will be to be stored
The storage condition of video passes through the virtual management system feedback to the video storage tube platform.
User can allow users to understand current storage condition in real time by video storage tube platform.
Second of improvement to above-mentioned technical proposal, each distributed file system include multiple back end and are stored with
The host node of the positional information of each back end；
Corresponding distributed file system is specifically used for performing under the triggering of the storage request：
Video to be stored is subjected to piecemeal, obtains some data blocks；
Each data block is stored on different back end according to the positional information stored on host node.
Since the size of each data block is much smaller relative to whole video file, so each data block is stored at the same time
Compared to whole video file is stored on corresponding back end, the storage used time is greatly reduced, improves storage efficiency.
The further of above-mentioned second of improvement project is improved, the host node is additionally operable to obtain its affiliated distributed document
The work state information of each back end in system, and the work state information is sent to the virtual management system；
The virtual management system further includes rm-cell；The rm-cell is used to receive each distribution
The work state information that the host node of file system is sent.
This programme obtains the work state information of back end by host node, and work state information is sent to resource
Administrative unit, rm-cell receive the work state information of back end, it is achieved thereby that monitoring each back end
Working status purpose, it follows that can back end carry out storage work, to judge whether that data block can be stored in
The back end.
Further, the rm-cell is additionally operable to monitor the resource distribution situation of each distributed file system,
Thus the size of the storage capacity of the current Video Storage System is estimated.
In conclusion video storage method and system of the present invention based on virtual management distributed file system can bases
The type and size of the video to be stored select suitable distributed file system, improve the storage efficiency of video, while
The memory space of distributed file system is set to have obtained effective utilization.
Although being described in conjunction with the accompanying embodiments of the present invention, those skilled in the art can not depart from this hair
Various modifications and variations are made in the case of bright spirit and scope, such modifications and variations are each fallen within by appended claims
Within limited range.
Claims (8)
- A kind of 1. video storage method based on virtual management distributed file system, it is characterised in that including：The type and size of video to be stored are obtained, and generate the storage request of the video to be stored；Selected accordingly from the distributed file system of at least two types according to the type and size of the video to be stored Distributed file system, and the storage request is sent to corresponding distributed file system；Wherein, each distributed text Part system includes multiple back end and is stored with the host node of the positional information of each back end；The video to be stored is carried out piecemeal by corresponding distributed file system according to the storage request, is obtained some A data block, and the positional information stored on the host node according to corresponding distributed file system is by each data Block is stored on the different back end of corresponding distributed file system.
- 2. video storage method according to claim 1, it is characterised in that further include：Corresponding distributed document The storage condition of video to be stored described in system Real-time Feedback.
- 3. video storage method according to claim 1, it is characterised in that further include：Monitor each distributed field system The working status of each back end in system.
- 4. video storage method according to claim 1, it is characterised in that further include：Monitor each distributed field system The resource distribution situation of system.
- A kind of 5. Video Storage System based on virtual management distributed file system, it is characterised in that including：Video storage tube The distributed file system of platform, virtual management system and at least two types；Wherein, the video storage tube platform, for obtaining the type and size of video to be stored, generates video to be stored Storage request, and the type of video to be stored, size and storage request are sent to the virtual management system；The virtual management system includes scheduling of resource unit, the scheduling of resource unit be used to receiving video to be stored class Type, size and storage request, select according to the type and size from the distributed file system of at least two type Corresponding distributed file system, and the storage request is sent to corresponding distributed file system；Wherein, each point Cloth file system includes multiple back end and is stored with the host node of the positional information of each back end；Corresponding distributed file system, is treated for receiving the storage request, and according to the storage request by described Store video and carry out piecemeal, obtain several data blocks, and deposited on the host node according to corresponding distributed file system Each data block is stored on the different back end of corresponding distributed file system by the positional information of storage.
- 6. Video Storage System according to claim 5, it is characterised in that corresponding distributed file system is also used The storage condition of video to be stored is fed back into described regard by the scheduling of resource unit in the virtual management system in real time Frequency storage management platform.
- 7. Video Storage System according to claim 5, it is characterised in thatThe host node is additionally operable to obtain the work state information of each back end in its affiliated distributed file system, and will The work state information is sent to the virtual management system；The virtual management system further includes rm-cell；The rm-cell is used to receive each distributed document The work state information that the host node of system is sent.
- 8. Video Storage System according to claim 7, it is characterised in that it is each that the rm-cell is additionally operable to monitoring The resource distribution situation of a distributed file system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372701.2A CN105025306B (en) | 2015-06-30 | 2015-06-30 | Video storage method and system based on virtual management distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372701.2A CN105025306B (en) | 2015-06-30 | 2015-06-30 | Video storage method and system based on virtual management distributed file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105025306A CN105025306A (en) | 2015-11-04 |
| CN105025306B true CN105025306B (en) | 2018-05-11 |
Family
ID=54414959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510372701.2A Expired - Fee Related CN105025306B (en) | 2015-06-30 | 2015-06-30 | Video storage method and system based on virtual management distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105025306B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101706783A (en) * | 2009-11-16 | 2010-05-12 | 中兴通讯股份有限公司 | Method and device for checking and repairing files in embedded equipment |
| CN102938784A (en) * | 2012-11-06 | 2013-02-20 | 无锡江南计算技术研究所 | Method and system used for data storage and used in distributed storage system |
| CN103856567A (en) * | 2014-03-26 | 2014-06-11 | 西安电子科技大学 | Small file storage method based on Hadoop distributed file system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100426256C (en) * | 2003-02-20 | 2008-10-15 | 松下电器产业株式会社 | Information recording medium and region management method thereof |
| US20080147667A1 (en) * | 2006-12-15 | 2008-06-19 | Samsung Electronics Co., Ltd. | Data management apparatus and data management method thereof |
-
2015
- 2015-06-30 CN CN201510372701.2A patent/CN105025306B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101706783A (en) * | 2009-11-16 | 2010-05-12 | 中兴通讯股份有限公司 | Method and device for checking and repairing files in embedded equipment |
| CN102938784A (en) * | 2012-11-06 | 2013-02-20 | 无锡江南计算技术研究所 | Method and system used for data storage and used in distributed storage system |
| CN103856567A (en) * | 2014-03-26 | 2014-06-11 | 西安电子科技大学 | Small file storage method based on Hadoop distributed file system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105025306A (en) | 2015-11-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105933376B (en) | A kind of data manipulation method, server and storage system | |
| CN110520853A (en) | The queue management of direct memory access | |
| CN105550345B (en) | File operation method and device | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| US20240031316A1 (en) | Managing message attachments | |
| US9137394B2 (en) | Systems and methods for obtaining a resource | |
| AU2015204742B2 (en) | Methods for generating an activity stream | |
| CN104537076A (en) | File reading and writing method and device | |
| CN103369048A (en) | Content sharing method and system based on dragging operations | |
| CN103905311A (en) | Flow table matching method and device and switch | |
| US11082517B2 (en) | Content transformations using a transformation node cluster | |
| US20150370870A1 (en) | Elastic scalability of a content transformation cluster | |
| CN103176849A (en) | Virtual machine clustering deployment method based on resource classification | |
| CN103491024A (en) | Job scheduling method and device for streaming data | |
| CN102882960B (en) | A kind of sending method of resource file and device | |
| CN107408239B (en) | Architecture for managing mass data in communication application through multiple mailboxes | |
| CN114924751A (en) | Method and device for distributing service access request flow | |
| CN106940712A (en) | Sequence generating method and equipment | |
| CN110413588A (en) | Distributed objects storage method, device, computer equipment and storage medium | |
| WO2023179784A1 (en) | Data processing method and apparatus | |
| CN105025306B (en) | Video storage method and system based on virtual management distributed file system | |
| CN108243207A (en) | A kind of date storage method of network cloud disk | |
| CN101415029B (en) | Method and apparatus for distributing files | |
| CN110471968A (en) | Dissemination method, device, equipment and the storage medium of ETL task | |
| CN104735149A (en) | Cloud computing resource management system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180511 Termination date: 20200630 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |