CN112241244A - Mass storage method of streaming media logs - Google Patents
Mass storage method of streaming media logs Download PDFInfo
- Publication number
- CN112241244A CN112241244A CN202011148033.2A CN202011148033A CN112241244A CN 112241244 A CN112241244 A CN 112241244A CN 202011148033 A CN202011148033 A CN 202011148033A CN 112241244 A CN112241244 A CN 112241244A
- Authority
- CN
- China
- Prior art keywords
- streaming media
- file
- index
- level index
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a mass storage method of a streaming media log, which comprises the following steps: the method comprises the steps of firstly, dividing log files, then selecting a placement method, then placing the log files in blocks, then writing the log files into a storage medium, then establishing an index, and finally calling the files; the invention pre-allocates the storage space of the streaming media server according to the size of each fragment file, and can prolong the service life of the storage hard disk to a certain extent compared with the prior art; the invention uses four levels of indexes, and can position more quickly and accurately.
Description
Technical Field
The invention relates to a streaming media log, in particular to a mass storage method of the streaming media log, belonging to the field of logs.
Background
The logs in life are used for recording the point and drop of your life and exposing the world of your heart to better explain the content of the world of your heart, the logs of the computer are used for recording event records called logs generated by the operation of a user when the user operates the computer, the description of relevant operations such as date, time, users and actions is recorded in each row of logs, and the streaming media logs are logs which can be streamed on the network, because the logs of the computer can be valuable information treasury and also can be worthless data puddle, and the logs of the computer can generate a large number of logs along with the accumulation of time, a large-amount storage method of the streaming media logs is needed.
The existing streaming media log storage method is characterized in that log files are directly and automatically stored in a corresponding server, the method easily causes log file storage disorder, occupies a large amount of memory, is not beneficial to quickly calling the corresponding log files, and can cause serious reduction of the performance of the whole system due to the existence of the internal access tendency characteristics of the files.
Disclosure of Invention
The present invention is directed to a method for mass storage of a streaming media log, which includes the following steps:
s1, dividing the log file, and uniformly dividing the streaming media log file to be stored into a plurality of fragment files from head to tail;
s2, selecting a placement method, selecting a symmetrical block placement method according to the access tendency of the streaming media log file, judging whether redundant data exist in the streaming media server for storage, selecting a symmetrical duplication factor pair method if the redundant data exist, and selecting a symmetrical duplication factor pair method if the redundant data do not exist;
s3, placing the log files in blocks, and reasonably storing each fragment file into the streaming media server according to the selected placement method and the number of the streaming media servers;
s4, writing the data into a storage medium, and writing the data of each fragment file into a mechanical hard disk or a solid state disk in the streaming media server;
s5, establishing indexes, namely establishing corresponding indexes according to storage positions of the fragment file data of related information of the fragment file data, and finally sending the corresponding indexes to a streaming media server and an index server for storage;
and S6, calling the file, finding the needed streaming media log file through the corresponding index, and calling to show the content of the streaming media log file to an operator.
Preferably, the S1 needs to pre-allocate the storage space of the streaming media server according to the size of each fragmented file.
Preferably, the access tendency in S2 is that people prefer to access the header of the streaming media log file.
Preferably, the symmetry factor in S2 positively correlates the load balancing capability of the method with the parity of the number of files.
Preferably, the load unevenness of the method by the symmetric replication factor in S2 is always zero.
Preferably, the writing method in S4 employs a large cache mechanism.
Preferably, the writing of the data in S4 adopts a process method of mutual exclusion lock.
Preferably, the index in S5 is a four-level index, and the index includes: the index management system comprises a first-level index, a second-level index, a third-level index and a fourth-level index, wherein the first-level index is stored in a resource library of an index server, the second-level index is stored in a file database of a streaming media server, the third-level index is stored at the tail of each piece of file data, and the fourth-level index is stored at the head of each piece of file data.
Preferably, the system marks the file in S6 every time the file is called.
Preferably, at least two streaming media servers and one index server are used in S1-S6.
The invention has the beneficial effects that:
(1) compared with the traditional log file placement strategy, the method can ensure the ordered storage of the log, increase the storage capacity, improve the expansion capability and the load balancing capability of the system and overcome the negative influence of the access tendency on the system performance;
(2) the storage space of the streaming media server is pre-allocated according to the size of each fragment file, and compared with the prior art, the method can prevent the disk fragments generated in the process of repeatedly creating the file by the storage hard disk in the streaming media server from influencing the read-write efficiency of the disk, and can prolong the service life of the storage hard disk to a certain extent;
(3) the invention uses four-level indexes, including the first-level index, the second-level index, the third-level index and the fourth-level index, and the four-level indexes are stored in the index server and the streaming media server.
Drawings
FIG. 1 is a flow chart of a storage method of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
a method for mass storage of a streaming media log, the method comprising the steps of:
s1, dividing the log file, and uniformly dividing the streaming media log file to be stored into a plurality of fragment files from head to tail;
s2, selecting a placement method, selecting a symmetrical block placement method according to the access tendency of the streaming media log file, judging whether redundant data exist in the streaming media server for storage, selecting a symmetrical duplication factor pair method if the redundant data exist, and selecting a symmetrical duplication factor pair method if the redundant data do not exist;
s3, placing the log files in blocks, and reasonably storing each fragment file into the streaming media server according to the selected placement method and the number of the streaming media servers;
s4, writing the data into a storage medium, and writing the data of each fragment file into a mechanical hard disk or a solid state disk in the streaming media server;
s5, establishing indexes, namely establishing corresponding indexes according to storage positions of the fragment file data of related information of the fragment file data, and finally sending the corresponding indexes to a streaming media server and an index server for storage;
and S6, calling the file, finding the needed streaming media log file through the corresponding index, and calling to show the content of the streaming media log file to an operator.
Before S1, the storage space of the streaming media server needs to be pre-allocated according to the size of each fragmented file.
The access tendency in S2 is that people prefer to access the header of the streaming media log file.
The symmetry factor in S2 positively correlates the load balancing capability of the method with the parity of the number of files.
The load non-uniformity of the symmetric copy factor pair method in S2 is always zero.
The writing method in S4 employs a large cache mechanism.
The writing of the data in the S4 adopts a process method of mutual exclusion lock.
The index in S5 is a four-level index, and the index includes: the index management system comprises a first-level index, a second-level index, a third-level index and a fourth-level index, wherein the first-level index is stored in a resource library of an index server, the second-level index is stored in a file database of a streaming media server, the third-level index is stored at the tail of each piece of file data, and the fourth-level index is stored at the head of each piece of file data.
The system will label each time a file is called in S6.
At least two streaming servers and one index server are used in the steps S1-S6.
The file placing method can fully utilize the storage space and also greatly improve the expansion capability and the load balancing capability of the system.
Example two:
a method for mass storage of a streaming media log, the method comprising the steps of:
s1, dividing the log file, and uniformly dividing the streaming media log file to be stored into a plurality of fragment files from head to tail;
s2, selecting a placement method, selecting a symmetrical block placement method according to the access tendency of the streaming media log file, judging whether redundant data exist in the streaming media server for storage, selecting a symmetrical duplication factor pair method if the redundant data exist, and selecting a symmetrical duplication factor pair method if the redundant data do not exist;
s3, placing the log files in blocks, and reasonably storing each fragment file into the streaming media server according to the selected placement method and the number of the streaming media servers;
s4, writing the data into a storage medium, and writing the data of each fragment file into a mechanical hard disk or a solid state disk in the streaming media server;
s5, establishing indexes, namely establishing corresponding indexes according to storage positions of the fragment file data of related information of the fragment file data, and finally sending the corresponding indexes to a streaming media server and an index server for storage;
and S6, calling the file, finding the needed streaming media log file through the corresponding index, and calling to show the content of the streaming media log file to an operator.
Before S1, the storage space of the streaming media server needs to be pre-allocated according to the size of each fragmented file.
The access tendency in S2 is that people prefer to access the header of the streaming media log file.
The symmetry factor in S2 positively correlates the load balancing capability of the method with the parity of the number of files.
The load non-uniformity of the symmetric copy factor pair method in S2 is always zero.
The writing method in S4 employs a large cache mechanism.
The writing of the data in the S4 adopts a process method of mutual exclusion lock.
The index in S5 is a four-level index, and the index includes: the index management system comprises a first-level index, a second-level index, a third-level index and a fourth-level index, wherein the first-level index is stored in a resource library of an index server, the second-level index is stored in a file database of a streaming media server, the third-level index is stored at the tail of each piece of file data, and the fourth-level index is stored at the head of each piece of file data.
The system will label each time a file is called in S6.
At least two streaming servers and one index server are used in the steps S1-S6.
The storage method is convenient for quick and accurate review of the streaming media log file.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (10)
1. A method for mass storage of streaming media logs, comprising: the mass storage method of the streaming media logs comprises the following steps:
s1, dividing the log file, and uniformly dividing the streaming media log file to be stored into a plurality of fragment files from head to tail;
s2, selecting a placement method, selecting a symmetrical block placement method according to the access tendency of the streaming media log file, judging whether redundant data exist in the streaming media server for storage, selecting a symmetrical duplication factor pair method if the redundant data exist, and selecting a symmetrical duplication factor pair method if the redundant data do not exist;
s3, placing the log files in blocks, and reasonably storing each fragment file into the streaming media server according to the selected placement method and the number of the streaming media servers;
s4, writing the data into a storage medium, and writing the data of each fragment file into a mechanical hard disk or a solid state disk in the streaming media server;
s5, establishing indexes, namely establishing corresponding indexes according to storage positions of the fragment file data of related information of the fragment file data, and finally sending the corresponding indexes to a streaming media server and an index server for storage;
and S6, calling the file, finding the needed streaming media log file through the corresponding index, and calling to show the content of the streaming media log file to an operator.
2. The method of claim 1, wherein the streaming media log comprises: before S1, the storage space of the streaming media server needs to be pre-allocated according to the size of each fragmented file.
3. The method of claim 1, wherein the streaming media log comprises: the access tendency in S2 is that people prefer to access the header of the streaming media log file.
4. The method of claim 1, wherein the streaming media log comprises: the symmetry factor in S2 positively correlates the load balancing capability of the method with the parity of the number of files.
5. The method of claim 1, wherein the streaming media log comprises: the load non-uniformity of the symmetric copy factor pair method in S2 is always zero.
6. The method of claim 1, wherein the streaming media log comprises: the writing method in S4 employs a large cache mechanism.
7. The method of claim 1, wherein the streaming media log comprises: the writing of the data in the S4 adopts a process method of mutual exclusion lock.
8. The method of claim 1, wherein the streaming media log comprises: the index in S5 is a four-level index, and the index includes: the index management system comprises a first-level index, a second-level index, a third-level index and a fourth-level index, wherein the first-level index is stored in a resource library of an index server, the second-level index is stored in a file database of a streaming media server, the third-level index is stored at the tail of each piece of file data, and the fourth-level index is stored at the head of each piece of file data.
9. The method of claim 1, wherein the streaming media log comprises: the system will label each time a file is called in S6.
10. The method of claim 1, wherein the streaming media log comprises: at least two streaming servers and one index server are used in the steps S1-S6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011148033.2A CN112241244A (en) | 2020-10-23 | 2020-10-23 | Mass storage method of streaming media logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011148033.2A CN112241244A (en) | 2020-10-23 | 2020-10-23 | Mass storage method of streaming media logs |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112241244A true CN112241244A (en) | 2021-01-19 |
Family
ID=74169546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011148033.2A Pending CN112241244A (en) | 2020-10-23 | 2020-10-23 | Mass storage method of streaming media logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112241244A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101141627A (en) * | 2007-10-23 | 2008-03-12 | 深圳市迅雷网络技术有限公司 | Storage system and method of stream media file |
| CN104021088A (en) * | 2014-06-24 | 2014-09-03 | 广东睿江科技有限公司 | Log storage method and device |
| CN111444154A (en) * | 2020-03-26 | 2020-07-24 | 涵涡智航科技(玉溪)有限公司 | Method for efficiently storing log files in single chip microcomputer system |
-
2020
- 2020-10-23 CN CN202011148033.2A patent/CN112241244A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101141627A (en) * | 2007-10-23 | 2008-03-12 | 深圳市迅雷网络技术有限公司 | Storage system and method of stream media file |
| CN104021088A (en) * | 2014-06-24 | 2014-09-03 | 广东睿江科技有限公司 | Log storage method and device |
| CN111444154A (en) * | 2020-03-26 | 2020-07-24 | 涵涡智航科技(玉溪)有限公司 | Method for efficiently storing log files in single chip microcomputer system |
Non-Patent Citations (2)
| Title |
|---|
| 吴松等: "一种流媒体文件的分块放置方法", 《计算机学报》, vol. 29, no. 3, 31 March 2006 (2006-03-31), pages 500 - 507 * |
| 张明等: "《操作系统习题解答与实验指导》", 28 February 2017, pages: 122 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| US8271462B2 (en) | Method for creating a index of the data blocks | |
| KR101207510B1 (en) | Cluster Data Management System And Method for Data Restoring Using Shared Read-Only Log in Cluster Data Management System | |
| CN106201771B (en) | Data-storage system and data read-write method | |
| CN109710572B (en) | HBase-based file fragmentation method | |
| CN109947363B (en) | Data caching method of distributed storage system | |
| CN109522283B (en) | Method and system for deleting repeated data | |
| CN103399823B (en) | The storage means of business datum, equipment and system | |
| CN111400083B (en) | Data storage method and system and storage medium | |
| CN110058822A (en) | A kind of disk array transverse direction expanding method | |
| US11385823B2 (en) | Method, electronic device and computer program product for rebuilding disk array | |
| CN110888837B (en) | Object storage small file merging method and device | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN101464901A (en) | Object search method in object storage device | |
| CN110007870A (en) | A kind of storage equipment write request processing method and relevant apparatus | |
| CN104216908A (en) | Internet data management system and reading and writing method thereof | |
| CN113704217A (en) | Metadata and data organization architecture method in distributed persistent memory file system | |
| CN111399761A (en) | Storage resource allocation method, device and equipment and storage medium | |
| CN107506466A (en) | A kind of small documents storage method and system | |
| CN114281989A (en) | Data deduplication method and device based on text similarity, storage medium and server | |
| CN106547484A (en) | It is a kind of that internal storage data reliability method and system realized based on RAID5 | |
| CN108959497A (en) | distributed file system log processing method, device, equipment and storage medium | |
| CN112241244A (en) | Mass storage method of streaming media logs | |
| CN109800208B (en) | Network traceability system and its data processing method, computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210119 |