CN110267060B - Video file storage injection method and device, computer equipment and storage medium - Google Patents
Video file storage injection method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110267060B CN110267060B CN201910669429.2A CN201910669429A CN110267060B CN 110267060 B CN110267060 B CN 110267060B CN 201910669429 A CN201910669429 A CN 201910669429A CN 110267060 B CN110267060 B CN 110267060B
- Authority
- CN
- China
- Prior art keywords
- storage
- downloading
- storage node
- queue
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23113—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving housekeeping operations for stored content, e.g. prioritizing content for deletion because of storage space restrictions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application relates to a video file storage injection method, a video file storage injection device, computer equipment and a storage medium. The method comprises the following steps: the method comprises the steps of receiving an injection request of a video file to be stored, sent by a video source, pushing the injection request to a cluster message queue of each storage cluster, obtaining a downloading task from the cluster message queue, distributing the downloading task to a downloading queue corresponding to a corresponding storage node according to available state information of each storage node in the storage cluster, and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue. By adopting the method, the transmission bandwidth during data recovery can be reduced when the storage center is damaged.
Description
Technical Field
The present application relates to the field of cloud storage technologies, and in particular, to a method and an apparatus for storing and injecting a video file, a computer device, and a storage medium.
Background
With the development of cloud storage technology, enterprises often store data on a cloud server, in practical application, cloud manufacturers need to receive file storage requirements of customers, particularly storage of video files, and at present, storage space requirements of video websites reach a PB level, so that certain difficulty exists in developing a cloud product suitable for video storage.
According to the traditional video storage cloud product, storage data are stored in a storage center, and once the storage center is damaged, due to the fact that the video data volume is large, huge transmission bandwidth expenditure is needed for data recovery.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a video file storage injection method, apparatus, computer device and storage medium capable of reducing transmission bandwidth during data recovery when a storage center is damaged.
A video file storage injection method, the method comprising:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
In one embodiment, the method further comprises the following steps: requesting to acquire a downloading task in a cluster message queue, and storing the acquired downloading task in a storage cluster center of the storage cluster; and distributing the downloading task to the corresponding downloading queue of the storage node according to the detection result of the storage cluster center on the available state information of each storage node.
In one embodiment, the available state information includes: survival state information, storage space information and download queue busy information; further comprising: detecting survival state information, storage space information and downloading queue busy information of each storage node in the storage cluster according to the storage cluster center; if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list; and distributing the downloading task to a downloading queue of a storage node in the white list.
In one embodiment, the method further comprises the following steps: receiving a client request sent by a storage node; detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node; and if so, distributing the downloading task to a downloading queue of the storage node.
In one embodiment, the available state information includes: a minimum usage store, the minimum usage store comprising: the download queue is minimum, the I/O read-write is minimum and the use of the storage space is minimum; further comprising: calculating minimum use storage for each storage node in the white list; and distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
In one embodiment, the method further comprises the following steps: according to the downloading task, sending a client downloading request to the cluster message queue; downloading the video file to be stored according to the client downloading request; performing file verification on the video file to be stored, and storing the video file to be stored passing the verification to a physical disk of a storage node; and detecting whether the physical disk of the storage node is abnormal or not, and if so, uploading abnormal information to the storage cluster center so that the storage cluster center distributes the downloading task to other storage nodes.
In one embodiment, the method further comprises the following steps: recording the access frequency of the video files in the storage nodes; and if the access frequency is less than a threshold value, deleting the video file in the storage node.
A video file storage injection apparatus, the apparatus comprising:
the request receiving module is used for receiving an injection request of a video file to be stored, which is sent by a video source;
the pushing module is used for pushing the injection request to a cluster message queue of each storage cluster;
an allocation module, configured to acquire a download task from the cluster message queue, and allocate the download task to a download queue corresponding to the corresponding storage node according to available state information of each storage node in the storage cluster
And the storage module is used for downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
According to the video file storage and injection method, device, computer equipment and storage medium, when a video file is injected into a video source, an injection request is pushed to the cluster message queue of each storage cluster, wherein the distributed storage system comprises a plurality of storage clusters, and each storage cluster comprises a plurality of storage nodes. And then acquiring a downloading task from the cluster message queue, distributing the downloading task to the downloading queue of each storage node, and storing the video file in the storage node. Therefore, when the video file is stored, the video file may be stored in a storage node in any one of the storage clusters, and the storage center only keeps one copy, so that when the storage center of one of the storage clusters is damaged, the data can be recovered only by performing data transmission from other clusters, and the bandwidth cost of data recovery is saved.
Drawings
FIG. 1 is a diagram illustrating an exemplary implementation of a video file storage injection method;
FIG. 2 is a flow diagram illustrating a method for video file storage injection in one embodiment;
FIG. 3 is a flowchart illustrating the step of assigning download tasks in one embodiment;
FIG. 4 is a flowchart illustrating the step of assigning download tasks in another embodiment;
FIG. 5 is a block diagram of an embodiment of a video file storage and injection apparatus;
FIG. 6 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The video file storage and injection method provided by the application can be applied to the application environment shown in fig. 1. The terminal 102 communicates with the cloud platform server 104 through a network. The terminal 102 is a video source, and the terminal 102 may send an injection request to the cloud platform server 104 to store a video file in the terminal 102 to the cloud platform server 104.
The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the cloud platform server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers.
In an embodiment, as shown in fig. 2, a video file storage injection method is provided, which is described by taking the method as an example applied to the cloud platform server in fig. 1, and includes the following steps:
The video source can be a personal computer, a notebook computer and the like, and the video source can actively send an injection request to the cloud platform server.
It should be noted that, in one injection request, the video source may request the cloud platform server to inject a plurality of video files, and therefore, the cloud platform server needs to process storage of the plurality of video files.
The cloud platform server is composed of a plurality of storage clusters, each storage cluster is controlled by a storage cluster center, a plurality of storage nodes are further distributed in the storage clusters, the storage nodes are responsible for storing video files to be stored, and the storage cluster centers are responsible for distributing storage tasks of the storage nodes.
Specifically, each storage cluster corresponds to a cluster message queue, and the storage cluster center can acquire the data of the video file corresponding to the injection request from the cluster message queue.
The cluster message queue is constructed by a cache mechanism, and the data in the cluster message queue can be ensured to be orderly acquired by the storage cluster center according to the first-in first-out principle.
And step 206, acquiring the download tasks from the cluster message queue, and distributing the download tasks to the download queues corresponding to the corresponding storage nodes according to the available state information of each storage node in the storage cluster.
The downloading task refers to a task capable of executing downloading operation, and the storage node can download the corresponding video file corresponding to the downloading task according to the downloading task. For batch injection, one or a group of video files corresponds to one downloading task. Therefore, after the storage cluster center acquires the video file to be stored in the cluster message queue, the storage cluster center correspondingly generates the downloading tasks, and detects the available state information of each storage node in the storage cluster to determine which storage nodes can execute the downloading tasks, so that the downloading tasks are distributed to the downloading queues corresponding to the storage nodes, the downloading queues are also constructed through a cache mechanism, the downloading tasks are pushed to the downloading queues, and the current working state of the storage nodes can be determined.
And step 208, downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
The storage node can download the video file to be stored according to the download task in the download queue, so that the video file is stored in the storage node.
In the video file storage and injection method, when a video source injects a video file, an injection request is pushed to a cluster message queue of each storage cluster, where a distributed storage system includes a plurality of storage clusters, and each storage cluster includes a plurality of storage nodes. And then acquiring a downloading task from the cluster message queue, distributing the downloading task to the downloading queue of each storage node, and storing the video file in the storage node. Therefore, when the video file is stored, the video file may be stored in a storage node in any one of the storage clusters, and the storage center only keeps one copy, so that when the storage center of one of the storage clusters is damaged, the data can be recovered only by performing data transmission from other clusters, and the bandwidth cost of data recovery is saved.
In one embodiment, the download task may be distributed in the following manner: requesting to acquire a download task in the cluster message queue, storing the acquired download task in a storage cluster center of a storage cluster, and distributing the download task to a download queue of a corresponding storage node according to a detection result of the storage cluster center on available state information of each storage node. In this embodiment, a storage cluster center in a storage cluster actively sends a request to a cluster message queue to obtain download tasks, where the number of the download tasks is determined according to the number of video files injected in batch by a video source, and therefore, after the storage cluster center obtains the download tasks and needs to store the download tasks in the storage cluster, the storage cluster center detects available state information of storage nodes below the storage cluster, and the download tasks are conveniently distributed.
In another embodiment, the available state information includes: survival status information, storage space information, and download queue busy information, as shown in fig. 3, a schematic flowchart of the step of allocating a download task is provided, and the specific steps are as follows:
The survival state information refers to whether the storage cluster center can establish connection with the storage node, if so, the survival state information is survival, and if not, the survival state information is non-survival. For the storage space information, the space of the physical memory of each storage node is limited, and when the storage space is smaller than the video file to be stored, the storage node cannot execute the download task, and the storage space information is the size of the remaining storage space. The downloading queue busy information is determined by the length of the queue in the downloading queue, and when the length of the downloading queue exceeds a threshold value, the downloading queue is busy.
And step 304, if the survival state information, the storage space information and the downloading queue busy information all reach the preset threshold condition, adding the storage node into a preset white list.
The white list is provided with storage nodes which can pass through, namely, a mode that the storage nodes can distribute the downloading tasks to the storage nodes when the storage nodes request the downloading tasks, and the white list is set, so that the downloading tasks can be effectively distributed to the storage nodes which can execute the downloading tasks, and the storage of the video data is ensured.
In the embodiment, the white list of the storage node is established by detecting the survival state information, the storage space information and the downloading queue busy information, and the accurate storage of the video file can be ensured through the white list.
For step 306, in one embodiment, the client request sent by the storage node may be received, and whether the storage node is in a white list is detected according to an internet protocol address corresponding to the storage node, and if so, the download task is allocated to a download queue of the storage node. In this embodiment, through a white list mechanism, a storage node for downloading a task can be quickly determined, so that task allocation efficiency is improved.
Specifically, the client request sent by the storage node is composed of an encryption algorithm and an IP address, the security of interaction is guaranteed through the encryption algorithm, and white list verification is performed through the IP address.
In addition, the client request may be an HTTP request, and the download task is obtained from the storage cluster center by the HTTP request.
In one embodiment, the available state information includes: a minimum usage store, the minimum usage store comprising: the download queue is minimal, the I/O read and write is minimal, and the storage space usage is minimal. In this embodiment, the minimum used storage of all storage nodes in the storage cluster needs to be detected, which is beneficial to load balancing.
Specifically, as shown in fig. 4, a schematic flowchart of downloading task allocation is provided, which includes the following specific steps:
And step 404, distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
In this embodiment, when downloading the video file, the video file has a size that is divided into two, and the video file is too large, which may cause congestion in a download queue of the storage node, congestion in I/O reading and writing, and uneven use of the storage space for the whole storage cluster.
Specifically, the distributed algorithm may be a consistent hashing algorithm, and specifically, before performing consistent hashing, the minimum used storage of each storage node is obtained, and then consistent hashing policy reallocation is performed on the minimum storage node. Thereby ensuring the balance of the storage nodes in the actual work.
In one embodiment, the storage node acquires a download task from the download queue, then sends a client download request to the cluster message queue according to the download task, downloads a video file to be stored according to the client download request, then performs file verification on the video file to be stored, and stores the video file to be stored, which passes the verification, in a physical disk of the storage node. In the process, whether the physical hard disk of the storage node is abnormal needs to be detected, and if so, the abnormal information needs to be uploaded to the storage cluster center, so that the river downloading task in the storage cluster is distributed to other storage nodes. In this embodiment, the video file is checked to avoid storing the damaged video file, so that the physical disk space is effectively utilized, the downloading task cannot be continuously executed on the storage node with the damaged physical disk, and the storage cluster center needs to allocate the downloading task to other storage nodes to ensure that the video file can be stored in the storage cluster.
It is worth noting that when the physical disk of one storage node is damaged, all the downloading tasks are stopped and redistributed to other storage nodes
Specifically, the client download request may be an HTTP or HTTPs download request, so as to download the video file from the storage cluster center.
In another embodiment, a sha1/md5 file verification algorithm can be used to verify the video file to be stored, so as to ensure that the downloaded file is correct.
In another embodiment, the storage cluster center further needs to record the access frequency of the video file in the storage node, and if the access frequency is smaller than a threshold, delete the video file in the storage node.
In this embodiment, when a video file is accessed, the storage cluster center records the time period of the video file being accessed, and may delete the video file in a day event unit, for example, if the video file is not accessed for 180 days, so as to ensure that the video file in the storage node is valid.
It should be understood that although the various steps in the flow charts of fig. 2-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-4 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 5, there is provided a video file storage injection apparatus including: a request receiving module 502, a pushing module 504, an allocating module 506, and a storing module 508, wherein:
a request receiving module 502, configured to receive an injection request of a video file to be stored, where the injection request is sent by a video source;
a pushing module 504, configured to push the injection request to a cluster message queue of each storage cluster;
an allocating module 506, configured to acquire a download task from the cluster message queue, and allocate the download task to a download queue corresponding to the corresponding storage node according to available state information of each storage node in the storage cluster
And the storage module 508 is configured to download the video file to be stored to the storage node according to the download task in the download queue.
In one embodiment, the allocating module 506 is further configured to request to acquire a download task in the cluster message queue, and store the acquired download task in a storage cluster center of the storage cluster; and distributing the downloading task to the corresponding downloading queue of the storage node according to the detection result of the storage cluster center on the available state information of each storage node.
In one embodiment, the available state information includes: survival state information, storage space information and download queue busy information; the allocation module 506 is further configured to detect, according to the storage cluster center, survival state information, storage space information, and download queue busy information of each storage node in the storage cluster; if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list; and distributing the downloading task to a downloading queue of a storage node in the white list.
In one embodiment, the allocating module 506 is further configured to receive a client request sent by a storage node; detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node; and if so, distributing the downloading task to a downloading queue of the storage node.
In one embodiment, the available state information includes: a minimum usage store, the minimum usage store comprising: the download queue is minimum, the I/O read-write is minimum and the use of the storage space is minimum; the allocation module 506 is further configured to calculate a minimum usage storage for each storage node in the white list; and distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
In one embodiment, the storage module 508 is further configured to send a client download request to the cluster message queue according to the download task; downloading the video file to be stored according to the client downloading request; performing file verification on the video file to be stored, and storing the video file to be stored passing the verification to a physical disk of a storage node; and detecting whether the physical disk of the storage node is abnormal or not, and if so, uploading abnormal information to the storage cluster center so that the storage cluster center distributes the downloading task to other storage nodes.
In one embodiment, the system further comprises an updating module, configured to record access frequency of the video file in the storage node; and if the access frequency is less than a threshold value, deleting the video file in the storage node.
For specific limitations of the video file storage and injection device, reference may be made to the above limitations of the video file storage and injection method, which are not described herein again. The modules in the video file storage and injection device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 6. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing video file data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a video file storage injection method.
Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is provided a computer device comprising a memory storing a computer program and a processor implementing the following steps when the processor executes the computer program:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
In one embodiment, the processor, when executing the computer program, further performs the steps of: requesting to acquire a downloading task in a cluster message queue, and storing the acquired downloading task in a storage cluster center of the storage cluster; and distributing the downloading task to the corresponding downloading queue of the storage node according to the detection result of the storage cluster center on the available state information of each storage node.
In one embodiment, the available state information includes: survival state information, storage space information and download queue busy information; the processor, when executing the computer program, further performs the steps of: detecting survival state information, storage space information and downloading queue busy information of each storage node in the storage cluster according to the storage cluster center; if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list; and distributing the downloading task to a downloading queue of a storage node in the white list.
In one embodiment, the processor, when executing the computer program, further performs the steps of: receiving a client request sent by a storage node; detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node; and if so, distributing the downloading task to a downloading queue of the storage node.
In one embodiment, the available state information includes: a minimum usage store, the minimum usage store comprising: the download queue is minimum, the I/O read-write is minimum and the use of the storage space is minimum; the processor, when executing the computer program, further performs the steps of: calculating minimum use storage for each storage node in the white list; and distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
In one embodiment, the processor, when executing the computer program, further performs the steps of: according to the downloading task, sending a client downloading request to the cluster message queue; downloading the video file to be stored according to the client downloading request; performing file verification on the video file to be stored, and storing the video file to be stored passing the verification to a physical disk of a storage node; and detecting whether the physical disk of the storage node is abnormal or not, and if so, uploading abnormal information to the storage cluster center so that the storage cluster center distributes the downloading task to other storage nodes.
In one embodiment, the processor, when executing the computer program, further performs the steps of: recording the access frequency of the video files in the storage nodes; and if the access frequency is less than a threshold value, deleting the video file in the storage node.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
and downloading the video file to be stored to the storage node according to the downloading task in the downloading queue.
In one embodiment, the computer program when executed by the processor further performs the steps of: requesting to acquire a downloading task in a cluster message queue, and storing the acquired downloading task in a storage cluster center of the storage cluster; and distributing the downloading task to the corresponding downloading queue of the storage node according to the detection result of the storage cluster center on the available state information of each storage node.
In one embodiment, the available state information includes: survival state information, storage space information and download queue busy information; the computer program when executed by the processor further realizes the steps of: detecting survival state information, storage space information and downloading queue busy information of each storage node in the storage cluster according to the storage cluster center; if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list; and distributing the downloading task to a downloading queue of a storage node in the white list.
In one embodiment, the computer program when executed by the processor further performs the steps of: receiving a client request sent by a storage node; detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node; and if so, distributing the downloading task to a downloading queue of the storage node.
In one embodiment, the available state information includes: a minimum usage store, the minimum usage store comprising: the download queue is minimum, the I/O read-write is minimum and the use of the storage space is minimum; the computer program when executed by the processor further realizes the steps of: calculating minimum use storage for each storage node in the white list; and distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
In one embodiment, the computer program when executed by the processor further performs the steps of: according to the downloading task, sending a client downloading request to the cluster message queue; downloading the video file to be stored according to the client downloading request; performing file verification on the video file to be stored, and storing the video file to be stored passing the verification to a physical disk of a storage node; and detecting whether the physical disk of the storage node is abnormal or not, and if so, uploading abnormal information to the storage cluster center so that the storage cluster center distributes the downloading task to other storage nodes.
In one embodiment, the computer program when executed by the processor further performs the steps of: recording the access frequency of the video files in the storage nodes; and if the access frequency is less than a threshold value, deleting the video file in the storage node.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (7)
1. A video file storage injection method, the method comprising:
receiving an injection request of a video file to be stored, which is sent by a video source;
pushing the injection request to a cluster message queue of each storage cluster;
acquiring a downloading task from the cluster message queue, and distributing the downloading task to a downloading queue corresponding to the corresponding storage node according to the available state information of each storage node in the storage cluster;
downloading the video file to be stored to the storage node according to the downloading task in the downloading queue;
acquiring a download task from the cluster message queue, and distributing the download task to a corresponding download queue of the storage node according to available state information of each storage node in the storage cluster, including:
requesting to acquire a downloading task in a cluster message queue, and storing the acquired downloading task in a storage cluster center of the storage cluster;
distributing the downloading tasks to the downloading queues of the corresponding storage nodes according to the detection result of the storage cluster center on the available state information of each storage node;
the available state information includes: survival state information, storage space information and download queue busy information;
according to the detection result of the storage cluster center on the available state information of each storage node, distributing the download task to the corresponding download queue of the storage node, including:
detecting survival state information, storage space information and downloading queue busy information of each storage node in the storage cluster according to the storage cluster center;
if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list;
distributing the downloading task to a downloading queue of a storage node in the white list;
the allocating the download task to the download queue of the storage node in the white list includes:
receiving a client request sent by a storage node;
detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node;
and if so, distributing the downloading task to a downloading queue of the storage node.
2. The method of claim 1, wherein the available state information comprises: a minimum usage store, the minimum usage store comprising: the download queue is minimum, the I/O read-write is minimum and the use of the storage space is minimum;
the allocating the download task to the corresponding download queue corresponding to the storage node according to the available state information of each storage node in the storage cluster includes:
calculating minimum use storage for each storage node in the white list;
and distributing the downloading tasks in the storage cluster center to each storage node through a preset distributed algorithm according to the minimum use storage.
3. The method according to claim 1, wherein the downloading the video file to be stored to the storage node according to the download task in the download queue comprises:
according to the downloading task, sending a client downloading request to the cluster message queue;
downloading the video file to be stored according to the client downloading request;
performing file verification on the video file to be stored, and storing the video file to be stored passing the verification to a physical disk of a storage node;
and detecting whether the physical disk of the storage node is abnormal or not, and if so, uploading abnormal information to the storage cluster center so that the storage cluster center distributes the downloading task to other storage nodes.
4. The method of claim 1, further comprising:
recording the access frequency of the video files in the storage nodes;
and if the access frequency is less than a threshold value, deleting the video file in the storage node.
5. A video file storage injection apparatus, the apparatus comprising:
the request receiving module is used for receiving an injection request of a video file to be stored, which is sent by a video source;
the pushing module is used for pushing the injection request to a cluster message queue of each storage cluster;
the distribution module is used for acquiring the download tasks from the cluster message queue and distributing the download tasks to the download queues corresponding to the storage nodes according to the available state information of the storage nodes in the storage cluster;
the storage module is used for downloading the video file to be stored to the storage node according to the downloading task in the downloading queue;
the distribution module is also used for requesting to acquire the downloading tasks in the cluster message queue and storing the acquired downloading tasks in a storage cluster center of the storage cluster; distributing the downloading tasks to the downloading queues of the corresponding storage nodes according to the detection result of the storage cluster center on the available state information of each storage node;
the available state information includes: survival state information, storage space information and download queue busy information; the distribution module is also used for detecting survival state information, storage space information and downloading queue busy information of each storage node in the storage cluster according to the storage cluster center; if the survival state information, the storage space information and the downloading queue busy information all reach preset threshold conditions, adding the storage node into a preset white list; distributing the downloading task to a downloading queue of a storage node in the white list;
the distribution module is also used for receiving a client request sent by the storage node; detecting whether the storage node is in the white list or not according to the Internet protocol address corresponding to the storage node; and if so, distributing the downloading task to a downloading queue of the storage node.
6. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910669429.2A CN110267060B (en) | 2019-07-24 | 2019-07-24 | Video file storage injection method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910669429.2A CN110267060B (en) | 2019-07-24 | 2019-07-24 | Video file storage injection method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110267060A CN110267060A (en) | 2019-09-20 |
| CN110267060B true CN110267060B (en) | 2021-03-26 |
Family
ID=67927962
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910669429.2A Active CN110267060B (en) | 2019-07-24 | 2019-07-24 | Video file storage injection method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110267060B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127219A (en) * | 2019-12-31 | 2021-07-16 | 中移(苏州)软件技术有限公司 | Data storage method, data acquisition device and storage medium |
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN112035064A (en) * | 2020-08-28 | 2020-12-04 | 浪潮云信息技术股份公司 | Distributed migration method for object storage |
| CN114238240B (en) * | 2022-02-14 | 2022-05-27 | 柏科数据技术（深圳）股份有限公司 | Distributed multi-cluster data storage method and device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106951456A (en) * | 2017-02-24 | 2017-07-14 | 广东广信通信服务有限公司 | A kind of memory database system and data handling system |
| CN107968802A (en) * | 2016-10-19 | 2018-04-27 | 华为技术有限公司 | The method, apparatus and filtering type scheduler of a kind of scheduling of resource |
| CN108241469A (en) * | 2016-12-27 | 2018-07-03 | 北京金山云网络技术有限公司 | A kind of date storage method, apparatus and system |
| CN109597567A (en) * | 2017-09-30 | 2019-04-09 | 网宿科技股份有限公司 | A kind of data processing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107295080B (en) * | 2017-06-19 | 2020-12-18 | 北京百度网讯科技有限公司 | Data storage method applied to distributed server cluster and server |
-
2019
- 2019-07-24 CN CN201910669429.2A patent/CN110267060B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107968802A (en) * | 2016-10-19 | 2018-04-27 | 华为技术有限公司 | The method, apparatus and filtering type scheduler of a kind of scheduling of resource |
| CN108241469A (en) * | 2016-12-27 | 2018-07-03 | 北京金山云网络技术有限公司 | A kind of date storage method, apparatus and system |
| CN106951456A (en) * | 2017-02-24 | 2017-07-14 | 广东广信通信服务有限公司 | A kind of memory database system and data handling system |
| CN109597567A (en) * | 2017-09-30 | 2019-04-09 | 网宿科技股份有限公司 | A kind of data processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110267060A (en) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110267060B (en) | Video file storage injection method and device, computer equipment and storage medium | |
| US11727006B2 (en) | Systems and methods for secure file management via an aggregation of cloud storage services | |
| US10848560B2 (en) | Aggregation and management among a plurality of storage providers | |
| US10404798B2 (en) | Systems and methods for third-party policy-based file distribution in an aggregation of cloud storage services | |
| US10356158B2 (en) | Systems and methods for aggregation of cloud storage | |
| CN108667721B (en) | Subscription report generation method and device, computer equipment and storage medium | |
| US20170331892A1 (en) | Systems and methods for processing-based file distribution in an aggregation of cloud storage services | |
| US10445495B2 (en) | Buffer overflow exploit detection | |
| US8972465B1 (en) | Burst buffer appliance with small file aggregation | |
| JP6330528B2 (en) | Data division control program, data division control method, and data division control device | |
| CN112115167B (en) | Cache system hot spot data access method, device, equipment and storage medium | |
| US20200266971A1 (en) | Re-encrypting data on a hash chain | |
| US11593270B1 (en) | Fast distributed caching using erasure coded object parts | |
| CN109376033B (en) | Snapshot backup method and device, computer equipment and storage medium | |
| CN111222821A (en) | Goods supplementing method and device, computer equipment and storage medium | |
| CN113886496A (en) | Data synchronization method and device of block chain, computer equipment and storage medium | |
| CN108628551B (en) | Data processing method and device | |
| CN107846429B (en) | File backup method, device and system | |
| CN115935414A (en) | Block chain based data verification method and device, electronic equipment and storage medium | |
| Henziger et al. | Delta encoding overhead analysis of cloud storage systems using client-side encryption | |
| US10635334B1 (en) | Rule based data transfer model to cloud | |
| US10528432B2 (en) | Off-site backup network disk | |
| KR20210083780A (en) | Method and system for improving android app security by DEX file segmentation and obfuscation | |
| CN111191082A (en) | Data management method and device, computer equipment and storage medium | |
| CN110633207A (en) | Operation request processing method and system based on gray level test and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |