CN108540382B - Network content storage system and routing method thereof - Google Patents
Network content storage system and routing method thereof Download PDFInfo
- Publication number
- CN108540382B CN108540382B CN201810168810.6A CN201810168810A CN108540382B CN 108540382 B CN108540382 B CN 108540382B CN 201810168810 A CN201810168810 A CN 201810168810A CN 108540382 B CN108540382 B CN 108540382B
- Authority
- CN
- China
- Prior art keywords
- storage
- content
- processing unit
- subunit
- subunits
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/14—Routing performance; Theoretical aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/74—Address processing for routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The network content storage system with separated control and storage comprises a storage subunit, a storage unit and a storage unit, wherein the storage subunit is mainly composed of common storage media and is used for storing network content; the storage processing unit is used for carrying out relevant processing on the network content; and the storage control unit is separated from the storage processing unit and is used for monitoring the working state of the storage processing unit and the storage state of the storage subunit, and configuring and managing the storage behavior of the storage processing unit according to the user requirement and the characteristics of the storage medium. The invention adopts the idea of separating control and storage, and the storage control unit monitors and manages the storage behavior of the storage processing unit so as to intelligently and uniformly store the network content into the corresponding storage sub-units. The storage control unit can dynamically collect the states of the storage processing unit and the storage subunit. According to the state information, a user can freely write storage algorithm software of the storage control unit, and the flexibility and the intelligence of the storage equipment are improved.
Description
Technical Field
The invention relates to the technical field of network content storage, in particular to a network content storage system and a routing method thereof.
Background
There is a great deal of data redundancy in the existing internet, and an efficient cache mechanism is urgently needed to reduce network redundancy. An intelligent collaborative network (SINET) adopts a design idea of 'separation of resources and positions', supports caching of data packets or data block granularity from an original design, obviously reduces network redundancy, and improves link utilization rate and user experience. SINET is no longer addressed with IP addresses, but instead is a uniquely determined new addressing space: SID (Service ID), each piece of network content has a unique SID. Meanwhile, the SINET introduces a cache concept, and adds a cache function into SINET basic network equipment, namely a network component, and the network component accesses in a cache system according to the SID of a network data packet, so that the network layer supports the way cache of the packet granularity. The SINET caching mechanism works in a network layer and is decoupled with an application layer, and the caching mechanism has the advantages of flexibility, high efficiency, independence from an application layer protocol and the like. The network component stores the data flowing through the network component according to the cache strategy, so that other users can conveniently acquire the data nearby, and the network efficiency is improved.
The storage system in the traditional network component generally directly adopts a high-speed storage medium such as a static or dynamic random access memory to store network contents so as to realize the linear speed store-and-forward, but the storage capacity of the system is limited, the price is high, and the large-scale deployment is inconvenient; another storage system in a network component employs distributed storage technology, and uses parallel processing of inexpensive storage media to implement wire-speed store-and-forward of the network component. Distributed storage technologies generally employ a storage management unit to distribute and manage the accessed content. Before accessing data, the client inquires the storage management unit about the position of the accessed data, determines the data position and then initiates access to the storage medium to which the data belongs. The storage management unit of the storage system is responsible for routing calculation and forwarding of the stored content, and the network content is routed and forwarded to different storage media through a unified routing algorithm such as Hash. The method enables the cheap storage media to be processed in parallel, improves the total bandwidth of the storage system and reduces the cost of the storage system. However, because the routing algorithm is over-bound to the forwarding function, its flexibility and scalability have drawbacks. As storage requirements continue to increase, such conventional techniques will not be able to efficiently handle large-scale data storage. For a large-scale storage system composed of a large number of content storage devices, the traditional storage device cannot intelligently and cooperatively sense the storage states of other devices because a control plane responsible for route calculation and forwarding responsible for route forwarding are bound on one hardware, so that intelligent cooperative storage of the large-scale network system cannot be realized.
Disclosure of Invention
The invention provides a network content storage system, which aims to improve the flexibility and intelligence of the storage system; the storage rate of the storage device is improved, and the production cost is reduced.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a network content storage system, which comprises a storage control unit, a storage processing unit and a storage subunit, wherein the storage control unit is used for controlling the storage processing unit to store network content; the storage control unit is in communication connection with the storage processing unit, and the storage processing unit is in communication connection with the storage subunit;
the storage control unit configures a routing method of the storage processing unit and monitors storage states of the storage processing unit and the storage subunit;
and the storage processing unit receives the storage content sent by the terminal and stores the storage content into the storage subunit according to the routing method.
Preferably, the storage control unit includes a first communication module, a control interface abstraction layer and a storage control application layer;
the first communication module is in communication connection with the storage processing unit, and the storage control application layer is in communication connection with the first communication module through the control interface abstraction layer;
and configuring a routing method of the storage processing unit through a storage control application layer, wherein the first communication module receives the storage content reported by the storage processing unit, the storage state of the storage subunit and sends the routing method.
Preferably, the storage processing unit comprises a fragment combination module, an index module, a cache module, a second communication module, a routing module and a pre-fetching module which are sequentially connected in a communication manner;
the fragmentation combination module stores the storage content fragments sent by the terminal to the storage subunit or aggregates the storage content of the storage subunit;
the index module provides global index information for the storage content of the storage subunit;
the cache module caches the hot spot storage content of the storage subunit;
the second communication module is respectively in communication connection with the storage control unit and the storage subunit, and receives the storage content sent by the terminal, the storage state reported by the storage subunit and the routing method sent by the storage control unit;
the routing module routes the storage content sent by the terminal to a corresponding storage subunit according to the routing method;
the prefetch module prefetches a data block associated with a currently accessed data block to the cache module.
Preferably, the number of the storage subunits is one or more, and the storage subunits are composed of storage media.
The invention also provides a routing method of the network content storage system, which comprises the following steps:
the storage processing unit reports the storage content sent by the terminal to the storage control unit;
the storage control unit judges whether the difference between the current reporting time and the last reporting time of the storage content is lower than a set threshold value or not;
if the value is larger than the set threshold value, updating the reporting time and discarding the stored content;
if the current write-in speed is smaller than a certain threshold value, arranging the storage subunits in ascending order according to the storage state information of the storage subunits reported by the storage processing unit;
selecting the storage subunits in the pre-set positions sorted in the ascending order, and sorting the storage subunits in the descending order according to the residual capacity of the storage subunits;
selecting the storage subunits in the preset positions sorted in the descending order, and sorting the storage subunits in the descending order according to the reading rate of the storage subunits;
and acquiring a storage subunit with the lowest reading rate, and issuing a route forwarding entry to write the storage content into the storage subunit with the lowest reading rate.
It can be seen from the above technical solutions provided by the present invention that, the present invention adopts the idea of separating control and storage, and separates the storage control unit from the storage processing unit, and different storage processing units can: the plurality of storage devices can form a storage group by connecting or disconnecting one or more same or different storage control units, and intelligent cooperative storage is realized. The storage processing unit is in software: a user can write specific storage control (algorithm) application on the storage control unit according to specific scenes, so that the storage algorithm is more flexible and convenient to realize. In addition, the invention utilizes the locality principle of network flow and the speed aggregation effect of the storage device to realize the SINET high-speed cache device with the cache speed far higher than that of the existing cache system, so as to improve the problems that the storage speed in the network component cannot catch up with the forwarding speed and the production cost of the network component is high, thereby obviously improving the network utilization rate and the user experience and promoting the deployment of the SINET in the range of the Internet.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a schematic structural diagram of a network content storage system according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a memory control unit according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a storage processing unit according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a memory sub-unit according to an embodiment of the present invention;
fig. 5 is a flowchart of a routing method configured by a storage control unit according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a network content storage system according to an embodiment of the present invention, in which a storage control unit controls a plurality of storage processing units.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the convenience of understanding the embodiments of the present invention, the following description will be further explained by taking several specific embodiments as examples in conjunction with the drawings, and the embodiments are not to be construed as limiting the embodiments of the present invention.
The basic idea of the invention is to separate the control of the content storage device from the storage processing unit, thereby improving the flexibility and intelligence of the storage device. The storage processing unit is responsible for basic storage behavior processing, such as table lookup, routing, distribution of storage contents to the storage subunits, monitoring of real-time states of the network, the storage processing unit, the storage subunits and the like, such as real-time storage throughput, storage residual capacity and the like, and reporting of monitoring results of the storage control unit. The storage control unit is in software, is separated from the storage processing unit, and establishes communication with the storage control unit through a control communication port (generally an Ethernet interface) of the storage processing unit. The storage control unit collects the storage states monitored by the storage processing unit, dynamically configures a storage forwarding table of the storage processing unit according to the state data, and automatically performs cooperative storage. The different storage processing units may: the plurality of storage devices can form a storage group by connecting or disconnecting one or more same or different storage control units, and intelligent cooperative storage is realized. A user can compile specific storage algorithm application software on the storage control unit according to the characteristics of an actual storage scene, flexibly and intelligently control storage behaviors, for example, storage forwarding tables of different storage processing units are set according to the characteristics of dynamic network flow so that flow data are efficiently and uniformly stored, and the like. The main design goals are listed as follows:
the functions of a general distributed storage system can be completed, including load balancing through a unified algorithm (such as a Hash algorithm), and load balancing according to the storage real-time flow and the storage equipment characteristics;
the production cost of network components in the intelligent cooperative network is reduced, and the storage rate is improved by utilizing the network flow locality principle and parallel processing of common storage equipment;
the expandability of the storage system is enhanced, and when the storage rate cannot keep up with the forwarding rate of the network component, one or more of a storage control unit, a storage processing unit or a storage subunit can be added in a hot-plug mode;
the intelligence and the flexibility of the storage system are improved, a user can dynamically and intelligently adjust a storage management algorithm according to the needs of the user and the change of an application scene, and the value of the storage equipment is maximized.
Example one
The embodiment provides a high-speed content storage device with separate control and storage, the general structure of which is shown in fig. 1 and mainly comprises three parts: the device comprises a storage control unit, a storage processing unit and a storage subunit.
In this embodiment, the storage control unit is separated from the storage processing unit and the storage subunit, and is used for monitoring the states of the storage processing unit and the storage subunit. The different storage processing units may: the plurality of storage devices can form a storage group by connecting or disconnecting one or more same or different storage control units, and intelligent cooperative storage is realized. The storage processing unit is in a software mode, and a user can write specific storage control (algorithm) application on the storage control unit according to specific scenes, so that the storage algorithm is more flexible and convenient to realize;
in this embodiment, the storage processing unit is responsible for routing the storage content to the storage subunits, distributing (or aggregating) the storage traffic, so that each storage subunit can store in parallel, thereby improving the total storage rate of the storage device; meanwhile, the storage processing unit collects the storage characteristics of the storage subunit, dynamically monitors the storage rate and the residual storage capacity of the storage subunit, and periodically or triggerably reports the collection result to the storage control unit, so that a basis is provided for the storage algorithm decision of the storage control unit.
In this embodiment, the storage subunit is responsible for storing, reading, updating, and deleting the content stored in the sub-module.
In this embodiment, the embodiment of the storage control unit is shown in fig. 2, which mainly includes three major parts:
a communication module: the Ethernet communication module is responsible for communicating with the storage processing unit and is used for receiving information such as storage content reported by the storage processing unit, storage rate of the storage subunit, residual capacity and the like; sending management and configuration information of the storage processing unit;
control interface abstraction layer: packaging and abstracting the bottom layer function, and providing uniform interface calling for upper storage policy application software (APP);
storage control Application (APP): the storage user can compile specific APP according to the collected storage information, the application scene and the own requirements, so that the storage scheme is more convenient, flexible and intelligent to realize.
In this embodiment, a specific implementation of the storage processing unit is shown in fig. 3, and it mainly includes six parts:
and (3) slicing and combining modules: the stored large storage content is fragmented (or combined), and the continuous small storage content fragments are dispersed to a plurality of storage subunits (or the contents in the plurality of storage subunits are aggregated), so that the plurality of storage subunits can be executed in parallel, and the access performance of the storage device is obviously improved;
an indexing module: global index information is provided for the content stored in each storage subunit. The index may only provide correspondence between the stored content and the storage device;
a cache module: caching the content fetched from the storage subunit, caching the hot data, and reducing the access to the storage subunit;
a communication module: the device is used for communicating with a storage control unit, reporting a storage state, receiving and storing routing configuration information and the like;
a routing module: routing the storage content to the corresponding storage sub-unit according to a flow table (or a default equalization algorithm, such as a hash algorithm) configured by the storage control unit;
a prefetch module: according to the locality principle of network flow, namely, a user in a network accesses a data block and then accesses the data block related to the data block immediately, the data block is prefetched to a cache module of a storage processing unit, and the bus bandwidth of a storage subunit is fully utilized.
The whole storage processing unit communicates with the network component through a PCIE 3.0 interface, which can provide a bandwidth of 128Gb/s at most. And the network sub-module communicates with the network sub-module through a 1Gb/s or 10Gb/s Ethernet port. The characteristic of loose coupling between the storage processing unit and the storage subunit improves the expandability of the system.
In this example, a specific implementation of the memory subunit is shown in fig. 4. The ARM is mainly used for processing an operation algorithm of contents stored in the solid state disk, such as a write-in operation algorithm, a read-out operation algorithm, an update operation algorithm and a delete operation algorithm, and managing a SATA 3.0 interface bus protocol, an ethernet protocol and a DDR memory. The ROM is used for storage of some code programs when offline. The solid state disk is used for storing network data contents.
In this embodiment, the content storing system flow is as follows: when a write request comes, the storage content is transmitted to the storage processing unit through a PCI E3.0 interface protocol, a fragmentation combination module of the storage processing unit divides the larger storage content into content blocks with specific sizes (or caches the smaller storage content in a cache module until the smaller storage content is combined into the content blocks with specific sizes), and a routing module disperses the content blocks into different storage subunits for storage through a 1Gb/s or 10Gb/s Ethernet port, so that the storage subunits store the content in parallel, and the storage bandwidth of the system is improved. The routing forwarding entry of the routing module stores the corresponding relation between the stored content and the number of the storage subunit, when new stored content arrives and the routing module does not have the corresponding entry of the content, the storage processing unit reports the storage control unit, the storage control unit issues a routing table through running control application, and the content is routed to the corresponding storage subunit. Meanwhile, the index module records the storage device number of the content for the reading request. And finally, the storage subunit stores the network content into the solid state disk under the control of the ARM, and stores the related index information into the memory DDR.
In this embodiment, the content reading system flow is as follows: when a reading request comes, obtaining a reading request content identification (SID) through a PCIE 3.0 communication interface; then searching a cache module through the SID, and if corresponding request content exists in the cache module, directly returning; if the cache module does not request the content, searching the index module and judging whether the content exists in a certain storage subunit; if the index module does not have the index, returning a communication interface search result; if the index of the content is found, the request data and the specific amount of related data are prefetched from the storage subunit to the cache module through the prefetching module, and the requested content is returned through the PCIE 3.0 interface.
In this embodiment, the storage control unit runs a storage Application (APP), which includes a network content storage algorithm, as shown in fig. 4, a network content routing method, or called a storage algorithm. In the system of the present invention, a user can write a storage application on a storage control unit as if writing a computer application. The user can deploy a new storage algorithm at any time at a high speed, and only the storage control unit API is called to write the storage application, so that the flexibility and the intelligence of the storage system are greatly improved;
the flow chart of the embodiment of the storage algorithm shown in fig. 4 describes the storage routing process of the content when the store-and-forward entry is not configured in the storage processing unit as follows:
the storage processing unit reports the content identification of the storage content to the storage control unit;
the storage algorithm on the storage control unit judges whether the difference between the current reporting time and the last reporting time of the content identification is lower than a certain threshold value;
if the content is larger than a certain threshold value, updating the content reporting time, and then discarding the content;
if the current writing speed is less than a certain threshold value, the storage sub-units are arranged in ascending order according to the current writing speed of the storage sub-units according to the storage sub-unit state information reported by the storage processing unit by the storage algorithm;
selecting the storage subunits in the first few bits of the sequence, and then arranging the storage modules in a descending order according to the residual capacity of the storage subunits;
selecting the storage subunits in the first bits of the sequence, and then arranging the storage modules in a descending order according to the reading speed of the storage subunits;
and obtaining the memory subunit with the lowest read rate, and issuing a route forwarding entry to write the memory content into the memory subunit with the lowest read rate.
In the present embodiment, the algorithm is a simple storage algorithm embodiment, but the advantages of the storage system can be fully illustrated. In this embodiment, a solid state disk is used as the storage medium. The service life of the solid state disk is limited, and the erasing times of the solid state disk are generally 10K-100K ten thousand times. Therefore, in the storage algorithm, the life limit of the solid state disk is considered, and in the step 2, a content reporting frequency filter is added. The system only stores the network contents with the request frequency within a certain range, and the erasing times of the solid state disk are reduced. In addition, the algorithm considers factors such as writing rate, residual capacity, reading rate and the like, so that the use efficiency of the storage subunit is maximized, and the factors such as wear leveling of the solid state disk and the like are also fully considered. Compared with a simple hash distributed storage algorithm, the system considers more factors and can be more intelligent.
In addition, the storage system of the invention adopts a structure with separated storage and control, and a plurality of storage processing units can be connected with the same controller (or controllers), thereby realizing the global cooperative storage of network contents by a plurality of storage systems. As shown in fig. 6, the basic control flow is as follows:
the client requests a certain content of the server, and the server returns the corresponding content to the client;
the content returned by the server is forwarded and stored to the client through the network component 1 and the network component 3 (or the network component 1 and the network component 2);
the storage control unit can store the content returned by the server in one or more network components of the network components according to the storage load of the storage system in the network components along the path and the like, so that the performance of a plurality of storage systems is optimized.
Example two
The embodiment provides a network content storage system, which comprises a storage control unit, a storage processing unit and a storage subunit; the storage control unit is in communication connection with the storage processing unit, and the storage processing unit is in communication connection with the storage subunit;
the storage control unit configures a routing method of the storage processing unit and monitors storage states of the storage processing unit and the storage subunit;
and the storage processing unit receives the storage content sent by the terminal and stores the storage content into the storage subunit according to the routing method.
In this embodiment, the storage control unit includes a first communication module, a control interface abstraction layer, and a storage control application layer;
the first communication module is in communication connection with the storage processing unit, and the storage control application layer is in communication connection with the first communication module through the control interface abstraction layer;
and configuring a routing method of the storage processing unit through a storage control application layer, wherein the first communication module receives the storage content reported by the storage processing unit, the storage state of the storage subunit and sends the routing method.
In this embodiment, the storage processing unit includes a fragment combination module, an index module, a cache module, a second communication module, a routing module, and a prefetch module, which are sequentially connected in a communication manner;
the fragmentation combination module stores the storage content fragments sent by the terminal to the storage subunit or aggregates the storage content of the storage subunit;
the index module provides global index information for the storage content of the storage subunit;
the cache module caches the hot spot storage content of the storage subunit;
the second communication module is respectively in communication connection with the storage control unit and the storage subunit, and receives the storage content sent by the terminal, the storage state reported by the storage subunit and the routing method sent by the storage control unit;
the routing module routes the storage content sent by the terminal to a corresponding storage subunit according to the routing method;
the prefetch module prefetches a data block associated with a currently accessed data block to the cache module.
In this embodiment, the number of the storage subunits is one or more, and the storage subunits are composed of storage media.
In summary, in the content storage system according to the embodiment of the present invention, one or more storage processing units can perform balanced, flexible and intelligent cooperative storage under the control of the storage control unit according to the characteristics of the storage sub-units, the network topology and the traffic characteristics. This system uses inexpensive storage media, but achieves the storage rate of expensive professional storage media; the system can improve the storage cooperation capacity of the storage equipment and reduce the production cost of the storage system; meanwhile, the flexibility and the intelligence of the storage system are improved.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (5)
1. A network content storage system is characterized by comprising a storage control unit, a storage processing unit and a storage subunit; the storage control unit is in communication connection with the storage processing unit, and the storage processing unit is in communication connection with the storage subunit;
the storage control unit configures a routing method of the storage processing unit and monitors storage states of the storage processing unit and the storage subunit;
the storage processing unit receives the storage content sent by the terminal and stores the storage content to the storage subunit according to the routing method;
the specific process of the routing method of the network content storage system comprises the following steps:
the storage processing unit reports the storage content sent by the terminal to the storage control unit;
the storage control unit judges whether the difference between the current reporting time and the last reporting time of the storage content is lower than a set threshold value or not;
if the value is larger than the set threshold value, updating the reporting time and discarding the stored content;
if the current write-in speed is smaller than a certain threshold value, arranging the storage subunits in ascending order according to the storage state information of the storage subunits reported by the storage processing unit;
selecting the storage subunits in the pre-set positions sorted in the ascending order, and sorting the storage subunits in the descending order according to the residual capacity of the storage subunits;
selecting the storage subunits in the preset positions sorted in the descending order, and sorting the storage subunits in the descending order according to the reading rate of the storage subunits;
and acquiring a storage subunit with the lowest reading rate, and issuing a route forwarding entry to write the storage content into the storage subunit with the lowest reading rate.
2. The network content storage system according to claim 1, wherein the storage control unit includes a first communication module, a control interface abstraction layer, and a storage control application layer;
the first communication module is in communication connection with the storage processing unit, and the storage control application layer is in communication connection with the first communication module through the control interface abstraction layer;
and configuring a routing method of the storage processing unit through a storage control application layer, wherein the first communication module receives the storage content reported by the storage processing unit, the storage state of the storage subunit and sends the routing method.
3. The network content storage system according to claim 1, wherein the storage processing unit comprises a fragment combination module, an index module, a cache module, a second communication module, a routing module and a pre-fetching module which are sequentially connected in a communication manner;
the fragmentation combination module stores the storage content fragments sent by the terminal to the storage subunit or aggregates the storage content of the storage subunit;
the index module provides global index information for the storage content of the storage subunit;
the cache module caches the hot spot storage content of the storage subunit;
the second communication module is respectively in communication connection with the storage control unit and the storage subunit, and receives the storage content sent by the terminal, the storage state reported by the storage subunit and the routing method sent by the storage control unit;
the routing module routes the storage content sent by the terminal to a corresponding storage subunit according to the routing method;
the prefetch module prefetches a data block associated with a currently accessed data block to the cache module.
4. The network content storage system according to claim 1, wherein the number of the storage sub-units is one or more, and the storage sub-units are composed of storage media.
5. A routing method of the network contents storage system according to any one of claims 1 to 4,
the storage processing unit reports the storage content sent by the terminal to the storage control unit;
the storage control unit judges whether the difference between the current reporting time and the last reporting time of the storage content is lower than a set threshold value or not;
if the value is larger than the set threshold value, updating the reporting time and discarding the stored content;
if the current write-in speed is smaller than a certain threshold value, arranging the storage subunits in ascending order according to the storage state information of the storage subunits reported by the storage processing unit;
selecting the storage subunits in the pre-set positions sorted in the ascending order, and sorting the storage subunits in the descending order according to the residual capacity of the storage subunits;
selecting the storage subunits in the preset positions sorted in the descending order, and sorting the storage subunits in the descending order according to the reading rate of the storage subunits;
and acquiring a storage subunit with the lowest reading rate, and issuing a route forwarding entry to write the storage content into the storage subunit with the lowest reading rate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810168810.6A CN108540382B (en) | 2018-02-28 | 2018-02-28 | Network content storage system and routing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810168810.6A CN108540382B (en) | 2018-02-28 | 2018-02-28 | Network content storage system and routing method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108540382A CN108540382A (en) | 2018-09-14 |
| CN108540382B true CN108540382B (en) | 2020-08-18 |
Family
ID=63486371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810168810.6A Active CN108540382B (en) | 2018-02-28 | 2018-02-28 | Network content storage system and routing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108540382B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103813336B (en) * | 2012-11-07 | 2017-08-18 | 华为技术有限公司 | WLAN transfer control method, equipment and system |
| US20150350154A1 (en) * | 2014-06-03 | 2015-12-03 | John Myla | Using Distributed Network Elements to Send Authoritative DNS Responses |
| CN104135539B (en) * | 2014-08-15 | 2018-03-16 | 华为技术有限公司 | Date storage method, SDN controllers and distributed network storage system |
| CN105516263B (en) * | 2015-11-28 | 2019-02-01 | 华为技术有限公司 | Data distributing method, device, calculate node and storage system in storage system |
| CN107147507A (en) * | 2016-03-01 | 2017-09-08 | 中卫大河云联网络技术有限公司 | The control plane framework and control method of a kind of software defined network |
| CN107682411A (en) * | 2017-09-14 | 2018-02-09 | 广州西麦科技股份有限公司 | A kind of extensive SDN controllers cluster and network system |
-
2018
- 2018-02-28 CN CN201810168810.6A patent/CN108540382B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108540382A (en) | 2018-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200278985A1 (en) | Content delivery network analytics management via edge stage collectors | |
| US10911530B2 (en) | Content delivery method, virtual server management method, cloud platform, and system | |
| EP2359536B1 (en) | Adaptive network content delivery system | |
| US8745262B2 (en) | Adaptive network content delivery system | |
| US8612668B2 (en) | Storage optimization system based on object size | |
| US7552235B2 (en) | Autonomic selection of a request routing policy based upon cache effectiveness | |
| US20160132541A1 (en) | Efficient implementations for mapreduce systems | |
| CN102118433A (en) | Multiple-tier distributed cluster system | |
| CN102439934A (en) | Method and system for managing multilevel caches of edge server in cdn | |
| CN102439580A (en) | Systems and methods for network acceleration and efficient indexing for caching file systems | |
| WO2003027856A1 (en) | Pooling and provisionig storage resources in a storage network | |
| US10326854B2 (en) | Method and apparatus for data caching in a communications network | |
| CN102137139A (en) | Method and device for selecting cache replacement strategy, proxy server and system | |
| US8402198B1 (en) | Mapping engine for a storage device | |
| CN107368608A (en) | The HDFS small documents buffer memory management methods of algorithm are replaced based on ARC | |
| WO2015155768A1 (en) | Improved method and system for caching of video files | |
| CN101729272B (en) | Method, system and device for content distribution, and media server | |
| US7962657B1 (en) | Selection of a data path based on operation type | |
| CN108540382B (en) | Network content storage system and routing method thereof | |
| CN109002260A (en) | A kind of data cached processing method and processing system | |
| Carofiglio et al. | Evaluating per-application storage management in content-centric networks | |
| CN114996023B (en) | Target cache device, processing device, network equipment and table item acquisition method | |
| KR101218574B1 (en) | Delivering contents by using storage of network | |
| KR20220078244A (en) | Method and edge server for managing cache file for content fragments caching | |
| US9069676B2 (en) | Mapping engine for a storage device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |