CN111708482B - Method, system, device and medium for limiting bandwidth in distributed storage - Google Patents
Method, system, device and medium for limiting bandwidth in distributed storage Download PDFInfo
- Publication number
- CN111708482B CN111708482B CN202010396240.3A CN202010396240A CN111708482B CN 111708482 B CN111708482 B CN 111708482B CN 202010396240 A CN202010396240 A CN 202010396240A CN 111708482 B CN111708482 B CN 111708482B
- Authority
- CN
- China
- Prior art keywords
- maximum capacity
- data
- capacity value
- bandwidth
- tokens
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000004044 response Effects 0.000 claims abstract description 24
- 239000012634 fragment Substances 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013467 fragmentation Methods 0.000 claims description 4
- 238000006062 fragmentation reaction Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 229920006395 saturated elastomer Polymers 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0634—Configuration or reconfiguration of storage systems by changing the state or mode of one or more devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for limiting bandwidth in distributed storage, wherein the method comprises the following steps: the method comprises the steps of fragmenting a data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as a maximum capacity value of a bandwidth token bucket; judging whether the data length value of each data sheet is larger than the current maximum capacity value or not in sequence every preset time; judging whether a token application of a data stream is received or not in response to the fact that the length value of the data sheet is not larger than the current maximum capacity value; responding to the received token application of the data stream, and judging whether the token quantity of the token application is greater than the maximum capacity value; and in response to the fact that the number of tokens of the token application is not larger than the maximum capacity value, subtracting the number of tokens required by the token application from the bandwidth token bucket, and issuing the data through the virtual disk.
Description
Technical Field
The present invention relates to the field of data processing, and more particularly, to a method, system, computer device and readable medium for limiting bandwidth in distributed storage.
Background
The storage system can set a bandwidth limit value for the virtual disk provided for the client, so as to improve the service quality of the storage. In performing bandwidth limiting, a token bucket algorithm is typically employed. When IO (input/output) data stream is transmitted, the token bucket is successfully applied for tokens, and then data can be issued to the bottom storage layer. In the traditional process of limiting the bandwidth according to the token bucket algorithm, if the data length of an IO at a certain time exceeds the maximum capacity of the token bucket, the IO at this time will cause infinite blocking because insufficient tokens are obtained.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method, a system, a computer device, and a computer readable storage medium for limiting bandwidth in distributed storage, in which data chunks are fragmented, the maximum capacity of the data chunks is reduced by drastic changes of the data chunks, and the maximum limit capacity of a bandwidth token bucket is dynamically adjusted according to the size of the data chunks and a bandwidth limit value, so as to reduce the occurrence of IO blocking problem on the basis of ensuring stable data transmission.
Based on the above object, an aspect of the embodiments of the present invention provides a method for limiting bandwidth in distributed storage, including the following steps: the method comprises the steps of fragmenting a data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as a maximum capacity value of a bandwidth token bucket; judging whether the data length value of each data sheet is larger than the current maximum capacity value or not in sequence every preset time; judging whether a token application of a data stream is received or not in response to the fact that the length value of the data sheet is not larger than the current maximum capacity value; responding to a received token application of a data stream, and judging whether the number of tokens of the token application is greater than the maximum capacity value; and in response to the fact that the number of tokens applied for the token is not larger than the maximum capacity value, subtracting the number of tokens required for the token application from the bandwidth token bucket, and issuing data through a virtual disk.
In some embodiments, further comprising: and in response to the fact that the data length value of the data piece is larger than the current maximum capacity value, updating the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judging whether a token application of a data stream is received or not.
In some embodiments, further comprising: and in response to the token number of the token application being larger than the maximum capacity value, pausing data output to wait for the maximum capacity value to be updated.
In some embodiments, further comprising: judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and in response to the fact that the number of tokens in the bandwidth token bucket is not the current maximum capacity value, adding tokens to the bandwidth token bucket according to a preset speed until the number of tokens in the bandwidth token bucket is equal to the current maximum capacity value.
In another aspect of the embodiments of the present invention, a system for limiting bandwidth in distributed storage is further provided, including: the fragmentation module is configured to fragment the data block to obtain a plurality of data fragments, and set a preset bandwidth limit value as a maximum capacity value of the bandwidth token bucket; the first judging module is configured to sequentially judge whether the data length value of each data sheet is greater than the current maximum capacity value every preset time; the second judgment module is configured to respond that the length value of the data sheet is not greater than the current maximum capacity value and judge whether a token application of the data stream is received; the third judging module is configured to respond to a received token application of a data stream and judge whether the number of tokens of the token application is greater than the maximum capacity value; and the execution module is configured to respond that the number of tokens applied for the token is not greater than the maximum capacity value, subtract the number of tokens required for the token application from the bandwidth token bucket, and issue data through a virtual disk.
In some embodiments, further comprising: and the updating module is configured to respond that the data length value of the data piece is larger than the current maximum capacity value, update the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judge whether a token application of a data stream is received.
In some embodiments, further comprising: and the suspending module is configured to respond to the condition that the number of tokens applied for the tokens is greater than the maximum capacity value, and suspend data output to wait for the maximum capacity value to be updated.
In some embodiments, the method further comprises adding the module configured to: judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and in response to the fact that the number of tokens in the bandwidth token bucket is not the current maximum capacity value, adding tokens to the bandwidth token bucket according to a preset speed until the number of tokens in the bandwidth token bucket is equal to the current maximum capacity value.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In a further aspect of the embodiments of the present invention, a computer-readable storage medium is also provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: by fragmenting the data blocks, reducing the drastic change of the maximum capacity of the data block fragments and dynamically adjusting the maximum limit capacity of the bandwidth token bucket according to the size of the data fragments and the bandwidth limit value, the IO blocking problem is reduced on the basis of ensuring the stability of data transmission.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a diagram illustrating an embodiment of a method for limiting bandwidth in distributed storage according to the present invention;
fig. 2 is a schematic hardware structure diagram of an embodiment of a computer device for limiting bandwidth in distributed storage according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a method for limiting bandwidth in distributed storage. Fig. 1 is a schematic diagram illustrating an embodiment of a method for limiting bandwidth in distributed storage according to the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, fragmenting the data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as the maximum capacity value of the bandwidth token bucket;
s2, sequentially judging whether the data length value of each data sheet is larger than the current maximum capacity value every preset time;
s3, responding to the fact that the data length value of the data sheet is not larger than the current maximum capacity value, and judging whether a token application of the data flow is received or not;
s4, responding to the received token application of the data stream, and judging whether the token number of the token application is larger than the maximum capacity value; and
and S5, in response to the fact that the number of tokens applied for the token is not larger than the maximum capacity value, subtracting the number of tokens required for the token application from the bandwidth token bucket, and issuing the data through the virtual disk.
And fragmenting the data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as the maximum capacity value of the bandwidth token bucket. The fragmentation of the data blocks may be performed according to a data protection policy. The data protection strategy comprises a copy redundancy strategy and an erasure code fault tolerance strategy, wherein the copy redundancy strategy is to copy original data into a plurality of copies, each copy is called as a copy, and the copies are respectively stored on different nodes in a cluster; the erasure code fault-tolerant strategy is to encode original data by using an erasure code algorithm to obtain redundancy, and store the data and the redundancy together. And dividing the data block into data slices with different sizes, performing redundancy operation on the data slices, and storing the data slices in different nodes respectively. The bandwidth limit may be set as desired, for example, the bandwidth limit may be set according to the size of the data pieces, the bandwidth limit may be set smaller if the data pieces are all smaller, and the bandwidth limit may be set larger if the data pieces are all larger.
And judging whether the data length value of each data sheet is larger than the current maximum capacity value or not in sequence every preset time. The bandwidth limit value is the most original maximum capacity value of the bandwidth token bucket, and whether the maximum capacity value of the bandwidth token bucket is updated or not can be determined by comparing the data length value of the data sheet with the size of the bandwidth limit value.
And judging whether a token application of the data stream is received or not in response to the fact that the data length value of the data piece is not larger than the current maximum capacity value. If the data length value of the data sheet is not larger than the current maximum capacity value, the current maximum capacity value can be reserved, and whether a token application from the data stream exists or not is judged in real time.
In some embodiments, further comprising: and in response to the fact that the data length value of the data piece is larger than the current maximum capacity value, updating the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judging whether a token application of a data stream is received or not. If the data length value of the data sheet is larger than the current maximum capacity value, the maximum capacity value needs to be updated to the data length value of the current data sheet, and meanwhile, whether a token application from a data stream exists or not is judged in real time.
And responding to the received token application of the data stream, and judging whether the token quantity of the token application is larger than the maximum capacity value. If a token application is received from the data stream, the number of tokens of the token application is compared to a maximum capacity value to determine whether the issuance of the data can be performed.
And in response to the fact that the number of tokens of the token application is not larger than the maximum capacity value, subtracting the number of tokens required by the token application from the bandwidth token bucket, and issuing the data through the virtual disk. If the number of tokens applied for the tokens is less than or equal to the maximum capacity value, the data can be issued, the number of tokens required for the token application can be subtracted from the bandwidth token bucket, and the data is issued through the virtual disk.
In some embodiments, further comprising: and in response to the number of tokens of the token application being greater than the maximum capacity value, pausing data output to wait for the maximum capacity value to update. If the number of tokens applied for the tokens is larger than the maximum capacity value, the data can not be issued, the updating of the maximum capacity value needs to be waited, and the data can be issued only when the maximum capacity value is larger than or equal to the number of tokens applied for the tokens.
In some embodiments, further comprising: judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and in response to the fact that the number of tokens in the bandwidth token bucket is not the current maximum capacity value, adding tokens to the bandwidth token bucket according to a preset speed until the number of tokens in the bandwidth token bucket is equal to the current maximum capacity value. And judging whether the number of tokens in the bandwidth token bucket is saturated in real time, if not, adding the tokens into the bandwidth token bucket at a preset speed until the bandwidth token bucket is saturated.
It should be particularly noted that, the steps in the embodiments of the method for limiting bandwidth in distributed storage described above can be mutually intersected, replaced, added, and deleted, so that these methods for limiting bandwidth in distributed storage, which are transformed by reasonable permutation and combination, should also belong to the scope of the present invention, and should not limit the scope of the present invention to the embodiments.
In view of the above object, a second aspect of the embodiments of the present invention provides a system for limiting bandwidth in distributed storage, including: the fragmentation module is configured to fragment the data block to obtain a plurality of data fragments, and set a preset bandwidth limit value as a maximum capacity value of the bandwidth token bucket; the first judging module is configured to sequentially judge whether the data length value of each data sheet is greater than the current maximum capacity value every preset time; the second judgment module is configured to respond that the length value of the data sheet is not greater than the current maximum capacity value and judge whether a token application of the data stream is received; the third judging module is configured to respond to a received token application of a data stream and judge whether the number of tokens of the token application is greater than the maximum capacity value; and the execution module is configured to respond that the number of tokens applied for the token is not greater than the maximum capacity value, subtract the number of tokens required for the token application from the bandwidth token bucket, and issue data through a virtual disk.
In some embodiments, further comprising: and the updating module is configured to respond that the data length value of the data piece is larger than the current maximum capacity value, update the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judge whether a token application of a data stream is received.
In some embodiments, further comprising: and the suspending module is configured to respond to the condition that the number of tokens applied for the tokens is greater than the maximum capacity value, and suspend data output to wait for the maximum capacity value to be updated.
In some embodiments, the method further comprises adding the module configured to: judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and in response to the fact that the number of tokens in the bandwidth token bucket is not the current maximum capacity value, adding tokens to the bandwidth token bucket according to a preset speed until the number of tokens in the bandwidth token bucket is equal to the current maximum capacity value.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, fragmenting the data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as the maximum capacity value of the bandwidth token bucket; s2, sequentially judging whether the data length value of each data sheet is larger than the current maximum capacity value every preset time; s3, responding to the fact that the data length value of the data sheet is not larger than the current maximum capacity value, and judging whether a token application of the data flow is received or not; s4, responding to the received token application of the data stream, and judging whether the token number of the token application is larger than the maximum capacity value; and S5, responding to the fact that the number of tokens of the token application is not larger than the maximum capacity value, subtracting the number of tokens required by the token application from the bandwidth token bucket, and issuing the data through the virtual disk.
In some embodiments, further comprising: and in response to the fact that the data length value of the data piece is larger than the current maximum capacity value, updating the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judging whether a token application of a data stream is received or not.
In some embodiments, further comprising: and in response to the number of tokens of the token application being greater than the maximum capacity value, pausing data output to wait for the maximum capacity value to update.
In some embodiments, further comprising: judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and in response to the fact that the number of tokens in the bandwidth token bucket is not the current maximum capacity value, adding tokens to the bandwidth token bucket according to a preset speed until the number of tokens in the bandwidth token bucket is equal to the current maximum capacity value.
Fig. 2 is a schematic hardware structural diagram of an embodiment of the computer device for limiting bandwidth in the distributed storage according to the present invention.
Taking the apparatus shown in fig. 2 as an example, the apparatus includes a processor 301 and a memory 302, and may further include: an input device 303 and an output device 304.
The processor 301, the memory 302, the input device 303 and the output device 304 may be connected by a bus or other means, and fig. 2 illustrates the connection by a bus as an example.
The memory 302, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for limiting bandwidth in distributed storage in the embodiments of the present application. The processor 301 executes various functional applications of the server and data processing, i.e. implementing the method of limiting bandwidth in distributed storage of the above-described method embodiments, by running non-volatile software programs, instructions and modules stored in the memory 302.
The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the method of limiting bandwidth in distributed storage, and the like. Further, the memory 302 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 302 optionally includes memory located remotely from processor 301, which may be connected to a local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 303 may receive information such as a user name and a password that are input. The output means 304 may comprise a display device such as a display screen.
Program instructions/modules corresponding to the method of limiting bandwidth in one or more distributed stores are stored in memory 302 and, when executed by processor 301, perform the method of limiting bandwidth in distributed stores in any of the method embodiments described above.
Any embodiment of a computer device that performs the method for limiting bandwidth in distributed storage described above may achieve the same or similar effects as any of the preceding method embodiments corresponding thereto.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the method as above.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for limiting bandwidth in distributed storage can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods as described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the above embodiments of the present invention are merely for description, and do not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (8)
1. A method for limiting bandwidth in distributed storage, comprising the steps of:
the method comprises the steps of fragmenting a data block to obtain a plurality of data fragments, and setting a preset bandwidth limit value as a maximum capacity value of a bandwidth token bucket;
judging whether the data length value of each data sheet is larger than the current maximum capacity value or not in sequence every preset time;
judging whether a token application of a data stream is received or not in response to the fact that the length value of the data sheet is not larger than the current maximum capacity value;
responding to a received token application of a data stream, and judging whether the number of tokens of the token application is larger than the maximum capacity value; and
in response to the fact that the number of tokens of the token application is not larger than the maximum capacity value, subtracting the number of tokens required by the token application from the bandwidth token bucket, and issuing data through a virtual disk;
the method further comprises the following steps: and in response to the fact that the data length value of the data piece is larger than the current maximum capacity value, updating the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judging whether a token application of a data stream is received or not.
2. The method of claim 1, further comprising:
and in response to the number of tokens of the token application being greater than the maximum capacity value, pausing data output to wait for the maximum capacity value to update.
3. The method of claim 1, further comprising:
judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and
and in response to the fact that the number of the tokens in the bandwidth token bucket is not the current maximum capacity value, adding the tokens to the bandwidth token bucket according to a preset speed until the number of the tokens in the bandwidth token bucket is equal to the current maximum capacity value.
4. A system for limiting bandwidth in distributed storage, comprising:
the fragmentation module is configured to fragment the data block to obtain a plurality of data fragments, and set a preset bandwidth limit value as a maximum capacity value of the bandwidth token bucket;
the first judging module is configured to sequentially judge whether the data length value of each data sheet is greater than the current maximum capacity value every preset time;
the second judgment module is configured to respond that the length value of the data sheet is not greater than the current maximum capacity value and judge whether a token application of the data stream is received;
the third judging module is configured to respond to a received token application of a data stream and judge whether the number of tokens of the token application is greater than the maximum capacity value; and
the execution module is configured to respond that the number of tokens applied for the token is not greater than the maximum capacity value, subtract the number of tokens required for the token application from the bandwidth token bucket, and issue data through a virtual disk;
the system further comprises: and the updating module is configured to respond that the data length value of the data piece is larger than the current maximum capacity value, update the maximum capacity value of the bandwidth token bucket to the data length value of the data piece and judge whether a token application of a data stream is received.
5. The system of claim 4, further comprising:
and the suspending module is configured to respond to the condition that the number of tokens applied for the tokens is greater than the maximum capacity value, and suspend data output to wait for the maximum capacity value to be updated.
6. The system of claim 4, further comprising an add module configured to:
judging whether the number of tokens in the bandwidth token bucket is the current maximum capacity value; and
and in response to the fact that the number of the tokens in the bandwidth token bucket is not the current maximum capacity value, adding the tokens to the bandwidth token bucket according to a preset speed until the number of the tokens in the bandwidth token bucket is equal to the current maximum capacity value.
7. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 3.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396240.3A CN111708482B (en) | 2020-05-12 | 2020-05-12 | Method, system, device and medium for limiting bandwidth in distributed storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396240.3A CN111708482B (en) | 2020-05-12 | 2020-05-12 | Method, system, device and medium for limiting bandwidth in distributed storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111708482A CN111708482A (en) | 2020-09-25 |
| CN111708482B true CN111708482B (en) | 2022-08-09 |
Family
ID=72537845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010396240.3A Active CN111708482B (en) | 2020-05-12 | 2020-05-12 | Method, system, device and medium for limiting bandwidth in distributed storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111708482B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112328613B (en) * | 2020-11-04 | 2022-07-22 | 迈普通信技术股份有限公司 | Online analysis processing method and device, electronic equipment and storage medium |
| CN113810306B (en) * | 2021-09-07 | 2024-07-26 | 山石网科通信技术股份有限公司 | Bandwidth allocation method, device, storage medium and processor |
| CN114143267B (en) * | 2021-11-26 | 2023-08-29 | 北京天融信网络安全技术有限公司 | Flow adjustment method, device, electronic equipment and computer readable storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804043A (en) * | 2018-06-26 | 2018-11-13 | 郑州云海信息技术有限公司 | Distributed block storage system bandwidth traffic control method, device, equipment and medium |
-
2020
- 2020-05-12 CN CN202010396240.3A patent/CN111708482B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804043A (en) * | 2018-06-26 | 2018-11-13 | 郑州云海信息技术有限公司 | Distributed block storage system bandwidth traffic control method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111708482A (en) | 2020-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111708482B (en) | Method, system, device and medium for limiting bandwidth in distributed storage | |
| CN111344706B (en) | Method and system for managing transactions on blockchain | |
| CN110620793B (en) | Method, device and medium for improving audio quality | |
| CN110519329B (en) | Method, device and readable medium for concurrently processing samba protocol request | |
| CN110597825B (en) | Data processing method and device based on block chain and node equipment | |
| CN111708484A (en) | Method, system, device and medium for controlling data brushing speed | |
| CN111708486B (en) | Method, system, equipment and medium for balanced optimization of main placement group | |
| US11210003B2 (en) | Method, device and computer program product for restoring data based on replacing child node identifiers with parent node identifier | |
| CN111240595A (en) | Method, system, equipment and medium for optimizing storage cache | |
| CN111104232A (en) | Method, device and medium for accelerating message writing of message queue | |
| CN112087530B (en) | Method, device, equipment and medium for uploading data to block chain system | |
| CN111488987B (en) | Method, system, equipment and medium for deep learning large model training | |
| CN110442601B (en) | Openstack mirror image data parallel acceleration method and device | |
| CN110989934B (en) | Block chain link point data storage method, block chain system and block chain node | |
| US11307790B2 (en) | Method, device, and computer program product for managing data placement | |
| CN111309264B (en) | Method, system, device and medium for making directory quota compatible with snapshot | |
| CN111314311A (en) | Method, system, equipment and medium for improving performance of switch | |
| CN111858538B (en) | Method, device, equipment and medium for configuring BeeGFS quota by cluster | |
| CN111090496B (en) | Virtual machine memory migration method and device based on automatic iteration time adjustment | |
| CN110764699B (en) | Online migration method and device for storage volume | |
| CN111130976B (en) | Method, equipment and medium for configuring virtual local area network of white box switch | |
| CN113419870A (en) | Volume reading and writing method and device | |
| CN111309255B (en) | Method, system, device and medium for converting volume type | |
| US10838660B2 (en) | Identifying and processing predefined dispersed storage network workflows | |
| CN111177066A (en) | Method, device and medium for improving efficiency of accessing off-chip memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |