CN110519310B - Storage task allocation method and device - Google Patents
Storage task allocation method and device Download PDFInfo
- Publication number
- CN110519310B CN110519310B CN201810486211.9A CN201810486211A CN110519310B CN 110519310 B CN110519310 B CN 110519310B CN 201810486211 A CN201810486211 A CN 201810486211A CN 110519310 B CN110519310 B CN 110519310B
- Authority
- CN
- China
- Prior art keywords
- storage
- parameter
- capacity
- task
- coefficient
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention provides a storage task allocation method and device, and relates to the technical field of load balancing. The method and the device calculate the service pressure coefficient of each front-end device according to each group of service parameters by obtaining a plurality of groups of service parameters, then group a plurality of front-end devices according to the service pressure coefficients to generate a plurality of task distribution groups, then determine the storage capacity coefficients of a plurality of storage devices corresponding to each front-end device to generate a storage capacity coefficient group, and finally respectively distribute the storage tasks initiated by the front-end devices in each task distribution group to the storage devices according to the plurality of storage capacity coefficient groups; when the storage tasks are distributed, the service volume of the front-end equipment and the task condition of the rear-end storage equipment are fully considered, so that the problem of load imbalance caused by distributing the storage tasks of the front-end equipment with larger task volume to the same storage equipment is avoided, and the stability of the storage task distribution system is improved.
Description
Technical Field
The invention relates to the technical field of load balancing, in particular to a storage task allocation method and device.
Background
Along with the fact that road monitoring is applied more and more widely in the security protection field, the storage capacity of the bayonet pictures is larger and larger, and the storage mode is changed from the original centralized storage to the currently popular distributed storage, namely cloud storage. The effect of the load balancing implementation comprises two aspects: the used capacity of each storage node is balanced, and the traffic pressure of each node is balanced.
However, in the prior art, the load balancing for storage is to allocate services according to different back-end storage capacities through the processing capacities such as a back-end storage controller and a bandwidth, but there is no good method for the differentiated processing of the front-end interface camera. Therefore, lots of bayonets of multi-lane large pictures of busy road sections are distributed on some storage nodes, and the actual writing pressure of the bayonets distributed on some storage nodes is very low, so that the real load balance is not achieved.
Disclosure of Invention
In view of the above, the present invention provides a method and an apparatus for allocating storage tasks to solve the above problems.
In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:
in a first aspect, an embodiment of the present invention provides a storage task allocation method, where the storage task allocation method is applied in a task processing module communicatively connected to a plurality of front-end devices and a plurality of storage devices, and the storage task allocation method includes:
determining a current picture capacity parameter, a current gate lane parameter and a current flow parameter of each front-end device to obtain a plurality of groups of service parameters;
calculating a service pressure coefficient of each front-end device according to each group of service parameters;
grouping the front-end equipment according to the service pressure coefficient to generate a plurality of task distribution groups;
determining storage capacity coefficients of a plurality of storage devices corresponding to each front-end device to generate a storage capacity coefficient group;
and distributing the storage tasks initiated by the front-end equipment in each task distribution group to the storage equipment according to the plurality of storage capacity coefficient groups.
In a second aspect, an embodiment of the present invention further provides a storage task allocation apparatus, where the storage task allocation method is applied in a task processing module communicatively connected to a plurality of front-end devices and a plurality of storage devices, and the storage task allocation apparatus includes:
a service parameter determining unit, configured to determine a current picture capacity parameter, a current gate lane parameter, and a current traffic parameter of each piece of front-end equipment to obtain multiple sets of service parameters;
the calculating unit is used for calculating the service pressure coefficient of each front-end device according to each group of service parameters;
the grouping unit is used for grouping the front-end equipment according to the business pressure coefficient to generate a plurality of task distribution groups;
a storage capability coefficient determination unit configured to determine storage capability coefficients of a plurality of storage devices corresponding to each of the head-end devices to generate a storage capability coefficient group;
and the allocation unit is used for allocating the storage tasks initiated by the front-end equipment in each task allocation group to the storage equipment according to the plurality of storage capacity coefficient groups.
The method and the device for allocating the storage tasks, provided by the embodiment of the invention, are characterized in that a plurality of groups of service parameters are obtained by determining a current picture capacity parameter, a current bayonet lane parameter and a current flow parameter of each front-end device, a service pressure coefficient of each front-end device is calculated according to each group of service parameters, then a plurality of front-end devices are grouped according to the service pressure coefficients to generate a plurality of task allocation groups, then storage capacity coefficients of a plurality of storage devices corresponding to each front-end device are determined to generate a storage capacity coefficient group, and finally the storage tasks initiated by the front-end devices included in each task allocation group are allocated to the storage devices according to the plurality of storage capacity coefficient groups; when the storage tasks are distributed, the service volume of the front-end equipment and the task condition of the rear-end storage equipment are fully considered, so that the problem of load imbalance caused by distributing the storage tasks of the front-end equipment with larger task volume to the same storage equipment is avoided, and the stability of the storage task distribution system is improved.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 shows a functional block diagram of a storage task allocation system according to an embodiment of the present invention.
Fig. 2 shows a functional block diagram of a task processing module provided in an embodiment of the present invention.
Fig. 3 is a flowchart illustrating a storage task allocation method according to an embodiment of the present invention.
Fig. 4 shows a detailed flowchart of step S301 in fig. 3.
Fig. 5 shows a detailed flowchart of sub-step S3012 in fig. 4.
Fig. 6 shows a detailed flowchart of sub-step S3013 in fig. 4.
Fig. 7 shows a detailed flowchart of step S303 in fig. 3.
Fig. 8 shows a detailed flowchart of step S304 in fig. 3.
Fig. 9 shows a task allocation diagram of the storage task allocation system according to an embodiment of the present invention.
Fig. 10 is a functional block diagram of a storage task allocating apparatus according to an embodiment of the present invention.
Icon: 100-a storage task allocation system; 110-front-end equipment; 120-a storage device; 130-task processing module; 131-a parameter acquisition submodule; 132-a calculation submodule; 133-task allocation submodule; 134-a storage submodule; 200-storage task allocation means; 210-a service parameter determination unit; 220-a computing unit; 230-a grouping unit; 240-storage capability coefficient determination unit; 250-allocation unit.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
First embodiment
The embodiment of the invention provides a storage task allocation system 100, which is used for balanced allocation of a large number of storage tasks generated in a road monitoring process. Referring to fig. 1, a functional block diagram of a storage task allocation system 100 according to an embodiment of the present invention is shown. The storage task allocation system 100 includes a plurality of front-end devices 110, a task processing module 130, and a plurality of storage devices 120, wherein the task processing module 130 is communicatively connected to the plurality of front-end devices 110 and the plurality of storage devices 120.
The front-end devices 110 are used for collecting the images of the passing vehicles on the road.
The task processing module 130 is used for evenly distributing a plurality of front-end devices 110 to generate a large amount of storage tasks in the road monitoring process. Referring to fig. 2, a functional block diagram of the task processing module 130 according to an embodiment of the present invention is shown. The task processing module 130 includes a parameter obtaining sub-module 131, a calculating sub-module 132, a task allocating sub-module 133, and a storing sub-module 134.
The storage sub-module 134 may be configured to store software programs and units, such as program instructions/units corresponding to the storage task allocation apparatus 200 and the method in the embodiment of the present invention, and the parameter obtaining sub-module 131, the calculating sub-module 132, and the task allocation sub-module 133 execute various functional applications and data processing, such as the storage task allocation method provided in the embodiment of the present invention, by running the software programs and units of the storage task allocation apparatus 200 and the method stored in the memory.
In a preferred embodiment, the task processing module 130 can be integrated into an intelligent terminal, such as a server, a computer, etc.
In another preferred embodiment, the task processing module 130 can be distributed on the front-end device 110 and the storage device 120. Specifically, the parameter obtaining sub-module 131 is distributed in the front-end device 110, and the calculating sub-module 132, the task allocating sub-module 133, and the storage sub-module 134 are distributed in the storage device 120.
It should be noted that, when the task processing module 130 is distributed on the front-end device 110 and the storage devices 120, each storage device 120 is integrated with the computing sub-module 132, the task allocation sub-module 133, and the storage sub-module 134, and at this time, one of the plurality of storage devices 120 is selected as a master device to execute the storage task allocation method provided in the embodiment of the present invention.
The plurality of storage devices 120 are used for storing the pictures acquired by the plurality of front-end devices 110.
Second embodiment
The embodiment of the present invention provides a storage task allocation method, which is applied to the task processing module 130 provided in the first embodiment, and is used for balanced allocation of a large number of storage tasks generated in a road monitoring process. Please refer to fig. 3, which is a flowchart illustrating a storage task allocation method according to an embodiment of the present invention. The storage task allocation method comprises the following steps:
step S301: the current picture capacity parameter, the current checkpoint lane parameter, and the current traffic parameter of each front-end device 110 are determined to obtain multiple sets of service parameters.
It should be noted that the current picture size parameter can be used to estimate the size of the maximum picture to be sent in the next storage period of the front-end device 110. Specifically, the larger the current picture capacity parameter is, the larger the maximum picture that may be sent in the next storage period of the front-end device 110 is; the smaller the current picture capacity parameter is, the smaller the maximum picture that may be transmitted in the next storage period of the front-end device 110 is.
It should be noted that, since the front-end device 110 may also have a thumbnail, the size of the maximum picture sent by the front-end device 110 is selected as a basis for determining the current picture capacity parameter more accurately and scientifically.
The current bayonet lane parameters may be used to reflect that the front-end device 110 includes several lanes; generally, the larger the current checkpoint lane parameter is, the more lanes the front-end device 110 includes; the smaller the current bayonet lane parameters, the fewer the number of lanes the front-end device 110 includes. It should be noted that, the current gate lane parameters generally do not change, and are consistent with the preset initial gate lane parameters.
The current traffic parameter may be used to predict the number of possible pictures to send in the next storage period of the headend device 110. It can be understood that the larger the current flow parameter is, the larger the number of pictures that may be sent by the front-end device 110 in the next storage period is; conversely, the smaller the current traffic parameter, the smaller the number of pictures that may be sent by the front-end device 110 in the next storage period.
It should be noted that, when the task processing module 130 does not perform the process of allocating the storage task for the first time, the current picture capacity parameter may also be used to reflect the size of the maximum picture sent by the front-end device 110 in the last storage period; the current traffic parameter may also be used to reflect the number of pictures sent by the head-end 110 during the last storage period.
Please refer to fig. 4, which is a flowchart illustrating the step S301. Step S301 includes:
substep S3011: the picture capacity and the number of pictures in the last storage period of each front-end device 110 are obtained.
It should be noted that, when the task processing module 130 executes the flow of allocating the storage task for the first time, the picture capacity and the number of pictures in the last storage period of each front-end device 110 are both 0. At this time, the current picture capacity parameter, the current bayonet lane parameter, and the current flow parameter of each front-end device 110 are respectively a preset initial picture capacity parameter, a preset initial bayonet lane parameter, and a preset initial flow parameter.
When the task processing module 130 does not execute the process of allocating the storage task for the first time, the picture capacity and the number of pictures in the last storage period of each front-end device 110 are obtained to determine the current picture capacity parameter, the current checkpoint lane parameter, and the current traffic parameter of each front-end device 110.
Substep S3012: the preset initial picture capacity parameter is adjusted according to the picture capacity to determine the current picture capacity parameter of the front-end device 110.
Please refer to fig. 5, which is a flowchart illustrating the sub-step S3012. Sub-step S3012 includes:
substep S30121: and determining the capacity threshold range where the picture capacity is located.
It can be understood that, a user may preset a plurality of capacity thresholds according to needs, so as to divide the picture capacity into a plurality of capacity threshold ranges, and by comparing the picture capacity with the plurality of capacity thresholds, the capacity threshold range in which the picture capacity is located can be determined.
Substep S30122: a first multiplication factor is determined in accordance with a capacity threshold range.
In a preferred embodiment, the task processing module 130 presets a first capacity threshold and a second capacity threshold. When the picture capacity is smaller than or equal to a preset first capacity threshold, the first product coefficient is a first preset multiple; when the picture capacity is larger than a preset first capacity threshold and smaller than or equal to a preset second capacity threshold, the first product coefficient is a second preset multiple; and when the picture capacity is larger than a preset second capacity threshold, the first product coefficient is a third preset multiple.
It should be noted that the second capacity threshold is greater than the first capacity threshold, and the first preset multiple, the second preset multiple, and the third preset multiple sequentially increase.
It is to be understood that the task processing module 130 may also set more capacity thresholds than the first capacity threshold and the second capacity threshold and the first multiplication coefficient corresponding to each capacity threshold range. Generally, the larger the picture capacity, the larger its corresponding first product coefficient.
Substep S30123: and determining the current picture capacity parameter according to the first product coefficient and the preset initial picture capacity parameter.
Specifically, the current picture capacity parameter is a product of a first product coefficient and a preset initial picture capacity parameter.
When the picture capacity is smaller than or equal to a preset first capacity threshold, determining that the current picture capacity parameter is a first preset multiple of the initial picture capacity parameter; when the picture capacity is larger than a preset first capacity threshold and smaller than or equal to a preset second capacity threshold, determining that the current picture capacity parameter is a second preset multiple of the initial picture capacity parameter; and when the picture capacity is larger than a preset second capacity threshold, determining that the current picture capacity parameter is a third preset multiple of the initial picture capacity parameter.
Substep S3013: the preset initial flow parameters are adjusted according to the number of pictures to determine the current flow parameters of the front-end device 110.
Please refer to fig. 6, which is a flowchart illustrating the sub-step S3013. Sub-step S3013 includes:
substep S30131: and determining the flow threshold range in which the number of pictures is positioned.
It can be understood that, a user may set a plurality of flow thresholds in the task processing module 130 in advance as needed to divide the number of pictures into a plurality of flow threshold ranges, and the flow threshold range in which the number of pictures is located can be determined by comparing the number of pictures with the size of the plurality of flow thresholds.
Substep S30132: a second product coefficient is determined based on the flow threshold range.
In a preferred embodiment, the task processing module 130 sets a first flow threshold, a second flow threshold, and a third flow threshold in advance. When the number of pictures is equal to a preset first flow threshold, determining that the second product coefficient is a fourth preset multiple; when the number of pictures is greater than the preset first flow threshold and less than or equal to the preset second flow threshold, the second product coefficient is a fifth preset multiple; when the number of pictures is greater than the preset second flow threshold and less than or equal to the preset third flow threshold, the second product coefficient is a sixth preset multiple; and when the number of pictures is greater than a preset third flow threshold, the second product coefficient is a seventh preset multiple.
It should be noted that the first flow threshold, the second flow threshold, and the third flow threshold are sequentially increased, and the fourth preset multiple, the fifth preset multiple, the sixth preset multiple, and the seventh preset multiple are sequentially increased.
It is to be understood that the task processing module 130 may further set more flow thresholds than the first flow threshold, the second flow threshold, and the third flow threshold, and a second multiplication coefficient corresponding to each flow threshold range. Generally, the larger the number of pictures, the larger its corresponding second product coefficient.
In this embodiment, the first flow rate threshold is 0.
Substep S30133: and determining the current flow parameter according to the second product coefficient and the preset initial flow parameter.
Specifically, the current flow parameter is a product of the second product coefficient and a preset initial flow parameter.
When the number of the pictures is equal to the first flow threshold, determining that the current flow parameter is a fourth preset multiple of the initial flow parameter; when the number of pictures is greater than a preset first flow threshold and less than or equal to a preset second flow threshold, determining that the current flow parameter is a fifth preset multiple of the initial flow parameter; when the number of the pictures is larger than a preset second flow threshold and smaller than or equal to a preset third flow threshold, determining that the current flow parameter is a sixth preset multiple of the initial flow parameter; and when the number of the pictures is larger than a preset third flow threshold, determining that the current flow parameter is a seventh preset multiple of the initial flow parameter.
Substep S3014: and determining the current checkpoint lane parameters as preset initial checkpoint lane parameters.
Because the current gate lane parameters generally do not change, the current gate lane parameters are consistent with the preset initial gate lane parameters.
Step S302: the traffic pressure coefficient of each front-end device 110 is calculated according to each set of traffic parameters.
Specifically, the product of the current picture capacity parameter, the current bayonet lane parameter and the current traffic parameter is used as a service pressure coefficient.
It can be understood that a larger traffic pressure coefficient of the front-end device 110 indicates a larger traffic volume of the front-end device 110; a smaller traffic pressure coefficient of the front-end 110 indicates a smaller traffic volume of the front-end 110.
It should be noted that the traffic pressure coefficient not only reflects the task amount of each front-end device 110 in the previous storage period, but also estimates the task amount of each front-end device 110 in the upcoming storage period, so as to more reasonably balance the load.
In addition, since the current picture capacity parameter and the current traffic parameter are statistically recalculated in each storage period along with the real-time change of the service of the front-end device 110, the service pressure coefficient is automatically allocated in each storage period according to the actual service pressure of the service of the front-end device 110.
Step S303: the plurality of head-end devices 110 are grouped according to traffic pressure coefficients to generate a plurality of task assignment groups.
Please refer to fig. 7, which is a flowchart illustrating the step S303. The step S303 includes:
substep S3031: and determining the service pressure threshold range in which the service pressure coefficient is positioned.
Substep S3032: the plurality of front-end devices 110 are grouped according to a traffic pressure threshold range to generate a plurality of task assignment groups.
Specifically, the set of front-end devices 110 corresponding to the traffic pressure coefficient smaller than or equal to the preset first traffic pressure threshold is used as the first task allocation group.
And taking the set of the front-end equipment 110 corresponding to the service pressure coefficient which is greater than the preset first service pressure threshold value and less than or equal to the preset second service pressure threshold value as a second task allocation group.
And taking the set of the front-end equipment 110 corresponding to the service pressure coefficient greater than the preset second service pressure threshold value as a third task allocation group.
It should be noted that the task processing module 130 may further set more traffic pressure thresholds than the first traffic pressure threshold and the second traffic pressure threshold, so as to divide the traffic pressure coefficient into more traffic pressure threshold ranges.
In addition, since the traffic pressure coefficient is changed once per storage period, the front-end devices 110 included in each divided task allocation group may be different, but it should be noted that the traffic pressure coefficients of the plurality of front-end devices 110 belonging to the same task allocation group are similar, that is, the traffic volumes of the plurality of front-end devices 110 of the group are similar.
Step S304: the storage capacity coefficients of the plurality of storage devices 120 corresponding to each front-end device 110 are determined to generate a storage capacity coefficient group.
As can be appreciated, the set of storage capability coefficients includes a plurality of storage capability coefficients, each of which is used to characterize a storage device 120 stress to storage tasks of the same head-end device 110.
The storage capability coefficient set may represent the stress on each storage device 120 for the storage task of the same front-end device 110.
Please refer to fig. 8, which is a flowchart illustrating the step S304. The step S304 includes:
substep S3041: the number of tasks per storage device 120 and the number of uses per head end 110 are obtained.
It should be understood that the task number refers to the number of front-end devices 110 in the task allocation group where a front-end device 110 is allocated to the storage node at the current time; the number of uses refers to the number of uses of the storage device 120 recorded by the head-end device 110 at the current time.
Substep S3042: the storage capacity coefficient of each storage device 120 is calculated in accordance with the number of tasks and the number of uses to generate a storage capacity coefficient group.
Specifically, the storage capability coefficient can be calculated by calculating the formula Val ═ task _ num × p + used _ num × q.
Wherein, Val is a storage capacity coefficient, task _ num is a task number, used _ num is a use frequency, p is a preset first scale coefficient, and q is a preset second scale coefficient.
As can be appreciated, the storage capability factor can be used to characterize the stress of a storage task assigned by a head end 110 to a storage device 120. The smaller the storage capability factor of a storage device 120, the less stress the front-end device 110 is assigned to the storage device 120 for the storage task.
For example, the storage task allocation system 100 includes 500 front-end devices 110 and 6 storage devices 120, and divides the 500 front-end devices 110 into a first task allocation group and a second task allocation group according to the traffic pressure coefficients of the 5 front-end devices 110; the first task allocation group includes 300 front-end devices 110, one of which is the first front-end device, the second task allocation group includes 200 front-end devices 110, and the 6 storage devices 120 include the first storage device, the second storage device, and the sixth storage device. When the first front-end device initiates a storage task request, it is necessary to determine the number of the first storage devices allocated to 300 front-end devices 110 in the first task allocation group at the current time, and assuming that 50 front-end devices 110 in the first task allocation group all allocate storage tasks to the first storage devices, the number of the tasks is 50; meanwhile, it is also required to determine the record of the number of times that the first front-end device uses the first storage device, and if the record is 3, the number of times of use is 3, and then the storage capacity coefficient Val of the first storage device corresponding to the first front-end device is 50 × p +3 × q; and similarly, calculating the storage capacity coefficients of the second storage device to the sixth storage device corresponding to the first front-end device, so as to form a storage capacity coefficient group corresponding to the first front-end device.
Step S305: and respectively allocating the storage tasks initiated by the front-end equipment 110 contained in each task allocation group to the storage equipment 120 according to the plurality of storage capacity coefficient groups.
Specifically, the task processing module 130 selects the storage device 120 to allocate the storage task in a manner that the smaller storage capability coefficient takes precedence (as shown in fig. 9). In addition, there are two cases where multiple storage devices 120 may be repeatedly selected and multiple storage devices 120 polled for selection, as described below.
For example, the task processing module 130 is communicatively connected to the first front end device, the second front end device, … … through the nth front end device, and the task processing module 130 is further communicatively connected to the first storage device, the second storage device, … … through the mth storage device. The task processing module 130 divides the front-end devices 110 into a first task allocation group, a second task allocation group, and a third task allocation group according to the traffic pressure coefficient of each front-end device 110. The first task allocation group includes a plurality of front-end devices 110 with similar traffic volumes, the second task allocation group includes a plurality of front-end devices 110 with similar traffic volumes, and the third task allocation group includes a plurality of front-end devices 110 with similar traffic volumes.
First, multiple storage devices 120 may be repeatedly selected. Assuming that the first front-end device belongs to the first task allocation group, when the first front-end device issues a storage task request to the task processing module 130, the task processing module 130 obtains a record of the number of times that the first front-end device has used each storage device 120, obtains the number of front-end devices 110 included in the first task allocation group allocated to each storage device 120 at the current time, and respectively calculates a storage capacity coefficient of each storage device 120 relative to the first front-end device, and after the calculation is completed, if the storage capacity coefficient of the first storage device is the minimum, the task processing module 130 allocates the storage task to the first storage device, and in a subsequent storage cycle, allocates the storage task initiated by the first front-end device to the first storage device; when the second storage period is reached, the task processing module 130 recalculates the storage capacity coefficient of each storage device 120 relative to the first front-end device, and if the storage capacity coefficient of the second storage device is the minimum at this time, the task processing module 130 allocates the storage task to the second storage device, and in the next storage period, allocates the storage task to the second storage device; by analogy, each time the storage period comes, the task processing module 130 recalculates the storage capacity coefficient of each storage device 120 relative to the first front-end device to select the storage device 120 with the smallest storage capacity coefficient to store the photo taken by the first front-end device.
Second, multiple storage device 120 polls are selected. Assuming that the first front-end device belongs to the first task allocation group, when the first front-end device issues a storage task request to the task processing module 130, the task processing module 130 obtains a record of the number of times that the first front-end device has used each storage device 120, obtains the number of front-end devices 110 included in the first task allocation group allocated to each storage device 120 at the current time, and respectively calculates a storage capacity coefficient of each storage device 120 relative to the first front-end device, and after the calculation is completed, if the storage capacity coefficient of the first storage device is the minimum, the task processing module 130 allocates the storage task to the first storage device, and in a subsequent storage cycle, allocates the storage task initiated by the first front-end device to the first storage device; when the second storage period is reached, the task processing module 130 recalculates the storage capacity coefficients of all the other storage devices 120 except the first storage device relative to the first front-end device, and if the storage capacity coefficient of the second storage device is the minimum at this time, the task processing module 130 allocates the storage tasks to the second storage device, and allocates the storage tasks to the second storage device in the next storage period; by analogy, each time the storage cycle arrives, the task processing module 130 recalculates the storage capacity coefficients of all other storage devices 120 that are not selected relative to the first front-end device, so as to select the storage device 120 with the smallest storage capacity coefficient to store the photo taken by the first front-end device.
It is understood that the plurality of front-end devices 110 are divided into a plurality of task allocation groups according to the traffic pressure coefficient, and the storage tasks are allocated based on the principle of capacity balance with each task allocation group as a unit. The problem of load imbalance caused by distributing the storage tasks of the front-end equipment 110 with larger traffic to the same storage equipment 120 and distributing the storage tasks of the front-end equipment 110 with smaller traffic to the same storage equipment 120 due to larger traffic difference of different front-end equipment 110 is avoided.
Second embodiment
Referring to fig. 10, fig. 10 is a functional block diagram of a storage task allocation apparatus 200 according to a preferred embodiment of the present invention. It should be noted that the basic principle and the generated technical effect of the storage task allocating device 200 provided in the present embodiment are the same as those of the above embodiments, and for the sake of brief description, no part of the present embodiment is mentioned, and corresponding contents in the above embodiments may be referred to. The storage task assigning apparatus 200 includes a service parameter determining unit 210, a calculating unit 220, a grouping unit 230, a storage capability coefficient determining unit 240, and an assigning unit 250.
The service parameter determining unit 210 is configured to determine a current picture capacity parameter, a current checkpoint lane parameter, and a current traffic parameter of each front-end device 110 to obtain multiple sets of service parameters.
Specifically, the service parameter determining unit 210 is configured to obtain a picture capacity and a picture number in a last storage period of each front-end device 110, and adjust a preset initial picture capacity parameter according to the picture capacity to determine a current picture capacity parameter of the front-end device 110, so as to adjust a preset initial flow parameter according to the picture number to determine a current flow parameter of the front-end device 110 and to determine that the current bayonet lane parameter is a preset initial bayonet lane parameter.
In a preferred embodiment, the service parameter determining unit 210 is configured to determine a capacity threshold range in which the picture capacity is located, determine a first product coefficient according to the capacity threshold range, and then determine a current picture capacity parameter according to the first product coefficient and a preset initial picture capacity parameter.
The service parameter determining unit 210 is further configured to determine a traffic threshold range in which the number of pictures is located, then determine a second multiplication coefficient according to the traffic threshold range, and determine a current traffic parameter according to the second multiplication coefficient and a preset initial traffic parameter.
It is to be understood that, in a preferred embodiment, the service parameter determination unit 210 is operable to execute step S301, sub-step S3011, sub-step S3012, sub-step S3013, sub-step S3014, sub-step S30121, sub-step S30122, sub-step S30123, sub-step S30131, sub-step S30132, and sub-step S30133.
The calculating unit 220 is configured to calculate a traffic pressure coefficient of each front-end device 110 according to each group of traffic parameters.
Specifically, the product of the current picture capacity parameter, the current bayonet lane parameter and the current traffic parameter is used as a service pressure coefficient.
It can be understood that a larger traffic pressure coefficient of the front-end device 110 indicates a larger traffic volume of the front-end device 110; a smaller traffic pressure coefficient of the front-end 110 indicates a smaller traffic volume of the front-end 110.
It should be noted that the traffic pressure coefficient not only reflects the task amount of each front-end device 110 in the previous storage period, but also estimates the task amount of each front-end device 110 in the upcoming storage period, so as to more reasonably balance the load.
It is to be understood that, in a preferred embodiment, the computing unit 220 is operable to perform step S302.
The grouping unit 230 is configured to group the plurality of front-end devices 110 according to the traffic pressure coefficients to generate a plurality of task allocation groups.
Specifically, the grouping unit 230 is configured to determine a traffic pressure threshold range in which the traffic pressure coefficient is located, and group the plurality of front-end devices 110 according to the traffic pressure threshold range to generate a plurality of task allocation groups.
It is to be understood that in a preferred embodiment, the grouping unit 230 is operable to perform step S303, sub-step S3031 and sub-step S3032.
The storage capability coefficient determination unit 240 is configured to determine storage capability coefficients of the plurality of storage devices 120 corresponding to each front-end device 110 to generate a storage capability coefficient group.
Specifically, the storage capability coefficient determining unit 240 is configured to obtain the number of tasks of each storage device 120 and the number of times of use of each front-end device 110, and calculate the storage capability coefficient of each storage device 120 according to the number of tasks and the number of times of use.
It is to be understood that, in a preferred embodiment, the storage capability coefficient determining unit 240 may be configured to perform the step S304, the sub-step S3041 and the sub-step S3042.
The allocating unit 250 is configured to allocate the storage tasks initiated by the front-end devices 110 included in each task allocation group to the storage devices 120 according to the storage capability coefficients.
It is to be understood that in a preferred embodiment, the allocation unit 250 is operable to perform step S305.
To sum up, in the storage task allocation method and apparatus provided in the embodiments of the present invention, a plurality of sets of service parameters are obtained by determining a current picture capacity parameter, a current bayonet lane parameter, and a current flow parameter of each front-end device, a service pressure coefficient of each front-end device is calculated according to each set of service parameters, then a plurality of front-end devices are grouped according to the service pressure coefficients to generate a plurality of task allocation groups, then storage capability coefficients of a plurality of storage devices corresponding to each front-end device are determined to generate a storage capability coefficient group, and finally, a storage task initiated by the front-end device included in each task allocation group is allocated to the storage device according to the plurality of storage capability coefficient groups; when the storage tasks are distributed, the service volume of the front-end equipment and the task condition of the rear-end storage equipment are fully considered, so that the problem of load imbalance caused by distributing the storage tasks of the front-end equipment with larger task volume to the same storage equipment is avoided, and the stability of the storage task distribution system is improved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A storage task allocation method is applied to a task processing module which is in communication connection with a plurality of front-end devices and a plurality of storage devices, and comprises the following steps:
determining a current picture capacity parameter, a current gate lane parameter and a current flow parameter of each front-end device to obtain a plurality of groups of service parameters;
calculating a service pressure coefficient of each front-end device according to each group of service parameters;
grouping the front-end equipment according to the service pressure coefficient to generate a plurality of task distribution groups;
determining storage capacity coefficients of a plurality of storage devices corresponding to each front-end device to generate a storage capacity coefficient group;
and distributing the storage tasks initiated by the front-end equipment in each task distribution group to the storage equipment according to the plurality of storage capacity coefficient groups.
2. The storage task allocation method according to claim 1, wherein the step of determining a current picture capacity parameter, a current checkpoint lane parameter, and a current traffic parameter of each of the front-end devices to obtain a plurality of sets of service parameters comprises:
acquiring the picture capacity and the picture quantity in a storage period on each front-end device;
adjusting preset initial picture capacity parameters according to the picture capacity to determine current picture capacity parameters of the front-end equipment;
adjusting preset initial flow parameters according to the number of the pictures to determine current flow parameters of the front-end equipment;
and determining the current gate lane parameter as a preset initial gate lane parameter.
3. The storage task allocation method according to claim 2, wherein the step of adjusting the preset initial picture capacity parameter according to the picture capacity to determine the current picture capacity parameter of the front-end device comprises:
determining a capacity threshold range in which the picture capacity is located;
determining a first multiplication coefficient according to the capacity threshold range;
and determining the current picture capacity parameter according to the first product coefficient and a preset initial picture capacity parameter.
4. The storage task allocation method according to claim 2, wherein the step of adjusting the preset initial flow parameter according to the number of pictures to determine the current flow parameter of the front-end device comprises:
determining a flow threshold range in which the number of pictures is located;
determining a second product coefficient according to the flow threshold range;
and determining the current flow parameter according to the second product coefficient and a preset initial flow parameter.
5. The storage task allocation method according to claim 1, wherein the step of grouping the plurality of front-end devices according to the traffic pressure coefficients to generate a plurality of task allocation groups comprises:
determining a service pressure threshold range in which the service pressure coefficient is positioned;
and grouping the plurality of front-end equipment according to the service pressure threshold value range to generate a plurality of task allocation groups.
6. The storage task allocation method according to claim 1, wherein the step of determining storage capacity coefficients of a plurality of storage devices corresponding to each front-end device to generate a storage capacity coefficient group comprises:
acquiring the task number of each storage device and the number of usage times of each front-end device, wherein the task number refers to the number of the front-end devices in a task allocation group where each front-end device is located, the number of the front-end devices being allocated to the storage device, and the usage times refers to the number of usage times of the storage device by each front-end device;
and calculating the storage capacity coefficient of each storage device according to the task number and the use number to generate a storage capacity coefficient group.
7. The storage task allocation method according to claim 6, wherein the step of calculating the storage capacity coefficient of each storage device according to the task number and the usage number comprises:
and calculating the storage capacity coefficient by the formula Val ═ task _ num × p + used _ num × q, wherein Val is the storage capacity coefficient, task _ num is the task number, used _ num is the use frequency, p is a preset first scale coefficient, and q is a preset second scale coefficient.
8. The storage task allocation method according to claim 1, wherein the step of calculating a traffic pressure coefficient of each of the front-end devices according to each set of the traffic parameters comprises:
and taking the product of the current picture capacity parameter, the current bayonet lane parameter and the current flow parameter as the service pressure coefficient.
9. A storage task allocation apparatus, wherein the storage task allocation apparatus is applied to a task processing module communicatively connected to a plurality of front-end devices and a plurality of storage devices, and the storage task allocation apparatus comprises:
a service parameter determining unit, configured to determine a current picture capacity parameter, a current gate lane parameter, and a current traffic parameter of each piece of front-end equipment to obtain multiple sets of service parameters;
the calculating unit is used for calculating the service pressure coefficient of each front-end device according to each group of service parameters;
the grouping unit is used for grouping the front-end equipment according to the business pressure coefficient to generate a plurality of task distribution groups;
a storage capability coefficient determination unit configured to determine storage capability coefficients of a plurality of storage devices corresponding to each of the head-end devices to generate a storage capability coefficient group;
and the allocation unit is used for allocating the storage tasks initiated by the front-end equipment in each task allocation group to the storage equipment according to the plurality of storage capacity coefficient groups.
10. The storage task allocation apparatus according to claim 9, wherein the service parameter determining unit is configured to obtain a picture capacity and a picture number in a storage period on each of the front-end devices;
the service parameter determining unit is further configured to adjust a preset initial picture capacity parameter according to the picture capacity to determine a current picture capacity parameter of the front-end device;
the service parameter determining unit is further configured to adjust a preset initial flow parameter according to the number of pictures to determine a current flow parameter of the front-end device;
the service parameter determining unit is further configured to determine that the current gate lane parameter is a preset initial gate lane parameter.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810486211.9A CN110519310B (en) | 2018-05-21 | 2018-05-21 | Storage task allocation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810486211.9A CN110519310B (en) | 2018-05-21 | 2018-05-21 | Storage task allocation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110519310A CN110519310A (en) | 2019-11-29 |
| CN110519310B true CN110519310B (en) | 2021-12-28 |
Family
ID=68621425
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810486211.9A Active CN110519310B (en) | 2018-05-21 | 2018-05-21 | Storage task allocation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110519310B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1288194A (en) * | 1999-06-18 | 2001-03-21 | 凤凰技术有限公司 | Device and method for updating image in nonvolatile memory |
| US7376743B1 (en) * | 2002-04-02 | 2008-05-20 | Cisco Technology, Inc. | Method and apparatus for load balancing in a virtual private network |
| CN103155482A (en) * | 2010-10-01 | 2013-06-12 | 皇家飞利浦电子股份有限公司 | Device and method for load balancing for data packet transmissions in wireless networks |
| CN106383668A (en) * | 2016-09-18 | 2017-02-08 | 浙江宇视科技有限公司 | Information storage method, storage management device and client |
| CN106547481A (en) * | 2016-09-29 | 2017-03-29 | 浙江宇视科技有限公司 | A kind of data method for pre-distributing and equipment |
| CN107169083A (en) * | 2017-05-11 | 2017-09-15 | 聚龙融创科技有限公司 | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140258062A1 (en) * | 2013-03-08 | 2014-09-11 | Bank Of America Corporation | Load balancing of financial institution channels |
-
2018
- 2018-05-21 CN CN201810486211.9A patent/CN110519310B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1288194A (en) * | 1999-06-18 | 2001-03-21 | 凤凰技术有限公司 | Device and method for updating image in nonvolatile memory |
| US7376743B1 (en) * | 2002-04-02 | 2008-05-20 | Cisco Technology, Inc. | Method and apparatus for load balancing in a virtual private network |
| CN103155482A (en) * | 2010-10-01 | 2013-06-12 | 皇家飞利浦电子股份有限公司 | Device and method for load balancing for data packet transmissions in wireless networks |
| CN106383668A (en) * | 2016-09-18 | 2017-02-08 | 浙江宇视科技有限公司 | Information storage method, storage management device and client |
| CN106547481A (en) * | 2016-09-29 | 2017-03-29 | 浙江宇视科技有限公司 | A kind of data method for pre-distributing and equipment |
| CN107169083A (en) * | 2017-05-11 | 2017-09-15 | 聚龙融创科技有限公司 | Public security bayonet socket magnanimity vehicle data storage and retrieval method and device, electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| "浅谈平安城市建设-视频存储技术";崔国进;《电子世界》;20140329;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110519310A (en) | 2019-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106375115B (en) | Resource allocation methods and device | |
| CN110300134B (en) | Storage space adjusting method and device of cloud storage resource pool and cloud storage system | |
| CN108924221B (en) | Method and device for allocating resources | |
| CN109194984B (en) | Video frame scheduling method based on edge calculation | |
| US20090055835A1 (en) | System and Method for Managing License Capacity in a Telecommunication Network | |
| CN109962856B (en) | Resource allocation method, device and computer readable storage medium | |
| CN110232537A (en) | Distribute method, apparatus, equipment and the storage medium of data object | |
| CN110196767A (en) | Service resource control method, device, equipment and storage medium | |
| CN106161485B (en) | A kind of resource regulating method of infrastructure service cluster, device and system | |
| WO2017147331A1 (en) | User behavior-based dynamic resource adjustment | |
| CN109426896A (en) | A kind of method and device of determining region transport power state, electronic equipment | |
| CN106101276B (en) | A kind of cluster load balancing method and device | |
| CN108023834A (en) | A kind of cloud resource auto-allocation method and device | |
| CN110377415A (en) | A kind of request processing method and server | |
| CN110109800A (en) | A kind of management method and device of server cluster system | |
| CN110839069A (en) | Node data deployment method, node data deployment system and medium | |
| CN110636388A (en) | Service request distribution method, system, electronic equipment and storage medium | |
| CN104679444A (en) | Dynamic adjustment method and device for virtualized storage resources | |
| US10666742B2 (en) | Software defined network controller and network service allocating system and method | |
| CN111861217A (en) | Vehicle allocation method and device and computer readable storage medium | |
| CN110519310B (en) | Storage task allocation method and device | |
| CN113079062B (en) | Resource adjusting method and device, computer equipment and storage medium | |
| CN109992408A (en) | A kind of resource allocation methods, device, electronic equipment and storage medium | |
| CN110557339B (en) | Flow planning method, device, computer equipment and storage medium | |
| CN109951506B (en) | Method and equipment for evaluating performance of storage cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |