Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
  • G06F3/0604—Improving or facilitating administration, e.g. storage management
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
  • H04N21/2408—Monitoring of the upstream path of the transmission network, e.g. client requests
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0629—Configuration or reconfiguration of storage systems
  • G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0638—Organizing or formatting or addressing of data
  • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
  • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
  • H04N21/23103—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion using load balancing strategies, e.g. by placing or distributing content on different disks, different memories or different servers
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
  • H04N21/2312—Data placement on disk arrays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
  • H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F2003/0697—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers device management, e.g. handlers, drivers, I/O schedulers
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems

Definitions

• This specification relates to resource allocation in a distributed system for data storage and access.
• Distributed network-based data storage for example accessible over the Internet, has various applications.
• One application is video storage and access.
• the invention features a computer-implemented method that includes receiving a description of a pattern of access to a first group of items of content; and partitioning a storage space based at least in part on the description of the pattern of access.
• the description of the pattern of access is a function of at least a first attribute that characterizes one or more operational objectives of a second group of items of content to be stored in a storage system.
• the partitioning the storage space includes defining each one of a plurality of categories in relation to a range of values of the first attribute; and associating each one of the plurality of categories with a respective one of a plurality of resource allocation arrangements.
• Embodiments may include one or more of the following features.
• the storage system may have a predetermined configuration.
• the computer- implemented method may further include determining each one of the plurality of resource allocation arrangements such that an aggregate of resource requirements associated with the plurality of resource allocation arrangements is compatible with the predetermined configuration of the storage system.
• the aggregate of resource requirements includes requirements for both access and storage capacities.
• the computer-implemented method may further include determining a configuration of the storage system for storing the second group of items of content based on a result of the partitioning the storage space.
• the step of determining the configuration of the storage system may include determining characteristics of each one of a set of storage components of the storage system.
• the step of determining the configuration of the storage system may also include iteratively generating a candidate configuration of the storage system based on the result of the partitioning the storage space; and selecting a preferred configuration of the storage system according to a predetermined design objective.
• the predetermined design objective may be characterized by a design objective function.
• the step of iteratively generating the candidate configuration includes providing an initial set of definitions to each one of the plurality of categories in relation to the range of values of the first attribute; and making adjustments to the initial set of definitions to optimize the design objective function.
• the design objective may include minimizing a level of resources requirements for the storage system, and/or minimizing a degree of contention between potential resource consumption by different items in the second group of items.
• the description of the pattern of access may be a function of a plurality of attributes that characterize one or more operational objectives of the second group of items of content.
• Each one of the plurality of categories is defined in relation to a respective range of values of each one of the plurality of attributes.
• the plurality of categories collectively represents a full range of values of the first attribute.
• the operational objectives of the second group of content may include one or more of the following: accessibility, integrity, resilience and power consumption.
• the description of the pattern of access to the first group of content may relate to a description of one or more of the following types: an access rate, a level of bandwidth consumption, a statistical usage distribution, a geographical usage distribution, and a projected usage distribution based on empirical data.
• the respective resource allocation arrangement may include a specification of the number of instances of an item associated with the category will be stored in the storage system.
• the first group of items may share a common characteristic with the second group of items of content, such that the pattern of access to the first group of items forms a reasonable basis for projecting a pattern of access to the second group of items.
• Each of the first and second group of items may be a collection of movie titles.
• each of the first and second group of items may be a collection of content of one or more of the following types: online video ads, user-generated content, music video, video of sport- event, news segments, and TV episodes.
• the second group of items may include at least a subgroup of items that is included in the first group of items.
• the second group of items may be substantially different in content from the first group of items. Further, the second group may not include any item in the first groups of items.
• the invention features a system that includes an interface for receiving a description of a pattern of access to a first group of items of content; and a machine-readable medium that stores instructions executable by a processor to partition a storage space based at least in part on the description of the pattern of access.
• the description of the pattern of access is a function of at least a first attribute that characterizes one or more operational objectives of a second group of items of content to be stored in a storage system.
• the instructions to partition the storage space include instructions to define each one of a plurality of categories in relation to a range of values of the first attribute; and to associate each one of the plurality of categories with a respective one of a plurality of resource allocation arrangements.
• the system further includes a database for storing the plurality of resource allocation arrangements.
• aspects of the invention may include one or more of the following features.
• the instructions to partition the storage space may further include instructions to determine a configuration of the storage system for storing the second group of items of content based on a result of the partitioning the storage space.
• the instructions to partition the storage space may further include instructions to define a design objective function characterizing a predetermined design objective of the storage system; and to compute a preferred configuration of the storage system that optimizes the design objective function.
• the design objective may include reducing a level of resource requirements for the storage system, and may further include minimizing the level of resource requirements for the storage system.
• the design objective may include reducing a degree of contention between potential resource consumption by different items in the second group of items, and may further include minimizing the degree of contention between potential resource consumption by different items in the second group of items.
• Other general aspects include other combinations of the aspects and features described above and other aspects and features expressed as methods, apparatus, systems, computer program products, and in other ways.
• the invention features a computer-implemented method that includes obtaining information characterizing a level of actual usage of a first item of content; based on the obtained information, determining whether a re-provisioning condition is satisfied and if so, generating a specification of a re-provisioning operation to be executed in association with the resources of a storage environment; and executing the re-provisioning operation.
• the first item of content is stored on a first set of elements of resources of the storage environment according to a first resource allocation arrangement.
• the re -provisioning operation includes identifying a second resource allocation arrangement for storing the first item of content; and allocating a second set of elements of the resources of the storage environment according to the second resource allocation arrangement.
• Executing the re -provisioning operation may include storing one or multiple new instances of the first item of content on the second set of elements of resources according to the second resource allocation arrangement; and removing one or multiple previously- stored instances of the first item of content from the first set of elements of resources of the storage environment.
• the re-provisioning operation may include a first type of anomaly -based re- provisioning operations.
• the computer-implemented method may further include detecting a discrepancy between the level of the actual usage of a first item of content and a provisioned level of usage, the provisioned level of usage being characterized by a first set of schema attributes associated with the first resource allocation arrangement; and comparing the discrepancy to an anomaly criterion to determine an occurrence of a provisioning anomaly.
• the anomaly criterion may include an anomaly threshold.
• the computer-implemented method may further include, upon determining the occurrence of the provisioning anomaly, creating a provisioning anomaly object associated with the first item of content; maintaining the provisioning anomaly object in a list of outstanding re-provisions; and determining an anomaly-based re-provisioning operation to be executed in response to the provisioning anomaly object.
• the list of outstanding re- provisions includes a plurality of provisioning anomaly objects, each one of the plurality of provisioning anomaly objects being associated with a respective one of a plurality of anomaly-based re-provisioning operations.
• the computer-implemented method may further include prioritizing the plurality of anomaly-based re-provisioning operations based on a relative urgency of each one of the plurality of re-provisioning operations and an availability of the resources of the storage environment.
• the anomaly-based re-provisioning operation may include identifying a second set of schema attributes that is representative of the level of actual usage of the first item of content.
• the second set of schema attributes is associated with the second resource allocation arrangement.
• the anomaly-based re-provisioning operation may further include: an up-provisioning operation to be executed when the level of actual usage of the first item of content exceeds the provisioned level of usage by a first predetermined threshold amount; and a down-provisioning operation to be executed when the provisioned level of usage of the first item of content exceeds the level of actual usage by a second predetermined threshold amount.
• the obtained information may include one or more of the following: a number of concurrent access requests to the first item of content at a given time, an aggregate of access requests to the first item during a detection period, and a number and frequency of access requests being denied due to resource unavailability.
• the invention features a computer-implemented technique that includes accepting a plan for managing resource allocation associated with a first item stored on resources of a storage environment, the plan including at least a first rule associated with a rule trigger; and applying the first rule upon activation of the rule trigger.
• the first rule associates an adjustment in resource allocation to the first item.
• the first item is characterized by an initial set of attributes, and is stored on resources of an storage environment according to one of a plurality of resource allocation arrangements that is compatible with the initial set of attributes.
• the plurality of resource allocation arrangements may be maintained in a first database.
• the first rule includes a specification of an adjustment applicable to the initial set of attributes characterizing the first item, and the method of applying the first rule includes computing an adjusted set of attributes of the first item according to the specification of the adjustment; selecting one from the plurality of resource allocation arrangements that is compatible with the adjusted set of attributes of the first item; and allocating resources to store one or multiple instances of the first item based on the selected one of the plurality of resource allocation arrangements.
• the adjustment applicable to the initial set of attributes includes applying a discount factor to a present value of at least one of the initial set of attributes.
• the first rule may include a description of sequentially selecting one from a list of candidate resource allocation arrangements for the first item.
• the rule trigger is defined in part by an onset point followed by an activation window.
• the onset point and the activation window may be determined based in part on an assessment of a trend of a pattern of access to the first item with respect to time.
• the rule trigger may be defined by a sequence of onset points. Each one of the sequence of onset points is followed by a respective activation window.
• the method of applying the first rule may include iteratively applying the first rule to the first item upon occurrence of each one of the sequence of onset points.
• the first item may be characterized by a respective initial set of attributes, and is stored on resources of the storage environment according to a respective one of a plurality of resource allocation arrangements.
• the method of applying the first rule may include applying the first rule to each one of a group of items according to a predetermined order.
• the first rule may include a plurality of components, each component being specific to a different item of a group of items.
• the predetermined order is specified in the plan.
• the plan may further include a plurality of rules, each rule associated with a respective rule trigger.
• the computer-implemented method may further include modifying the plan for manage resource allocation in response to a change in an operating condition of the storage environment.
• the operating condition of the storage environment may include a hardware configuration of the resources of the storage environment.
• the operating condition of the storage environment may also include a pattern of access to one or multiple items of the group of items stored on resources of the storage environment.
• the invention features a computer-implemented method that includes detecting an actual workload representative of a pattern of access of a plurality of items of content; comparing the actual workload against a prescriptive workload to determine an occurrence of a substantial deviation from the prescriptive workload; and upon determining the occurrence of the substantial deviation, revising the prescriptive workload based at least in part on the actual workload.
• the plurality of items is stored on resources of a storage environment according to one of a plurality of resource allocation arrangements.
• the prescriptive workload including a plurality of categories, each category being associated with a respective one of the plurality of resource allocation arrangements.
• Each one of the plurality of categories may be defined in relation to a respective range of values of a first attribute.
• the first attribute characterizes one or more operational objectives of the plurality of items of content.
• the method of revising the prescriptive workload includes modifying at least a first category by changing the definition of the first category to a new range of values of the first attribute.
• the method of revising the prescriptive workload may further include associating the modified first category with a proper one of the plurality of resource allocation arrangements, such that a result of the association is compatible with an availability of the resources in the storage environment.
• the resources in the storage environment include storage resources and access resources.
• the associated respective resource allocation arrangement may include a first specification of the number of instances of an item associated with the category is stored in the storage system, and a second specification of a partially defined destination for elements of the item.
• the resources of the storage environment may include at least a first storage component having a plurality of distinct regions, and the second specification of the partially defined destination for elements of the item associates elements of the item with one of the plurality of distinct regions.
• a respective level of resource capacity is allocated to the item in the category by applying the respective one of the resource allocation arrangements.
• the respective level of resource capacity includes at least a storage capacity that relates to the number of instances of the item in the category stored on resources of the storage environment; and an access capacity that relates to an allowable access bandwidth provided to the item in the category.
• the method of revising the prescriptive workload may include, for at least some of the plurality of categories, changing the respective level of resource capacity by changing the respective one of the resource allocation arrangements associated with the category.
• the method of changing the respective one of resource allocation arrangements may include changing the first specification of the number of instances of the item associated with the category is stored in the storage system, or alternatively, include changing the second specification of the partially defined destination for elements of the item.
• the prescriptive workload may be maintained in a first database.
• the computer- implemented method may further include generating a new prescriptive workload based on a result of revising the prescriptive workload.
• the new prescriptive workload enables re- allocation of resources of the storage environment to the plurality of items, the re-allocation being reflective of the actual workload. The substantial deviation occurs when a deviation of the actual workload from the prescriptive workload exceeds a deviation threshold.
• the method of detecting the actual workload may include obtaining information on one or more of the following: a number of concurrent access requests to each one of the plurality of items at a given time, an aggregate of access requests to each one of plurality of items during a detection period, and a number and frequency of access requests being denied with respective to each one of plurality of items due to resource unavailability.
• the invention features a computer-implemented method that includes detecting an actual workload representative of a pattern of access of a plurality of items of content, the plurality of items being stored on resources of an storage environment according to one of a plurality of resource allocation arrangements; comparing the actual workload against a first workload model to determine an occurrence of a substantial deviation from the first workload model,; and upon determining the occurrence of the substantial deviation, determining a second workload model based at least in part on the actual workload.
• the first workload model includes a first set of schemas, each schema being associated with a respective one of a first set of resource allocation arrangements.
• the second workload model includes a second set of schemas, each schema being associated with a respective one of a second set of resource allocation arrangements.
• FIG. 1 is a block diagram of an exemplary distributed system for data storage and access.
• FIG. 2 is a flow chart illustrating the operation of model-based content provisioning.
• FIG. 3 is a flow diagram illustrating an exemplary approach for attribute -based modeling.
• FIGs. 4A and 4B illustrate one example of modeling using a single attribute.
• FIG. 5 illustrates another example of modeling using multiple attributes.
• FIG. 6 is a diagram illustrating an exemplary approach of anomaly-based content re- provisioning.
• FIG. 7 is a flow chart illustrating an exemplary implementation of schedule-based content re -provisioning.
• FIGs. 8 A and 8B illustrate one example of re-modeling.
• a library of content characterized by a broad range of attributes (e.g., data type, length, popularity and access patterns) and operational objectives (e.g., performance, integrity, and resilience), such that each piece of content may involve the use of system resources on different levels.
• attributes e.g., data type, length, popularity and access patterns
• operational objectives e.g., performance, integrity, and resilience
• each piece of content may involve the use of system resources on different levels.
• content of higher popularity may generally need more access resources (e.g., readout bandwidth) in to enhance their streaming performance
• content of greater importance may need more storage space (e.g., to store redundancy data) to ensure data integrity.
• Managing a finite amount of system resources in a way that can well serve the operational objectives of content usage can be a complex problem. Moreover, the level of difficulty can grow progressively as the system increases in size. In some systems, it is advantageous to apply domain-specific knowledge in managing system resources across multiple dimensions (such as storage and access resources).
• a distributed system 100 consists of three basic types of computing platforms for content storage and delivery: storage server platforms 130, access server platforms 120, and client platforms 110.
• the term "platform” is used to refer to a collection of components, which may, for example, be hosted on a single computer, or distributed over multiple computers.
• the storage server platforms 130 may host multiple storage servers 132, which contain storage and provide the capability to read and write data.
• each server may be hosted on one computer, or its function may itself be distributed on multiple hardware elements.
• the access server platforms 120 provide access services that collectively provide applications with one or more methods of accessing data/content with respect to storage servers of the system. Some access services handle the data according to a prescribed access protocol/regime/interface (e.g., HTTP, RTSP, NFS, etc). Other access services manage the resources of the system and regulate access to content accordingly. Services that manage resources of the system include for example, a content provisioning service 124, which allocates resources of the system to store and deliver content, and an admission service 126, which admits sessions when called upon by various session requests in the system.
• a content provisioning service 124 which allocates resources of the system to store and deliver content
• an admission service 126 which admits sessions when called upon by various session requests in the system.
• the access server platforms 120 are interconnected on one side with the storage server platforms 130 by a distribution network 160, and on the other side with the client platforms 110 by an access network 170.
• the access network 170 allows various types of clients to communicate with the access server platforms 120 via one or multiple channels.
• operations client 112 e.g., administrators of online video stores
• access clients 114 e.g., customers of online video stores
• download server 122 may interact with the rest of the system to handle various client requests such as uploading and downloading content.
• the content provisioning service 124 determines a good way of arranging data across available storage resources that can satisfy operational objectives of the content.
• operational objectives include performance objectives (such as accessibility), integrity, resilience, and power consumption, each of which can be characterized by one or multiple attributes.
• performance objectives such as accessibility
• integrity such as integrity
• resilience such as resilience
• power consumption each of which can be characterized by one or multiple attributes.
• a popularity attribute associated with a viewing frequency of a particular title can represent a desired level of content accessibility — one type of operational objective that the system may intend to serve.
• the process of making good provisioning decisions that take into account the objectives of each individual in the system can represent an NP Complete problem, which is particularly complex for large systems.
• One approach to solving this problem involves applying domain knowledge to organize the problem space and to form partial solutions (e.g., models) that can guide the search for optimal solutions in a more efficient manner. For example, a pattern of access to titles of a library can be used to establish a model workload that contains a finite set of categories that are each mapped to a partially determined allocation scheme that represents a way of resource arrangement.
• content provisioning can be then approached as a two-step process — firstly, attribution of each piece of content to a proper category, and secondly, resource allocation within the local scheme of that category.
• Model-based content provisioning will be described in greater detail in the following sections.
• the content provisioning service makes use of model-based partial solutions, which are obtained during a configuration stage 210, to pre-compute information that is later used during a provisioning stage 220 for determining a good arrangement of content on the storage resources of the system.
• empirical data 212 are used to generate a model workload 214 that represents a type of workload the system aims to support.
• Empirical data can range from a verbal description of the overall usage of video, through a detailed weblog of all events from a running system including video viewing events.
• empirical data 212 is a statistical usage distribution provided by video store operators, characterizing the number of views per each title of a library over a 24-hour service window.
• Another example is a geographic profile of viewing habits among consumers across a wide range of areas. In situations where empirical data is unavailable or insufficient, other information such as customer projections of content usage and service objectives can also be used for creating the model workload 214.
• the model workload 214 can be manually generated prior to being introduced to the system, or automated (e.g., in a heuristic way) by the system itself. Based on the model workload 214, a prescriptive workload 218 is created. Very generally, the prescriptive workload 218 offers partial solutions to the problem of resource allocation by partitioning the problem space in a discrete way.
• a prescriptive workload 218 can be generated for example, by mapping the model workload 214 onto a particular system configuration — either 1) the system size is fixed and the workload is scaled to fit the system; 2) the workload is fixed and the system size is scaled to match the workload; or 3) a combination or both.
• the prescriptive workload 218 can be viewed as a partitioning of attribute space, defining a set of attribute-based categories (referred to herein as schemas) in which each individual can be mapped to a type of resource arrangement associated with that category.
• schemas a set of attribute-based categories
• the prescriptive workload 218 can be represented in the system as a collection of configuration data (i.e., schema database 219) that describe how to provision and access objects in the system.
• the schema database 219 includes schemas and provision tables, which together provide a description of a definition of each schema and the resource arrangement associated with that schema. Take popularity-based content provisioning for example.
• One schema can be defined to include titles in the top 5% of the most frequently viewed in the library. For a system that aims to provide a maximum of 800 Mbps bandwidth for each title in this category with an aggregate limit of 800 Mbps or more for all titles in the category, the schema database 219 describes the set of arrangement combinations that can support such a bandwidth goal, including for example, distributing content data across a span of eight disks.
• the prescriptive workload 218 and associated schemas are also designed to reduce a degree of inter-title contention in the system.
• Examples of a prescriptive workload that creates a high level of inter-title contention include storing four copies each of "Shrek I" and "Shrek II" on one single server that has a maximum readout rate of 100 Mbps.
• “Shrek I” is at peak usage (e.g., streamed at 90 Mbps) that occupies most or all of the server's access capacity, the remaining data on this server (including "Shrek II") are rendered inaccessible, while resources of other servers in the system may be left idle.
• a less contentious prescriptive workload stores one copy of "Shrek I" and "Shrek II” per server across four different servers, so that the peak usage of one object does not exceed 25% of a server's output capacity, without blocking access to other data on these servers.
• content provisioning service 124 makes use of information in the schema database 219 to provision content in the following way.
• the content provisioning service Upon receiving a request 222 for uploading a piece of content, the content provisioning service first determines a proper schema for this content, by matching content attribute(s) against the partitioning results in the schema database (step 224). If, for example, this content ranks among the top 5% popular titles, the exemplary schema described above is selected. Accordingly, content data will be stripped across eight disks. Given this eight-disk requirement, there still exists multiple ways of mapping content to the storage. For a system consisting of 16 disks of storage, one option is to map content to disks number 1 through number 8, and a second option is to use disk number 9 through number 16 (and possibly other eight-disk combinations).
• content provisioning service 124 selects a best mapping, for example, the one with the lowest server occupancy (step 226). If there are sufficient resources available in this best mapping (step 228), content provisioning service 124 proceeds to allocate a set of physical blocks on each of the eight disk drives according to the best mapping (step 230) and subsequently, the administrative server 121 delivers the content data to these locations (step 232). If content provisioning service 124 fails to locate sufficient resources in the best mapping, it selects the next mapping to repeat the evaluation process 228 until an available mapping has been found and committed. In some situations when multiple copies of a piece of content are desired on the system, the content provisioning service may determine for each individual copy a best available mapping and store data accordingly. After data delivery completes, results of the mapping (including the disk locations of the data) are documented in the object directory 129, so that content can be conveniently accessed in the future by other services (e.g., admission service 126)
• other services e.g., admission service 126)
• model and prescriptive workloads can help content provisioning service 124 to manage resources across multiple dimensions in a more efficient manner.
• One approach, for example, is to perform &-ary subdivision in n- space, where & is a subdivision factor (e.g., k 2 is binary subdivision) and n is the number of independent attributes.
• an attribute can be a characterization of one of operational objectives (such as popularity, resilience, and integrity), independent resource pools and their characteristics (such arrays of disks and the disks' model(s)), and/or other aspects in resource management.
• STEP 1 Obtain empirical usage distribution that represents a pattern of access of titles in a library.
• Usage distribution can be a description of access rate (e.g., views per second, average number of concurrent views, Megabits per second) as a function of one or multiple attributes (e.g., title rank).
• access rate e.g., views per second, average number of concurrent views, Megabits per second
• attributes e.g., title rank
• Such information can be obtained either from actual usage data (e.g., store records from Netflix, Blockbuster, Hollywood Video, etc.), or based on projections (e.g., using video rental data to extrapolate their usage characteristics).
• a continuous mathematical model is formulated to describe usage data.
• STEP 2 Partition the attribute space based on usage distribution.
• a set of schemas are defined, each being associated with ranges of one or multiple attributes (e.g., top 5 percentile of title rank).
• a schema generally describes how a title with a given set of attributes is assigned resources in the system.
• Each schema is mapped to a pattern of storage allocation, including for example, a combination of disks on which titles in this schema will be located and the number of copies that will be stored for each title.
• STEP 3 Compute resource utilization (e.g., total amount of storage) by integrating resource requirements of each title of the library based on the partitioning result. In some applications, the aggregate resource requirements serve as a basis for determining the number and types of storage servers that will be used in the system.
• steps 2 and 3 can be iteratively performed to determine a good (or best) way of partitioning that yields the least amount of utility cost while satisfying certain constraints that limits some aspects of provisioning (e.g., the total amount of storage and maximum readout rate per disk). If the process at a particular step is not converging effectively, the previous step can be reentered and modified.
• objective functions e.g., cost, resource utilization, inter-title contention, power consumption
• models for content provisioning are created based on a single attribute —popularity.
• popularity is an expression of anticipated demand for a piece of content. For example, if content is popular, demand for it will likely be high, and thus the provisioning system will attempt to provision the content with the resources necessary to meet the expected demand.
• Graph 410 is a distribution of the access rate (e.g., views per day) of 1 million titles in a user generated video library plotted against popularity rank on a log-log scale. As shown in the graph, access rate (and therefore the demand for access bandwidth) varies widely from title to title. While some of the most popular titles are viewed more than a million counts per day, titles in the lowest rank are accessed no more than once per day. Generating a popularity-based model workload can thus allow system resources to be allocated in ways that are consistent with service objectives (such as title accessibility).
• service objectives such as title accessibility
• FIG. 420 One way of creating a model workload that takes popularity into account is shown in graph 420.
• a set of quanta are created as a result of sub-dividing the continuous usage distribution curve 410.
• Each quantum is associated with a range of title rank within which all titles will share a partially determined allocation schema.
• This model workload provides a framework that later guides the process of content provisioning, e.g., by concentrating more system resources on individuals that have a greater demand.
• a full variety of quantization schemes e.g., binary subdivision or ternary subdivision or even non-uniform and non-formulaic subdivisions
• Different quantization schemes may have advantages or disadvantages with respect to different sets of empirical data, specific applications (e.g., providing content in distinct data categories such as movies or online ads), or system configuration that will be used to support the workload.
• FIG. 4B illustrates one set of schemas that are applicable in this example.
• each quantum that spans over a given range along the axis of title rank corresponds to a specific schema.
• Titles in the same schema share a set of pre-determined arrangements and service characteristics, including for example, the number of disks and disk region where each title will be stored, the number of primary copies of each title, and performance characteristics such as the maximum access bandwidth the system is able to offer, as illustrated in provision table 440.
• provision table 440 For instance, a title that falls within the top 5 percentile will be provisioned according to Schema 1 (Sl) — that is, having 12 copies on the storage with each copy stripped on disk region 1 (Rl) across a span of 96 disks.
• Schema 1 (Sl) Schema 1 (Sl) — that is, having 12 copies on the storage with each copy stripped on disk region 1 (Rl) across a span of 96 disks.
• Sl Schema 1
• Rl disk region 1
• the amount of storage capacity needed for the entire library can be computed and used for determining hardware requirements for the system. For example, if the amount of storage needed for IM titles is approximately 90TB according to the prescriptive workload, the system can be configured to include 96 ITB disks organized as 4 servers each with 24 disk drives. Further, if it is desired to provide an access bandwidth of 36Gbps for servicing 4OK simultaneous sessions (each session encoded at 900Kbs) on the system, each disk drive is selected to have at least a sustained readout rate of 400 Mbps.
• resilience relates to a predicted level of service maintained to an information object in the presence of component failure. For example, in accessing a resilient object, a customer encountering a failed read on disk A can continue to be serviced through reading an alternative copy of data on disk B. Generally, the more resilient an object needs to be, the more resources (and possibly the more types of resources) the content provisioning system needs to allocate to that object.
• Graph 510 illustrates an exemplary subdivision of a two-dimensional attribute space.
• each schema is now defined by combinations of ranges of each of the two attributes.
• tittles in each schema are assigned with a set of pre-determined arrangements and service characteristics.
• titles in Schema 1 correspond to both high resilience and high popularity, and are therefore each stored with 4 primary copies in the fast region (Rl) of 12 disks to provide good resilience and accessibility.
• titles in Schema 5 correspond to both low popularity and low resilience, and are therefore each stored with only one primary copy in the slow region (R5) of 4 disks.
• Each schema can be defined by specified ranges of attribute values.
• Schema 2 is defined by popularity attribute value in the range of [A 1 S2 mm , A' S 2 nid ⁇ ] and resilience attribute value in the range of [A 2 s2 mm , A 2 s2 ma ⁇ ]-
• This schema provides that each of the three primary copies of the title will be written in R2 across a span of 8 disks. This arrangement gives rise to a 600 Mbps access capacity supportable by the system for each title in S2.
• each schema will be defined by combinations of value ranges of each one of these attributes. Once a title is allocated to a proper schema based on its attribute values, content provisioning is performed according to the pre-determined arrangements assigned to this specific schema.
• content integrity relates to the ability to recover content after a component failure, even a catastrophic failure of the system. For example, if a disk fails completely and is unrecoverable, the portions of content contained on the disk can be recovered from an alternate copy of the content within the system or across systems and/or from an encoded reconstruction method.
• attributes include geographical affinity and exclusion, topological affinity and exclusion, and power consumption.
• manual or automated optimization can be incorporated in modeling to achieve one or multiple design objectives.
• One design objective can be, for example, to make best use of available resources, given that the number and types of storage servers have been determined prior to modeling.
• Another design objective can be to minimize the overall system resource requirements while satisfying the service goals of individual or an aggregate of titles, if hardware components of the system are yet to be determined.
• an objective function is first defined and the inputs that can minimize or maximize its value are determined. (Sometimes, a group of constrains are imposed on the values that the inputs can take.)
• the definitions of schemas can be viewed as inputs to an objective function of utility cost that needs to be minimized.
• an optimized set of schema definitions may be found by making small steps of adjustments in the inputs (such as moving the boundaries of Schema 2 along each attribute axis) to arrive at conditions that yield the lowest system cost. (See Re-Modeling below for adjusting model and schemas to optimize a running system.)
• one advantage of using a model-based approach to manage resource allocation is that the performance and behavior of components and subsystems can be tested prior to building the entire system.
• the model can be refined to the subsystem or component level, and tests can be devised for the subsystem or component prior to incorporating it into the overall system.
• multilayer test scripts can be used to test and qualify subsystems on various levels (e.g., an individual disk drive, a group of disk drives, and a complete storage server) and ultimately be extended to the entire system (which includes access servers, storage servers, disk scheduler, and etc).
• the test results are analyzed and used to determine a desirable system configuration or modification. These results may also provide a verifiable performance objective for the next level of system development.
• the actual usage of the content may be tracked by collecting content statistics (such as session counts and bandwidth consumption) from storage and access servers.
• content statistics such as session counts and bandwidth consumption
• the actual usage of an object may deviate from its level of provisioning, and large deviations can affect overall efficiency of resource utilization. For example, over- provisioned objects may be consuming resources that are not put in active use yet nonetheless are prevented from being allocated for other objects, whereas in the mean time, under-pro vision objects are assigned insufficient resources resulting in incoming access requests being rejected. Therefore, in order to enable the system to continue to operate in a globally optimal state, a content re-provisioning service is provided.
• a first situation occurs when a title is initially provisioned without attributes (e.g., placed in a default schema and provisioned accordingly), the actual usage of that tile is later used to determine which schema is appropriate to use and whether the title needs to be re- provisioned.
• a second situation occurs when a title is initially provisioned using attributes that are not representative of the actual usage (e.g., when marketing promotion boosts a movie's near term popularity), the system detects the discrepancy ("anomaly") between the actual and provisioned levels and orders the title to be re-provisioned by selecting a schema most reflective of the title's actual usage.
• a third situation of content-repositioning occurs when there is a planned lifecycle management of attributes (e.g., "aging"), and titles are scheduled to be moved between schemas on a pre-determined basis, for example, providing progressively less bandwidth for a title over the lifetime of its release cycle.
• attributes e.g., "aging”
• the content re-provisioning service provides at least two types of operations: 1) up-provisioning, which upgrades provisioning to a higher-level schema that will generally assign more resources to the content; and 2) down-provisioning, which downgrades provisioning to a lower-level schema to release some amounts of resources back to the system.
• up-provisioning which upgrades provisioning to a higher-level schema that will generally assign more resources to the content
• down-provisioning which downgrades provisioning to a lower-level schema to release some amounts of resources back to the system.
• anomaly-based and threshold-based content re-provisioning deals (1) with content that has been provisioned using attributes that are not representative of its actual usage or (2) with content whose usage has unpredictably changed such that the provisioning is no longer representative of and appropriate for its actual usage.
• "anomaly" can be caused by many other factors, including for example, certain marketing initiatives and unpredictable events (such as the death of an actor) that results in a change in content's near term popularity.
• the content re-provisioning service 125 interacts with other services in the system, including the admission service 126 and content monitor 123, to provide the following functionalities.
• the admission service 126 gathers and maintains a set of running statistics for each piece of content that is currently being accessed (step 612). Examples of running statistics include the number of admitted concurrent sessions for each object and the total usage across all objects at any given time and over time.
• the admission service 126 also maintains provisioning anomaly threshold on a per schema basis as part of a service configuration file (step 614).
• a provisioning anomaly object is created (step 616) and provided to the content monitor 123.
• the content monitor 123 maintains a work queue (step 622) of provisioning anomaly objects submitted by the admission service 126, and determines and preferably prioritizes a list of outstanding re-provisions (step 624). For example, based on the type of each provisioning anomaly, the content monitor 123 determines whether the anomaly can be remedied by reprovisioning. For content that requires up-provisioning, but is already at the top level of provisioning, a "Provisioning Anomaly Not Repaired" event will be sent and no further processing is required for that content. In some examples, the content monitor 123 places content to be up-provisioned at higher priority in order for the system to be able to admit as many future service requests related to the content as possible.
• requests to down-provision content are honored prior to requests to up-provision in order to free up system resources for up-provisioning.
• the content monitor 123 determines a set of new attributes that are representative of its actual usage and subsequently generates a re-provisioning command (step 626) to instruct the re-provisioning service 125 to re-provision the item accordingly.
• content re-provisioning service 125 After content re-provisioning service 125 receives the re -provisioning command (step 632), content is re-provisioned based on the new attributes (step 634). Similar to a general content provisioning procedure described earlier, the re-provisioning service 125 selects a schema corresponding to the new attributes, and assigns resources to write new copies of the content on the storage according to the schema. When content has been given its new provisioning, the re -provisioning service 125 updates content information in the object directory (step 636) and removes old instances of content data from the storage.
• threshold-based re-provisioning uses a threshold mechanism to make re-provisioning decisions by taking into account short-term and/or long-term variability (e.g., a viewing trend) of title usage.
• short-term and/or long-term variability e.g., a viewing trend
• threshold-based re -provisioning is described below.
• a trend of usage of a title is monitored, for example, by computing the first derivative of the title's viewing rates with respect to time). This trend is then measured against a threshold to determine whether the usage of the title follows a projected trend of this title.
• This projected trend can be configured on a per schema basis (i.e., every title in the same schema is associated with a same/similar projected trend), or alternatively, configured specifically for each title based on its own characteristics (e.g., depending on attributes such as the type and content of titles).
• the title will be placed in a re-provisioning list to be re-provisioned.
• re -provisioning actions initiated by threshold-based re-provisioning decisions are conducted by the system at convenient time (e.g., during times of light service load) to reduce disturbance to other operations of the system.
• a first implementation is based on a non-weighted backwards looking time window and usage threshold. By discrete sampling of continuous time events, usage distribution of a title is estimated and compared with threshold to determine whether re- provisioning actions need to take place.
• a second implementation is based on a time- sensitive weighted moving average/sum of usage of a title. More specifically, in recognizing that empirical usage events inherently exhibit a statistical probability that fit a distribution function, a distribution function is used to weight the trailing moving average/sum. This second implementation can reduce sampling errors and aliasing that may occur in a non-weighted implementation, and improve overall system efficiency.
• schedule-based content re-provisioning is performed as a result of lifecycle management, for example, a title that is expected to receive decreasing usage over time can be provided progressively less bandwidth over time in a planned manner.
• schedule-based re-provisioning process runs in the background at low priority to keep its impacts on other operations at minimum.
• schedule-based content re-provisioning proceeds as follows.
• re -provisioning schedules are associated with the titles in the system (step 702).
• a re-provisioning schedule can be determined manually (e.g., planned by operators based on domain knowledge), or automated (e.g., using a content aging function modeled by gathering content statistics and analyzing a trend of access pattern).
• the schedule may contain a beginning reference point for the lifecycle and a series of planned re-provisioning actions to be taken (e.g. re-provision a title from its current schema to another specified schema or make a specified change in the title's attributes) and when to take these actions (including, for example, a set of clock time that triggers the actions).
• the re-provisioning schedule can be later revised based on the object's actual usage over time and overall system objectives.
• a title's new attributes are computed based on the schedule (step 706).
• the title is then re-provisioned using these new attributes (step 708).
• the re-provisioning service waits for the next start point to begin another round of re-provisioning actions (step 710).
• the re-provisioning schedules are configured on a contractual base.
• a schedule includes the start dates/time for a set of re-provisioning actions, and the specifics of the actions to be taken at the time (e.g., a specified provisioning level of the title in each action).
• the re -provisioning schedules further include a set of usage models configured for re-provisioning purposes, where each usage model corresponds to a particular set of re-provisioning actions.
• each usage model corresponds to a particular set of re-provisioning actions.
• the re-provisioning service then re-provisions the title according to the actions corresponding to the best-matching model.
• the title will be re-provisioned, again, according to the actions corresponding to this model.
• the services involved in content re-provisioning can be provided by one or multiple physical servers. These servers may be one or more access server platforms 120 (as shown in FIG. 1), or one or more external re-provisioning platforms, or a combination of both.
• the content re- provisioning service is used to re-assign the content to an appropriate schema that can more accurately represent the actual usage.
• a significant number of re -provisioning requests may occur in the system or the re-provisioning requests can be satisfied, which can cause re-provisioning storms that can unbalance the system and threaten its performance and stability.
• the prescriptive workload itself may no longer be appropriate for the system achieving desired performances and behavior.
• system updates e.g. loss of a failed server or addition of new servers to the system
• system reconfiguration and/or remodeling on a global scale is recommended.
• the system's actual usage is now used as new empirical data (a new 212) to create a new model workload, a new prescriptive workload, new schemas, etc.; these may represent a minor or major change to the system, depending on the extent of differences between previous and new empirical data and system objectives.
• One way to recover such a loss is to rebalance the relative allocation of resources among schemas, as illustrated in the new prescriptive workload 820.
• the access bandwidth for titles in the first two quanta is raised to 950 Mbps and 800 Mbps, respectively, whereas the rest of the library has been decreased to lower levels accordingly.
• a new set of schemas (Sl ', S2', S3', S4', and etc) now replaces the previous set (Sl, S2, S3, S4, and etc) to reflect the changes in resource allocation.
• Another example of re-modeling occurs with the addition of new resources, e.g., servers, to the system.
• new resources e.g., servers
• the above examples represent a system of 4 storage servers and 4 access servers
• expanding the system to, for example, 6 storage servers and 6 access servers can result in a 50% increase in system resources to be allocated. (Here, the number of storage and access servers need not be the same.)
• at least three changes to the prescriptive workload can be made. (1) If the system is being increased because the systems is becoming filled with more lower-popularity titles, then one or more lower-popularity schemas can be added. (2) If overall system usage is increasing and the proportion of usage across titles is as before, then all schemas are proportionally provided more system resources. (3) If the system is being increased because more higher-popularity titles are being added, then one or more higher-popularity schemas can be added. Note that these three scenarios are not exhaustive.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Information Transfer Between Computers (AREA)
• Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Applications Claiming Priority (4)

Publications (1)

Family

ID=40996612

Family Applications (1)

Country Status (3)

Families Citing this family (15)

Family Cites Families (5)

• 2009
  • 2009-07-09 CN CN200980132893.2A patent/CN102132577B/zh not_active Expired - Fee Related
  • 2009-07-09 WO PCT/US2009/050051 patent/WO2010006127A1/en active Application Filing
  • 2009-07-09 EP EP09790199A patent/EP2311250A1/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events