CN111443870A - Data processing method, device and storage medium - Google Patents
Data processing method, device and storage medium Download PDFInfo
- Publication number
- CN111443870A CN111443870A CN202010222821.5A CN202010222821A CN111443870A CN 111443870 A CN111443870 A CN 111443870A CN 202010222821 A CN202010222821 A CN 202010222821A CN 111443870 A CN111443870 A CN 111443870A
- Authority
- CN
- China
- Prior art keywords
- data
- state
- routing table
- priority
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0665—Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data processing method, equipment and a storage medium, capacity expansion or capacity reduction is reasonably carried out on data storage based on a scheduling decision strategy, data storage configuration is correspondingly increased or reduced, storage cost is reduced, a data routing table is determined based on a scheduling decision strategy, the data routing table can be dynamically transferred to different data storages based on the data routing table, access resources are obtained from different data storages, storage memory of a data storage device is reduced, resource redundancy and waste are avoided, the access requirement of upper-layer services on data is responded timely, and stable service services are provided. The foregoing data processing method includes: acquiring a target service state; performing data resource scheduling processing on the target service state to obtain a scheduling decision strategy; determining a data routing table based on a scheduling decision policy; and sending the data routing table to the data storage device so that the data storage device determines the access resource matched with the target service state based on the data routing table.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a data processing method, data processing equipment and a storage medium.
Background
The data storage is used as the infrastructure of the bottom layer, and on the basis of a layered architecture, the data storage can be decoupled with the upper-layer service, so that horizontal expansion and independent upgrade maintenance are facilitated. At present, the conventional method is to dynamically adjust the configuration resources required by the upper layer service through migration, capacity expansion, capacity reduction and the like based on the historical use conditions of online responses, such as data of request peak values, average consumed time and the like, on the basis of a layered architecture.
However, when the current method is used for processing the personalized service requirement of the B-side (also called as an enterprise user or a merchant), due to the fact that the personalized service requirement, the response of emergency and the change of the requirement of the B-side service have the service characteristics of randomness, violent fluctuation of the change of the requirement and the like, the processing method based on the historical use condition improves the problems of resource waste and response delay to a certain extent, but due to the design of the layered architecture, the data storage of the bottom layer cannot sense the service requirement of the upper layer service, so that the data storage of the bottom layer has redundancy and waste of a large amount of resources in order to meet the access requirement of the upper layer service on the data, and stable service is difficult to provide.
Disclosure of Invention
The embodiment of the application provides a data processing method, equipment and a storage medium, capacity expansion is carried out on frequently-used data storage and corresponding data storage configuration is increased on the basis of a scheduling decision strategy, capacity reduction is carried out on infrequently-used data storage and data storage configuration is reduced, storage cost is reduced, a data routing table is determined on the basis of a scheduling decision strategy, and the data routing table can be dynamically transferred to different data storages on the basis of the data routing table, so that access resources are obtained from different data storages, storage memory of a data storage device is reduced, redundancy and waste of resources are avoided, the access requirement of upper-layer services on data is responded timely, and stable service services are provided.
In view of this, the embodiments of the present application provide the following solutions:
in a first aspect, an embodiment of the present application provides a data processing method, where the method may include:
acquiring a target service state;
performing data resource scheduling processing on the target service state to obtain a scheduling decision strategy;
determining a data routing table based on the scheduling decision policy;
sending the data routing table to a data storage device to enable the data storage device to determine an access resource matched with the target service state based on the data routing table.
In a second aspect, an embodiment of the present application provides a data processing method, where the method may include:
receiving a data routing table sent by an adaptive scheduling device, wherein the data routing table is determined by a scheduling decision strategy obtained by the adaptive scheduling device after performing data resource scheduling processing on a target service state;
and determining the access resource matched with the target service state based on the data routing table.
In a third aspect, an embodiment of the present application provides an adaptive scheduler, where the adaptive scheduler may include:
the acquisition unit is used for acquiring a target service state;
the processing unit is used for carrying out data resource scheduling processing on the target service state so as to obtain a scheduling decision strategy;
a determining unit, configured to determine a data routing table according to the scheduling decision policy;
a sending unit, configured to send the data routing table to a data storage device, so that the data storage device determines, based on the data routing table, an access resource that matches the target service state.
Optionally, with reference to the third aspect, in a first possible implementation manner, the target service state includes a service plan state, a service start state, a service progress state, or a service completion state.
Optionally, with reference to the first possible implementation manner of the third aspect, in a second possible implementation manner, the determining unit is further configured to determine, after obtaining a target service state, a first priority of the service plan state, a second priority of the service start state, a third priority of the service progress state, or a fourth priority of the service completion state, where the third priority is greater than the second priority, the second priority is greater than the first priority, the first priority is greater than the fourth priority, the first priority is used to indicate a priority requirement of the service plan state for the scheduling decision policy, the second priority is used to indicate a priority requirement of the service start state for the scheduling decision policy, and the third priority is used to indicate a priority requirement of the service progress state for the scheduling decision policy, the fourth priority is used to indicate a priority requirement of the service completion status for the scheduling decision policy.
Optionally, with reference to the third aspect and the first to second possible implementation manners of the third aspect, in a third possible implementation manner, the processing unit includes:
the decomposition module is used for decomposing the target service state to obtain service storage requirement information;
and the determining module is used for determining a scheduling decision strategy according to the service storage requirement information.
Optionally, with reference to the third aspect and the third possible implementation manner of the third aspect, in a fourth possible implementation manner, the determining module is configured to process the target service state through an access amount prediction model or a space prediction model to calculate a data storage location and a data replication number;
the determining module is used for processing the target service state through the access amount pre-estimation model or the time pre-estimation model so as to calculate data migration time and data online time;
the determining module is used for determining the data storage position, the data copying quantity, the data migration time and the data online time as the service storage requirement information.
In a fourth aspect, an embodiment of the present application provides a data storage device, including:
the receiving unit is used for receiving a data routing table sent by the self-adaptive scheduling device, and the data routing table is determined by a scheduling decision strategy obtained by the self-adaptive scheduling device after data resource scheduling processing is carried out on a target service state;
and the determining unit is used for determining the access resource matched with the target service state based on the data routing table.
In a fifth aspect, an embodiment of the present application provides a computer device, including:
the method comprises the following steps: an input/output (I/O) interface, a processor and a memory,
the memory stores program instructions;
the processor is adapted to execute program instructions stored in the memory for implementing the method according to any one of the first aspect, the second aspect, any one of the first aspect and any one of the possible implementations of the second aspect as described above.
A sixth aspect of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for performing a method as described in the first aspect, the second aspect, any one of the first aspect and any one of the possible implementations of the second aspect.
A seventh aspect of embodiments of the present application provides a computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to perform the method of any of the above aspects.
According to the technical scheme, the embodiment of the application has the following advantages:
in the embodiment of the application, after the adaptive scheduling device performs data resource scheduling processing on the target service state to obtain a scheduling decision strategy, the adaptive scheduling device can determine a data routing table based on the scheduling decision strategy and send the data routing table to the data storage device, so that the data storage device determines the access resource matched with the target service state based on the data routing table. In the embodiment, based on the scheduling decision strategy, the capacity of the frequently-used data storage is expanded and the corresponding data storage configuration is increased, and for the data storage which is not frequently used, the capacity is reduced and the data storage configuration is reduced, so that the storage cost is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below, and it is apparent that the drawings in the following description are only some embodiments of the present application.
FIG. 1 is a block diagram of an architecture of a data processing system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an embodiment of a method for data processing provided by an embodiment of the present application;
FIG. 3 is a flowchart of a data resource scheduling process provided in an embodiment of the present application;
FIG. 4 is a schematic diagram of a data routing table provided in an embodiment of the present application;
FIG. 5 is a flow chart of data processing provided in an embodiment of the present application;
fig. 6 is a schematic diagram of an embodiment of an adaptive scheduling apparatus provided in an embodiment of the present application;
FIG. 7 is a schematic diagram of an embodiment of a data storage device provided by an embodiment of the present application;
fig. 8 is a schematic structural diagram of a computer device provided in an embodiment of the present application.
Detailed Description
The embodiment of the application provides a data processing method, equipment and a storage medium, capacity expansion is carried out on frequently-used data storage and corresponding data storage configuration is increased on the basis of a scheduling decision strategy, capacity reduction is carried out on infrequently-used data storage and data storage configuration is reduced, storage cost is reduced, a data routing table is determined on the basis of a scheduling decision strategy, and the data routing table can be dynamically transferred to different data storages on the basis of the data routing table, so that access resources are obtained from different data storages, storage memory of a data storage device is reduced, redundancy and waste of resources are avoided, the access requirement of upper-layer services on data is responded timely, and stable service services are provided.
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.
Cloud technology (Cloud technology) is based on a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied in a Cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.
A distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work by using functions such as cluster application, grid technology, and distributed storage file system, and provides a data storage function and a service access function to the outside.
At present, a storage method of a storage system is as follows: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data Identification (ID), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided into stripes in advance according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.
Currently, the conventional method for adjusting configuration resources is to dynamically adjust configuration resources required by the upper layer service through migration, capacity expansion, capacity reduction and the like based on historical usage of online responses, such as data of peak request values, average consumed time and the like, on the basis of a layered architecture. When the existing method is adopted to process the personalized service requirement of the B end (also called as an enterprise user merchant), due to the fact that the personalized service requirement, the response of emergent situations and the requirement change of the service of the B end have the service characteristics of randomness, violent requirement change and the like, the problem of resource waste and response delay is improved to a certain extent by the processing method based on the historical use condition, but due to the design of a layered architecture, the service requirement of upper-layer services cannot be sensed by data storage of a bottom layer, so that the data storage of the bottom layer has the redundancy and waste of a large amount of resources for meeting the access requirement of the upper-layer services to the data, stable service is difficult to provide, and the emergency situation is difficult to respond in time.
In order to solve the above problem, an embodiment of the present application provides a data processing method, which is applied to the data processing system described in fig. 1. Please refer to fig. 1, which is a schematic diagram of a system architecture of a data processing system according to an embodiment of the present application. As shown in fig. 1, the system architecture diagram includes a service-side access device, an adaptive scheduler, and a data storage device. The service side access device is used for providing a target service requirement which needs to apply for data storage resources, the target service requirement comprises a target service state, and the target service state is a service plan state, a service starting state, a service proceeding state and a service finishing state, so that the service side access device can send the target service state to the adaptive scheduling device. Therefore, the self-adaptive scheduling device can perform data resource scheduling processing on the change of the service storage requirement when the target service state is in different states, so that a dynamic data routing table is determined based on a scheduling decision strategy and is distributed to a data storage device at the bottom layer of data storage, and the data storage device determines access resources matched with the target service state based on the dynamic data routing table, deploys in advance and accurately meets the service requirement of service side access equipment for data storage.
It should be understood that the service-side access device mentioned above also needs to provide a state output interface corresponding to the target service state. In addition, as can also be seen from fig. 1, the adaptive scheduling apparatus may specifically include a traffic state receiver for receiving a target state traffic, an adaptive scheduling processor for performing data resource scheduling processing on a target traffic state, and a data scheduling controller, so as to determine a data routing table. The aforementioned data storage device may include, but is not limited to, a data physical storage device, a cloud storage, and the like, and is not limited in this embodiment.
The data processing method in this embodiment may be applied to the system architecture shown in fig. 1, and may also be applied to other system architectures, which are not limited herein.
For convenience of understanding, a method for data processing is provided in the embodiments of the present application, and please refer to fig. 2, which is a schematic diagram illustrating an embodiment of the method for data processing provided in the embodiments of the present application.
As shown in fig. 2, a method for data processing provided in an embodiment of the present application may include:
201. the adaptive scheduling device acquires a target service state.
In this embodiment, when the service-side access device applies for a data storage resource, a corresponding target service requirement is generated, and the target service requirement carries a corresponding target service state, such as: which are a service plan state, a service start state, a service progress state, and a service completion state. Therefore, the service side access device can send the target service state to the adaptive scheduling device according to the target service requirement, so that the adaptive scheduling device can obtain the corresponding target service state.
Optionally, in some embodiments, the target business state includes a business plan state, a business start state, a business progress state, or a business completion state.
Different traffic states include different traffic demands. For example, a service plan state, a service start state, a service progress state or a service completion state may be understood as that, assuming that map data is made as an example, a service party plans to implement a traffic light data acquisition project of a certain area in 3 months and 15 days to 3 months and 16 days, before 3 months and 15 days, service side access equipment uploads service information such as acquisition time, data volume and the like in the plan to an adaptive scheduling device, which is called a service plan state at this time; and the service is started in 3 months and 15 days, the service side access equipment informs the self-adaptive scheduling device of the starting time, the response time delay of the service and other starting information, which is called as a service starting state at this moment; traffic light data are collected in 15-3-16 days in 3 months, and the service side access equipment informs the self-adaptive scheduling device of specific collection time, service indexes and other information, which is called as a service proceeding state at the moment; and when the service is finished in 17 days after 3 months, namely the traffic light data acquisition is finished, the service is called as a service finished state. It should be understood that the aforementioned traffic light data collection items implemented in a certain area from 15 days in 3 months to 16 days in 3 months are only an illustrative description, and the service will not be limited in the embodiment of the present application.
Optionally, in other embodiments, because the priority requirements and policies for the data storage resources in different service states are different, after the adaptive scheduling apparatus acquires the target service state, the adaptive scheduling apparatus may further determine a first priority of the service plan state, a second priority of the service start state, a third priority of the service progress state, or a fourth priority of the service completion state.
That is to say, after acquiring different service states, the adaptive scheduling apparatus configures a corresponding scheduling decision policy based on the priority priorities of the different service states, that is, the first priority is used to indicate the priority requirement of the service plan state for the scheduling decision policy, the second priority is used to indicate the priority requirement of the service start state for the scheduling decision policy, the third priority is used to indicate the priority requirement of the service progress state for the scheduling decision policy, and the fourth priority is used to indicate the priority requirement of the service completion state for the scheduling decision policy.
Generally, the data storage resources required by the service progress state are greater than the data storage resources required by the service start state, the data storage resources required by the service start state are greater than the data storage resources required by the service meter state, and the data storage resources required by the service plan state are greater than the data storage resources required by the service completion state, so that the third priority corresponding to the service progress state is greater than the second priority corresponding to the service start state, the second priority corresponding to the service start state is greater than the first priority corresponding to the service plan state, and the first priority corresponding to the service plan state is greater than the fourth priority corresponding to the service completion state. That is to say, when the adaptive scheduling apparatus determines the corresponding scheduling decision policy based on the target service state, it may also determine which service state is to be processed preferentially according to the priorities corresponding to different service states in the target service state, and preferentially configure the corresponding scheduling decision policy for which service state.
202. And the self-adaptive scheduling device performs data resource scheduling processing on the target service state to obtain a scheduling decision strategy.
In this embodiment, after obtaining the target service state, the adaptive scheduling apparatus may perform data resource scheduling processing on the target service state, so as to obtain a scheduling decision policy matched with the target service state, where the scheduling decision policy may be used to schedule the data storage configuration information, such as performing operations of capacity expansion, offline, new increase, capacity reduction, or migration on the data storage configuration information.
Optionally, in other embodiments, the data resource scheduling processing is performed on the target service state by the adaptive scheduling apparatus, so that when the scheduling decision policy is used, the scheduling decision policy may be performed in the following manner, that is:
the self-adaptive scheduling device decomposes the target service state to obtain service storage requirement information;
and the self-adaptive scheduling device determines a scheduling decision strategy based on the service storage requirement information.
In this embodiment, the process of decomposing the target service state by the adaptive scheduling apparatus may be understood as mainly digitizing the service demand included in the target service state, so as to decompose the service demand into information that can be processed by a computer and converted into service storage demand information, for example: data storage location, data migration time, data online time, or number of data copies, etc. The adaptive scheduling apparatus can generate a scheduling decision policy based on the service storage requirement information, so as to perform a scheduling decision on the data storage configuration information, for example: the adaptive scheduling device may monitor the service index in real time, for example: when data storage resources are insufficient, the adaptive scheduling device can generate a capacity-expanded scheduling decision strategy based on the service storage demand information, so that data storage capacity expansion is performed on data storage configuration information based on the capacity-expanded scheduling decision strategy; of course, when the data storage resources are greatly satisfied, the data storage configuration information may also be subjected to data storage capacity reduction and the like based on a capacity reduction scheduling decision policy.
It should be understood that the scheduling decision policy may include, in addition to the aforementioned capacity expansion and capacity reduction, a new increase, a new offline, and the like, and is not limited in this embodiment of the application.
Optionally, in another embodiment, the decomposing the target service state by the adaptive scheduling apparatus to obtain service storage requirement information includes:
the self-adaptive scheduling device processes the target service state through an access amount pre-estimation model or a space pre-estimation model so as to calculate the data storage position and the data copy number;
the self-adaptive scheduling device processes the target service state through the access amount pre-estimation model or the time pre-estimation model so as to calculate data migration time and data online time;
and the self-adaptive scheduling device determines the data storage position, the data copying quantity, the data migration time and the data online time as service storage requirement information.
That is, it is understood that, in the process of decomposing the target service state, the adaptive scheduling apparatus may decompose based on three processing manners, i.e., the access amount, the space, and the time, to obtain the estimated amount of the data storage resource of the target service requirement in different dimensions, so that the estimated amount of the data storage resource of different dimensions is used as the service storage requirement information.
Each processing mode has a corresponding pre-estimation model, and the self-adaptive scheduling device can process the target service state through the access amount pre-estimation model and the space pre-estimation model so as to calculate the data storage position and the data copy number corresponding to the target service state; and the data migration time and the data online time corresponding to the target service state can be calculated through the access quantity estimation model and the time estimation model. Thus, the adaptive scheduling device can determine the data storage positions, the data copy number, the data migration time and the data online time as the service storage requirement information. Specifically, fig. 3 may be referred to as a processing flow chart of the data resource scheduling process provided in the embodiment of the present application. As can be seen from fig. 3, after the adaptive scheduling apparatus obtains the target service state, the adaptive scheduling apparatus may first decompose the target service state, so as to calculate the data storage location and the data copy number based on the access amount estimation and the space estimation, and calculate the data migration time and the data online time based on the access amount estimation and the time estimation, so that the scheduling decision policy may be determined based on the data storage location, the data copy number, the data migration time, and the data online time.
It can be understood that the access quantity pre-estimation model, the space pre-estimation model and the time pre-estimation model are pre-set pre-estimation models, wherein the access quantity pre-estimation model mainly determines data storage positions, data copying quantity, data migration time and data online time from the dimensionality of access quantity, the space pre-estimation model mainly determines the data storage positions and the data copying quantity from the dimensionality of space, and the time pre-estimation model mainly determines the data migration time and the data online time from the dimensionality of time.
203. The adaptive scheduling device determines a data routing table based on a scheduling decision policy.
In this embodiment, because the scheduling decision policy may be used to schedule the data storage configuration information, for example, the data storage configuration information may be subjected to operations such as capacity expansion, offline, new increase, capacity reduction, or migration, and the scheduling process of the data storage configuration information may be reflected in a change of the data routing table, after the adaptive scheduling apparatus obtains the scheduling decision policy, the adaptive scheduling apparatus may determine the data routing table, where the data routing table is a dynamic routing table and includes the data storage configuration information, and the data storage configuration information may be used by the data storage apparatus to determine the access resource matching the target service state.
204. The adaptive scheduling means transmits the data routing table to the data storage means.
In this embodiment, after obtaining the data routing table, the adaptive scheduling apparatus may send the data routing table to the data storage apparatus through the data scheduling instruction, so that the data storage apparatus can receive the corresponding data routing table. Reference may be made to fig. 4, which is a schematic diagram of a data routing table provided in the embodiment of the present application. It should be understood that, in practical application, a data routing table corresponding to other target service requirements may also be included, and the specific details are not limited in this embodiment of the present application.
205. The data storage device determines an access resource matching the target traffic state based on the data routing table.
In this embodiment, since only the data storage configuration information is recorded in the data routing table, and the corresponding access resource is not stored, after receiving the data routing table, the data storage device may dynamically switch to a different data storage based on the data storage configuration information in the data routing table, so as to obtain the corresponding access resource from the different data storage. In addition, fig. 5 is also referred to, which is a data processing flow chart provided in the embodiment of the present application. As can be seen from fig. 5, the adaptive scheduler receives the service plan state and the service start state. After the state and the service completion state are also performed, the data routing table is obtained by performing adaptive data resource scheduling processing on the service states, and the data routing table is issued to the data storage device through a data scheduling instruction, for example: physical storage of data or cloud storage.
In the embodiment of the application, after the adaptive scheduling device performs data resource scheduling processing on the target service state to obtain a scheduling decision strategy, the adaptive scheduling device can determine a data routing table based on the scheduling decision strategy and send the data routing table to the data storage device, so that the data storage device determines the access resource matched with the target service state based on the data routing table. In the embodiment, based on the scheduling decision strategy, the capacity of the frequently-used data storage is expanded and the corresponding data storage configuration is increased, and for the data storage which is not frequently used, the capacity is reduced and the data storage configuration is reduced, so that the storage cost is reduced.
The scheme provided by the embodiment of the application is mainly introduced from the perspective of a method. It is to be understood that the hardware structure and/or software modules for performing the respective functions are included to realize the above functions. Those of skill in the art will readily appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiment of the present application, functional modules of the apparatus may be divided according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and there may be another division manner in actual implementation.
Referring to fig. 6, fig. 6 is a schematic diagram of an embodiment of the adaptive scheduler 60 provided in the embodiment of the present application, where the adaptive scheduler 60 in the embodiment of the present application is described in detail below, and the adaptive scheduler 60 may include:
an obtaining unit 601, configured to obtain a target service state;
a processing unit 602, configured to perform data resource scheduling processing on the target service state to obtain a scheduling decision policy;
a determining unit 603, configured to determine a data routing table according to the scheduling decision policy;
a sending unit 604, configured to send the data routing table to a data storage device, so that the data storage device determines, based on the data routing table, an access resource matching the target service status.
Optionally, on the basis of the embodiment corresponding to fig. 6, in another embodiment of the adaptive scheduling apparatus 60 provided in the embodiment of the present application, the target service state includes a service plan state, a service start state, a service progress state, or a service completion state.
Optionally, on the basis of the optional embodiment corresponding to fig. 6, in another embodiment of the adaptive scheduling device 60 provided in this embodiment of the present application, the determining unit 603 is further configured to determine, after obtaining a target service state, a first priority of the service plan state, a second priority of the service starting state, a third priority of the service proceeding state, or a fourth priority of the service completing state, where the third priority is greater than the second priority, the second priority is greater than the first priority, the first priority is greater than the fourth priority, the first priority is used to indicate a priority requirement of the service plan state for the scheduling decision policy, and the second priority is used to indicate a priority requirement of the service starting state for the scheduling decision policy, the third priority is used for indicating the priority requirement of the service progress state for the scheduling decision policy, and the fourth priority is used for indicating the priority requirement of the service completion state for the scheduling decision policy.
Optionally, on the basis of fig. 6 and the optional embodiment, in another embodiment of the adaptive scheduling apparatus 60 provided in this embodiment of the present application, the processing unit 602 includes:
the decomposition module is used for decomposing the target service state to obtain service storage requirement information;
and the determining module is used for determining a scheduling decision strategy according to the service storage requirement information.
Optionally, on the basis of fig. 6 and the optional embodiment, in another embodiment of the adaptive scheduling apparatus 60 provided in this embodiment of the present application, the determining module is configured to process the target service state through an access amount prediction model or a space prediction model to calculate a data storage location and a data replication number;
the determining module is used for processing the target service state through the access amount pre-estimation model or the time pre-estimation model so as to calculate data migration time and data online time;
the determining module is used for determining the data storage position, the data copying quantity, the data migration time and the data online time as the service storage requirement information.
The adaptive scheduling apparatus 60 in the embodiment of the present application is mainly described above from the perspective of a modular functional entity, and the data storage apparatus 70 in the embodiment of the present application will be described below from the perspective of a modular functional entity. Referring to fig. 7, a schematic diagram of an embodiment of a data storage device 70 according to the present application is provided. The data storage device 70 may include:
a receiving unit 701, configured to receive a data routing table sent by an adaptive scheduling apparatus, where the data routing table is determined by a scheduling decision policy obtained by the adaptive scheduling apparatus after performing data resource scheduling processing on a target service state;
a determining unit 702, configured to determine, based on the data routing table, an access resource matching the target service state.
The adaptive scheduling apparatus 60 and the data storage apparatus 70 in the embodiment of the present application are described above from the perspective of a modular functional entity, and the computer device in the embodiment of the present application is described below from the perspective of hardware processing. Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure, where the computer device may include the adaptive scheduling apparatus 60 or the data storage apparatus 70 described above, and the computer device may have a relatively large difference due to different configurations or performances, and the computer device may include at least one processor 801, a communication line 807, a memory 803, and at least one communication interface 804.
The processor 801 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more ICs for controlling the execution of programs in accordance with the present invention.
The communication link 807 may include a path that conveys information between the aforementioned components.
The communication interface 804 may be any device, such as a transceiver, for communicating with other devices or communication networks, such as ethernet, Radio Access Network (RAN), wireless local area networks (W L AN), etc.
The memory 803 may be a read-only memory (ROM) or other type of static storage device that may store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that may store information and instructions, which may be separate and coupled to the processor via a communication line 807. The memory may also be integral to the processor.
The memory 803 is used for storing computer-executable instructions for executing the present invention, and is controlled by the processor 801. The processor 801 is configured to execute computer-executable instructions stored in the memory 803, thereby implementing the data processing method provided by the above-described embodiments of the present application.
Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application program codes, which are not specifically limited in the embodiments of the present application.
In particular implementations, the computer device may include multiple processors, such as processor 801, processor 802 in fig. 8, for example, as an embodiment. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
In particular implementations, the computer device may also include an output device 805 and an input device 806, as one embodiment. The output device 805 is in communication with the processor 801 and may display information in a variety of ways. The input device 806 is in communication with the processor 801 and may receive user input in a variety of ways. For example, the input device 806 may be a mouse, a touch screen device, or a sensing device, among others.
The computer apparatus described above may be a general-purpose device or a special-purpose device. In particular implementations, the computer device may be a desktop, laptop, nas server, wireless end device, embedded device, or a device with a similar structure as in fig. 8. The embodiment of the application does not limit the type of the computer equipment.
In the embodiment of the present application, the processor 801 included in the computer device further has the following functions:
acquiring a target service state;
performing data resource scheduling processing on the target service state to obtain a scheduling decision strategy;
determining a data routing table based on the scheduling decision policy;
sending the data routing table to a data storage device to enable the data storage device to determine an access resource matched with the target service state based on the data routing table.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the unit is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A method of data processing, comprising:
acquiring a target service state;
performing data resource scheduling processing on the target service state to obtain a scheduling decision strategy;
determining a data routing table based on the scheduling decision policy;
sending the data routing table to a data storage device to enable the data storage device to determine an access resource matched with the target service state based on the data routing table.
2. The method of claim 1, wherein the target business state comprises a business plan state, a business initiation state, a business progress state, or a business completion state.
3. The method of claim 2, wherein after obtaining the target traffic state, the method further comprises:
determining a first priority of the service plan status, a second priority of the service activation status, a third priority of the service progress status, or a fourth priority of the service completion status, wherein the third priority is greater than the second priority, which is greater than the first priority, the first priority is greater than the fourth priority, the first priority indicating a priority requirement of the business plan state for the scheduling decision policy, the second priority is used to indicate a priority requirement of the traffic initiation state for the scheduling decision policy, the third priority is used to indicate the priority requirement of the traffic progress status for the scheduling decision policy, the fourth priority is used to indicate a priority requirement of the service completion status for the scheduling decision policy.
4. The method according to any of claims 1-3, wherein the performing data resource scheduling processing on the target traffic state to obtain a scheduling decision policy comprises:
decomposing the target service state to obtain service storage requirement information;
and determining a scheduling decision strategy based on the service storage requirement information.
5. The method of claim 4, wherein decomposing the target service state to obtain service storage requirement information comprises:
processing the target service state through an access amount estimation model or a space estimation model to calculate a data storage position and a data copying number;
processing the target service state through the access amount pre-estimation model or the time pre-estimation model to calculate data migration time and data online time;
and determining the data storage position, the data copying quantity, the data migration time and the data online time as service storage requirement information.
6. A method of data processing, comprising:
receiving a data routing table sent by an adaptive scheduling device, wherein the data routing table is determined by a scheduling decision strategy obtained by the adaptive scheduling device after performing data resource scheduling processing on a target service state;
and determining the access resource matched with the target service state based on the data routing table.
7. An adaptive scheduling apparatus, comprising:
the acquisition unit is used for acquiring a target service state;
the processing unit is used for carrying out data resource scheduling processing on the target service state so as to obtain a scheduling decision strategy;
a determining unit, configured to determine a data routing table according to the scheduling decision policy;
a sending unit, configured to send the data routing table to a data storage device, so that the data storage device determines, based on the data routing table, an access resource that matches the target service state.
8. A data storage device, comprising:
the receiving unit is used for receiving a data routing table sent by the self-adaptive scheduling device, and the data routing table is determined by a scheduling decision strategy obtained by the self-adaptive scheduling device after data resource scheduling processing is carried out on a target service state;
and the determining unit is used for determining the access resource matched with the target service state based on the data routing table.
9. A computer device, characterized in that the computer device comprises: an input/output (I/O) interface, a processor and a memory,
the memory has stored therein program instructions;
the processor is configured to execute program instructions stored in the memory to perform the method of any of claims 1-5 or 6.
10. A computer-readable storage medium comprising instructions that, when executed on a computer device, cause the computer device to perform the method of one of claims 1-5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010222821.5A CN111443870B (en) | 2020-03-26 | 2020-03-26 | Data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010222821.5A CN111443870B (en) | 2020-03-26 | 2020-03-26 | Data processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111443870A true CN111443870A (en) | 2020-07-24 |
| CN111443870B CN111443870B (en) | 2021-08-03 |
Family
ID=71649051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010222821.5A Active CN111443870B (en) | 2020-03-26 | 2020-03-26 | Data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111443870B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112394882A (en) * | 2020-11-17 | 2021-02-23 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN113269514A (en) * | 2021-05-13 | 2021-08-17 | 企家有道网络技术(北京)有限公司 | Enterprise health degree measuring method, device and system |
| CN114710334A (en) * | 2022-03-23 | 2022-07-05 | 平安付科技服务有限公司 | Access policy adjustment method and system for server |
| CN115102898A (en) * | 2022-06-01 | 2022-09-23 | 中国联合网络通信集团有限公司 | Communication method, apparatus and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1588946A (en) * | 2004-09-14 | 2005-03-02 | 北京邮电大学 | Managing system for providing service quality assuranced on internet and its realizing method |
| CN102487397A (en) * | 2010-12-02 | 2012-06-06 | 中兴通讯股份有限公司 | Method and node for storing and routing data on basis of node bottom layer security level |
| CN103067433A (en) * | 2011-10-24 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method, device and system of data migration of distributed type storage system |
| US20150067252A1 (en) * | 2013-08-29 | 2015-03-05 | International Business Machines Corporation | Communicating outstanding maintenance tasks to improve disk data integrity |
| CN105516242A (en) * | 2015-11-23 | 2016-04-20 | 华为技术有限公司 | Storage resource distribution method and storage resource distribution system |
| CN107885594A (en) * | 2016-09-30 | 2018-04-06 | 腾讯科技（深圳）有限公司 | Distributed resource scheduling method, scheduling node and access node |
| CN108768877A (en) * | 2018-07-20 | 2018-11-06 | 网宿科技股份有限公司 | A kind of distribution method of burst flow, device and proxy server |
| CN109684078A (en) * | 2018-12-05 | 2019-04-26 | 苏州思必驰信息科技有限公司 | Resource dynamic distributing method and system for spark streaming |
| CN110636632A (en) * | 2019-09-29 | 2019-12-31 | 京信通信系统（中国）有限公司 | User scheduling method, device, base station and storage medium |
-
2020
- 2020-03-26 CN CN202010222821.5A patent/CN111443870B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1588946A (en) * | 2004-09-14 | 2005-03-02 | 北京邮电大学 | Managing system for providing service quality assuranced on internet and its realizing method |
| CN102487397A (en) * | 2010-12-02 | 2012-06-06 | 中兴通讯股份有限公司 | Method and node for storing and routing data on basis of node bottom layer security level |
| CN103067433A (en) * | 2011-10-24 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method, device and system of data migration of distributed type storage system |
| US20150067252A1 (en) * | 2013-08-29 | 2015-03-05 | International Business Machines Corporation | Communicating outstanding maintenance tasks to improve disk data integrity |
| CN105516242A (en) * | 2015-11-23 | 2016-04-20 | 华为技术有限公司 | Storage resource distribution method and storage resource distribution system |
| CN107885594A (en) * | 2016-09-30 | 2018-04-06 | 腾讯科技（深圳）有限公司 | Distributed resource scheduling method, scheduling node and access node |
| CN108768877A (en) * | 2018-07-20 | 2018-11-06 | 网宿科技股份有限公司 | A kind of distribution method of burst flow, device and proxy server |
| CN109684078A (en) * | 2018-12-05 | 2019-04-26 | 苏州思必驰信息科技有限公司 | Resource dynamic distributing method and system for spark streaming |
| CN110636632A (en) * | 2019-09-29 | 2019-12-31 | 京信通信系统（中国）有限公司 | User scheduling method, device, base station and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112394882A (en) * | 2020-11-17 | 2021-02-23 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN113269514A (en) * | 2021-05-13 | 2021-08-17 | 企家有道网络技术(北京)有限公司 | Enterprise health degree measuring method, device and system |
| CN114710334A (en) * | 2022-03-23 | 2022-07-05 | 平安付科技服务有限公司 | Access policy adjustment method and system for server |
| CN115102898A (en) * | 2022-06-01 | 2022-09-23 | 中国联合网络通信集团有限公司 | Communication method, apparatus and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111443870B (en) | 2021-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111443870B (en) | Data processing method, device and storage medium | |
| US11003492B2 (en) | Virtual machine consolidation | |
| US10069749B1 (en) | Method and apparatus for disaggregated overlays via application services profiles | |
| KR101994506B1 (en) | Decoupling paas resources, jobs, and scheduling | |
| KR101976234B1 (en) | Paas hierarchial scheduling and auto-scaling | |
| JP2020064676A (en) | Opportunistic resource migration to optimize resource placement | |
| US20180052714A1 (en) | Optimized resource metering in a multi tenanted distributed file system | |
| US20140025828A1 (en) | Extending processing capacity of server | |
| KR101471749B1 (en) | Virtual machine allcoation of cloud service for fuzzy logic driven virtual machine resource evaluation apparatus and method | |
| JP2010277589A (en) | Data center batch job quality of service control | |
| US20050132379A1 (en) | Method, system and software for allocating information handling system resources in response to high availability cluster fail-over events | |
| Adhikary et al. | Quality of service aware cloud resource provisioning for social multimedia services and applications | |
| JP2021504780A (en) | Application Prioritization for Automatic Diagonal Scaling in a Distributed Computing Environment | |
| EP3274859B1 (en) | Cluster computing service assurance apparatus and method | |
| Alarifi et al. | A fault-tolerant aware scheduling method for fog-cloud environments | |
| KR20200080458A (en) | Cloud multi-cluster apparatus | |
| CN114461376A (en) | Sharing enterprise clouds | |
| JPWO2005116832A1 (en) | Computer system, method and program for controlling job execution in a distributed processing environment | |
| Li et al. | PageRankVM: A pagerank based algorithm with anti-collocation constraints for virtual machine placement in cloud datacenters | |
| Yakubu et al. | Service level agreement violation preventive task scheduling for quality of service delivery in cloud computing environment | |
| CN116414518A (en) | Data locality of big data on Kubernetes | |
| US20130219230A1 (en) | Data center job scheduling | |
| Pandya et al. | Dynamic resource allocation techniques in cloud computing | |
| KR101125038B1 (en) | Method and apparatus for providing grid resource management in grid computing middleware system | |
| JP2021504777A (en) | Automatic diagonal scaling of workloads in a distributed computing environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |