CN110460682A - A kind of data processing method, device, system and storage medium - Google Patents
A kind of data processing method, device, system and storage medium Download PDFInfo
- Publication number
- CN110460682A CN110460682A CN201910883498.3A CN201910883498A CN110460682A CN 110460682 A CN110460682 A CN 110460682A CN 201910883498 A CN201910883498 A CN 201910883498A CN 110460682 A CN110460682 A CN 110460682A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- data file
- service equipment
- deployment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
Abstract
The invention discloses a kind of data processing method, device, system and storage mediums.The step of this method includes: the work quality parameter for collecting service equipment, and carries out analysis to work quality parameter and generate status information；The file temperature information of data file is obtained, and calculates the deployment requirements of data file according to file temperature information；The destination service equipment for meeting deployment requirements is chosen according to status information；Data file is deployed to destination service equipment.Data file is accurately deployed in the service equipment that matches by this method according to the availability of service equipment, and then opposite can ensure that the ability of service equipment offer data file is consistent with requirements for access of the user to data file.In addition, the present invention also provides a kind of data processing equipment, system and storage medium, beneficial effect is same as above.
Description
Technical field
The present invention relates to field of cloud calculation, more particularly to a kind of data processing method, device, system and storage medium.
Background technique
With the continuous development of cloud computing, the cloud server framework based on CDN mode constantly obtains essence in application aspect
Property progress, be currently based on the main purposes of cloud server framework of CDN mode according to the requirements for access of user first is that provide
Corresponding data file.In order to further increase the magnitude of back end under CDN network mode, it is currently based on CDN network
The Cloud Server frame that the service equipment of mode is constituted, is increasingly becoming current development trend.
In cloud server framework, how to ensure that data file is accurately deployed in the service equipment that matches, and then ensure
The demand to data file of ability and user that service equipment provides data file is coincide, and is asking of mainly facing of current this field
Topic.
Summary of the invention
The object of the present invention is to provide a kind of data processing method, device, system and storage mediums, to ensure data file
It is accurately deployed in the service equipment that matches, and then the ability for ensuring service equipment offer data file and user are to data text
The demand of part is coincide.
In order to solve the above technical problems, the present invention provides a kind of data processing method, comprising:
The work quality parameter of service equipment is collected, and analysis is carried out to work quality parameter and generates status information；
The file temperature information of data file is obtained, and calculates the deployment requirements of data file according to file temperature information；
The destination service equipment for meeting deployment requirements is chosen according to status information；
Data file is deployed to destination service equipment.
Preferably, work quality parameter includes survival quality parameter, TCP connection mass parameter and the mass parameter that burrows；
Status information includes the service status information and file deployment status information of service equipment.
Preferably, the file temperature information of data file is obtained, comprising:
Receive the file request that user equipment is passed in real time；
Statistics file request, generates the file temperature information of data file.
Preferably, statistics file is requested, comprising:
The file request of continuous duration before counting current time and current time.
Preferably, data file is deployed to destination service equipment, comprising:
Data file is deployed to destination service equipment according to the priority orders of data file accessed.
Preferably, the file temperature information of data file is obtained, and calculates the portion of data file according to file temperature information
Administration's demand, comprising:
Receive the traffic characteristic data of the data file of user equipment to report；Traffic characteristic data include the stream of data file
Measure the relationship information between fluctuation information and data file；
Using traffic characteristic data as file temperature information input to preset characteristic matching deployment model, data text is calculated
The deployment requirements of part.
Preferably, the deployment requirements of data file are calculated according to file temperature information, comprising:
The deployment requirements of data file are calculated with weighting scheme according to file temperature information.
In addition, device includes memory, processor and bus, memory the present invention also provides a kind of data processing equipment
On be stored with the data processor that can be run to processor and on a processor by bus transfer, data processor is processed
Such as above-mentioned data processing method is realized when device executes.
In addition, the present invention also provides a kind of data processing system, system includes:
Statistic unit is analyzed for collecting the work quality parameter of service equipment, and to work quality parameter
Generate status information；
Demand computing unit calculates number for obtaining the file temperature information of data file, and according to file temperature information
According to the deployment requirements of file；
Equipment chooses module, for choosing the destination service equipment for meeting deployment requirements according to status information；
Data distributing module, for data file to be deployed to destination service equipment.
In addition, being stored with number on computer readable storage medium the present invention also provides a kind of computer readable storage medium
According to processing routine, data processor can be executed by one or more processor, to realize such as above-mentioned data processing method.
Data processing method provided by the present invention collects the work quality parameter of service equipment first, and then to work
Mass parameter carries out the status information that analysis generates service equipment, and then obtains the file temperature information of data file, and according to
File temperature information calculates the deployment requirements of the data file, and then meets deployment need according to the selection of the status information of service equipment
The destination service equipment asked, and data file is deployed to destination service equipment.Since this method is first to the work of service equipment
It is collected and analyzes as mass parameter, generate the status information of characterization service equipment entirety operating condition, and then according to service equipment
Status information selection be able to carry out the destination service equipment of data file deployment, and carry out the deployment of data file, be equivalent to
According to the availability of service equipment, data file is accurately deployed in the service equipment that matches, and then opposite can be ensured
The ability that service equipment provides data file is consistent with requirements for access of the user to data file.In addition, the present invention also provides one
Kind data processing equipment, system and storage medium, beneficial effect are same as above.
Detailed description of the invention
In order to illustrate the embodiments of the present invention more clearly, attached drawing needed in the embodiment will be done simply below
It introduces, it should be apparent that, drawings in the following description are only some embodiments of the invention, for ordinary skill people
For member, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow chart of data processing method provided in an embodiment of the present invention；
Fig. 2 is the flow chart of another data processing method provided in an embodiment of the present invention；
Fig. 3 is the flow chart of another data processing method provided in an embodiment of the present invention；
Fig. 4 is the configuration diagram of the scheduling deployment system under a kind of concrete scene provided in an embodiment of the present invention；
Fig. 5 is a kind of structure chart of data processing equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, rather than whole embodiments.Based on this
Embodiment in invention, those of ordinary skill in the art are without making creative work, obtained every other
Embodiment belongs to the scope of the present invention.
In cloud server framework, how to ensure that data file is accurately deployed in the service equipment that matches, and then ensure
The demand to data file of ability and user that service equipment provides data file is coincide, and is asking of mainly facing of current this field
Topic.
Core of the invention is to provide a kind of data processing method, device, system and storage medium, to ensure data file
It is accurately deployed in the service equipment that matches, and then the ability for ensuring service equipment offer data file and user are to data text
The demand of part is coincide.
In order to enable those skilled in the art to better understand the solution of the present invention, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.
Fig. 1 is a kind of flow chart of data processing method provided in an embodiment of the present invention.Referring to FIG. 1, at the data
Reason method can be applied to the deployment services device in CDN, also can be applied in the equipment that other have deployment function.The data
The specific steps of processing method include:
Step S10: the work quality parameter of service equipment is collected, and analysis generation state letter is carried out to work quality parameter
Breath.
It should be noted that the service equipment in this step refers to that data file service can be provided to user equipment
Equipment can store the data file of deployment server disposition specifically, the service equipment can be the node device of CDN,
And the user terminal that the transmitting data file of storage can be serviced to needs.The service equipment may include play objective cloud, light cat,
Router, TV box, smart television, IPFS mine machine, network attached storage (NAS), PC and server etc. are all types of
Electronic equipment.Since the difference of service equipment hardware performance itself decides the data file that service equipment itself is capable of providing
Service ability it is also different, and the work quality parameter of this collection step service equipment, it is therefore an objective to be joined according to work quality
The work state information that number generates service equipment carries out, and what work state information characterized is the working performance of service equipment.
Step S11: obtaining the file temperature information of data file, and the portion of data file is calculated according to file temperature information
Administration's demand.
The purpose of this step is to obtain the file temperature information of data file, and then calculate needs according to file temperature information
To the quantity that data file is disposed, the i.e. deployment requirements of data file.The file temperature information obtained in this step can be with
For the amount of access information for counting generation for the access of data file according to user equipment, file temperature information representation is to use
Family equipment is for the demand of data file, and the data file high to file demand is disposed in service equipment, can
It is opposite to ensure user equipment for the access hit rate of data file in service equipment.
Step S12: the destination service equipment for meeting deployment requirements is chosen according to status information.
It is that service equipment can carry deployment for data file due to status information characterization, and deployment requirements then table
Sign is requirements for access of the user equipment for data file, therefore status information decides that can service equipment dispose data text
Part is to meet user equipment for the requirements for access of data file, and then this step meets deployment requirements according to status information selection
Destination service equipment.For example, the requirements for access of data file added up to be 100,000 times within the unit time, then to data file into
Row deployment when it is ensured that service equipment can normal response user terminal data file is carried out 100,000 times within the unit time
Access, it is therefore desirable to according to the status information of service equipment judge service equipment whether have to user terminal provide unit when
The abilities of interior 100,000 services, and then select the destination service equipment that can satisfy deployment requirements.
Step S13: data file is deployed to destination service equipment.
Data file is deployed to by this step to be met in the destination service equipment for meeting data file deployment requirements, really with this
It protects destination service equipment and normally provides the data file to user terminal.
The execution object of each step can be scheduling deployment equipment in the present embodiment, and controlling equipment and service equipment are established
There is communication connection, can realize the deployment of data file to service equipment with this.
Data processing method provided by the present invention collects the work quality parameter of service equipment first, and then to work
Mass parameter carries out the status information that analysis generates service equipment, and then obtains the file temperature information of data file, and according to
File temperature information calculates the deployment requirements of the data file, and then meets deployment need according to the selection of the status information of service equipment
The destination service equipment asked, and data file is deployed to destination service equipment.Since this method is first to the work of service equipment
It is collected and analyzes as mass parameter, generate the status information of characterization service equipment entirety operating condition, and then according to service equipment
Status information selection be able to carry out the destination service equipment of data file deployment, and carry out the deployment of data file, be equivalent to
According to the availability of service equipment, data file is accurately deployed in the service equipment that matches, and then opposite can be ensured
The ability that service equipment provides data file is consistent with requirements for access of the user to data file.
As a preferred embodiment, work quality parameter may include survival quality parameter, TCP connection quality ginseng
The several and mass parameter that burrows.Status information includes the service status information and file deployment status information of service equipment.
The survival quality parameter characterization of service equipment be service equipment self-operating reliability, the TCP of service equipment
Quality of connection parameter characterization be between service equipment and user equipment carry out transmitting data file when reliability；Burrow quality
Parameter characterization and between service equipment and user equipment when progress transmitting data file reliability, burrow connection and TCP
Connection is the two ways connected between service equipment and SDK, and SDK refers to the software being set in user equipment applications program
Development kit, burrow the connection type referred between service equipment and SDK based on udp protocol.Survival quality parameter can be with
The communication success rate of service equipment or the probability that resource deadlock or delay machine occurs of service equipment are specifically included, that is, service is set
The probability of standby normal sending and receiving data；TCP connection mass parameter can be specifically included in the data transmission efficiency under TCP connection state
And correctness of transmitted data etc.；The mass parameter that burrows can be specifically included in the biography of the data under udp protocol connection status
Defeated efficiency and the correctness of transmitted data etc..According to the survival quality parameter of service equipment, TCP connection mass parameter and
The mass parameter that burrows is able to reflect service status information and logarithm of the service equipment in terms of carrying out communication service to data file
The file deployable state in local acquisition unit management side face is carried out according to file, present embodiment is further directed to service equipment to data text
The local of part obtains deployment and communication service capabilities, has refined the particular content of work quality parameter, has improved status information
Accuracy, and then improve according to status information choose destination service equipment reliability.
Fig. 2 is the flow chart of another data processing method provided in an embodiment of the present invention.Referring to FIG. 2, data processing
The specific steps of method include:
Step S20: the work quality parameter of service equipment is collected, and analysis generation state letter is carried out to work quality parameter
Breath.
Step S21: the file request that user equipment is passed in real time is received.
Step S22: statistics file request generates the file temperature information of data file.
It should be noted that the generating mode for focusing on file temperature information of the present embodiment is based on to user equipment
Incoming file request carries out real-time statistics and generates, that is to say, that in the present embodiment, controlling equipment is set with user simultaneously
The foundation of standby and service equipment has communication connection, and then user equipment asks the file of data file to controlling equipment initiation
Ask, controlling equipment when receiving file request of the user equipment for data file, i.e., to the amount of access of the data file into
The corresponding statistics of row is believed with this according to the file temperature that file request amount of the user equipment for data file generates data file
Breath.Due to being the file request for receiving user equipment and being passed in real time, file request is carried out to count file generated
Temperature information real-time with higher.
Step S23: the deployment requirements of data file are calculated according to file temperature information.
Step S24: the destination service equipment for meeting deployment requirements is chosen according to status information.
Step S25: data file is deployed to destination service equipment.
In the present embodiment, when each user equipment is passed to file request, i.e., phase is requested according to file request pair and this document
The data file answered carries out the statistics of file temperature information, therefore real-time update when the file temperature information of data file, into
And the deployment requirements for the data file being calculated according to file temperature information are also real-time update, timeliness with higher
Property, it being capable of the opposite overall accuracy improved when being disposed for data file.
On the basis of the above embodiments, as a preferred embodiment, statistics file is requested, comprising:
The file request of continuous duration before counting current time and current time.
It should be noted that the emphasis of present embodiment is when counting to file request, it is to current statistics
The file request in continuous duration before moment and current statistics moment carries out common statistics, that is to say, that this reality
The mode of applying is counted to the file request in a period of time, and the whole feelings for having repeated file request in history cycle are equivalent to
Condition, with this learn current time and current time for the previous period in each data file by user equipment access entirety
Situation, opposite can avoid changing greatly because of the temperature of data file in the short time or due to relative distribution caused by file temperature believe
Cease the situation of inaccuracy, it is ensured that the temperature for the reflection data file that the file temperature information of data file can be relatively accurate,
Further improve the deployment accuracy for data file.
Specifically, the request number of times of statistical data file can be carried out as measurement period according to grade time second, and according to working as
The request number of times of preceding measurement period and the request number of times in several periods before current statistic assess the temperature information of file.Example
Such as, measurement period can be set as 1 second, then count the access times of data file in current 1 second, and T second data text before
The requested number of part, to obtain the temperature information of data file.The T, which can according to need, to be configured.Pass through above-mentioned side
Formula may be implemented the temperature variation of second grade granularity perception data file, promote the timeliness of data file deployment.
Further, as a preferred embodiment, data file is deployed to destination service equipment, comprising:
Data file is deployed to destination service equipment according to the priority orders of data file accessed.
It should be noted that present embodiment, which allows for, often has different significance levels between data file, this
The significance level of the data file at place refers to a possibility that data file is accessed by user.For example, watching video in user
In scene, since the viewing habit of user is often first watched since the initial time of video track, but not necessarily can be complete
Whole finishes watching all videos, that is to say, that if when video file is divided into multiple data files, video beginning pair
The significance level height for the data file answered data file corresponding with video trailer part, should first ensure that can be efficient to user
There is provided video beginning corresponding data file.
Data file higher for significance level needs user equipment that can access in the shortest time to it, number
According to file the significance level in overall data file should with priority of the data file in overall data file at just
Than, such as the significance level of tri- data files of A, B, C is divided into for A is most important, B importance is inferior to A, C importance is inferior to B, then
A, the priority orders between tri- data files of B, C are A highest priority, and B priority is inferior to A, and C priority is inferior to B.At this
In embodiment, the priority orders accessed, and then being accessed according to data file are previously provided with for data file
Priority orders data file is deployed to destination service equipment, that is, will be corresponding according to the sequence of priority from high to low
Data file be deployed to service equipment, therefore data file higher for priority orders can be visited preferentially by user equipment
It asks, opposite service equipment can be ensured for the efficiency of service of data file.
Fig. 3 is the flow chart of another data processing method provided in an embodiment of the present invention.Referring to FIG. 3, data processing
The specific steps of method include:
Step S30: the work quality parameter of service equipment is collected, and analysis generation state letter is carried out to work quality parameter
Breath.
Step S31: the traffic characteristic data of the data file of user equipment to report are received；Traffic characteristic data include data
Relationship information between the flowed fluctuation information and data file of file.
It should be noted that the present embodiment is the operation carried out by the way of offline to deployment requirements, deployment facility is connect
By the traffic characteristic data about data file of user equipment to report, traffic characteristic data include the flowed fluctuation of data file
Relationship information between information and data file, flowed fluctuation information refer to data file in a period of time by user
The whole amount of access situation of change of equipment access, the relationship information between data file refer between different data file
The incidence relation of subordinate relation or file content.
Step S32: using traffic characteristic data as file temperature information input to preset characteristic matching deployment model, meter
Calculate the deployment requirements of data file.
And then after the traffic characteristic data for the data file for receiving user equipment to report, by by traffic characteristic data
As file temperature information input to preset characteristic matching deployment model, the deployment requirements of data file are calculated.
It should be noted that the lower deployment cost that can be that the characteristic matching deployment model in the present embodiment characterizes is received with deployment
Corresponding relationship between benefit, and then mould is disposed using traffic characteristic data as file temperature information input to preset characteristic matching
Type, it is therefore an objective to using traffic characteristic data as under the premise of generating deployment requirements, calculate corresponding lower deployment cost and deployment
Income chooses lower deployment cost and disposes deployment requirements when income is in certain threshold range.
Step S33: the destination service equipment for meeting deployment requirements is chosen according to status information.
Step S34: data file is deployed to destination service equipment.
Controlling equipment receives the traffic characteristic data that user equipment is generated and reported in the present embodiment, and then is based on feature
Traffic characteristic data are analyzed with deployment model, generate corresponding deployment requirements.The present embodiment can be by generating offline
The mode of deployment requirements can be realized mentioning for data file higher for certain temperatures and the biggish data file of relevance
Preceding deployment further ensures that the accuracy to administration, data Documents Department.
Further, as a preferred embodiment, calculating the deployment of data file according to file temperature information
Demand, comprising:
The deployment requirements of data file are calculated with weighting scheme according to file temperature information.
Present embodiment calculates the deployment requirements of data file by weighting scheme according to file temperature information, being capable of basis
Corresponding weight is arranged when calculating deployment requirements in the significance level of file temperature information, with this it is more flexible realize for
The calculating of deployment requirements further ensures the accuracy disposed to data file.
In order to deepen the understanding for technical solution of the present invention, below according to the scene embodiment under a kind of concrete scene into
Row explanation:
It is as shown in Figure 4 in the configuration diagram of concrete scene dispatching deployment system.
The framework of scheduling deployment system relies on:
Data platform: being responsible for SDK (user equipment) and Peer (service equipment) and scheduling deployment system, index burrow
The data such as system report, log statistic；
Data statistics: on the basis of data platform, each system and each module related data are counted；
Data analysis: conventional index, data on flows preliminary analysis；
Data visualization: statistical data and analysis data visualization are shown；
Monitoring alarm: the monitoring of configuration index of correlation and alarm strategy, push notice and warning message etc.；
O&M console: including the related operation management platform such as classification publication, program management；
Run console: including data such as service parameter configuration, the business manually dispose triggering provided, business correlative flows
It checks, Peer resource operation etc.；
Charging: proprietary accounting module is responsible for statistics cost benefit data；
It logs in: platform of user management；
Index: index maintenance；
It burrows: P2P being assisted to burrow；
Scheduling deployment system includes 3 major parts:
Service equipment resource management:
Peer information is collected: collecting the relevant information of the Peer of scheduling and deployment index；
Peer Information Statistics: convergence statistics is carried out to Peer information；
Control information issues: issuing scheduling, deployment system and deletes control, Peer to the control command of Peer, such as index
Self-protection state modulator etc.；
Testing, which is visited, lives: Periodic probe Peer survival quality, TCP connection quality, the quality that burrows etc.；
Peer information analysis: the sharing ability and service quality, downloading deployment ability of Peer are assessed, to refine scheduling
And deployment；
On-line scheduling deployment system:
P2P scheduling: SDK on demand to scheduler module request meet demand Peer information, so as to SDK suitable opportunity into
Enter P2P sharing model；
Temperature information collection: the request record of scheduler module is collected；
Temperature converging information: using the second as granularity sliding time window maintenance request information；Various flow converges the period, converges in real time
The request record of poly- one section of continuous time in the past.To realize that second grade perception meets the file deployment requirements of temperature, at the same it is multiple
The convergence period guarantees that deployment requirements of the temperature compared with dispersion will not be missed；
Deployment requirements calculate: according to real-time temperature, weighting redundancy coefficient, calculate current reconnaissance demand for file；
Reconnaissance deployment: according to Peer information, in conjunction with multiple reconnaissance strategy, such as according to disk space reconnaissance, according to sharing
Ability reconnaissance, according to characteristic matching reconnaissance, according to configuration weight reconnaissance etc.；
Deployment task issues: will be to deployment task information, real-time & is periodical, after sorting by priority, is issued to Peer；
Index/cache management: according to Peer deployment success and failure information, index is established；Processing Peer caching, which is deleted, asks
It asks；According to disk space, global visual angle maintenance indexed cache is deleted；
Dispose resource management: control CDN returns source lower deployment cost, on the basis of not influencing Peer sharing and deployment efficiency,
Resource is disposed using P2P as far as possible；
Service traffics are collected: collecting the service traffics characteristic that SDK is reported；
Service traffics statistics: the service traffics characteristic that statistics SDK is reported；
Traffic characteristics analysis: analysis flow heat symptom-complex, the statistical information of the flowed fluctuation including All Files；Phase between file
Guan Xing；Peer service condition etc., the deployment parameters such as redundancy coefficient are arranged；Generate pre- deployment file linked character data；Instruction
Practice SDK-Peer characteristic matching deployment scheduling reconnaissance model etc.；
Pre- deployment requirements management: according to flow correlations feature, integration meets deployment requirements (this of the file of linked character
Class file may not up to dispose temperature requirement, but its associated with has met high temperature requirement, thinks associated documents at this time again
Next probability that may be hot is higher, then the deployment number that can refer to corresponding hot file is disposed in advance in advance, deployment redundancy, by
Interdependence effects, relevance is bigger, and the discount that deployment number is beaten is smaller)；
Deployment parameters training: according to the data such as traffic characteristic, Peer service condition, control deployment system redundancy coefficient (industry
Business arrange discharge model & cost control) adaptive adjustment；Characteristic matching rule etc. when control scheduling, deployment reconnaissance；
Service equipment resource management: as service equipment type increases, service differences are also increasing between service equipment,
The problems such as such as service equipment race amount caused by the scheduling deployment strategy of " making no exception " is unbalanced, and the level of resources utilization is low is more next
It is more serious；
Service equipment resource management is except conventional service equipment region-operator, type, disk, the maintenances of information such as bandwidth
Etc. outside functions, frame is newly-increased:
Reversed testing, which is visited, lives: guaranteeing Peer availability, the integrity service ability and service quality assessment of Peer, to support to tie
The service ability and service quality for closing Peer do fining scheduling and deployment；
Caching, self-protection centralization control: scheduling perception global service situation, control Peer cache cleaning with
And share upper limit control；
Online deployment system:
Online deployment system main target is guaranteed timely the file for having deployment to be worth, rapid deployment to target Peer
In；
The mode of the temperature convergence of update realizes (the more dispersed file deployment requirements of the perception temperature) second in more convergence periods
Rank responds (granularity per second converges forward)；
More reasonable HTTP and P2P coordinates deployment, balance cost and deployment efficiency；
Offline deployment system: offline deployment system has been increased newly；
To adapt to current program request business tine multiplicity, lacks temperature centrality and characteristics of SSTA persistence, use analysis business
Correlation between request, to certain hotter files, the larger problem of correlation is disposed in advance, so that sharing ratio is improved,
And according to flow heat symptom-complex, weighs quality and cost, adaptive adjustment deployment control parameter, and then analyze the request flow of business
Feature, in conjunction with cost benefit model, control deployment redundancy coefficient etc..
Hereinbefore the embodiment of data processing method is described in detail, the present invention also provides one kind and is somebody's turn to do
The corresponding data processing equipment of method, since the embodiment of data processing equipment part and the embodiment of method part are mutually right
It answers, the description of the embodiment of data processing equipment part wouldn't repeat here.
Fig. 5 is a kind of structure chart of data processing equipment provided in an embodiment of the present invention.
Referring to FIG. 5, a kind of data processing equipment 1 provided in an embodiment of the present invention includes memory 11,12 and of processor
Bus 13 is stored with the data processing journey that can be transmitted to processor 12 by bus 13 and be run on the processor 12 on memory 11
Sequence realizes such as above-mentioned data processing method when data processor is executed by processor 12.
Wherein, memory 11 includes at least a type of readable storage medium storing program for executing, and readable storage medium storing program for executing includes flash memory, hard
Disk, multimedia card, card-type memory (for example, SD or DX memory etc.), magnetic storage, disk, CD etc..Memory 11 exists
It can be the internal storage unit of data processing equipment 1, such as the hard disk of the data processing equipment 1 in some embodiments.Storage
Device 11 is also possible to match on the External memory equipment of data processing equipment 1, such as data processing equipment 1 in further embodiments
Standby plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD)
Card, flash card (Flash Card) etc..Further, memory 11 can also both include the storage inside of data processing equipment 1
Unit also includes External memory equipment.Memory 11 can be not only used for the application software that storage is installed on data processing equipment 1
And Various types of data, such as the code of video code conversion program etc., it can be also used for temporarily storing and exported or will export
Data.
Processor 12 can be in some embodiments a central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips, the program for being stored in run memory 11
Code or processing data, such as execute video code conversion program etc..
The bus 13 can be Peripheral Component Interconnect standard (peripheral component interconnect, abbreviation
PCI) bus or expanding the industrial standard structure (extended industry standard architecture, abbreviation EISA)
Bus etc..The bus can be divided into address bus, data/address bus, control bus etc..For convenient for indicating, in Fig. 5 only with one slightly
Line indicates, it is not intended that an only bus or a type of bus.
Data processing equipment provided by the present invention collects the work quality parameter of service equipment first, and then to work
Mass parameter carries out the status information that analysis generates service equipment, and then obtains the file temperature information of data file, and according to
File temperature information calculates the deployment requirements of the data file, and then meets deployment need according to the selection of the status information of service equipment
The destination service equipment asked, and data file is deployed to destination service equipment.Since the present apparatus is first to the work of service equipment
It is collected and analyzes as mass parameter, generate the status information of characterization service equipment entirety operating condition, and then according to service equipment
Status information selection be able to carry out the destination service equipment of data file deployment, and carry out the deployment of data file, be equivalent to
According to the availability of service equipment, data file is accurately deployed in the service equipment that matches, and then opposite can be ensured
The ability that service equipment provides data file is consistent with requirements for access of the user to data file.
The present invention also provides a kind of data processing system, system includes:
Statistic unit is analyzed for collecting the work quality parameter of service equipment, and to work quality parameter
Generate status information；
Demand computing unit calculates number for obtaining the file temperature information of data file, and according to file temperature information
According to the deployment requirements of file；
Equipment chooses module, for choosing the destination service equipment for meeting deployment requirements according to status information；
Data distributing module, for data file to be deployed to destination service equipment.
Data processing system provided by the present invention collects the work quality parameter of service equipment first, and then to work
Mass parameter carries out the status information that analysis generates service equipment, and then obtains the file temperature information of data file, and according to
File temperature information calculates the deployment requirements of the data file, and then meets deployment need according to the selection of the status information of service equipment
The destination service equipment asked, and data file is deployed to destination service equipment.Since this system is first to the work of service equipment
It is collected and analyzes as mass parameter, generate the status information of characterization service equipment entirety operating condition, and then according to service equipment
Status information selection be able to carry out the destination service equipment of data file deployment, and carry out the deployment of data file, be equivalent to
According to the availability of service equipment, data file is accurately deployed in the service equipment that matches, and then opposite can be ensured
The ability that service equipment provides data file is consistent with requirements for access of the user to data file.
In addition, being stored with number on computer readable storage medium the present invention also provides a kind of computer readable storage medium
According to processing routine, data processor can be executed by one or more processor, to realize such as above-mentioned data processing method.
Computer readable storage medium provided by the present invention collects the work quality parameter of service equipment, in turn first
The status information that analysis generates service equipment is carried out to work quality parameter, and then obtains the file temperature information of data file,
And the deployment requirements of the data file are calculated according to file temperature information, and then meet according to the selection of the status information of service equipment
The destination service equipment of deployment requirements, and data file is deployed to destination service equipment.Due to this computer-readable storage medium
Matter is collected and analyzes to the work quality parameter of service equipment first, generates the state letter of characterization service equipment entirety operating condition
Breath, and then the destination service equipment that data file is disposed, line number of going forward side by side are able to carry out according to the selection of the status information of service equipment
According to the deployment of file, it is equivalent to the availability according to service equipment, data file is accurately deployed to the service equipment that matches
In, and then opposite can ensure that the ability of service equipment offer data file is consistent with requirements for access of the user to data file.
In the above-described embodiments, can come wholly or partly by software, hardware, firmware or any combination thereof real
It is existing.When implemented in software, it can entirely or partly realize in the form of a computer program product.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and module, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed system, device and method can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the division of module,
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple module or components can be with
In conjunction with or be desirably integrated into another system, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING of device or module or
Communication connection can be electrical property, mechanical or other forms.
Module may or may not be physically separated as illustrated by the separation member, show as module
Component may or may not be physical module, it can and it is in one place, or may be distributed over multiple networks
In module.Some or all of the modules therein can be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It should be noted that the serial number of the above embodiments of the invention is only for description, do not represent the advantages or disadvantages of the embodiments.And
The terms "include", "comprise" herein or any other variant thereof is intended to cover non-exclusive inclusion, so that packet
Process, device, article or the method for including a series of elements not only include those elements, but also including being not explicitly listed
Other element, or further include for this process, device, article or the intrinsic element of method.Do not limiting more
In the case where, the element that is limited by sentence "including a ...", it is not excluded that including process, device, the article of the element
Or there is also other identical elements in method.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data processing method characterized by comprising
The work quality parameter of service equipment is collected, and analysis is carried out to the work quality parameter and generates status information；
The file temperature information of data file is obtained, and calculates the deployment need of the data file according to the file temperature information
It asks；
The destination service equipment for meeting the deployment requirements is chosen according to the state information；
The data file is deployed to destination service equipment.
2. data processing method according to claim 1, which is characterized in that the work quality parameter includes survival quality
Parameter, TCP connection mass parameter and the mass parameter that burrows；
The status information includes the service status information and file deployment status information of the service equipment.
3. data processing method according to claim 1, which is characterized in that the file temperature letter for obtaining data file
Breath, comprising:
Receive the file request that user equipment is passed in real time；
The file request is counted, the file temperature information of the data file is generated.
4. data processing method according to claim 3, which is characterized in that the statistics file request, comprising:
The file request of continuous duration before counting current time and the current time.
5. data processing method according to claim 4, which is characterized in that described that the data file is deployed to target
Service equipment, comprising:
The data file is deployed to the destination service equipment according to the priority orders of the data file accessed.
6. data processing method according to claim 1, which is characterized in that the file temperature letter for obtaining data file
It ceases, and calculates the deployment requirements of the data file according to the file temperature information, comprising:
Receive the traffic characteristic data of the data file of user equipment to report；The traffic characteristic data include the data
Relationship information between the flowed fluctuation information and the data file of file；
Using the traffic characteristic data as the file temperature information input to preset characteristic matching deployment model, institute is calculated
State the deployment requirements of data file.
7. according to claim 1 to data processing method described in 6 any one, which is characterized in that described according to the file
Temperature information calculates the deployment requirements of the data file, comprising:
The deployment requirements of the data file are calculated with weighting scheme according to the file temperature information.
8. a kind of data processing equipment, which is characterized in that described device includes memory, processor and bus, the memory
On be stored with the data processor that can be run to the processor and on the processor by bus transfer, at the data
Reason program realizes the data processing method as described in any one of claims 1 to 7 when being executed by the processor.
9. a kind of data processing system, which is characterized in that the system comprises:
Statistic unit is analyzed for collecting the work quality parameter of service equipment, and to the work quality parameter
Generate status information；
Demand computing unit calculates institute for obtaining the file temperature information of data file, and according to the file temperature information
State the deployment requirements of data file；
Equipment chooses module, for choosing the destination service equipment for meeting the deployment requirements according to the state information；
Data distributing module, for the data file to be deployed to destination service equipment.
10. a kind of computer readable storage medium, which is characterized in that be stored at data on the computer readable storage medium
Program is managed, the data processor can be executed by one or more processor, to realize as any in claim 1 to 7
Data processing method described in.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910883498.3A CN110460682A (en) | 2019-09-18 | 2019-09-18 | A kind of data processing method, device, system and storage medium |
| PCT/CN2020/090810 WO2021051839A1 (en) | 2019-09-18 | 2020-05-18 | Data processing method, apparatus and system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910883498.3A CN110460682A (en) | 2019-09-18 | 2019-09-18 | A kind of data processing method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110460682A true CN110460682A (en) | 2019-11-15 |
Family
ID=68492402
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910883498.3A Pending CN110460682A (en) | 2019-09-18 | 2019-09-18 | A kind of data processing method, device, system and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110460682A (en) |
| WO (1) | WO2021051839A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111770180A (en) * | 2020-06-29 | 2020-10-13 | 百度在线网络技术（北京）有限公司 | Deployment method, device, equipment and storage medium |
| WO2021051839A1 (en) * | 2019-09-18 | 2021-03-25 | 深圳市网心科技有限公司 | Data processing method, apparatus and system and storage medium |
| CN113312329A (en) * | 2020-02-26 | 2021-08-27 | 阿里巴巴集团控股有限公司 | Data file scheduling method, device and equipment |
| CN114244903A (en) * | 2021-11-01 | 2022-03-25 | 网宿科技股份有限公司 | Resource scheduling method, system, server and storage medium |
| CN114881802A (en) * | 2022-07-11 | 2022-08-09 | 湖南三湘银行股份有限公司 | Metadata-based data asset management method and system |
| CN116170476A (en) * | 2023-04-20 | 2023-05-26 | 北京华医网科技股份有限公司 | Cloud service information management method and platform for medical relay teaching |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113485973B (en) * | 2021-07-02 | 2023-05-16 | 中国联合网络通信集团有限公司 | Data synchronization method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102687112A (en) * | 2009-11-03 | 2012-09-19 | 皮斯佩斯有限公司 | Apparatus and method for managing a file in a distributed storage system |
| US20150112951A1 (en) * | 2013-10-23 | 2015-04-23 | Netapp, Inc. | Data management in distributed file systems |
| CN105049268A (en) * | 2015-08-28 | 2015-11-11 | 东方网力科技股份有限公司 | Distributed computing resource allocation system and task processing method |
| US20160173620A1 (en) * | 2014-12-11 | 2016-06-16 | International Business Machines Corporation | Time-based data placement in a distributed storage system |
| CN107302561A (en) * | 2017-05-23 | 2017-10-27 | 南京邮电大学 | A kind of hot spot data Replica placement method in cloud storage system |
| CN110035306A (en) * | 2019-04-23 | 2019-07-19 | 深圳市网心科技有限公司 | Dispositions method and device, the dispatching method and device of file |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110460682A (en) * | 2019-09-18 | 2019-11-15 | 深圳市网心科技有限公司 | A kind of data processing method, device, system and storage medium |
-
2019
- 2019-09-18 CN CN201910883498.3A patent/CN110460682A/en active Pending
-
2020
- 2020-05-18 WO PCT/CN2020/090810 patent/WO2021051839A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102687112A (en) * | 2009-11-03 | 2012-09-19 | 皮斯佩斯有限公司 | Apparatus and method for managing a file in a distributed storage system |
| US20150112951A1 (en) * | 2013-10-23 | 2015-04-23 | Netapp, Inc. | Data management in distributed file systems |
| US20160173620A1 (en) * | 2014-12-11 | 2016-06-16 | International Business Machines Corporation | Time-based data placement in a distributed storage system |
| CN105049268A (en) * | 2015-08-28 | 2015-11-11 | 东方网力科技股份有限公司 | Distributed computing resource allocation system and task processing method |
| CN107302561A (en) * | 2017-05-23 | 2017-10-27 | 南京邮电大学 | A kind of hot spot data Replica placement method in cloud storage system |
| CN110035306A (en) * | 2019-04-23 | 2019-07-19 | 深圳市网心科技有限公司 | Dispositions method and device, the dispatching method and device of file |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021051839A1 (en) * | 2019-09-18 | 2021-03-25 | 深圳市网心科技有限公司 | Data processing method, apparatus and system and storage medium |
| CN113312329A (en) * | 2020-02-26 | 2021-08-27 | 阿里巴巴集团控股有限公司 | Data file scheduling method, device and equipment |
| CN113312329B (en) * | 2020-02-26 | 2024-03-01 | 阿里巴巴集团控股有限公司 | Scheduling method, device and equipment for data files |
| CN111770180A (en) * | 2020-06-29 | 2020-10-13 | 百度在线网络技术（北京）有限公司 | Deployment method, device, equipment and storage medium |
| CN111770180B (en) * | 2020-06-29 | 2023-06-30 | 百度在线网络技术（北京）有限公司 | Deployment method, device, equipment and storage medium |
| CN114244903A (en) * | 2021-11-01 | 2022-03-25 | 网宿科技股份有限公司 | Resource scheduling method, system, server and storage medium |
| CN114881802A (en) * | 2022-07-11 | 2022-08-09 | 湖南三湘银行股份有限公司 | Metadata-based data asset management method and system |
| CN114881802B (en) * | 2022-07-11 | 2022-10-04 | 湖南三湘银行股份有限公司 | Metadata-based data asset management method and system |
| CN116170476A (en) * | 2023-04-20 | 2023-05-26 | 北京华医网科技股份有限公司 | Cloud service information management method and platform for medical relay teaching |
| CN116170476B (en) * | 2023-04-20 | 2023-11-03 | 北京华医网科技股份有限公司 | Cloud service information management method and platform for medical relay teaching |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021051839A1 (en) | 2021-03-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110460682A (en) | A kind of data processing method, device, system and storage medium | |
| CN102347864B (en) | System for monitoring service quality of content distribution networks | |
| EP1887732B1 (en) | A method and system for content charging | |
| CN107995283B (en) | A kind of data bury the method, equipment and system of point analysis | |
| US7801985B1 (en) | Data transfer for network interaction fraudulence detection | |
| CN105024872B (en) | The method and device of applied in network performance test | |
| EP4234060A1 (en) | Service processing method and apparatus, and storage medium | |
| US9231982B2 (en) | Data transfer for network interaction fraudulence detection | |
| CN103377077A (en) | Method and system for evaluating the resiliency of a distributed computing service by inducing a latency | |
| CN109474487A (en) | Network performance monitoring method, the network equipment and network performance monitoring system | |
| CN104301161A (en) | Computing method, computing device and communication system for business quality index | |
| US20170364833A1 (en) | Ranking video delivery problems | |
| CN101159635A (en) | Parameter collecting method and system for service quality monitoring of IPTV service | |
| CN102187357A (en) | Apparatus and method for measuring advertising metrics | |
| CN109194545B (en) | Network test platform flow generation system, method and device and electronic equipment | |
| CN110191362B (en) | Data transmission method and device, storage medium and electronic equipment | |
| CN103780446A (en) | Method and system for monitoring network bandwidth resource in real time | |
| CN105491167A (en) | Realizing method of sensing operation experience of browser terminal user in real time | |
| US20220182444A1 (en) | Peer-managed content distribution network | |
| Marques et al. | Proactive resource management for cloud of services environments | |
| CN111314743A (en) | Interface data playback method and device | |
| CN204836205U (en) | Network monitoring system based on cloud calculates | |
| CN111147664B (en) | Mobile terminal big data processing method and device and storage medium | |
| CN109491862A (en) | A kind of service requesting information collection method and device based on cloud platform | |
| WO2023041051A1 (en) | User perception evaluation method and device, storage medium, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191115 |
|
| RJ01 | Rejection of invention patent application after publication |