CN104378396B - Data administrator and method - Google Patents
Data administrator and method Download PDFInfo
- Publication number
- CN104378396B CN104378396B CN201310355089.9A CN201310355089A CN104378396B CN 104378396 B CN104378396 B CN 104378396B CN 201310355089 A CN201310355089 A CN 201310355089A CN 104378396 B CN104378396 B CN 104378396B
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- persistence
- client
- present
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of data administrator, including：Data mirroring management unit, data high-speed Dispatching Unit, data buffer storage unit, data persistence unit etc..Data mirroring management unit carries out data existence inquiry judging to receive and distribute mass data access request；Data high-speed Dispatching Unit reaches the most short data transmission path of end user to calculate and provide；Data buffer storage unit is accelerating the read or write speed of data, and the result data that caching is handled through data processing unit；The data that data persistence unit obtains to permanent storage data mirror image administrative unit from exterior client data system, and the data of original form are provided for the data administrator.The present invention greatly reduces data persistence unit and the request pressure of data processing unit, is accessed and handled the data in the self-defined unit of client in a manner of complete non-intrusion type, gradually reduces until the dependence to client data system is completely eliminated.
Description
Technical field
The present invention is on a kind of cloud computing technology, more particularly to a kind of data administrator and method.
Background technology
By the development of nearly 20 years, Internet technology was nowadays highly developed, and special with equipment is moved in recent years
It is the data generation threshold decline caused by the quick popularization of smart mobile phone, the total amount of data of the whole society quickly increases.2011
The total amount of data in the whole world reaches 1800EB, it is contemplated that will reach 7900EB within 2015, and keep the speed that double about every two years.
With being rapidly incremented by for total amount of data, society is more and more stronger for the demand of cloud storage service.Cloud storage service
Can be with the data of cheap unit carrying cost management magnanimity, and access performance will not be caused with the development of data volume
It is gradually reduced.
For for the enterprise of mass data is managed using traditional approach before cloud storage service is popularized, how
Under the premise of keeping service unbroken it is smooth and safety historical data is migrated into cloud storage system become them it
Preceding one maximum of obstacle.
Existing technology is in the industry cycle widely used that content distributing network.By by all to source data system
Request is routed to content distributing network, and content distributing network, which can perceive frequently accessed content and store, arrives content delivery network
The cache node of network, source station is accessed pressure and raising access performance so as to achieve the purpose that to reduce at the same time.But content delivery network
Network can not completely disengage the dependence to source data system, so as to can not reach very extensive universality.
The storage of mass data, access, handle this three major issues usually while occur, and only solves the problems, such as wherein one to two
It can not solve the problems, such as the data management technique of complexity.With increasing sharply for data volume, the data processing basis originally built
Facility can't bear to use, and when frequently occurring user's visit capacity and increasing, it is slow in reacting to frequently result in data handling system, Wu Faji
When respond access of the user to data.Because the invention one kind can be on the premise of not exclusively original base facility be overthrown by data
The method of smooth migration to advanced cloud storage system has become very urgent.
The content of the invention
To overcome above-mentioned the shortcomings of the prior art, the present invention proposes a kind of data administrator, including：Data image
Administrative unit, data high-speed Dispatching Unit, data buffer storage unit, data persistence unit and data processing unit, wherein,
Data mirroring management unit carries out the inquiry of data existence and sentences to receive and distribute mass data access request
It is disconnected；
Data high-speed Dispatching Unit reaches the most short data transmission path of end user to calculate and provide；
Data buffer storage unit is accelerating the read or write speed of data, and the number of results that caching is handled through data processing unit
According to；
Data persistence unit obtains to permanent storage data mirror image administrative unit from exterior client data system
The data taken, and the data of original form are provided for the data administrator；
Data processing unit is to balanced according to load progress of the pressure between the calculate node of magnanimity is calculated, by access
Ask subsidiary data processing needs to carry out respective handling to the initial data of input, and target location is written to as output.
Further, data processing unit includes a data processing carry module, to the specific data processing of carry
Program, realizes and remains able to perform data handling procedure with stable efficiency under high concurrent scene.
Further, the data high-speed Dispatching Unit, which is also supported to upload, downloads transmitted in both directions acceleration function, to by number
According to progress high speed distribution.
Further, the data persistence unit obtains customer data and persistence to be disposable.
Further, the data mirroring management unit is also captured from exterior client data system and there is no in number
According to the corresponding data in permanent location and it is put into data persistence unit.
The present invention also provides a kind of management method of data, include the following steps：
1）Use environment is configured in data mirroring management unit；
2）Respond access request of the user to data.
Further, step 1）Including：
The memory space of migrating data is configured in data mirroring management unit；
The access module and access address of the client data system of outside are configured in data mirroring management unit；
The former domain name mapping of exterior client data system is directed toward to the data mirroring management unit of configuration completion, will be original
The data access for being pointing directly at the client data system of outside is adjusted to unify to access by data mirroring management unit.
Further, step 2）Including：
Judge whether the data of access request are already present in data buffer storage unit, if in the presence of directly reading and return
Back to user；If being not present,
Inquire about data persistence unit, and backward reference request results.
Further, data persistence unit is inquired about, and backward reference request results step includes：
If the target data of access request is present in data persistence unit, backward reference request results；
If target data is not present in data persistence unit, data mirroring management unit is by outside remote inquiry
The client data system in portion, please by accessing if the original form of target data is present in the client data system of outside
Ask result to return to user, and the data of return are stored into data persistence unit；
If target data is not present in the client data system of outside, data mirroring management unit is returned to user
The relevant information that data are not present.
Further, if the original form of the target data of access request is present in data persistence unit, but do not deposit
It is data buffer storage unit, whether is described according in access request comprising data processing rule, it is determined whether by target data
First become to swap out via data processing unit before from data persistence unit lifting to data buffer storage unit and meet final demand
Data shape.
Compared with prior art, technique effect of the invention is as follows：
1st, there is provided data mirroring management unit, data management function is added for data persistence unit.
2nd, data buffer storage unit accelerates the read-write of data, greatly reduces data persistence unit and data processing list
The request pressure of member.
3rd, data stewardship program is introduced, reaches to access and handle in a manner of complete non-intrusion type and is located at the self-defined unit of client
In data, and realize data access accelerate target, and gradually reduce until be completely eliminated to client data system according to
Lai Xing.
Brief description of the drawings
Fig. 1 is a kind of system architecture diagram of data administrator of the present invention；
Fig. 2 is a kind of step flow chart of data managing method of the present invention.
Embodiment
Below by way of specific instantiation and embodiments of the present invention are described with reference to the drawings, those skilled in the art can
Understand the further advantage and effect of the present invention easily by content disclosed in the present specification.The present invention can also pass through other differences
Instantiation implemented or applied, the various details in this specification also can be based on different viewpoints with application, without departing substantially from
Various modifications and change are carried out under the spirit of the present invention.
Fig. 1 is the system architecture diagram of the data administrator of the present embodiment.The data administrator 1 includes data image
Administrative unit 13, data processing unit 14, data high-speed Dispatching Unit 15, data buffer storage unit 12 and data permanently store list
Member 11, wherein, data mirroring management unit 13 combines above-mentioned some units, to receive and distribute mass data access request, and
Carry out data existence inquiry judging.Complete to capture corresponding data from client data system 2 if necessary and to be put into data permanent
In storage unit 11.
When using the data administrator, configuration is used on the memory space of migrating data；The visit of client data system 2
Ask pattern and access address；The former domain name mapping of client data system 2 is directed toward to the data mirroring management unit 13 of configuration completion,
The data access for being originally pointing directly at client data system 2 is adjusted to access by data mirroring management unit 13.When
After the user of outside initiates access request to data administrator 1, data mirroring management unit 13, which can start to perform to access, appoints
Business.
Data high-speed Dispatching Unit 15 reaches the most short data transmission path of end user to calculate and provide, at the same time also
Support to upload and download transmitted in both directions acceleration function, data can be uploaded without causing data inconsistent in multiple nodes by solving
Sex chromosome mosaicism.
Data buffer storage unit 12 is accelerating the read or write speed of data, and the result that caching is handled through data processing unit
Data.Data buffer storage unit 12 can substantially reduce the request pressure for data persistence unit 11 and data processing unit 14
Power.
Data persistence unit 11 fills to obtain data from exterior client data system for the data management
Put and original form data are provided.In actual use, can be used for permanently storing the number disposably obtained from client data system 2
According to.The data stored in data persistence unit 11 will not be as traditional content distributing network, can be because data buffer storage
The memory space limitation of unit 12 and cause the deletion of stage, and be to maintain it is permanent effectively, substantially reduce and finally elimination pair
The dependence of client data system 2.
In order to further improve the calculated performance of data administrator 1 in the present embodiment, data administrator 1 can also be set
It is equipped with data processing unit 14, there is provided a kind of machine for carrying out automatic load balancing between magnanimity calculate node according to calculating pressure
System, respective handling is carried out by the subsidiary data processing needs of access request to the initial data of input, and is written to as output
Target location.And has a data processing carry module（It is not shown）, to the specific data processor of carry, such as to meet
Standard interface protocol defined in data processing unit 14 realizes the conventional or customized data processing algorithm of client, by data
Processing unit 14 is scheduled and performs, and completes required data handling procedure.Still can so as to fulfill under high concurrent scene
It is enough that data are handled with stable efficiency.
Fig. 2 is referred to, Fig. 2 is the step flow chart of the data managing method of the present embodiment, in order to further spread out explanation
The technique effect of the present embodiment, below carries out the workflow of access request with the data administrator 1 in the present embodiment detailed
Thin explanation.
Step S1, first, configures use environment in data mirroring management unit.Wherein, include the following steps：
1. the memory space of migrating data is configured in data mirroring management unit；
2. the access module and access address of the client data system of outside are configured in data mirroring management unit；
3. the former domain name mapping of exterior client data system is directed toward to the data mirroring management unit of configuration completion, by original
The data access for being first pointing directly at the client data system of outside is adjusted to access by data mirroring management unit.
Step S2, user initiate access request to data administrator, and data administrator please according to the access of user
Ask, it is complete to perform mirror image data access service flow.
Step S0, accesses and completes.
The step of user access request being performed the following detailed description of data administrator：
Step S20：First, user asks to access data mirroring management unit 13 and data buffer storage unit 12；
Step S21：If the target data accessed is present in data buffer storage unit 12, step S210 is performed, will
Digital independent in data buffer storage unit 12, returns to user, enters step S0, reads and completes.
In the step s 21, if the target data accessed is not present in data buffer storage unit 12, S is entered step
22。
Step S22 and step S23, accesses data persistence unit 11 and is judged, if the target of access request
Data are present in data persistence unit 11, then enter step S230.
Step S230, handles data, describes, determines according to data processing rule whether is included in access request
Whether first become before target data is lifted to data buffer storage unit from data persistence unit via data processing unit
Swap out and meet the data shape of final demand, such as video file transcoding, picture scaling and shearing, journal file compression and decompression
Etc..Handled if desired, then enter step S231, data are handled, be disposed, enter step S232.
Step S232, data buffer storage unit is write by the data having access to or by the processed data of step S231
12 and user is returned to, enter step S0 and access and complete.
In step S23, if the target data of access request is not present in data persistence unit 11, into step
Rapid S24 and step S240, data mirroring management unit 13 is by the client data system 2 outside remote inquiry.If target
The original form of data is present in the client data system 2 of outside, then enters step S242, access request results are stored
Into data persistence unit 11, subsequently into step S230.
In step S240, if target data is not present in the client data system 2 of outside, S is entered step
241, the relevant information that data mirroring management unit 13 is not present to user's returned data, for example return to the mistake of " file is not present "
False information, finally accesses and completes.
In conclusion a kind of its effect of data management system and method for the present invention is including as follows：
1st, there is provided data mirroring management unit, data management function is added for data persistence unit.
2nd, data buffer storage unit accelerates the read-write of data, greatly reduces data persistence unit and data processing list
The request pressure of member.
3rd, data stewardship program is introduced, reaches to access and handle in a manner of complete non-intrusion type and is located at the self-defined unit of client
In data, and realize data access accelerate target, and gradually reduce until be completely eliminated to client data system according to
Lai Xing.
The above-described embodiments merely illustrate the principles and effects of the present invention, not for the limitation present invention.Any
Field technology personnel can modify above-described embodiment and changed under the spirit and scope without prejudice to the present invention.Therefore,
The scope of the present invention, should be as listed by claims.
Claims (8)
- A kind of 1. data administrator, it is characterised in that including：Data mirroring management unit, data high-speed Dispatching Unit, data Buffer unit, data persistence unit and data processing unit, wherein,Data mirroring management unit carries out data existence inquiry judging to receive and distribute mass data access request；Data high-speed Dispatching Unit reaches the most short data transmission path of end user to calculate and provide；Data buffer storage unit is accelerating the read or write speed of data, and the result data that caching is handled through data processing unit；Data persistence unit obtains to permanent storage data mirror image administrative unit from exterior client data system Data, and the data of original form are provided for the data administrator；Data processing unit is to balanced according to load progress of the pressure between the calculate node of magnanimity is calculated, by access request Subsidiary data processing needs carry out respective handling to the initial data of input, and are written to target location as output.
- 2. data administrator as claimed in claim 1, it is characterised in that：Data processing unit includes data processing extension Module is carried, to the specific data processor of carry, realizes and is remained able under high concurrent scene with stable efficiency execution Data handling procedure.
- 3. data administrator as claimed in claim 1, it is characterised in that：The data high-speed Dispatching Unit is also supported to upload Transmitted in both directions acceleration function is downloaded, data are carried out high speed distribution.
- 4. data administrator as claimed in claim 1, it is characterised in that：The data persistence unit is disposably to obtain Take customer data and persistence.
- 5. data administrator as claimed in claim 1, it is characterised in that：The data mirroring management unit is also from outside Crawl, which there is no the corresponding data in data persistence unit and be put into data, in client data system permanently stores list Member.
- 6. a kind of management method of data, it is characterised in that include the following steps：1) use environment is configured in data mirroring management unit；2) access request of the user to data is responded,Wherein：Step 1) includes：The memory space of migrating data is configured in data mirroring management unit；The access module and access address of the client data system of outside are configured in data mirroring management unit；The former domain name mapping of exterior client data system is directed toward to the data mirroring management unit of configuration completion, will be originally direct The data access of client data system outside being directed toward is adjusted to unified and is accessed by data mirroring management unit,Step 2) includes：Judge whether the data of access request are already present in data buffer storage unit, if in the presence of directly reading and return to User；If being not present,Inquire about data persistence unit, and backward reference request results.
- A kind of 7. management method of data as claimed in claim 6, it is characterised in that inquiry data persistence unit, and Backward reference request results step includes：If the target data of access request is present in data persistence unit, backward reference request results；If target data is not present in data persistence unit, data mirroring management unit is by outside remote inquiry Client data system, if the original form of target data is present in the client data system of outside, by access request knot Fruit returns to user, and the data of return are stored into data persistence unit；If target data is not present in the client data system of outside, data mirroring management unit is to user's returned data The relevant information being not present.
- 8. a kind of management method of data as claimed in claim 7, it is characterised in that if the original of the target data of access request Beginning form is present in data persistence unit, but is not present in data buffer storage unit, and whether number is included according in access request According to processing rule description, it is determined whether before target data is lifted to data buffer storage unit from data persistence unit first Become to swap out via data processing unit and meet the data shape of final demand.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310355089.9A CN104378396B (en) | 2013-08-15 | 2013-08-15 | Data administrator and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310355089.9A CN104378396B (en) | 2013-08-15 | 2013-08-15 | Data administrator and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104378396A CN104378396A (en) | 2015-02-25 |
| CN104378396B true CN104378396B (en) | 2018-05-15 |
Family
ID=52557050
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310355089.9A Active CN104378396B (en) | 2013-08-15 | 2013-08-15 | Data administrator and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104378396B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017166298A1 (en) * | 2016-04-01 | 2017-10-05 | Nokia Technologies Oy | Method, apparatus and computer program product for data distribution |
| CN107135265B (en) * | 2017-05-17 | 2020-05-29 | 郑州云海信息技术有限公司 | Cloud OS system-based secondary storage buffer area data management method and device |
| CN115657954B (en) * | 2022-11-01 | 2023-06-20 | 昆仑数智科技有限责任公司 | Data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170416A (en) * | 2006-10-26 | 2008-04-30 | 阿里巴巴公司 | Network data storage system and data access method |
| CN101937467A (en) * | 2010-09-17 | 2011-01-05 | 北京开心人信息技术有限公司 | High-efficiency caching method and system of server |
| CN102611745A (en) * | 2011-12-31 | 2012-07-25 | 成都市华为赛门铁克科技有限公司 | On-line file moving method, device and system |
| CN102799485A (en) * | 2012-07-12 | 2012-11-28 | 北京恒华伟业科技股份有限公司 | Historical data migration method and device |
| CN103221921A (en) * | 2010-11-23 | 2013-07-24 | 国际商业机器公司 | Direct migration of software images with streaming technique |
-
2013
- 2013-08-15 CN CN201310355089.9A patent/CN104378396B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170416A (en) * | 2006-10-26 | 2008-04-30 | 阿里巴巴公司 | Network data storage system and data access method |
| CN101937467A (en) * | 2010-09-17 | 2011-01-05 | 北京开心人信息技术有限公司 | High-efficiency caching method and system of server |
| CN103221921A (en) * | 2010-11-23 | 2013-07-24 | 国际商业机器公司 | Direct migration of software images with streaming technique |
| CN102611745A (en) * | 2011-12-31 | 2012-07-25 | 成都市华为赛门铁克科技有限公司 | On-line file moving method, device and system |
| CN102799485A (en) * | 2012-07-12 | 2012-11-28 | 北京恒华伟业科技股份有限公司 | Historical data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104378396A (en) | 2015-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10515058B2 (en) | Unified file and object data storage | |
| US12056194B2 (en) | Systems and methods for creating and delivering augmented reality content | |
| CN113010818A (en) | Access current limiting method and device, electronic equipment and storage medium | |
| CN106817388B (en) | Method and device for acquiring data by virtual machine and host machine and system for accessing data | |
| CN103369002A (en) | A resource downloading method and system | |
| CN104378396B (en) | Data administrator and method | |
| WO2020215833A1 (en) | Offline cache method and apparatus, and terminal and readable storage medium | |
| CN105897754A (en) | Data processing system, data pulling method and client | |
| CN110365587A (en) | Communication between devices method, apparatus, equipment and storage medium | |
| US20090327303A1 (en) | Intelligent allocation of file server resources | |
| US10423785B2 (en) | Data scanning within distributed computing components | |
| US10327133B2 (en) | Making subscriber data addressable as a device in a mobile data network | |
| CN110347656A (en) | The management method and device requested in document storage system | |
| CN105487929A (en) | Method for managing shared data of lens in cluster rendering process | |
| US11153388B2 (en) | Workflow engine framework for cross-domain extension | |
| CN112688980B (en) | Resource distribution method and device, and computer equipment | |
| US10516723B2 (en) | Distributing subscriber data in a mobile data network | |
| CN101557424A (en) | Point-to-point download system and resource caching method thereof | |
| CN109600407A (en) | Request distribution method and device based on virtual server | |
| CN113342837B (en) | Data transmission method, device, electronic equipment and computer readable medium | |
| CN103747081A (en) | Operating method, operating device and operating system of Cloud system | |
| CA3061403A1 (en) | Systems and methods for creating and delivering augmented reality content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |