CN110309215A - A kind of data processing method, system and metadata updates method, system - Google Patents
A kind of data processing method, system and metadata updates method, system Download PDFInfo
- Publication number
- CN110309215A CN110309215A CN201910334730.8A CN201910334730A CN110309215A CN 110309215 A CN110309215 A CN 110309215A CN 201910334730 A CN201910334730 A CN 201910334730A CN 110309215 A CN110309215 A CN 110309215A
- Authority
- CN
- China
- Prior art keywords
- operation log
- primary database
- cluster
- database cluster
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a kind of data processing method, system and metadata updates methods, system, wherein, the data processing method includes: the processing result for receiving object storage device and being directed to data processing request response, and it is based on the processing result, generate the corresponding operation log of the data processing request；Judge whether current primary database cluster can be used, if available, the operation log of generation is written in the primary database cluster；If unavailable, the operation log of generation is written in standby data-base cluster；After the primary database cluster recovery is available, untreated operation log is imported into the primary database cluster in the standby data-base cluster.Technical solution provided by the present application can be improved the stability and availability of distributed memory system.
Description
Technical field
The present invention relates to Internet technical field, in particular to a kind of data processing method, system and metadata updates side
Method, system.
Background technique
In existing distributed memory system, metadata is usually separated into storage with object data.Wherein, metadata
The attribute information of object data can be characterized.Current distributed memory system is when handling the request of user, it usually needs complete
At the processing of object data and the processing of corresponding metadata.
However, being just unable to complete the place of metadata if the Metadata Service in distributed memory system is unavailable
Reason process, and then cannot respond to the request of user.Therefore, the stability and availability of existing distributed memory system need to be mentioned
It is high.
Summary of the invention
The application's is designed to provide a kind of data processing method, system and metadata updates method, system, Neng Gouti
The stability and availability of high distributed memory system.
To achieve the above object, on the one hand the application provides a kind of data processing method, and the method is applied to business system
In system, which comprises receive the processing result that object storage device is directed to data processing request response, and be based on the place
Reason is as a result, generate the corresponding operation log of the data processing request；Judge whether current primary database cluster can be used, if can
With the operation log of generation is written in the primary database cluster；If unavailable, the operation log of generation is write
Enter in standby data-base cluster；After the primary database cluster recovery is available, untreated operation in the standby data-base cluster
Log is imported into the primary database cluster.
To achieve the above object, on the other hand the application also provides a kind of operation system, and the operation system includes: operation
Log generation unit, the processing result for being directed to data processing request response for receiving object storage device, and it is based on the place
Reason is as a result, generate the corresponding operation log of the data processing request；Active-standby switch writing unit, for judging current main number
Whether can be used according to library cluster, if available, the operation log of generation is written in the primary database cluster；If unavailable,
The operation log of generation is written in standby data-base cluster；After the primary database cluster recovery is available, the standby number
It is imported into the primary database cluster according to untreated operation log in the cluster of library.
To achieve the above object, on the other hand the application also provides a kind of metadata updates method, which comprises sentences
Whether the primary database cluster for breaking current can be used, if available, read untreated operation log from the primary database cluster,
And to the untreated operation log completion processing after, it is associated in the primary database cluster to update the operation log
Metadata；If the primary database cluster is unavailable, stop reading untreated operation log from the primary database cluster；When
After the primary database cluster recovery is available, untreated operation log is imported into the main number in the standby data-base cluster
According in the cluster of library；After the primary database cluster recovery is available, and the operation log quilt in the standby data-base cluster
After importing in the primary database cluster, untreated operation log is read from the primary database cluster, in the master
Metadata is updated in data-base cluster.
To achieve the above object, on the other hand the application also provides a kind of metadata updates system, the metadata updates
System includes: log read unit, and whether the primary database cluster for judging current can be used, if available, from the master data
Read untreated operation log in the cluster of library, and to the untreated operation log completion processing after, in the master data
The associated metadata of the operation log is updated in the cluster of library；Pause unit is read, if can not for the primary database cluster
With stopping reads untreated operation log from the primary database cluster；After the primary database cluster recovery is available,
Untreated operation log is imported into the primary database cluster in the standby data-base cluster；Unit is continued with, is used for
After the primary database cluster recovery is available, and the operation log in the standby data-base cluster is imported into the main number
After in the cluster of library, untreated operation log is read from the primary database cluster, in the primary database cluster
Metadata is updated.
Therefore technical solution provided by the present application, it can be the master data for storing metadata and operation log
Library cluster configuration is for data-base cluster.In this way, when primary database cluster is available, operation system can will be at the data that received
The object storage device (Object Storage Device, OSD) that is transmitted in distributed memory system of reason request, then by
The treatment process of OSD completion object data.Subsequent, operation system can receive the operation log of OSD feedback, thus by the behaviour
Make log to be written in available primary database cluster.If primary database cluster is unavailable, operation system then can should
Operation log is written in standby data-base cluster.In this way, operation system can be normally complete whether no matter primary database cluster is available
At the writing process of operation log.After primary database cluster recovery is available, untreated operation in the standby data-base cluster
Log can be imported into the primary database cluster.
In addition, metadata updates system can read untreated operation log from primary database cluster, and to reading
Operation log completion processing after, metadata can be updated in primary database cluster.If primary database cluster is not
It can use, metadata updates system, which can be temporarily ceased, reads untreated behaviour from primary database cluster or standby data-base cluster
Make log.The reason of handling in this way is: when primary database cluster is unavailable, may there is part in primary database cluster
Untreated operation log, subsequent newly-generated operation log all have been written in standby data-base cluster.These newly-generated behaviour
Make log, is temporally located in primary database cluster after untreated operation log.It is subsequent to work as primary database cluster recovery
It can use, and after carrying out the merging of operation log, the processing sequence of these newly-generated operation logs should be in primary database cluster
In after original untreated operation log.However, if metadata updates system when primary database cluster is unavailable, continues
If reading newly-generated operation log from standby data-base cluster and handled, then available in primary database cluster recovery
Afterwards, it will lead to the processing logic error of operation log.Therefore, when primary database cluster is unavailable, metadata updates system is answered
When stopping reading the operation log that does not operate, and available in primary database cluster recovery, and after completing the merging of operation log, then
Untreated operation log is sequentially read from primary database cluster.
Therefore the processing of the data processing of existing system and metadata is separated, by setting primary database cluster and
Standby data-base cluster, and the reading strategy of setting metadata updates system, guarantee the integrality and correctness of metadata, can be with
So that operation system is able to respond request always, solve existing system metadata processing part it is unavailable when whole system without
The problem of method externally services, to improve the stability and availability of distributed memory system.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, the accompanying drawings in the following description is only this hair.Bright some embodiments are right
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings
His attached drawing.
Fig. 1 is the structural schematic diagram of distributed memory system in embodiment of the present invention；
Fig. 2 is the step schematic diagram of data processing method in embodiment of the present invention；
Fig. 3 is the importing schematic diagram of operation log in embodiment of the present invention；
Fig. 4 is the step schematic diagram of metadata updates method in embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
The application provides a kind of data processing method and metadata updates method, above-mentioned method and can be applied to such as Fig. 1
Shown in distributed memory system.The distributed memory system may include metadata updates system, primary database cluster, standby
Multiple component units such as data-base cluster, operation system, data forwarding gateway, object data storage system.Wherein, the business
System can carry out data interaction with user client, and the data processing request that user client is sent can be passed through described
Data forwarding gateway forwards are to object data storage system.The object data storage system, can be to the data processing request
The object data of direction is handled, and to the operation system feedback processing result.Correspondence can be generated in the operation system
Operation log, and operation log is written in the primary database cluster or standby data-base cluster.The metadata updates
System can read untreated operation log from primary database cluster, and carry out more in primary database cluster to metadata
Newly.
In practical applications, the distributed memory system can be improves for existing distributed memory system
System afterwards.For example, the distributed memory system can be improved ceph system.The primary database cluster and standby number
It can be the cluster for having data persistence store function according to library cluster.For example, the primary database cluster and the standby data
Library cluster can be mongo cluster.The object data storage system can also be clever according to the requirement of distributed memory system
Setting living.For example, the object data storage system can be rados (Reliable, Autonomic in ceph system
Distributed Object Store, reliable autonomous distributed objects storage) system.Correspondingly, the data forwarding net
Pass can be rados gateway (Rados GateWay, RGW).
Referring to Fig. 2, the data processing method provided by the present application applied in above-mentioned operation system, may include following
Step.
S11: the processing result that object storage device is directed to data processing request response is received, and is tied based on the processing
Fruit generates the corresponding operation log of the data processing request.
In the present embodiment, operation system can receive the data processing request that user client is sent, and will be described
Data processing request is forwarded at object storage device, so that the object storage device is directed toward the data processing request
Object data handled.The user client can be the terminal device that user uses, and be also possible to run on described
Software on terminal device.The user client can be communicated with operation system, so as to send to operation system
Data processing request.
In practical applications, the data processing request can have a variety of request types.For example, the request type can
It can also include FastCGI protocol type to include http protocol type.Wherein, the actual type of the data processing request,
It can be determined according to the communication protocol used between user client and operation system.The data processing request may include
At least one of file upload request, file modification request, file deletion requests and file path change request.Wherein,
The file upload request and file modification request, and can be used as write request, it needs into the object data storage system
Data are written.The file deletion requests can then delete stored data in the object data storage system.Institute
File path change request is stated, then can be moved to from old path file new in the object data storage system
Path.
In practical applications, may include OSD in the object storage system, object data can store in OSD, and
And for every request of object data, can also be completed in OSD.
In the present embodiment, in above-mentioned every kind request, the description information of object data can be carried.Specifically,
For file upload request and file modification request, the description information of the object data of carrying may include object data
Title, the size of data of object data, the content of object data, the information such as destination path for storing object data.Wherein, due to
The content of object data is generally configured with biggish data volume, therefore in practical application scene, usually according to object data
Content calculates the cryptographic Hash of regular length, to indicate the content of object data using the content Hash value of object data.
For file deletion requests, the description information of the object data of carrying may include object data to be deleted title,
The information such as the destination path of object data to be deleted in the OSD.For file path change request, may include
The information such as the old path of object data to be changed and modified new destination path.
In this way, the description information of the object data, may include target road of the object data in the OSD
At least one of diameter, the size of data of the object data, the content Hash value of object data information.Wherein, for
For file amendment request, destination path of the object data in the OSD may include original path and new target
Path can also only include new destination path, specifically can according to practical application scene flexible setting, the application to this simultaneously
Without limitation.
In the present embodiment, operation system can pass through data forwarding net after receiving the data processing request
It closes at the OSD being forwarded to the data processing request in object data storage system, thus by OSD to the data processing request
The object data of direction is handled.
In practical applications, it in distributed memory system locating for OSD or object data storage system, can set in advance
An order identification is set, which can be used for each request that identity user client is sent, and can characterize institute
State the processing sequence of the corresponding operation log of processing result.In practical applications, the order identification can be a global sequence
Row number is also possible to current system time.For global sequence number, which can have in initialization
A standby initial value.The initial value being increasing and correspond to update with data processing request quantity.
Specifically, after the data processing request that the OSD receives operation system forwarding, the distributed storage
Current order identification can be fed back to the OSD by system or object data storage system.In this way, OSD is to the number
After the object data completion processing being directed toward according to processing request, the current order identification can be added in processing result, and will
The processing result for carrying the current order identification feeds back to operation system.The data processing request is directed in the OSD
After feedback carries the processing result of current order identification, the distributed memory system or object data storage system can be with
The current order identification is updated.Specific update mode can be configured according to the actual situation, for example, can be with
Add 1 on the basis of current global sequence, to obtain updated global sequence number.Or it can be by original system
Time is updated to current system time.The updated order identification can be used for receiving next number in the OSD
After processing request, the object storage device is distributed to.In this way, each data processing request can be with unique order mark
Sensible correspondence.
In the present embodiment, operation system receive OSD feedback carrying order identification processing result after, Ke Yigen
According to the processing result, the corresponding operation log of the data processing request is generated.The operation log, which can have, to be preset
Format, may include multinomial information in the format.For example, may include being carried in the processing result in the operation log
Current order identification, the description information of the type of the data processing request and the object data.Wherein, the number
According to the type of processing request, above-mentioned HTTP type or FastCGI type can be, the description information of the object data,
It can be the description information identified from the data processing request.Specifically, the description information at least may include institute
State the size of data of object data.In addition, the description information can also include that the object data is set in the object storage
The content Hash value of destination path and/or the object data in standby.
Certainly, in practical applications, the operation log can also have more or less information.For example, described
Can also include log identifier in operation log, the log identifier can be operation system generation for distinguishing different behaviour
Make the number of log.
S13: judging whether current primary database cluster can be used, if available, institute is written in the operation log of generation
It states in primary database cluster；If unavailable, the operation log of generation is written in standby data-base cluster；In the master data
After library cluster recovery is available, untreated operation log is imported into the primary database cluster in the standby data-base cluster.
In the present embodiment, operation system is after generating operation log, it can be determined that current primary database cluster is
It is no available.If primary database cluster is currently available, primary database can be written in the operation log of generation by operation system
In cluster.And if primary database cluster is currently unavailable, standby number can be written in the operation log of generation by operation system
According in the cluster of library.
In practical applications, operation system can also be added when generating the operation log for the operation log
Status indicator, the status indicator can be used for characterizing the operation log and be in untreated state.In this way, operation system is written
Operation log in primary database cluster or standby data-base cluster, can carry the state mark of the untreated state of the characterization
Know.The status indicator can be by metadata updates system identification, so that metadata updates system can distinguish primary database
In cluster, which operation log is not processed.It is subsequent, when in untreated state operation log by metadata more
After new system processing, metadata updates system can modify the status indicator, so that modified status indicator characterizes
The operation log, which is in, completes processing status.In this way, place would not be repeated in the operation log for completing processing status
Reason.
In one embodiment, operation system, can also be in life in addition to adding status indicator to the operation log generated
At the operation log in mark generation time of the operation log.It is subsequent that the generation time can be used as operation log
Sequence foundation when processed generates time more early operation log, should be by more early processing.By generating the time to operation day
Will is ranked up, and can cause the logic of metadata that mistake occurs to avoid due to operation order disorder.In this way, operation system
It can be labelled in the operation log write-in primary database cluster or the standby data-base cluster for generating the time.
It certainly, in another embodiment, can be from the master data when the order identification is global sequence
In the cluster of library read the smallest untreated operation log of order identification, and to the operation log of reading complete processing and then
From remaining untreated operation log, reads the smallest operation log of order identification and handled.
In one embodiment, operating mode, the original state of the operating mode can be pre-configured in operation system
It can be set to holotype, which, which can characterize the operation note that the operation system generates, should be written into the master
In data-base cluster.So after operation system determines that current primary database cluster is unavailable, the work of the operation system
Mode can be adjusted to standby mode, and the standby mode is then used to characterize the operation note that the operation system generates and should be write
Enter in the standby data-base cluster.In this way, operation system can be determined operation log dynamically by the switching of active-standby mode
It is written in primary database cluster or in standby data-base cluster.
In practical applications, when primary database cluster from it is unavailable revert to available after, operation system should will operate day
Will is written in primary database cluster, and in data-base cluster, there is also primary database cluster unavailable periods by business system at this time
The operation log of system write-in, this part operation log is also all untreated.Therefore, after primary database cluster recovery is available,
In order to enable metadata updates system can normally be handled this part operation log, need by the standby data-base cluster not
The operation log of processing imports in the primary database cluster.And this part operation log imported, can be from standby database
Deleted in cluster, or in standby data-base cluster status indicator is revised as it is processed, to avoid next time from standby database
In cluster when import operation log, this part operation log is repeated to import.
Referring to Fig. 3, assume that primary database cluster before unavailable, still remains untreated operation log A and B,
After primary database cluster is unavailable, the operation log that standby data-base cluster is written is C, D, E, then when primary database cluster is extensive
After can be used again, in import operation log C, D, E, need to be ranked up according to the generation time of operation log.Specifically, as schemed
Shown in 3, earlier than operation log C, D, E, therefore after importing, correctly sequence should be A, B, C, D, E to operation log A and B, after
Continuous metadata updates system can successively handle each operation log according to the sequence, mixed to avoid the logic of operation log
Disorderly.
The application also provides a kind of operation system, and the operation system includes:
Operation log generation unit, the processing result for being directed to data processing request response for receiving object storage device,
And it is based on the processing result, generate the corresponding operation log of the data processing request；
Whether active-standby switch writing unit, the primary database cluster for judging current can be used, if available, by the institute of generation
Operation log is stated to be written in the primary database cluster；If unavailable, standby data base set is written into the operation log of generation
In group；After the primary database cluster recovery is available, untreated operation log is imported into institute in the standby data-base cluster
It states in primary database cluster.
Referring to Fig. 4, the application also provides a kind of metadata updates method applied in metadata updates system, it is described
Method includes the following steps.
S21: judging whether current primary database cluster can be used, if available, reads not from the primary database cluster
The operation log of processing, and to the untreated operation log completion processing after, update institute in the primary database cluster
State the associated metadata of operation log.
S23: if the primary database cluster is unavailable, stop reading untreated operation day from the primary database cluster
Will；After the primary database cluster recovery is available, untreated operation log is imported into the master in standby data-base cluster
In data-base cluster.
S25: after the primary database cluster recovery is available, and the operation log quilt in the standby data-base cluster
After importing in the primary database cluster, untreated operation log is read from the primary database cluster, in the master
Metadata is updated in data-base cluster.
In the present embodiment, metadata updates system can be successively read each untreated from primary database cluster
Operation log, and the operation log of reading is handled, to update corresponding metadata in primary database cluster.Therefore,
In read operation log money, metadata updates system needs to judge whether current primary database cluster can be used.If it is available, then
Can be handled according to above-mentioned step, normal read operation log, and to untreated operation log completion processing after,
The associated metadata of untreated operation log is updated in the primary database cluster.
However, if the primary database cluster is unavailable, then it represents that untreated operation day in current primary database cluster
Will can not be read, and current newly-generated operation log has been written in standby data-base cluster.In this case, metadata updates
System should stop reading untreated operation log from the primary database cluster or standby data-base cluster, that is,
It says, when primary database cluster is unavailable, metadata updates system should also be as stopping more new metadata.The reason of handling in this way exists
In, referring to Fig. 3, also left operation log A and B when primary database cluster is unavailable and do not handled by metadata updates system,
It is subsequent to be written with new operation log C, D, E again in standby data-base cluster.At this point, if metadata updates system is normally from standby
Read operation log in data-base cluster, then will priority processing operation log C.However, from the point of view of sequentially, operation log
C should be located at the post-processing of operation log B, subsequent to work as main number if metadata updates system is first handled operation log C
After can be used according to library cluster recovery, then operation log A and B are handled, then necessarily will cause the confusion of metadata.Therefore,
After primary database cluster is unavailable, metadata updates system is also required to stop more new metadata.
It is subsequent, can be according to mode shown in Fig. 3 after primary database cluster recovery is available, it will standby data-base cluster
In operation log be directed into primary database cluster.Due to not locating in the primary database cluster and the standby data-base cluster
The operation log of reason is provided with log and generates time and global sequence number, then untreated behaviour in the standby data-base cluster
After being imported into the primary database cluster as log, the time or complete can be generated according to log in the primary database cluster
The sequencing of office's sequence number is ranked up current untreated each operation log, wherein current untreated each behaviour
Making log includes original not processed operation log in primary database cluster, and the operation day that standby data-base cluster imports
Will.In this way, metadata updates system can be according to ranking results successively to untreated each in the primary database cluster
Operation log is handled.Specifically, metadata updates system reads untreated operation day from the primary database cluster
When will, can be read from the primary database cluster log generate the time earliest or the smallest untreated behaviour of global sequence number
Make log, and processing is completed to the operation log of reading and then from remaining untreated operation log, it is raw to read log
At the time earliest or the smallest operation log of global sequence number.In this manner it is ensured that the logic of metadata updates meets operation day
The successive logic that will generates.
The application also provides a kind of metadata updates system, and the metadata updates system includes:
Whether log read unit, the primary database cluster for judging current can be used, if available, from the primary database
Read untreated operation log in cluster, and to the untreated operation log completion processing after, in the primary database
The associated metadata of the operation log is updated in cluster；
Pause unit is read, if unavailable for the primary database cluster, stops reading from the primary database cluster
Untreated operation log；After the primary database cluster recovery is available, untreated operation day in standby data-base cluster
Will is imported into the primary database cluster；
Unit is continued with, is used for after the primary database cluster recovery is available, and the standby data-base cluster
In operation log be imported into the primary database cluster after, untreated operation day is read from the primary database cluster
Will, to be updated in the primary database cluster to metadata.
Therefore technical solution provided by the present application, it can be the master data for storing metadata and operation log
Library cluster configuration is for data-base cluster.In this way, when primary database cluster is available, operation system can will be at the data that received
The OSD that reason request is transmitted in distributed memory system, is then completed the treatment process of object data by the OSD.It is subsequent, business
System can receive the operation log of OSD feedback, so that the operation log is written in available primary database cluster.If main
Data-base cluster is unavailable, then operation system the operation log can then be written in standby data-base cluster.In this way, no matter leading
Whether data-base cluster can be used, operation system can normally complete the writing process of operation log.When primary database cluster is extensive
After can be used again, untreated operation log can be imported into the primary database cluster in the standby data-base cluster.
In addition, metadata updates system can read untreated operation log from primary database cluster, and to reading
Operation log completion processing after, metadata can be updated in primary database cluster.If primary database cluster is not
It can use, metadata updates system, which can be temporarily ceased, reads untreated behaviour from primary database cluster or standby data-base cluster
Make log.The reason of handling in this way is: when primary database cluster is unavailable, may there is part in primary database cluster
Untreated operation log, subsequent newly-generated operation log all have been written in standby data-base cluster.These newly-generated behaviour
Make log, is sequentially being located in primary database cluster after untreated operation log.It is subsequent to work as primary database cluster recovery
It can use, and after carrying out the merging of operation log, the processing sequence of these newly-generated operation logs should be in primary database cluster
In after original untreated operation log.However, if metadata updates system when primary database cluster is unavailable, continues
If reading newly-generated operation log from standby data-base cluster and handled, then available in primary database cluster recovery
Afterwards, it will lead to the processing logic error of operation log.Therefore, when primary database cluster is unavailable, metadata updates system is answered
When stopping reading the operation log that does not operate, and available in primary database cluster recovery, and after completing the merging of operation log, then
Untreated operation log is sequentially read from primary database cluster.
Therefore the processing of the data processing of existing system and metadata is separated, by setting primary database cluster and
Standby data-base cluster, and the reading strategy of setting metadata updates system, guarantee the integrality and correctness of metadata, can be with
So that operation system is able to respond request always, solve existing system metadata processing part it is unavailable when whole system without
The problem of method externally services, to improve the stability and availability of distributed memory system.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It realizes by means of software and necessary general hardware platform, naturally it is also possible to be realized by hardware.Based on such
Understand, substantially the part that contributes to existing technology can embody above-mentioned technical proposal in the form of software products in other words
Out, which may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, packet
Some instructions are included to use so that a computer equipment (can be personal computer, server or the network equipment etc.) executes
Method described in certain parts of each embodiment or embodiment.
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (12)
1. a kind of data processing method, which is characterized in that the method is applied in operation system, which comprises
The processing result that object storage device is directed to data processing request response is received, and is based on the processing result, generates institute
State the corresponding operation log of data processing request；
Judge whether current primary database cluster can be used, if available, the master data is written into the operation log of generation
In the cluster of library；If unavailable, the operation log of generation is written in standby data-base cluster；It is extensive in the primary database cluster
After can be used again, untreated operation log is imported into the primary database cluster in the standby data-base cluster.
2. the method according to claim 1, wherein the data processing request includes file upload request, text
At least one of part modification request, file deletion requests and file path change request.
3. the method according to claim 1, wherein distributed memory system locating for the object storage device
In preset an order identification；Wherein, after the object storage device receives the data processing request, described point
Current order identification is fed back to the object storage device in cloth storage system；
Correspondingly, the object storage device is described current for carrying in the processing result of data processing request response
Order identification；
Wherein, the order identification is used to characterize the processing sequence of the corresponding operation log of the processing result.
4. according to the method described in claim 3, it is characterized in that, the method also includes:
After the object storage device feeds back the processing result for the data processing request, the distributed storage system
System is updated the current order identification；Wherein, updated order identification in the object storage device for connecing
After receiving next data processing request, the object storage device is distributed to.
5. according to the method described in claim 3, it is characterized in that, including current time in the operation log generated
The description information of sequence mark, the type of the data processing request and the object data；Wherein, the object data is retouched
State the size of data that information includes at least the object data.
6. the method according to claim 1, wherein generating the corresponding operation log of the data processing request
When, the method also includes:
Status indicator is added for the operation log, the status indicator is in untreated shape for characterizing the operation log
State；
Correspondingly, after the operation log in untreated state is processed, the status indicator is modified, so that after modification
Status indicator characterize the operation log and be in and complete processing status.
7. the method according to claim 1, wherein based on the operating mode of the operation system is set in advance
Mode, the holotype are used to characterize the operation note that the operation system generates and are written into the primary database cluster；
After determining that current primary database cluster is unavailable, the operating mode of the operation system is adjusted to standby mode, institute
Standby mode is stated to be written into the standby data-base cluster for characterizing the operation note that the operation system generates.
8. the method according to claim 1, wherein the primary database is written in the operation log of generation
Include: in cluster or the standby data-base cluster
The generation time of the operation log is marked in the operation log of generation, and will be labelled with the operation for generating the time
Log is written in the primary database cluster or the standby data-base cluster；
Wherein, if the operation system runs on holotype, the operation log is written in the primary database cluster and is located
Manage the operation log；If the operation system runs on standby mode, the standby data-base cluster is written into the operation log
In, and suspend the treatment process to operation log；
If the primary database cluster recovery, by the operation system from standby pattern switching be holotype, and by the standby data
Operation log in the cluster of library imports in the primary database cluster according to the time is generated, and in the standby data-base cluster
After the completion of operation log imports, the treatment process of recovery operation log.
9. a kind of operation system, which is characterized in that the operation system includes:
Operation log generation unit, the processing result for being directed to data processing request response for receiving object storage device, and base
In the processing result, the corresponding operation log of the data processing request is generated；
Whether active-standby switch writing unit, the primary database cluster for judging current can be used, if available, by the behaviour of generation
Make log to be written in the primary database cluster；If unavailable, the operation log of generation is written in standby data-base cluster；
After the primary database cluster recovery is available, untreated operation log is imported into the main number in the standby data-base cluster
According in the cluster of library.
10. a kind of metadata updates method, which is characterized in that the described method includes:
Judge whether current primary database cluster can be used, if available, reads untreated behaviour from the primary database cluster
Make log, and to the untreated operation log completion processing after, the operation day is updated in the primary database cluster
The associated metadata of will；
If the primary database cluster is unavailable, stop reading untreated operation log from the primary database cluster；Work as institute
State primary database cluster recovery it is available after, untreated operation log is imported into the primary database collection in standby data-base cluster
In group；
After the primary database cluster recovery is available, and described in the operation log in the standby data-base cluster is imported into
After in primary database cluster, untreated operation log is read from the primary database cluster, in the primary database collection
Metadata is updated in group.
11. according to the method described in claim 10, it is characterized in that, the primary database cluster and the standby data-base cluster
In operation log be provided with order identification；
Correspondingly, after the operation log in the standby data-base cluster is imported into the primary database cluster, according to order mark
Know sequence from small to large, untreated each operation log current in the primary database cluster is ranked up；
The smallest untreated operation log of order identification is read from the primary database cluster, and to the operation log of reading
It completes processing and then from remaining untreated operation log, reads the smallest operation log of order identification and handled.
12. a kind of metadata updates system, which is characterized in that the metadata updates system includes:
Whether log read unit, the primary database cluster for judging current can be used, if available, from the primary database cluster
It is middle to read untreated operation log, and to the untreated operation log completion processing after, in the primary database cluster
It is middle to update the associated metadata of operation log；
Pause unit is read, if unavailable for the primary database cluster, is stopped from primary database cluster reading not
The operation log of reason；After the primary database cluster recovery is available, untreated operation log quilt in standby data-base cluster
It imports in the primary database cluster；
Unit is continued with, is used for after the primary database cluster recovery is available, and in the standby data-base cluster
After operation log is imported into the primary database cluster, untreated operation log is read from the primary database cluster,
To be updated in the primary database cluster to metadata.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910334730.8A CN110309215A (en) | 2019-04-24 | 2019-04-24 | A kind of data processing method, system and metadata updates method, system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910334730.8A CN110309215A (en) | 2019-04-24 | 2019-04-24 | A kind of data processing method, system and metadata updates method, system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110309215A true CN110309215A (en) | 2019-10-08 |
Family
ID=68074566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910334730.8A Pending CN110309215A (en) | 2019-04-24 | 2019-04-24 | A kind of data processing method, system and metadata updates method, system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309215A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807064A (en) * | 2019-10-28 | 2020-02-18 | 北京优炫软件股份有限公司 | Data recovery device in RAC distributed database cluster system |
| CN112000735A (en) * | 2020-08-05 | 2020-11-27 | 中国邮政储蓄银行股份有限公司 | Data processing method, device and system |
| CN113590040A (en) * | 2021-07-29 | 2021-11-02 | 郑州阿帕斯数云信息科技有限公司 | Data processing method, device, equipment and storage medium |
| CN115718739A (en) * | 2023-01-09 | 2023-02-28 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium based on database back-end storage |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120089625A1 (en) * | 2010-10-12 | 2012-04-12 | Sap Ag | Incremental conversion of database objects during upgrade of an original system |
| CN103268351A (en) * | 2013-05-31 | 2013-08-28 | 网易（杭州）网络有限公司 | Method and device for data synchronism |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN107016027A (en) * | 2016-12-08 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The method and apparatus for realizing business information fast search |
| CN107943619A (en) * | 2017-12-08 | 2018-04-20 | 四川金英科技有限责任公司 | A kind of method to guarantee data integrity under the conditions of database failure |
-
2019
- 2019-04-24 CN CN201910334730.8A patent/CN110309215A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120089625A1 (en) * | 2010-10-12 | 2012-04-12 | Sap Ag | Incremental conversion of database objects during upgrade of an original system |
| CN103268351A (en) * | 2013-05-31 | 2013-08-28 | 网易（杭州）网络有限公司 | Method and device for data synchronism |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN107016027A (en) * | 2016-12-08 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The method and apparatus for realizing business information fast search |
| CN107943619A (en) * | 2017-12-08 | 2018-04-20 | 四川金英科技有限责任公司 | A kind of method to guarantee data integrity under the conditions of database failure |
Non-Patent Citations (1)
| Title |
|---|
| 廖国琼: "《嵌入式移动实时数据库事务处理技术》", 30 June 2010, 中国铁道出版社 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807064A (en) * | 2019-10-28 | 2020-02-18 | 北京优炫软件股份有限公司 | Data recovery device in RAC distributed database cluster system |
| CN110807064B (en) * | 2019-10-28 | 2022-08-26 | 北京优炫软件股份有限公司 | Data recovery device in RAC distributed database cluster system |
| CN112000735A (en) * | 2020-08-05 | 2020-11-27 | 中国邮政储蓄银行股份有限公司 | Data processing method, device and system |
| CN113590040A (en) * | 2021-07-29 | 2021-11-02 | 郑州阿帕斯数云信息科技有限公司 | Data processing method, device, equipment and storage medium |
| CN113590040B (en) * | 2021-07-29 | 2024-03-19 | 郑州阿帕斯数云信息科技有限公司 | Data processing method, device, equipment and storage medium |
| CN115718739A (en) * | 2023-01-09 | 2023-02-28 | 苏州浪潮智能科技有限公司 | Method, system, equipment and storage medium based on database back-end storage |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309215A (en) | A kind of data processing method, system and metadata updates method, system | |
| CN106998297B (en) | A kind of virtual machine migration method and device | |
| EP3223165B1 (en) | File processing method, system and server-clustered system for cloud storage | |
| US9025602B2 (en) | Data redistribution in data replication systems | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| CN109714239B (en) | Management message issuing method, VNFM (virtual network management frequency) equipment and server | |
| CN105701096A (en) | Index generation method, data inquiry method, index generation device, data inquiry device and system | |
| CN105141672B (en) | A kind of date storage method, apparatus and system | |
| CN102708165A (en) | Method and device for processing files in distributed file system | |
| WO2021017422A1 (en) | Index creation method in block chain type account book, device and apparatus | |
| CN110334094B (en) | Data query method, system, device and equipment based on inverted index | |
| CN113076290B (en) | File deletion method, device, equipment, system and storage medium | |
| US20170090856A1 (en) | Method and terminal for controlling grouping play of playing system | |
| CN107580032A (en) | Data processing method, device and equipment | |
| CN109981768A (en) | I/o multipath planning method and equipment in distributed network storage system | |
| US8683121B2 (en) | Storage system | |
| CN106789597A (en) | Instant communication information retransmission method and device, display methods and device | |
| US10142415B2 (en) | Data migration | |
| CN104426968A (en) | Data management method and device | |
| CN106470171B (en) | The delivery method and device of multicast forwarding list item | |
| CN103164172B (en) | Data flow storage method and device | |
| CN103888424A (en) | Cluster-type data encryption system and data processing method thereof | |
| CN107094085B (en) | Signaling transmission method and device | |
| WO2021137768A1 (en) | Method, apparatus and device for consuming messages based on message queue | |
| CN110168513B (en) | Partial storage of large files in different storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191008 |