CN112035292A - Method, device and equipment for processing data writing exception and storage medium - Google Patents
Method, device and equipment for processing data writing exception and storage medium Download PDFInfo
- Publication number
- CN112035292A CN112035292A CN202010760544.3A CN202010760544A CN112035292A CN 112035292 A CN112035292 A CN 112035292A CN 202010760544 A CN202010760544 A CN 202010760544A CN 112035292 A CN112035292 A CN 112035292A
- Authority
- CN
- China
- Prior art keywords
- data
- writing
- abnormality
- exception
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000005856 abnormality Effects 0.000 claims abstract description 62
- 230000008569 process Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000003672 processing method Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Retry When Errors Occur (AREA)
Abstract
The present disclosure relates to a method, an apparatus, a device and a storage medium for processing data write-in exception, wherein the method comprises: when detecting that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset standby database; and when the state of the system is determined to be writable, acquiring the data from the preset standby database, and writing the data into the system to finish the writing of the data. According to the method and the device, the hardware level of the system does not need to be improved, the data writing exception can be processed to ensure the data reliability of the system, and the economic cost is saved to a great extent.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a data write exception.
Background
Based on hardware basic limitation of the system, each system has a write bottleneck when data is written, data write exception may occur once the write bottleneck of the system is reached, and the situation of data write exception needs to be handled in order to avoid a data loss phenomenon caused by data write exception.
At present, the solution for the data write exception is usually realized by raising the hardware level of the system. However, as the data writing requirement is further increased, the hardware level of the system needs to be continuously increased to solve the problem of data writing exception, and obviously, the method for solving the data writing exception by continuously increasing the hardware level of the system is high in cost.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, the present disclosure provides a method, an apparatus, a device, and a storage medium for processing a data write exception.
In a first aspect, the present disclosure provides a method for handling a data write exception, where the method includes:
when detecting that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset standby database;
and when the state of the system is determined to be writable, acquiring the data from the preset standby database, and writing the data into the system to finish the writing of the data.
In an optional implementation manner, when it is determined that the state of the system is writable, the obtaining the data from the preset backup database, and writing the data into the system, before the writing of the data is completed, the method further includes:
polling whether the preset standby database stores data or not at regular time;
and after the data are determined to be stored in the preset standby database, detecting whether the state of the system is writable or not.
In an optional implementation manner, when it is detected that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset backup database, includes:
when detecting that the data writing abnormality occurs in the system, retrying the data corresponding to the data writing abnormality to write into the system;
and after the system fails to be written again, storing the data corresponding to the abnormal data writing into a preset standby database.
In an alternative embodiment, the data has a time attribute, and the writing the data into the system and completing the writing of the data includes:
and writing the data into a corresponding position in the system based on the time attribute to finish the writing of the data.
In an alternative embodiment, the data write exception includes a batch write-induced exception, and/or a system exception-induced write exception.
In an optional implementation manner, when it is detected that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset backup database, includes:
and when the abnormality caused by the batch writing is detected to occur in the system, storing the data of the current batch corresponding to the abnormality caused by the batch writing in a preset standby database.
In an optional implementation manner, when it is detected that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset backup database, includes:
in the process that a Kafka distributed message system writes data into an Elasticissearch cluster system, when data writing abnormality of the Elasticissearch cluster system is detected, data corresponding to the data writing abnormality are stored in a preset standby database.
In an alternative embodiment, the pre-defined backup database comprises a distributed database.
In a second aspect, the present disclosure provides an apparatus for handling a data write exception, the apparatus comprising:
the storage module is used for storing data corresponding to data writing abnormity in a preset standby database when the data writing abnormity of the system is detected;
the acquisition module is used for acquiring the data from the preset standby database when the state of the system is determined to be writable;
and the writing module is used for writing the data into the system to finish the writing of the data.
In a third aspect, the present disclosure provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the instructions cause the terminal device to implement the method according to any one of the above.
In a fourth aspect, the present disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, when executing the computer program, implementing the method as in any one of the above.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
when the data writing abnormality of the system is detected, the data which is being written is stored in the preset standby database, and when the system is writable, the data is taken out from the preset standby database, and the data is compensated and written into the system so as to ensure the reliability of the data in the system. According to the embodiment of the disclosure, the hardware level of the system is not required to be improved, the data writing exception can be processed to ensure the data reliability of the system, and the economic cost is saved to a great extent.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of a method for handling a data write exception according to an embodiment of the present disclosure;
FIG. 2 is a flow chart of another method for handling a data write exception according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of an application scenario system architecture according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a data write exception handling apparatus according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of a processing structure of a data write exception according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Due to the limitation of system hardware, the writable data of each system has a bottleneck, and once the write bottleneck of the system is reached, data write exception may occur. Data writing exception can cause data loss and other phenomena, so that an effective means is needed to process the data writing exception of the system to ensure the reliability of data in the system.
Therefore, the disclosure provides a processing method for data write-in abnormity, when detecting that data write-in abnormity occurs in a system, storing the data being written in a preset standby database, and when the system is in a writable state, taking out the data from the preset standby database, and performing compensation write-in on the data in the operation of the system so as to ensure the reliability of the data in the system.
According to the data writing exception processing method, the hardware level of the system is not required to be improved, the data writing exception can be processed to ensure the data reliability of the system, and the economic cost is saved to a great extent.
The following is a method for processing a data write exception according to an embodiment of the present disclosure, and referring to fig. 1, fig. 1 is a flowchart of a method for processing a data write exception according to an embodiment of the present disclosure, where the method for processing a data write exception specifically includes:
s101: when the data writing abnormality of the system is detected, the data corresponding to the data writing abnormality is stored in a preset standby database.
The method for processing data writing exception provided by the embodiment of the disclosure can be applied to intelligent equipment, such as a server, a computer, a certain computing node in a cluster, and the like.
In practical applications, when data is written into a system, data writing abnormality may occur for some reason, and in order to avoid loss of the data being written, the data writing abnormality that occurs needs to be processed in order to ensure reliability of the data in the system.
In the embodiment of the disclosure, the data writing exception may include an exception caused by batch writing and/or a writing exception caused by a system exception.
The batch write data refers to writing a large amount of data into the system at one time, which has the advantage of faster data writing speed, but due to the hardware limitation of the system and the large amount of data written at a time, a data write exception, i.e., an exception caused by batch write according to the embodiment of the present disclosure, is more likely to occur during batch write of data. In this case, the system may indicate that a data write exception is currently occurring by issuing an exception response or the like.
In an application scenario, when an abnormality caused by batch writing of a system is detected, the data of the batch corresponding to the abnormality caused by the batch writing can be stored in a preset standby database. For example, when 1000 pieces of data are written in a batch to a system, an exception caused by batch writing may occur in the system due to, for example, insufficient current resources of the system, and at this time, all 1000 pieces of data of the current batch that are being written may be stored in the database for the default device.
In another application scenario, the system itself may also cause a data write exception due to an exception of the system, that is, the write exception is caused by the system exception according to the embodiment of the present disclosure. For example, if the system is in a state of being unwritable for some reason, in this case, if data is written into the system, the system may indicate that data writing is abnormal currently by sending an abnormal response or the like, and at this time, data corresponding to the data writing abnormality needs to be stored in the preset backup database.
It is to be noted that the embodiments of the present disclosure do not limit what causes the system to generate the data write exception.
In an optional implementation manner, because a system has a certain instability, when detecting that data writing abnormality occurs in the system, the embodiment of the present disclosure may rewrite data corresponding to the data writing abnormality into the system. Specifically, the data corresponding to the data write exception may be rewritten into the system through a predetermined number of trial rewrite operations (e.g., 3 times). If all the rewriting operations fail, the data corresponding to the data writing exception can be stored in a preset standby database.
S102: and when the state of the system is determined to be writable, acquiring the data from the preset standby database, and writing the data into the system to finish the writing of the data.
In the embodiment of the disclosure, the data stored in the preset backup database can be written into the system in an attempt when the state of the system is writable, so that the reliability of the data in the system is ensured.
In practical application, when the state of the system is determined to be writable, data is acquired from a preset standby database and written into the system, and the writing of the data is completed. Therefore, the embodiment of the disclosure solves the problem of abnormal data writing through a data compensation writing mode, and ensures the reliability of data in the system.
In practical applications, even if the state of the system is determined to be writable, there may be a case where data acquired from the preset backup database cannot be written into the system, for example, the current remaining resources of the system are insufficient to receive the data written this time, and therefore, the embodiment of the present disclosure may wait for the next data compensation writing operation after the data compensation writing fails.
In an optional implementation manner, whether the preset backup database stores data may be polled regularly, and after it is determined that the preset backup database stores data, whether the state of the system is writable may be detected. And if the state of the system is writable at the moment, acquiring the data from a preset standby database, and writing the data into the system to finish the writing of the data.
When data acquired from a preset backup database is written into a system, in order to ensure the accuracy of the data written into the system, the data can be written into a corresponding position in the system based on the time attribute of the data, and the data writing is completed. For example, the time attribute of the data may include the time when the data is abnormally written, and the data is written into the corresponding position in the system in a compensation manner based on the time when the data is abnormally written, so that the data in the system does not have the problems of storage sequence and the like caused by data compensation writing, and the accuracy and reliability of the data in the system are ensured.
In the method for processing data write-in abnormality provided by the embodiment of the disclosure, when data write-in abnormality of a system is detected, data being written in is stored in a preset backup database, and when the system is writable, the data is taken out from the preset backup database, and the data is written in the system in a compensating manner, so as to ensure reliability of the data in the system. According to the embodiment of the disclosure, the hardware level of the system is not required to be improved, the data writing exception can be processed to ensure the data reliability of the system, and the economic cost is saved to a great extent.
In order to further understand the data write exception processing method provided by the present disclosure, the embodiment of the present disclosure is introduced with reference to a specific application scenario, and with reference to fig. 2, a flowchart of another data write exception processing method provided by the embodiment of the present disclosure is shown. Specifically, the method for processing the data write exception includes:
s201: in the process that a Kafka distributed message system writes data into an Elasticissearch cluster system, when data writing abnormality of the Elasticissearch cluster system is detected, data corresponding to the data writing abnormality are stored in a preset standby database.
The Kafka distributed message system is a distributed publish-subscribe message system, and can provide services such as rapid, scalable, distributed, partitioned and reproducible submission logs. The Elasticissearch cluster system is a distributed search engine and a data analysis engine.
In practical application, the Kafka distributed message system has a requirement for writing data into the Elasticsearch cluster system, and since the data written into the Elasticsearch cluster system is T +1 data, that is, the data written into the Elasticsearch cluster system is usually used on the next natural day, the data with data writing abnormality in the Elasticsearch cluster system can be written in a compensation manner, so as to ensure the reliability of the data in the Elasticsearch cluster system.
In an application scenario, referring to fig. 3, a schematic diagram of an application scenario system architecture provided by the embodiment of the present disclosure is shown, where a Kafka distributed message system may receive data of an open-API identifying service from a service layer, where the open-API is an open API, which is also called an open platform. The Kafka distributed message system writes data from the open platform into an Elasticsearch cluster system, also called an ES cluster, and in the process of writing data into the ES cluster by the Kafka distributed message system, if it is detected that data writing abnormality occurs in the Elasticsearch cluster system, the data writing abnormality corresponding to the data writing abnormality may be stored in a preset backup database, as shown in fig. 3, the ES cluster stores the data written with abnormality as abnormal data into a corresponding database, that is, a distributed database of a data layer.
In practical application, in the process that the Kafka distributed message system writes data into the Elasticsearch cluster system, if it is detected that data writing abnormality occurs in the Elasticsearch cluster system, the data corresponding to the data writing abnormality can be stored in the preset standby database. Specifically, the preset backup database may be a distributed database. In addition, the embodiments of the present disclosure are not limited to other types of databases.
S202: and when the state of the Elasticissearch cluster system is determined to be writable, acquiring the data from the preset standby database, and writing the data into the Elasticissearch cluster system to finish the writing of the data.
In practical application, whether data is stored in the preset standby database can be polled regularly, if the preset standby database is determined to store the data, whether the state of the Elasticissearch cluster system is writable or not is further detected, and if the state of the Elasticissearch cluster system is writable, the data can be obtained from the preset standby database and written into the Elasticissearch cluster system, so that the data is written.
As shown in fig. 3, polling the distributed database regularly to determine whether data is stored in the distributed database, if it is determined that data is stored in the distributed database, further detecting whether the state of the ES system is writable, and if the state of the ES system is writable, acquiring data from the distributed database and writing the data into the ES system, thereby completing the writing of the data.
In an optional implementation manner, when the data acquired from the preset backup database is written into the Elasticsearch cluster system, in order to ensure the accuracy of the data written into the Elasticsearch cluster system, the data may be written into the Elasticsearch cluster system in a compensated manner based on the time when the data is abnormally written into the Elasticsearch cluster system.
The processing method for the data write-in exception provided by the embodiment of the disclosure combines a specific application scenario, solves the problem of data write-in exception occurring when the Kafka distributed message system writes data into the Elasticsearch cluster system, does not need to improve the hardware level of the Elasticsearch cluster system, and greatly saves the economic cost.
Corresponding to the foregoing embodiment, an embodiment of the present disclosure further provides a device for processing a data write exception, and with reference to fig. 4, a schematic structural diagram of the device for processing a data write exception provided in the embodiment of the present disclosure is shown, where the device for processing a data write exception specifically includes:
the storage module 401 is configured to, when it is detected that data writing abnormality occurs in the system, store data corresponding to the data writing abnormality in a preset backup database;
an obtaining module 402, configured to obtain the data from the preset backup database when it is determined that the state of the system is writable;
a writing module 403, configured to write the data into the system, and complete the writing of the data.
In an alternative embodiment, the apparatus further comprises:
the polling module is used for polling whether the preset standby database stores data or not at regular time;
and the detection module is used for detecting whether the state of the system is writable or not after the data stored in the preset standby database is determined.
In an optional implementation, the storage module 401 includes:
the retry submodule is used for retrying and writing the data corresponding to the data writing abnormity into the system when the data writing abnormity of the system is detected;
and the storage submodule is used for storing the data corresponding to the abnormal data writing into a preset standby database after the retry writing into the system fails.
In another optional implementation, the data has a time attribute, and the writing module 403 is specifically configured to:
and writing the data into a corresponding position in the system based on the time attribute to finish the writing of the data.
In an alternative embodiment, the data write exception includes a batch write-induced exception, and/or a system exception-induced write exception.
In an optional implementation manner, the storage module 401 is specifically configured to:
and when the abnormality caused by the batch writing is detected to occur in the system, storing the data of the current batch corresponding to the abnormality caused by the batch writing in a preset standby database.
In an application scenario, the storage module 401 is specifically configured to:
in the process that a Kafka distributed message system writes data into an Elasticissearch cluster system, when data writing abnormality of the Elasticissearch cluster system is detected, data corresponding to the data writing abnormality are stored in a preset standby database.
Optionally, the preset backup database includes a distributed database.
In the processing apparatus for data write-in abnormality provided by the embodiment of the present disclosure, when it is detected that data write-in abnormality occurs in a system, data being written in is stored in a preset backup database, and when the system is in a writable state, the data is taken out from the preset backup database, and the data is written in the operation of the system in a compensation manner, so as to ensure reliability of the data in the system. According to the embodiment of the disclosure, the hardware level of the system is not required to be improved, the data writing exception can be processed to ensure the data reliability of the system, and the economic cost is saved to a great extent.
In addition, an embodiment of the present disclosure further provides a device for processing a data write exception, as shown in fig. 5, where the device may include:
a processor 501, a memory 502, an input device 503, and an output device 504. The number of processors 501 in the data write exception handling apparatus may be one or more, and one processor is taken as an example in fig. 5. In some embodiments of the present invention, the processor 501, the memory 502, the input device 503 and the output device 504 may be connected by a bus or other means, wherein the connection by the bus is exemplified in fig. 5.
The memory 502 may be used to store software programs and modules, and the processor 501 executes various functional applications of the data write exception handling device and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like. Further, the memory 502 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The input device 503 may be used to receive input numeric or character information and generate signal inputs related to user settings and function control of the processing apparatus for data write exceptions.
Specifically, in this embodiment, the processor 501 loads the executable file corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 501 runs the application program stored in the memory 502, thereby implementing various functions of the above-mentioned data write exception handling device.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in various embodiments according to the present disclosure as described in the "exemplary methods" section of this specification above.
The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in various embodiments according to the present disclosure as described in the "exemplary methods" section above of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (11)
1. A method for handling data write exceptions, the method comprising:
when detecting that data writing abnormality occurs in the system, storing data corresponding to the data writing abnormality in a preset standby database;
and when the state of the system is determined to be writable, acquiring the data from the preset standby database, and writing the data into the system to finish the writing of the data.
2. The method for handling the data write exception according to claim 1, wherein when the state of the system is determined to be writable, the method further includes, before the data is written into the system, acquiring the data from the preset backup database and writing the data into the system, and completing the writing of the data:
polling whether the preset standby database stores data or not at regular time;
and after the data are determined to be stored in the preset standby database, detecting whether the state of the system is writable or not.
3. The method for processing the data writing abnormality according to claim 1, wherein when the data writing abnormality of the system is detected, storing data corresponding to the data writing abnormality in a preset backup database, including:
when detecting that the data writing abnormality occurs in the system, retrying the data corresponding to the data writing abnormality to write into the system;
and after the system fails to be written again, storing the data corresponding to the abnormal data writing into a preset standby database.
4. The method for handling the data write exception according to claim 1, wherein the data has a time attribute, and the writing the data into the system is completed by:
and writing the data into a corresponding position in the system based on the time attribute to finish the writing of the data.
5. The method for processing the data writing exception according to any one of claims 1 to 4, wherein the data writing exception comprises an exception caused by batch writing and/or a writing exception caused by a system exception.
6. The method for processing the data writing abnormality according to claim 5, wherein when the data writing abnormality is detected in the system, storing the data corresponding to the data writing abnormality in a preset backup database, includes:
and when the abnormality caused by the batch writing is detected to occur in the system, storing the data of the current batch corresponding to the abnormality caused by the batch writing in a preset standby database.
7. The method for processing the data writing abnormality according to claim 1, wherein when the data writing abnormality of the system is detected, storing data corresponding to the data writing abnormality in a preset backup database, including:
in the process that a Kafka distributed message system writes data into an Elasticissearch cluster system, when data writing abnormality of the Elasticissearch cluster system is detected, data corresponding to the data writing abnormality are stored in a preset standby database.
8. The method for handling the data write exception as recited in claim 7, wherein the predetermined backup database comprises a distributed database.
9. An apparatus for handling data write exceptions, the apparatus comprising:
the storage module is used for storing data corresponding to data writing abnormity in a preset standby database when the data writing abnormity of the system is detected;
the acquisition module is used for acquiring the data from the preset standby database when the state of the system is determined to be writable;
and the writing module is used for writing the data into the system to finish the writing of the data.
10. A computer-readable storage medium having stored therein instructions which, when run on a terminal device, cause the terminal device to implement the method of any one of claims 1-8.
11. An apparatus, comprising: memory, a processor, and a computer program stored on the memory and executable on the processor, when executing the computer program, implementing the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010760544.3A CN112035292A (en) | 2020-07-31 | 2020-07-31 | Method, device and equipment for processing data writing exception and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010760544.3A CN112035292A (en) | 2020-07-31 | 2020-07-31 | Method, device and equipment for processing data writing exception and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112035292A true CN112035292A (en) | 2020-12-04 |
Family
ID=73583756
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010760544.3A Pending CN112035292A (en) | 2020-07-31 | 2020-07-31 | Method, device and equipment for processing data writing exception and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035292A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632184A (en) * | 2020-12-15 | 2021-04-09 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113392098A (en) * | 2021-06-11 | 2021-09-14 | 北京沃东天骏信息技术有限公司 | Data writing method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110078392A1 (en) * | 2009-09-28 | 2011-03-31 | International Business Machines Corporation | Writing to memory using adaptive write techniques |
| CN108234465A (en) * | 2017-12-26 | 2018-06-29 | 创新科存储技术有限公司 | Abnormal redundancy approach and device are coped in a kind of distributed file system |
| CN108520569A (en) * | 2018-03-30 | 2018-09-11 | 深圳市华宝电子科技有限公司 | Storage method, device, system and the storage medium of video data |
| CN109614270A (en) * | 2018-12-14 | 2019-04-12 | 深圳前海微众银行股份有限公司 | Data read-write method, device, equipment and storage medium based on Hbase |
| CN110232054A (en) * | 2019-06-19 | 2019-09-13 | 北京百度网讯科技有限公司 | Log transmission system and streaming log transmission method |
| US20200097355A1 (en) * | 2018-09-26 | 2020-03-26 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method and apparatus for outputting information |
-
2020
- 2020-07-31 CN CN202010760544.3A patent/CN112035292A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110078392A1 (en) * | 2009-09-28 | 2011-03-31 | International Business Machines Corporation | Writing to memory using adaptive write techniques |
| CN108234465A (en) * | 2017-12-26 | 2018-06-29 | 创新科存储技术有限公司 | Abnormal redundancy approach and device are coped in a kind of distributed file system |
| CN108520569A (en) * | 2018-03-30 | 2018-09-11 | 深圳市华宝电子科技有限公司 | Storage method, device, system and the storage medium of video data |
| US20200097355A1 (en) * | 2018-09-26 | 2020-03-26 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method and apparatus for outputting information |
| CN109614270A (en) * | 2018-12-14 | 2019-04-12 | 深圳前海微众银行股份有限公司 | Data read-write method, device, equipment and storage medium based on Hbase |
| CN110232054A (en) * | 2019-06-19 | 2019-09-13 | 北京百度网讯科技有限公司 | Log transmission system and streaming log transmission method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632184A (en) * | 2020-12-15 | 2021-04-09 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113392098A (en) * | 2021-06-11 | 2021-09-14 | 北京沃东天骏信息技术有限公司 | Data writing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9104796B2 (en) | Correlation of source code with system dump information | |
| CN107968841B (en) | Dynamic routing method, device, storage medium and client of network page | |
| US11562078B2 (en) | Assessing and managing computational risk involved with integrating third party computing functionality within a computing system | |
| CN110502357A (en) | A kind of stack retrogressive method, device, medium and equipment | |
| US20140068325A1 (en) | Test case result processing | |
| CN103150506A (en) | Method and device for detecting rogue program | |
| CN112035292A (en) | Method, device and equipment for processing data writing exception and storage medium | |
| CN112486552A (en) | Server hot update method, device, equipment and storage medium | |
| CN112231403B (en) | Consistency verification method, device, equipment and storage medium for data synchronization | |
| US20110265072A1 (en) | Dynamic Installation of Files for Running Programs | |
| CN109062602B (en) | Data processing method and device for application program and electronic equipment | |
| CN114003269A (en) | Component processing method and device, electronic equipment and storage medium | |
| US7127675B1 (en) | Method and system for automatically revising software help documentation | |
| CN106897224B (en) | Method and device for determining software testing range | |
| CN108280355A (en) | A kind of data desensitization method and device | |
| CN112631763A (en) | Program changing method and device of host program | |
| CN108197041B (en) | Method, device and storage medium for determining parent process of child process | |
| CN104317645A (en) | Method and device of monitoring port by virtue of application program embodiment | |
| CN112783683B (en) | Data processing method, device, equipment and storage medium | |
| CN113760631A (en) | Page loading duration determination method, device, equipment and storage medium | |
| CN112434013A (en) | Data table migration method and device, electronic equipment and storage medium | |
| CN111124459A (en) | Method and device for updating service logic of FPGA cloud server | |
| CN112506592A (en) | Method, device, equipment and storage medium for determining page loading duration | |
| CN110968754A (en) | Detection method and device for crawler page turning strategy | |
| CN111240928B (en) | Automatic detection method, device, equipment and storage medium for equipment drive |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |