CN111125738A - Method and device for preventing core data from being lost, computer equipment and storage medium - Google Patents
Method and device for preventing core data from being lost, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111125738A CN111125738A CN201911368283.4A CN201911368283A CN111125738A CN 111125738 A CN111125738 A CN 111125738A CN 201911368283 A CN201911368283 A CN 201911368283A CN 111125738 A CN111125738 A CN 111125738A
- Authority
- CN
- China
- Prior art keywords
- data
- core data
- information
- data information
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a device, computer equipment and a storage medium for preventing core data from being lost, wherein the method comprises the following steps: executing related tasks of the task scheduling system; acquiring data information in an execution task; judging whether the data information is core data; if yes, storing the data information in a cluster type database; if not, the data information is stored in a relational database. According to the method and the device, the data information of the related tasks executed in the scheduling task system is judged, if the data information is the core data, the core data is stored in the ZooKeeper, if the data information is not the core data, the core data is stored in the MySQL, the ZooKeeper is a cluster type database, the reliability of the data is guaranteed, even if the server where the MySQL is located is down, the data such as system logs and the like are lost, the worry is not needed, the core data can still be read from the ZooKeeper, and the core data are effectively prevented from being lost.
Description
Technical Field
The present invention relates to the field of data processing, and more particularly, to a method, an apparatus, a computer device, and a storage medium for preventing core data loss in a task scheduling system.
Background
In the existing task scheduling system, items, tasks and various data in the system are usually stored in a relational database, and the relational database can provide high-efficiency reading and writing efficiency, but because a server where the relational database is located has a risk of single-point failure, once a server where the relational database is located has a disk damage condition, various core data in the task scheduling system are lost, and the resulting consequences are hard to bear.
Therefore, a new scheme is needed to store the core data of the task scheduling system, so as to prevent the core data from being lost when the server where the relational database is located has a disk damage condition.
Disclosure of Invention
The present invention is directed to overcome the deficiencies of the prior art and to provide a method, an apparatus, a computer device and a storage medium for preventing core data loss.
In order to achieve the purpose, the invention adopts the following technical scheme:
in a first aspect, a method for preventing core data from being lost is applied to a task scheduling system, and the method includes:
executing related tasks of the task scheduling system;
acquiring data information in an execution task;
judging whether the data information is core data;
if yes, storing the data information in a cluster type database;
if not, the data information is stored in a relational database.
The further technical scheme is as follows: the cluster type database is ZooKeeper.
The further technical scheme is as follows: the relational database is MySQL.
The further technical scheme is as follows: in the step of judging whether the data information is core data, the core data includes task information, scheduling information and connection information.
In a second aspect, a device for preventing core data from being lost is applied to a task scheduling system, and comprises an execution unit, an acquisition unit, a judgment unit, a first storage unit and a second storage unit;
the execution unit is used for executing related tasks of the task scheduling system;
the acquisition unit is used for acquiring data information in the execution task;
the judging unit is used for judging whether the data information is core data;
the first storage unit is used for storing the data information in a cluster type database;
and the second storage unit is used for storing the data information in a relational database.
The further technical scheme is as follows: the cluster type database is ZooKeeper.
The further technical scheme is as follows: the relational database is MySQL.
The further technical scheme is as follows: the core data includes task information, scheduling information, and connection information.
In a third aspect, a computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method steps for preventing core data loss as described above when executing the computer program.
In a fourth aspect, a storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method steps of preventing loss of core data as described above.
Compared with the prior art, the invention has the beneficial effects that: according to the method for preventing the core data from being lost, the data information of the related tasks executed in the scheduling task system is judged, if the data information is the core data, the core data is stored in the ZooKeeper, if the data information is not the core data, the core data is stored in MySQL, the ZooKeeper is a cluster type database, the reliability of the data is ensured, even if the server where the MySQL is located is down, the data such as system logs and the like are lost, the core data can still be read from the ZooKeeper, and further the core data are effectively prevented from being lost.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical means of the present invention more clearly understood, the present invention may be implemented according to the content of the description, and in order to make the above and other objects, features, and advantages of the present invention more apparent, the following detailed description will be given of preferred embodiments.
Drawings
FIG. 1 is a flow chart of an embodiment of a method for preventing core data loss according to the present invention;
FIG. 2 is a block diagram schematically illustrating the structure of an embodiment of the method for preventing core data loss according to the present invention;
FIG. 3 is a schematic block diagram of a computer device of the present invention.
Detailed Description
In order to more fully understand the technical content of the present invention, the technical solution of the present invention will be further described and illustrated with reference to the following specific embodiments, but not limited thereto.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
The present invention provides a method for preventing core data from being lost, which is applied in a task scheduling system, please refer to fig. 1, and the method specifically includes the following steps:
and S10, executing the related tasks of the task scheduling system.
Specifically, the invention is used in a task scheduling system, and the task scheduling system generally has a Master (responsible for scheduling) and a Worker (responsible for executing tasks). When executing the relevant tasks of the task scheduling system, a large amount of data information is processed.
And S20, acquiring data information in the executed task.
Specifically, the acquired data information is a basis for dividing the importance of the data, and in the scheme, the data is divided into two types, namely, core data of one type, such as task information, scheduling information, connection information and the like; the other is general data, e.g., system logs, execution history, etc.
And S30, judging whether the data information is core data or not, if so, S301, storing the data information in a cluster type database, and if not, S302, storing the data information in a relational database.
Specifically, according to the divided categories, the core data and the general data are separately stored, the core data is stored in a cluster database, and the general data is stored in a relational database. The relational database is MySQL. The purpose of this is: since the ZooKeeper is deployed in a clustered manner, the stored data can not be lost when one server fails, so that the storage of the core data in the ZooKeeper is highly reliable, while the storage of the general data in MySQL can facilitate the processing and reading of the data, and even if the server in which MySQL is located is down, the data such as system logs and the like are lost, there is no need to worry about the loss of the data because the general data are not important as long as the core data are still available. Therefore, the scheme can effectively prevent the core data from being lost in the task scheduling system.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Corresponding to the above method for preventing the core data from being lost, the present invention further provides a device for preventing the core data from being lost, referring to fig. 2, the device includes an execution unit 1, an obtaining unit 2, a determining unit 3, a first storage unit 4 and a second storage unit 5;
the execution unit 1 is used for executing related tasks of the task scheduling system;
an acquisition unit 2, configured to acquire data information in an execution task;
a judging unit 3, configured to judge whether the data information is core data;
a first storage unit 4 for storing the data information in a cluster type database;
and a second storage unit 5 for storing the data information in a relational database.
Wherein, the cluster type database is ZooKeeper, and the relational database is MySQL; the core data includes task information, scheduling information, and connection information.
As shown in fig. 3, the present embodiment also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the method steps for preventing the core data from being lost are implemented.
The computer device 700 may be a terminal or a server. The computer device 700 includes a processor 720, memory, and a network interface 750, which are connected by a system bus 710, where the memory may include non-volatile storage media 730 and internal memory 740.
The non-volatile storage medium 730 may store an operating system 731 and computer programs 732. The computer programs 732, when executed, may cause the processor 720 to perform any of a number of methods for preventing core data loss.
The processor 720 is used to provide computing and control capabilities, supporting the operation of the overall computer device 700.
The internal memory 740 provides an environment for the execution of the computer program 732 in the non-volatile storage medium 730, and when the computer program 732 is executed by the processor 720, the processor 720 may be caused to perform any of the methods for preventing the loss of core data.
The network interface 750 is used for network communication such as sending assigned tasks and the like. Those skilled in the art will appreciate that the configuration shown in fig. 3 is a block diagram of only a portion of the configuration relevant to the present teachings and is not intended to limit the computing device 700 to which the present teachings may be applied, and that a particular computing device 700 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components. Wherein the processor 720 is configured to execute the program code stored in the memory to perform the following steps:
executing related tasks of the task scheduling system;
acquiring data information in an execution task;
judging whether the data information is core data;
if yes, storing the data information in a cluster type database;
if not, the data information is stored in a relational database.
The further technical scheme is as follows: the cluster type database is ZooKeeper.
The further technical scheme is as follows: the relational database is MySQL.
The further technical scheme is as follows: in the step of judging whether the data information is core data, the core data includes task information, scheduling information and connection information.
It should be understood that, in the embodiment of the present Application, the Processor 720 may be a Central Processing Unit (CPU), and the Processor 720 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that the configuration of computer device 700 depicted in FIG. 3 is not intended to be limiting of computer device 700 and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present invention may be implemented in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional units is merely illustrated, and in practical applications, the above distribution of functions may be performed by different functional units according to needs, that is, the internal structure of the apparatus may be divided into different functional units to perform all or part of the functions described above. Each functional unit in the embodiments may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the application. For the specific working process of the units in the above-mentioned apparatus, reference may be made to the corresponding process in the foregoing method embodiment, which is not described herein again.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the units is only one type of logical function division, and other division manners may be available in actual implementation, for example, a plurality of units or components may be combined or integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The technical contents of the present invention are further illustrated by the examples only for the convenience of the reader, but the embodiments of the present invention are not limited thereto, and any technical extension or re-creation based on the present invention is protected by the present invention. The protection scope of the invention is subject to the claims.
Claims (10)
1. The method for preventing the core data from being lost is applied to a task scheduling system, and is characterized by comprising the following steps:
executing related tasks of the task scheduling system;
acquiring data information in an execution task;
judging whether the data information is core data;
if yes, storing the data information in a cluster type database;
if not, the data information is stored in a relational database.
2. The method of claim 1, wherein the clustered database is ZooKeeper.
3. The method for preventing core data loss according to claim 1, wherein the relational database is MySQL.
4. The method of claim 1, wherein in the step of determining whether the data information is core data, the core data includes task information, scheduling information and connection information.
5. The device for preventing the core data from being lost is applied to a task scheduling system and is characterized by comprising an execution unit, an acquisition unit, a judgment unit, a first storage unit and a second storage unit;
the execution unit is used for executing related tasks of the task scheduling system;
the acquisition unit is used for acquiring data information in the execution task;
the judging unit is used for judging whether the data information is core data;
the first storage unit is used for storing the data information in a cluster type database;
and the second storage unit is used for storing the data information in a relational database.
6. The apparatus of claim 5, wherein the clustered database is a ZooKeeper.
7. The apparatus for preventing core data loss according to claim 5, wherein the relational database is MySQL.
8. The apparatus of claim 5, wherein the core data comprises task information, scheduling information and connection information.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method steps of preventing core data loss as claimed in any one of claims 1 to 4 when executing the computer program.
10. A storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of preventing core data loss according to any one of claims 1 to 4.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911368283.4A CN111125738A (en) | 2019-12-26 | 2019-12-26 | Method and device for preventing core data from being lost, computer equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911368283.4A CN111125738A (en) | 2019-12-26 | 2019-12-26 | Method and device for preventing core data from being lost, computer equipment and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111125738A true CN111125738A (en) | 2020-05-08 |
Family
ID=70503225
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911368283.4A Pending CN111125738A (en) | 2019-12-26 | 2019-12-26 | Method and device for preventing core data from being lost, computer equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111125738A (en) |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6453426B1 (en) * | 1999-03-26 | 2002-09-17 | Microsoft Corporation | Separately storing core boot data and cluster configuration data in a server cluster |
CN106991002A (en) * | 2017-02-08 | 2017-07-28 | 网易无尾熊(杭州)科技有限公司 | Method and apparatus for realizing Portable Batch System |
CN107341051A (en) * | 2016-05-03 | 2017-11-10 | 北京京东尚科信息技术有限公司 | Cluster task coordination approach, system and device |
CN108132833A (en) * | 2017-12-08 | 2018-06-08 | 广州视源电子科技股份有限公司 | Distributed timing task scheduling method and device based on zookeeper technology |
US20180210834A1 (en) * | 2017-01-20 | 2018-07-26 | Mz Ip Holdings, Llc | Systems and methods for reconstructing cache loss |
CN109409818A (en) * | 2018-09-07 | 2019-03-01 | 安徽恒科信息技术有限公司 | The small micro- set-off guarantor of one kind opens up management system |
US20190095293A1 (en) * | 2016-07-27 | 2019-03-28 | Tencent Technology (Shenzhen) Company Limited | Data disaster recovery method, device and system |
CN110083457A (en) * | 2018-01-26 | 2019-08-02 | 北京京东尚科信息技术有限公司 | A kind of data capture method, device and data analysing method, device |
CN110334091A (en) * | 2019-05-09 | 2019-10-15 | 重庆天蓬网络有限公司 | A kind of data fragmentation distributed approach, system, medium and electronic equipment |
CN110569113A (en) * | 2018-06-06 | 2019-12-13 | 海通证券股份有限公司 | Method and system for scheduling distributed tasks and computer readable storage medium |
-
2019
- 2019-12-26 CN CN201911368283.4A patent/CN111125738A/en active Pending
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6453426B1 (en) * | 1999-03-26 | 2002-09-17 | Microsoft Corporation | Separately storing core boot data and cluster configuration data in a server cluster |
CN107341051A (en) * | 2016-05-03 | 2017-11-10 | 北京京东尚科信息技术有限公司 | Cluster task coordination approach, system and device |
US20190095293A1 (en) * | 2016-07-27 | 2019-03-28 | Tencent Technology (Shenzhen) Company Limited | Data disaster recovery method, device and system |
US20180210834A1 (en) * | 2017-01-20 | 2018-07-26 | Mz Ip Holdings, Llc | Systems and methods for reconstructing cache loss |
CN106991002A (en) * | 2017-02-08 | 2017-07-28 | 网易无尾熊(杭州)科技有限公司 | Method and apparatus for realizing Portable Batch System |
CN108132833A (en) * | 2017-12-08 | 2018-06-08 | 广州视源电子科技股份有限公司 | Distributed timing task scheduling method and device based on zookeeper technology |
CN110083457A (en) * | 2018-01-26 | 2019-08-02 | 北京京东尚科信息技术有限公司 | A kind of data capture method, device and data analysing method, device |
CN110569113A (en) * | 2018-06-06 | 2019-12-13 | 海通证券股份有限公司 | Method and system for scheduling distributed tasks and computer readable storage medium |
CN109409818A (en) * | 2018-09-07 | 2019-03-01 | 安徽恒科信息技术有限公司 | The small micro- set-off guarantor of one kind opens up management system |
CN110334091A (en) * | 2019-05-09 | 2019-10-15 | 重庆天蓬网络有限公司 | A kind of data fragmentation distributed approach, system, medium and electronic equipment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108829560B (en) | Data monitoring method and device, computer equipment and storage medium | |
CN111343267B (en) | Configuration management method and system | |
CN110069925B (en) | Software monitoring method, system and computer readable storage medium | |
CN110888354A (en) | Method and device for realizing automatic identification of access equipment, computer equipment and medium | |
CN112346775A (en) | General index data processing method, electronic device and storage medium | |
CN109918221B (en) | Hard disk error reporting analysis method, system, terminal and storage medium | |
CN108021486B (en) | The method and storage medium that electronic device, collage-credit data are handled | |
CN110347471B (en) | Hierarchical display component system, display component calling method and device | |
CN111752715A (en) | System optimization method, device, equipment and storage medium under large quantity of requests | |
CN111125738A (en) | Method and device for preventing core data from being lost, computer equipment and storage medium | |
CN111078384A (en) | Method and device for migrating core data, computer equipment and storage medium | |
CN110993014A (en) | Behavior test method and device of SSD in idle state, computer equipment and storage medium | |
WO2019062087A1 (en) | Attendance check data testing method, terminal and device, and computer readable storage medium | |
CN109739565B (en) | Method and device for ensuring normal operation of Bootrom, computer equipment and storage medium | |
CN110008291B (en) | Data early warning method and device, storage medium and electronic equipment | |
CN113297149A (en) | Method and device for monitoring data processing request | |
CN112288990A (en) | Method, system, medium and device for generating internet of things event based on internet of things data | |
CN112907257A (en) | Risk threshold determining method, abnormality detecting device and electronic equipment | |
CN111857604A (en) | Method, apparatus, device and medium for quickly reconstructing packet management mapping reverse lookup table | |
CN112231290A (en) | Method, device and equipment for processing local log and storage medium | |
CN112214394A (en) | Memory leak detection method, device and equipment | |
CN111953569A (en) | State information reporting method, device, equipment and medium | |
CN112578995A (en) | Data set processing method, device and system | |
CN111767299A (en) | Database operation method, device and system, storage medium and electronic equipment | |
CN111581207B (en) | File generation method and device of Azkaban project and terminal equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200508 |