CN110134548A - A kind of rapid restarting method of distributed memory big data processing system - Google Patents
A kind of rapid restarting method of distributed memory big data processing system Download PDFInfo
- Publication number
- CN110134548A CN110134548A CN201910370338.9A CN201910370338A CN110134548A CN 110134548 A CN110134548 A CN 110134548A CN 201910370338 A CN201910370338 A CN 201910370338A CN 110134548 A CN110134548 A CN 110134548A
- Authority
- CN
- China
- Prior art keywords
- storage process
- memory
- processing system
- data
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1438—Restarting or rejuvenating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Techniques For Improving Reliability Of Storages (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention discloses a kind of rapid restarting methods of distributed memory big data processing system, receive reset command including the storage process in response to memory node, data is copied to shared drive by storage process, and storage process exits after copy；It is restarted in response to storage process, storage process restores data from shared drive.The present invention is by data copy to shared drive, and when restarting, data are read from shared drive, are restarted faster.
Description
Technical field
The present invention relates to a kind of rapid restarting methods of distributed memory big data processing system, belong to big data processing skill
Art field.
Background technique
Patent 201810605777.9 discloses a kind of distributed memory big data processing system and its data processing method,
The memory node of system stores data in memory, and with the growth of time, insertion, inquiry request cumulative amount are more next
It is more, centainly along with a large amount of memory application and release, also imply that the generation and accumulation of memory fragmentation, system is only restarted
It just can thoroughly solve the problems, such as memory fragmentation, when restarting, each memory node needs to read data from disk, in data
When measuring big, the read or write speed of hard disk will become reboot time bottleneck, restart and need the time long.
Summary of the invention
The present invention provides a kind of rapid restarting method of distributed memory big data processing system, solves and existing restart
Method time long problem.
In order to solve the above-mentioned technical problem, the technical scheme adopted by the invention is that:
A kind of rapid restarting method of distributed memory big data processing system, including,
Reset command is received in response to the storage process of memory node, data is copied to shared drive by storage process,
Storage process exits after copy；
It is restarted in response to storage process, storage process restores data from shared drive.
If shared drive is unavailable or restores data exception from shared drive, storage process restores number from disk
According to.
Reset command is received in response to storage process, storage process requests the insertion being currently executing, inquiry is asked
It asks and is finished, refuse new insertion request and inquiry request, the removal request in request queue is deleted, finally by itself
In all data copies to shared drive.
Storage process will be copied to shared drive after Data Serialization.
Data Serialization is that all data with spatial hierarchy that will be stored in storage process are converted into linear order
Column.
All memory nodes are divided into several pieces, first restart a copy of it memory node, when this part of memory node has been restarted
Cheng Houzai restarts another memory node, and all parts of memory nodes of traversal restart completion until all memory nodes.
Every part of memory node accounts for the 2% of all memory nodes.
Advantageous effects of the invention: 1, the present invention by data copy to shared drive, when restarting, from shared drive
Middle reading data, restart faster；2, the present invention is restarted using gradual, guarantees that a part of memory node can normally make when restarting
With making system can also normal use in the case where restarting.
Detailed description of the invention
Fig. 1 is flow chart of the invention；
Fig. 2 is Data Serialization location mode；
Data table status when Fig. 3 is data copy；
Fig. 4 is gradual restarting process figure.
Specific embodiment
The invention will be further described below in conjunction with the accompanying drawings.Following embodiment is only used for clearly illustrating the present invention
Technical solution, and not intended to limit the protection scope of the present invention.
As shown in Figure 1, a kind of rapid restarting method of distributed memory big data processing system, comprising the following steps:
Step 1, reset command is received in response to the storage process of memory node, data is copied to shared by storage process
Memory, storage process exits after copy.
Storage process will be copied to shared drive after Data Serialization, Data Serialization is the institute that will be stored in storage process
There are the data with spatial hierarchy to be converted into linear order, it is therefore an objective to can direct copying to shared drive or from shared
It is restored in memory, as shown in Fig. 2, recording current version number first, the byte length after serializing connects specific Data Serialization
Getting off is the information of every tables of data.It include the head information of every tables of data in one tables of data, the quantity of row block, followed by
The head information of each row block, the corresponding tables of data of the row block (title-data type), is pressed each again followed by each row block
The head information and corresponding data of column piecemeal storage.
Shared drive: shared drive refers in the computer system of multiprocessor, can be by different central processing units
(CPU) the large capacity memory accessed.
As shown in figure 3, receiving reset command in response to storage process, storage process will from the point of view of the state of tables of data
The insertion request that is currently executing, inquiry request are finished, and refuse new insertion request and inquiry request, then will ask
It asks the removal request in queue to delete, while log being written into disk, finally by data copies all in itself in sharing
It deposits.
Step 2, it is restarted in response to storage process, storage process first restores data from shared drive, if in shared
It deposits unavailable or restores data exception from shared drive, then storage process restores data from disk.
The above method is by data copy to shared drive, and when restarting, data are read from shared drive, are restarted faster.
In memory due to data storage, traditional restarting disposably restarts all memory nodes, is restarting
The operation such as insertion, inquiry of data can not be carried out in the process, will affect the normal work of system；Therefore here using gradual heavy
Open, guarantee a part of memory node energy normal use when restarting, make system in the case where restarting can also normal use, specifically such as
Shown in Fig. 4, all memory nodes are divided into several pieces, first restart a copy of it memory node, remaining memory node can normally make
With restarting another memory node again after the completion of this part of memory node is restarted, all parts of memory nodes of traversal are deposited until all
Storage node restarts completion.Optimal in conjunction with long-term practical experience, every part of memory node accounts for the 2% of all memory nodes, i.e.,
Guarantee the memory node normal use for having 98% all the time, meets the normal work of system.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, without departing from the technical principles of the invention, several improvement and deformations can also be made, these improvement and deformations
Also it should be regarded as protection scope of the present invention.
Claims (7)
1. a kind of rapid restarting method of distributed memory big data processing system, it is characterised in that: including,
Reset command is received in response to the storage process of memory node, data is copied to shared drive by storage process,
Storage process exits after copy；
It is restarted in response to storage process, storage process restores data from shared drive.
2. a kind of rapid restarting method of distributed memory big data processing system according to claim 1, feature exist
In: if shared drive is unavailable or restores from shared drive data exception, storage process restores data from disk.
3. a kind of rapid restarting method of distributed memory big data processing system according to claim 1, feature exist
In: reset command is received in response to storage process, storage process executes the insertion request being currently executing, inquiry request
It finishes, refuses new insertion request and inquiry request, the removal request in request queue is deleted, will finally own in itself
Data copy is to shared drive.
4. a kind of rapid restarting method of distributed memory big data processing system according to claim 1 or 3, feature
Be: storage process will be copied to shared drive after Data Serialization.
5. a kind of rapid restarting method of distributed memory big data processing system according to claim 4, feature exist
In: Data Serialization is that all data with spatial hierarchy that will be stored in storage process are converted into linear order.
6. a kind of rapid restarting method of distributed memory big data processing system according to claim 1, feature exist
In: all memory nodes are divided into several pieces, first restart a copy of it memory node, after the completion of this part of memory node is restarted again
Restart another memory node, all parts of memory nodes of traversal restart completion until all memory nodes.
7. a kind of rapid restarting method of distributed memory big data processing system according to claim 6, feature exist
In: every part of memory node accounts for the 2% of all memory nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910370338.9A CN110134548B (en) | 2019-05-06 | 2019-05-06 | Quick restarting method of distributed memory big data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910370338.9A CN110134548B (en) | 2019-05-06 | 2019-05-06 | Quick restarting method of distributed memory big data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110134548A true CN110134548A (en) | 2019-08-16 |
| CN110134548B CN110134548B (en) | 2023-08-11 |
Family
ID=67576259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910370338.9A Active CN110134548B (en) | 2019-05-06 | 2019-05-06 | Quick restarting method of distributed memory big data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134548B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198122A (en) * | 2013-04-02 | 2013-07-10 | 华为技术有限公司 | Method and device for restarting in-memory database |
| CN106354531A (en) * | 2016-08-25 | 2017-01-25 | 杭州华为数字技术有限公司 | Physical node upgrading method and device |
-
2019
- 2019-05-06 CN CN201910370338.9A patent/CN110134548B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198122A (en) * | 2013-04-02 | 2013-07-10 | 华为技术有限公司 | Method and device for restarting in-memory database |
| CN106354531A (en) * | 2016-08-25 | 2017-01-25 | 杭州华为数字技术有限公司 | Physical node upgrading method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110134548B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3726364B1 (en) | Data write-in method and solid-state drive array | |
| US8356150B2 (en) | Systems and methods for providing nonlinear journaling | |
| JP3808007B2 (en) | Caching method and system for storage device | |
| US8380689B2 (en) | Systems and methods for providing nonlinear journaling | |
| CN106055431B (en) | The method and device of backup and recovery based on the online rollback of snapshot | |
| JP2718031B2 (en) | History information acquisition method | |
| CN105549905A (en) | Method for multiple virtual machines to access distributed object storage system | |
| US20060143412A1 (en) | Snapshot copy facility maintaining read performance and write performance | |
| CN106126658B (en) | A kind of database auditing point construction method based on virtual memory snapshot | |
| US20070266203A1 (en) | Storage control method and system for performing backup and/or restoration | |
| US20080183988A1 (en) | Application Integrated Storage System Volume Copy and Remote Volume Mirror | |
| JP2006323826A (en) | System for log writing in database management system | |
| US6658541B2 (en) | Computer system and a database access method thereof | |
| KR20000022716A (en) | Efficient volume copy using pre-configuration of log structured target storage | |
| JP2005301499A (en) | Disk array device and control method for disk array device | |
| WO2023015866A1 (en) | Data writing method, apparatus and system, and electronic device and storage medium | |
| CN100504800C (en) | Method for snapshot of magnetic disc | |
| WO2023277819A9 (en) | Data processing method, system, device, computer program product, and storage function | |
| CN110515705A (en) | Expansible persistence transaction internal memory and its working method | |
| CN110309077A (en) | The flash translation layer (FTL) construction method and device of host and equipment collaboration work | |
| CN110989924A (en) | Metadata storage performance optimization method and storage server | |
| JP4915365B2 (en) | Data storage method and apparatus, duplicate data storage method and apparatus, and control program thereof | |
| JP4189342B2 (en) | Storage apparatus, storage controller, and write-back cache control method | |
| US20230273751A1 (en) | Resiliency and performance for cluster memory | |
| JP2006099802A (en) | Storage controller, and control method for cache memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230717 Address after: 1015 Huatong Building, Sungang East Road, Guiyuan Street, Luohu District, Shenzhen, Guangdong 518000 Applicant after: Shenzhen Foresight Network Technology Co.,Ltd. Address before: Room 521, Yuanhe building, 959 Jiayuan Road, Yuanhe street, Xiangcheng District, Suzhou City, Jiangsu Province Applicant before: SUZHOU NIEVA INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |