KR102024719B1 - Method and apparatus for journaling of file-based database - Google Patents
Method and apparatus for journaling of file-based database Download PDFInfo
- Publication number
- KR102024719B1 KR102024719B1 KR1020170123770A KR20170123770A KR102024719B1 KR 102024719 B1 KR102024719 B1 KR 102024719B1 KR 1020170123770 A KR1020170123770 A KR 1020170123770A KR 20170123770 A KR20170123770 A KR 20170123770A KR 102024719 B1 KR102024719 B1 KR 102024719B1
- Authority
- KR
- South Korea
- Prior art keywords
- database
- information
- file
- change
- journaling
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Discuss a journaling method of a file-based database. According to an embodiment of the present invention, a method of journaling a database by using a replica of a replica of an original database may include: reflecting changes of the original database by a plurality of applications in the replicated database; Generating a second original database and a second replica database by switching the original database and the replica database; And recording completion information, which is information indicating that the change by the plurality of applications is completed, in the second copy database.
Description
The present invention relates to a method and apparatus for journaling a file-based database, and more particularly, to a method and apparatus for journaling a file-based database using an original database and a replicated database replicating the original database.
The database adopts a data journaling method to ensure data integrity. SQLite, one of the mobile databases, provides PERSIST and WAL modes for journaling. In addition, it basically provides DELETE, TRUNCATE, MEMORY, and NONE modes. PERSIST mode guarantees integrity by writing to a journal file before changing data in the database, but requires heavy system calls (eg fdatasync). DELETE and TRUNCATE modes differ in the creation / deletion of journal files. MEMORY and NONE modes do not guarantee data integrity. WAL mode is faster and lighter than writing directly to the data file because it appends the update to the last position of the journal file.
However, although WAL has the advantage of being efficient in write requests among journal modes of SQLite, it has the disadvantage that it should refer to the current log file and data file when executing read request. In addition, when used with EXT4 in a mobile environment, additional input / output performance degradation occurs due to a design problem. The PERSIST mode requires five system calls per transaction, which contributes to overall database performance degradation.
Accordingly, there is a need for a method and apparatus for journaling a file-based database that supports a journaling function while minimizing performance degradation.
Related prior art is Korean Patent Publication No. 10-1258589 (Invention name: information storage medium recording data according to the journaling file system, method and apparatus for recording / recovering data using the journaling file system, date of publication: May 2, 2013).
The present invention is applicable to a file-based database, to provide a journaling method and apparatus that can guarantee the integrity of data by supporting a journaling function with a low performance degradation.
The problem to be solved by the present invention is not limited to the problem (s) mentioned above, and other object (s) not mentioned will be clearly understood by those skilled in the art from the following description.
In order to solve the above problems, a method of journaling a database using a replicated database that duplicates the original database provided by the present invention includes the steps of reflecting the changes of the original database by a plurality of applications in the replicated database; Generating a second original database and a second replica database by switching the original database and the replica database; And recording completion information, which is information indicating that the change by the plurality of applications is completed, in the second copy database.
Preferably, before the recording of the completion information, the method may further include updating, according to a predetermined criterion, change information, which is information indicating a change state of the database, to the duplicated database.
Preferably, in the recording of the completion information, the completion information may be recorded by initializing and recording the change information.
Preferably, when the change information includes a state value indicating a change state of the database, the change information of the replica database is updated to include a state value larger by a predetermined size than the change information of the original database, 2 The state value included in the change information of the replica database may be initialized to a predetermined initial value.
Preferably, at each step, a synchronization function, which is a function for synchronizing changes of the original database or the replica database file with the storage device, may be called.
Preferably, the step of collectively reflecting the duplicated database includes collecting page information which is information on at least one page of the original database file to be changed by each of the plurality of applications; Analyzing the page information for each of the plurality of applications to generate duplicate page information which is information about a page to be changed by two or more of the plurality of applications, and general page information about the remaining pages; And reflecting the change of the original database on the duplicated database based on the duplicated page information and the general page information.
In addition, in order to solve the above problems, an apparatus for journaling a database using a replicated database in which the original database provided by the present invention is duplicated is a file that reflects the change of the original database by a plurality of applications to the duplicated database An update unit; And a file exchanger configured to switch the original database and the replicated database to generate a second original database and a second replicated database, wherein the file update unit is changed by the plurality of applications in the second replicated database. Complete information, which is information indicating that the data is completed, is further recorded.
Preferably, the file update unit may further update change information, which is information indicating a change state of the database, to the copy database according to a predetermined criterion.
Preferably, the file updating unit may record the completion information by initializing and recording the change information.
Preferably, when the change information includes a state value indicating a change state of the database, the change information of the replica database is updated to include a state value larger by a predetermined size than the change information of the original database, 2 The state value included in the change information of the replica database may be initialized to a predetermined initial value.
Preferably, when each of the file update unit and the file exchange unit changes the original database or the duplicate database file, a synchronization function, which is a function for synchronizing the changed file with the storage device, may be called.
Preferably, when the file update unit reflects the change of the original database in the replica database, collecting page information which is information on at least one page of the original database file to be changed by each of the plurality of applications and Analyzing the page information for each of the plurality of applications, generating duplicate page information which is information about a page to be changed by two or more of the plurality of applications, and general page information about the remaining pages; And based on the general page information, the change of the original database may be reflected in the duplicate database.
Specific details of other embodiments are included in the detailed description and the accompanying drawings.
The present invention introduces a journaling mode using a source database and a replicated database in which the original database is replicated, thereby ensuring the integrity of the data while preventing performance degradation.
1 is a flowchart illustrating a journaling method of a file-based database according to an embodiment of the present invention.
2 is a flowchart illustrating a method of collectively reflecting a change of an original database in a replica database according to an embodiment of the present invention.
3 is a diagram illustrating a journaling apparatus of a file-based database according to an embodiment of the present invention.
4 is a view for explaining a method of reflecting the changes of the original database combined with the replica database according to an embodiment of the present invention.
5 is a view for explaining a method of exchanging an original database and a replica database according to an embodiment of the present invention.
6 and 7 are diagrams for explaining the I / O characteristics of the PERSIST mode and WAL mode of SQLite, respectively.
As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the drawings, similar reference numerals are used for similar elements.
Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.
When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.
The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.
Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
1 is a flowchart illustrating a journaling method of a file-based database according to an embodiment of the present invention.
Journaling file system is a file system that can maintain the integrity of the disk in case of system crash or power failure. More specifically, if a conflict occurs while keeping a log of everything that happens in the data area of the disk, the metadata in the log recreates the lost data and returns the data before the crash, as well as storing unsaved data in place. You can do that.
The present invention relates to a journaling method and apparatus for recovering a database in the event of a system crash or power failure for a file-based database, such as a journaling file system. At this time, the journaling method and apparatus of the present invention can be applied or mounted on an Android, iOS or other OS-based smart phones, mobile devices, tablet devices and wearable devices, as well as a variety of notebooks, desktop PCs and servers, etc. It can be applied or mounted to a computer device.
More specifically, the journaling method and apparatus of the present invention may operate in a manner of recording changes to be applied to the original database in a replica database, and swapping the original database and the replica database when the recording is completed. In this case, each of the original database and the replica database may include change information which is information indicating a change state of the database, and this change state may be represented by a state value.
For example, initially, the status values of the original database and the replica database may be <n, 0>. Then, when the change is written to the replicate database (that is, the transaction is complete), the status can be <n, n + 1>. Next, after exchanging files corresponding to the original database and the replica database, the status value may be <n + 1, n>. Lastly, the state values of the new second original database (the existing replica database) and the second replica database replicating the second original database are <n + 1, 0>, so that journaling may be performed again. Meanwhile, the journaling method of the present invention described above may be referred to as a SHADOW mode.
In step S110, the database journaling device reflects the changes of the original database by the plurality of applications in a replicated database.
For example, referring to FIG. 4, the change of the original database by each of the application A and the application B may be collectively reflected in a replica file in a file page unit. In other words, if there is a change that each of Application A and Application B wants to change in the original database, the database journaling device can integrate the change and reflect it in the replicate database.
In this case, each of the applications A and B may lock and unlock the replication database in order to prevent a situation where a plurality of applications simultaneously change the replication database. That is, each of the applications A and B can change the replication database only when the replication database is not in the locked state, and if it is locked, it may have to wait until the lock is released.
On the other hand, a detailed method of reflecting the changes of the original database in the replica database will be described later in detail in FIG.
In another embodiment, the database journaling device may update the change database, which is information indicating a change state of the database, according to a predetermined criterion.
For example, the database journaling apparatus may access the original database and the replica database, which are file-based databases, and record the change information regarding the change state of each of the original database and the replica database according to a predetermined criterion. That is, the database journaling device may update the change information to indicate that the change of the original database is reflected in the replica database. On the other hand, updating the change information may be performed at any time before or after reflecting the change of the original database to the replica database.
More specifically, in order to update the change information, the database journaling device may update by adding a constant size (eg, +1) to a change count existing in the header of the database file. In this way, the database journaling apparatus updates the database server with the change information, and when the system crash and power interruption occur in the future, the change can be used to recover the database.
In step S120, the database journaling device switches the original database and the replica database to generate a second original database and a second replica database.
For example, referring to FIG. 5, the database journaling device renames a replica file to generate a second original database file, and renames a file of the original database file to a second replicate database. You can create a file.
Finally, in step S130, the database journaling device records the completion information, which is information indicating that the change by the plurality of applications has been completed, in the second replica database.
That is, the database journaling device may record completion information indicating that at least one transaction included in the change by the plurality of applications is completed in the second replica database file. The database journaling device may determine whether the application of the change by the plurality of applications is completed according to whether the completion information is recorded in the newly created second replica database file.
In another embodiment, the database journaling device may record the completion information by initializing and recording the change information of the second replica database.
For example, the database journaling device may record the completion information by initializing and recording the change state value present in the header of the database file to zero. That is, when 0 is stored in the changed state value, the database journaling device may consider that completion information is recorded in the database file.
In another embodiment, when the change information includes a state value indicating a change state of the database, the change information of the clone database is updated to include a state value larger by a predetermined size than the change information of the original database, and the second clone is updated. The state value included in the change information of the database may be initialized to a predetermined initial value.
For example, when the change of the original database whose change state value existing in the header of the database file is n is reflected in the replicate database, the change state value of the replicate database can be increased by 1 to n + 1.
In addition, when the exchange of the original database and the replica database is completed and a new second source database and a second replica database that duplicates the second source database are created, the change state value existing in the header of the second replica database file is zero. It can be initialized to indicate that the journaling can be performed again.
In another embodiment, at each step, a synchronization function, which is a function for synchronizing changes of the original database or replica database file with the storage device, may be called.
For example, when the database journaling device is based on Linux or Unix, the synchronization function may be the fdatasync () function, and the fdatasync () function is called only once each time steps S110 to S130 are performed, so that the total synchronization function is performed three times. Compared to the conventional technique (SQLite), which is called five times, the file I / O can be reduced.
In particular, only one fdatasync () function is called for a database change made by a plurality of applications in S110, thereby minimizing performance degradation by reducing file I / O of the system.
As described above, the journaling method of the file-based database according to an embodiment of the present invention can introduce a journaling mode using a source database and a replica database that replicates the source database, thereby guaranteeing data integrity and preventing performance degradation. It works.
2 is a flowchart illustrating a method of collectively reflecting a change of an original database in a replica database according to an embodiment of the present invention.
In step S210, the database journaling device collects page information which is information on at least one page of the original database file that each of the plurality of applications is to change.
For example, referring to FIG. 4, application A attempts to modify pages {B1, B2, B9} of the original database. Therefore, the page information may include information of {B 1 , B 2 , B 9 }. Application B also wants to modify the {B 2 , B 4 } page of the original database. Therefore, the page information may include information of {B 2 , B 4 }. Therefore, the database journaling device may collect page information of {B 1 , B 2 , B 9 } and page information of {B 2 , B 4 } from applications A and B.
In step S220, the database journaling apparatus analyzes page information about each of the plurality of applications, and generates duplicate page information and general page information about the remaining pages, which are information about a page to be changed by two or more of the plurality of applications. .
For example, referring to FIG. 4, the apparatus for journaling a database may include duplicate page information of {B 2 }, which is a page to be changed by both applications A and B, and {B 1 , B 4 , B which is a remaining page to change only one of applications A and B. 9 } General page information can be created.
Finally, in step S230, the database journaling device reflects the change of the original database in the duplicate database based on the duplicate page information and the general page information.
At this time, the database journaling device reflects only one time for the page {B 2 } included in the duplicate page information in the replication database, and only one time for the pages {B 1 , B 4 , B 9 } included in the general page information. Can be reflected in the replica database.
As described above, the method of reflecting the change of the original database in a duplicate database according to an embodiment of the present invention includes only one case of the page included in the duplicate page information in the duplicate database, thereby causing unnecessary file I / It is effective to minimize O.
Meanwhile, referring to FIG. 6, the PERSIST mode of SQLite, a mobile-based file database, frequently calls fdatasync (), which is a heavy system call, and generates a lot of additional file I / O for journaling. Figure 6 shows that when inserting 100 bytes of data into a database using SQLite's PERSIST mode, the data-related update is only 8KB, but additional 20KB of I / O occurs for journaling. In addition, five calls to fdatasync () are made for each PERSIST mode transaction.
In addition, referring to FIG. 7, the WAL mode of SQLite is not efficient because additional file I / O occurs due to a difference between the journaling design of the Linux-based file system EXT4 and the journaling method of SQLite. For example, inserting 100 bytes of data into a SQLite database requires 8 KB of additional file I / O for WAL file logging and 20 KB of EXT4 journal area.
3 is a diagram illustrating a journaling apparatus of a file-based database according to an embodiment of the present invention.
Referring to FIG. 3, the
The
In another embodiment, the
The
At this time, the
In another embodiment, the
In another embodiment, when the change information includes a state value indicating a change state of the database, the change information of the clone database is updated to include a state value larger by a predetermined size than the change information of the original database, and the second clone is updated. The state value included in the change information of the database may be initialized to a predetermined initial value.
In another embodiment, when each of the
In another embodiment, when the
Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium.
The computer-readable recording medium may include a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.) and an optical reading medium (for example, a CD-ROM, DVD, etc.).
So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.
Claims (12)
Reflecting, by the journaling device of the database, the change of the original database by a plurality of applications in the replica database;
Generating a second original database and a second replica database by switching between the original database and the replica database by a journaling device of the database; And
And recording, by the journaling device of the database, complete information which is information indicating that the change by the plurality of applications is completed in the second replicate database.
Comprehensive reflecting in the replica database
Collecting, by the journaling device of the database, page information which is information on at least one page of a file of the original database to be changed by each of the plurality of applications;
The journaling device of the database analyzes the page information for each of the plurality of applications, and generates duplicate page information and general page information about the remaining pages, which are information about a page to be changed by two or more of the plurality of applications. step; And
The journaling device of the database reflecting the change of the original database to the duplicated database based on the duplicated page information and the general page information;
Journaling method of a file-based database comprising a.
Prior to recording the completion information,
Updating, by the journaling device of the database, change information, which is information indicating a change state of the database, to the replicate database according to a predetermined criterion;
Journaling method of a file-based database, characterized in that it further comprises.
The recording of the completion information
And recording the completion information by initializing and recording the change information by the journaling device of the database.
When the change information includes a state value indicating a change state of the database,
The journaling device of the database updates the change information of the replica to include a state value larger than the change information of the original database by a predetermined size,
The journaling method of the file-based database, characterized in that for initializing the state value contained in the change information of the second replica database to a predetermined initial value.
Journaling method of the file-based database, characterized in that for each step of the journaling device of the database calls a synchronization function that is a function for synchronizing the change of the file of the original database or the replica database with the storage device.
A file updating unit which reflects the change of the original database by a plurality of applications in the duplicate database; And
A file exchange unit for generating a second original database and a second replica database by switching between the original database and the replica database.
Including,
The file update unit
Wherein the completion information which is information indicating that the change is completed by the plurality of applications in the second replication database is further recorded,
When the file update unit reflects the change of the original database in the replica database,
Collecting page information which is information about at least one page of a file of the original database to be changed by each of the plurality of applications,
Analyzing the page information for each of the plurality of applications, generating duplicate page information which is information about a page to be changed by two or more of the plurality of applications, and general page information about the remaining pages,
And a change of the original database is reflected in the duplicated database based on the duplicated page information and the general page information.
The file update unit
And the change information, which is information indicating a change state of the database, is further updated with respect to the duplicated database according to a predetermined criterion.
The file update unit
And recording the completion information by initializing and recording the change information.
When the change information includes a state value indicating a change state of the database,
The change information of the replica database is updated to include a state value larger by a predetermined size than the change information of the original database,
The state value included in the change information of the second replica database is initialized to a predetermined initial value.
When each of the file update unit and the file exchange unit changes a file of the original database or the duplicate database,
And a synchronization function, which is a function for synchronizing the changed file with the storage device, is called.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020160152141 | 2016-11-15 | ||
| KR20160152141 | 2016-11-15 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| KR20180054422A KR20180054422A (en) | 2018-05-24 |
| KR102024719B1 true KR102024719B1 (en) | 2019-09-25 |
Family
ID=62299216
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020170123770A KR102024719B1 (en) | 2016-11-15 | 2017-09-25 | Method and apparatus for journaling of file-based database |
Country Status (1)
| Country | Link |
|---|---|
| KR (1) | KR102024719B1 (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001236256A (en) * | 2000-02-23 | 2001-08-31 | Hitachi Ltd | Distributed arrangement system for electronic information and distributed arrangement system for data base and remote management system |
| JP2012226453A (en) * | 2011-04-15 | 2012-11-15 | Toshiba Corp | Database device and database reorganization method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100913196B1 (en) * | 2007-12-11 | 2009-08-24 | 한국전자통신연구원 | System and method for updating file |
-
2017
- 2017-09-25 KR KR1020170123770A patent/KR102024719B1/en active IP Right Grant
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001236256A (en) * | 2000-02-23 | 2001-08-31 | Hitachi Ltd | Distributed arrangement system for electronic information and distributed arrangement system for data base and remote management system |
| JP2012226453A (en) * | 2011-04-15 | 2012-11-15 | Toshiba Corp | Database device and database reorganization method |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20180054422A (en) | 2018-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10452271B2 (en) | Reconstructing in-memory indices in a distributed data storage system | |
| JP6553822B2 (en) | Dividing and moving ranges in distributed systems | |
| US10936547B2 (en) | Filesystem replication using a minimal filesystem metadata changelog | |
| US10248356B2 (en) | Using scratch extents to facilitate copying operations in an append-only storage system | |
| US10140189B2 (en) | Database recovery and index rebuilds | |
| JP6309103B2 (en) | Snapshot and clone replication | |
| US9547706B2 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| US20150213100A1 (en) | Data synchronization method and system | |
| US10296518B2 (en) | Managing distributed deletes in a replicated storage system | |
| US8433863B1 (en) | Hybrid method for incremental backup of structured and unstructured files | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| US9772783B2 (en) | Constructing an index to facilitate accessing a closed extent in an append-only storage system | |
| US11397749B2 (en) | Asynchronous replication of in-scope table data | |
| EP2562657B1 (en) | Management of update transactions and crash recovery for columnar database | |
| US10628298B1 (en) | Resumable garbage collection | |
| US9720607B2 (en) | Append-only storage system supporting open and closed extents | |
| US9619322B2 (en) | Erasure-coding extents in an append-only storage system | |
| KR102139087B1 (en) | Method, server, and computer readable medium for index recovery using index redo log | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| US20100293143A1 (en) | Initialization of database for synchronization | |
| KR102024719B1 (en) | Method and apparatus for journaling of file-based database | |
| US11481372B1 (en) | Systems and methods for indexing multi-versioned data | |
| CN107145501B (en) | File updating method, file copying method and corresponding system | |
| US20240281546A1 (en) | Continuous Data Protection System and Method for Modern Applications | |
| KR102111555B1 (en) | Method and apparatus for journaling of file-based database which is used by one application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A201 | Request for examination | ||
| E902 | Notification of reason for refusal | ||
| AMND | Amendment | ||
| E601 | Decision to refuse application | ||
| AMND | Amendment | ||
| X701 | Decision to grant (after re-examination) | ||
| GRNT | Written decision to grant |