CN113641709A - Database data writing method, database data reading method and database - Google Patents
Database data writing method, database data reading method and database Download PDFInfo
- Publication number
- CN113641709A CN113641709A CN202110920712.5A CN202110920712A CN113641709A CN 113641709 A CN113641709 A CN 113641709A CN 202110920712 A CN202110920712 A CN 202110920712A CN 113641709 A CN113641709 A CN 113641709A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- database
- cache block
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000013500 data storage Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0868—Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0871—Allocation or management of cache space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1056—Simplification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/16—General purpose computing application
- G06F2212/163—Server or database system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data writing method and a data reading method for a database, wherein when data to be written is acquired, the data to be written is written into a cache block, and then the data in the cache block is written into a pre-established file through a pre-established file channel, so that the data is stored in the database; the method comprises the steps of searching data to be read from a file, reading the data to be read from the file through a pre-established file channel, writing the data into a cache block, further obtaining the data to be read from the cache block, and reading the data from a database. The invention stores or reads data in a file mode, can realize fast data writing or data reading, and does not need complex database deployment and control programs. The invention also discloses a database.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a database data writing method and a database data reading method. The invention also relates to a database.
Background
The essence of the database is that the data is stored in a structured form on the hard disk. With the improvement of software business requirements, various database products are developed in order to meet the storage requirements of business data and solve the storage problem of different business data, wherein the various database products include a relational database and a non-relational database.
In the existing database product, an independent database program needs to be installed, and the application is connected and communicated with the database program through a database driver, however, additional database deployment occupies more hardware resources, and a series of control processes need to be performed in the data reading and writing process.
Disclosure of Invention
The invention aims to provide a data writing method and a data reading method for a database, which can realize fast data writing or data reading and do not need complex database deployment. The invention also provides a database.
In order to achieve the purpose, the invention provides the following technical scheme:
a database write data method, comprising:
when data to be written is acquired, presetting a cache block according to the data to be written;
writing the data to be written into the cache block;
and writing the data in the cache block into a pre-established file through a pre-established file channel, wherein the file is established in a pre-established folder.
Preferably, presetting the cache block according to the data to be written includes: and establishing a cache block with the length consistent with that of the data to be written according to the data to be written.
Preferably, the data to be written includes a plurality of data records, and the length of the data to be written is the sum of the lengths of the data records.
Preferably, writing the data to be written into the cache block includes: writing the data to be written into the cache block through a write pointer of the cache block;
writing the data in the cache block into a pre-established file through a pre-established file channel comprises: and the file channel reads the data in the cache block through the read pointer of the cache block and writes the data into a pre-established file.
Preferably, it is applied to streaming data generated by application services.
A database reading method, comprising:
searching data to be read from a file, presetting a cache block according to the data to be read, wherein the file is established in a pre-established folder;
reading the data to be read from the file through a pre-established file channel, and writing the data into the cache block;
and acquiring the data to be read from the cache block.
Preferably, presetting the cache block according to the data to be read includes: and setting the length of the cache block to be consistent with the length of the data to be read.
Preferably, reading the data to be read from the file through a pre-established file channel includes:
calculating an offset according to the length of the data to be read and the position of the file tail pointer;
and controlling the position of the file tail pointer to move according to the offset, and reading the data to be read from the file through the file channel.
Preferably, the obtaining the data to be read from the cache block includes: the data to be read includes a plurality of data records, each of which is sequentially and respectively acquired from the cache block.
A database adopts the database data writing method or the database data reading method.
According to the technical scheme, the data writing method for the database comprises the steps of establishing a folder, establishing a file in the folder, storing and writing data into the file, specifically, when the data to be written is obtained, presetting a cache block according to the data to be written, writing the data to be written into the cache block, and then writing the data in the cache block into the file which is established in advance through a file channel which is established in advance, so that the data are stored in the database.
The invention provides a data reading method for a database, which comprises the steps of establishing a folder, establishing a file in the folder, storing data into the file, specifically searching data to be read from the file, presetting a cache block according to the data to be read, reading the data to be read from the file through a pre-established file channel, writing the data into the cache block, further acquiring the data to be read from the cache block, and reading the data from the database.
The database data writing method and the database data reading method of the invention store or read data in a file mode, can realize fast data writing or data reading, and do not need complex database deployment and control programs.
The database provided by the invention can achieve the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a data writing method for a database according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for reading data from a database according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a database according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the database data writing method of the embodiment, a folder is established, a file is established in the folder, and a data block is stored in the file, so that the data is stored in the database. Referring to fig. 1, fig. 1 is a flowchart of a data writing method for a database according to the present embodiment, as shown in the figure, the data writing method for a database includes the following steps:
s10: and when the data to be written is acquired, presetting a cache block according to the data to be written.
When data to be written is acquired, a cache block is preset according to the data to be written, and the cache block is used for temporarily storing the data. The set cache block is to be matched with the data to be written.
S11: and writing the data to be written into the cache block.
S12: and writing the data in the cache block into a pre-established file through a pre-established file channel, wherein the file is established in a pre-established folder.
And reading data to be written from the buffer block through a pre-established file channel, and writing the data into a pre-established file, so that the data is stored in a database.
According to the database data writing method, the folder is established, the file is established in the folder, the data storage is written into the file, the data storage is realized, the data is stored in a file mode, the data can be quickly written, and compared with the existing database data storage mode, the database data writing method does not need complex database deployment and control procedures, and a lightweight database scheme is realized.
Optionally, the step S10 of presetting the cache block according to the data to be written includes: and establishing a cache block with the length consistent with that of the data to be written according to the data to be written. The length of the cache block is set to be consistent with that of the data to be written, so that on the premise of ensuring that the data to be written can be completely written into the cache block, occupation of more storage space is avoided, and system resources are saved.
Specifically, if the data to be written includes one data record, the length of the data to be written is the length of the data record. If the data to be written includes a plurality of data records, the length of the data to be written is the sum of the lengths of the included data records.
Alternatively, the length of the data to be written may be determined according to the type of the data to be written. Illustratively, the types of data include a double-precision floating point type, a single-precision floating point type, a long integer type, an integer type, a short integer type, and a byte type, and the byte length corresponding to each basic type is: double precision floating point type-64 bytes, single precision floating point type-32 bytes, long integer type-64 bytes, integer type-32 bytes, short integer type-16 bytes, byte type-8 bytes.
Data records generated by an application may be abstracted into objects, i.e., data records may be defined as objects. For example, a generated data record is defined as an object a, and the internal data of the object a defines: the first field is of integer type and 32 bytes in length, the second field is of double precision floating point type and 64 bytes in length, and the third field is of byte type and 8 bytes in length. The length of the data record is 32+64+ 8-104 bytes.
If the length of each data record generated by the application is constant, the length of the data to be written is the length of the data record and the number of the data records. The length of the cache block is consistent with the length of the data to be written.
Optionally, the writing the data to be written into the cache block in step S11 includes: and writing the data to be written into the cache block through a write pointer of the cache block. The cache block has a write pointer and a read pointer, the default write pointer and read pointer point to the head position of the cache block, and after data is written into the cache block through the write pointer, the write pointer points to the tail position of the cache block at the moment.
And the data in the cache block can be read through the file channel, and the data is written into the file. Writing data in the cache block to the pre-established file through the pre-established file channel in step S12 includes: and the file channel reads the data in the cache block through the read pointer of the cache block and writes the data into a pre-established file. And the file channel reads the data in the cache block through the read pointer of the cache block and writes the data into the tail part of the pre-established file. The file channel can acquire the tail pointer position of the file, so that data is written into the tail of the file through the file channel according to the tail pointer position of the file.
And after the data to be written in the cache block is written into the file, finishing the writing operation of the data, wherein the read pointer and the write pointer of the cache block are overlapped at the tail part of the cache block. At this time, the data in the cache block is completely emptied, and the read pointer and the write pointer can be reset to point to the head position of the cache block for the next reading and writing. Optionally, after the data in the cache block is written into the pre-established file, the cache block may be retained or deleted.
The database data writing method of the embodiment can be used for storing service data generated by application and storing streaming data generated by application service. The streaming data is a data block generated in an application business process, the size of each type of data block is the same, and the data attributes stored by the same offset are consistent. The method is applied to streaming data, and realizes the rapid data writing of the database.
The embodiment also provides a database data reading method, wherein a folder is established, a file is established in the folder, and a data block is stored in the file, so that the data is stored in the database. Referring to fig. 2, fig. 2 is a flowchart of a database data reading method according to this embodiment, as shown in the figure, the database data reading method includes the following steps:
s20: searching data to be read from a file, and presetting a cache block according to the data to be read, wherein the file is established in a pre-established folder.
When the reading request is acquired, searching a file where the data to be read is located, and searching the data to be read from the file. And pre-sets a buffer block for temporarily storing the data to be read, according to the data to be read. The set cache block is to match the data to be read.
S21: and reading the data to be read from the file through a pre-established file channel, and writing the data into the cache block.
And reading data to be read from the file through a pre-established file channel, and writing the data into a cache block.
S22: and acquiring the data to be read from the cache block. And acquiring data to be read from the buffer block, thereby reading the data from the database.
According to the database data reading method, the folder is established, the file is established in the folder, the data is stored in the file, the data is stored, the data is read in a file mode, the data can be read quickly, compared with the existing database data storage mode, complex database deployment and control programs are not needed, and a lightweight database scheme is achieved.
Optionally, the presetting of the cache block according to the data to be read in step S20 includes: and setting the length of the cache block to be consistent with the length of the data to be read. Therefore, on the premise of ensuring that the data to be read can be completely written into the cache block, the occupation of more storage space is avoided, and system resources are saved.
Specifically, if the data to be read includes one data record, the length of the data to be read is the length of the data record. If the data to be read includes a plurality of data records, the length of the data to be read is the sum of the lengths of the included data records. The specific implementation of determining the data length or the data record length may refer to corresponding descriptions in the database data writing method, and is not described herein again.
Optionally, the reading the data to be read from the file through the pre-established file channel in step S21 includes: calculating an offset according to the length of the data to be read and the position of the file tail pointer; and controlling the position of the file tail pointer to move according to the offset, and reading the data to be read from the file through the file channel.
And calculating the offset required for reading the data to be read from the file and moving the position of the file tail pointer according to the length of the data to be read and the position of the file tail pointer. And controlling the file tail pointer to move according to the obtained offset, reading data to be read from the tail pointer position in the file through the file channel, and writing the data into the cache block.
Preferably, the acquiring data to be read from the buffer block in step S22 includes: the data to be read includes a plurality of data records, each of which is sequentially and respectively acquired from the cache block. If the data to be read includes a plurality of data records, the cache block may be read in segments, and specifically, the data in the cache block may be segmented according to the length of the data record, where the length of the segment is consistent with the length of the data record.
Optionally, the cache block may be deleted or retained as needed after the read operation is completed.
The database data reading method of the embodiment can be used for reading service data generated by application and reading streaming data generated by application service. The streaming data is a data block generated in an application business process, the size of each type of data block is the same, and the data attributes stored by the same offset are consistent. The method is applied to streaming data, and realizes the rapid data reading of the database.
Correspondingly, the embodiment also provides a database, and the database data writing method or the database data reading method is adopted.
The database of the embodiment establishes the folder and the file in the folder, writes the data storage into the file, realizes the data storage, stores or reads the data in a file mode, can realize fast data writing or reading, does not need complex database deployment and control programs compared with the existing database data storage mode, and realizes a lightweight database scheme.
Referring to fig. 3, fig. 3 is a schematic diagram of a database provided in this embodiment, as shown in the figure, a folder 30 is established, a file 31 is established in the folder 30, and a data block is stored in the file 31. The database of this embodiment stores or reads data in a file manner, and its hierarchical structure corresponds to the hierarchical structure of a conventional database as follows: folder-database, file-table, data block-row, database header attribute-row ID.
The database data writing method, the database data reading method and the database provided by the invention are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
1. A database data writing method, comprising:
when data to be written is acquired, presetting a cache block according to the data to be written;
writing the data to be written into the cache block;
and writing the data in the cache block into a pre-established file through a pre-established file channel, wherein the file is established in a pre-established folder.
2. The database data writing method according to claim 1, wherein presetting a cache block according to the data to be written comprises: and establishing a cache block with the length consistent with that of the data to be written according to the data to be written.
3. A database data writing method according to claim 1, wherein the data to be written comprises a plurality of data records, and the length of the data to be written is the sum of the lengths of the data records.
4. The database data writing method according to claim 1, wherein writing the data to be written into the cache block comprises: writing the data to be written into the cache block through a write pointer of the cache block;
writing the data in the cache block into a pre-established file through a pre-established file channel comprises: and the file channel reads the data in the cache block through the read pointer of the cache block and writes the data into a pre-established file.
5. The database data writing method according to claim 1, applied to streaming data generated by application services.
6. A method for reading data from a database, comprising:
searching data to be read from a file, presetting a cache block according to the data to be read, wherein the file is established in a pre-established folder;
reading the data to be read from the file through a pre-established file channel, and writing the data into the cache block;
and acquiring the data to be read from the cache block.
7. The database data reading method according to claim 6, wherein presetting a cache block according to the data to be read comprises: and setting the length of the cache block to be consistent with the length of the data to be read.
8. The database data reading method according to claim 6, wherein reading the data to be read from the file through a pre-established file channel comprises:
calculating an offset according to the length of the data to be read and the position of the file tail pointer;
and controlling the position of the file tail pointer to move according to the offset, and reading the data to be read from the file through the file channel.
9. The database data reading method according to claim 6, wherein the obtaining the data to be read from the cache block comprises: the data to be read includes a plurality of data records, each of which is sequentially and respectively acquired from the cache block.
10. A database, characterized in that the database data writing method of any one of claims 1 to 5 or the database data reading method of any one of claims 6 to 9 is adopted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110920712.5A CN113641709A (en) | 2021-08-11 | 2021-08-11 | Database data writing method, database data reading method and database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110920712.5A CN113641709A (en) | 2021-08-11 | 2021-08-11 | Database data writing method, database data reading method and database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113641709A true CN113641709A (en) | 2021-11-12 |
Family
ID=78420847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110920712.5A Withdrawn CN113641709A (en) | 2021-08-11 | 2021-08-11 | Database data writing method, database data reading method and database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641709A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192226A (en) * | 2006-12-01 | 2008-06-04 | 中兴通讯股份有限公司 | Data access method based on fixed length record database |
| CN103220167A (en) * | 2013-03-25 | 2013-07-24 | 中标软件有限公司 | Distributed monitoring system and data storing method thereof |
| CN109359118A (en) * | 2018-08-31 | 2019-02-19 | 阿里巴巴集团控股有限公司 | A kind of method for writing data and device |
-
2021
- 2021-08-11 CN CN202110920712.5A patent/CN113641709A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101192226A (en) * | 2006-12-01 | 2008-06-04 | 中兴通讯股份有限公司 | Data access method based on fixed length record database |
| CN103220167A (en) * | 2013-03-25 | 2013-07-24 | 中标软件有限公司 | Distributed monitoring system and data storing method thereof |
| CN109359118A (en) * | 2018-08-31 | 2019-02-19 | 阿里巴巴集团控股有限公司 | A kind of method for writing data and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9678969B2 (en) | Metadata updating method and apparatus based on columnar storage in distributed file system, and host | |
| WO2020041928A1 (en) | Data storage method and system and terminal device | |
| US20190361609A1 (en) | Data storage method, apparatus and storage medium | |
| CN106570018B (en) | Serialization and deserialization method, device and system and electronic equipment | |
| CN108388569B (en) | System and establishing method for rapid key value database | |
| CN112286941B (en) | Big data synchronization method and device based on Binlog + HBase + Hive | |
| CN107766374B (en) | Optimization method and system for storage and reading of massive small files | |
| CN111324665A (en) | Log playback method and device | |
| WO2022257575A1 (en) | Data processing method, apparatus, and device | |
| CN107506466B (en) | Small file storage method and system | |
| CN106909623B (en) | A kind of data set and date storage method for supporting efficient mass data to analyze and retrieve | |
| CN113641709A (en) | Database data writing method, database data reading method and database | |
| US20170286442A1 (en) | File system support for file-level ghosting | |
| CN110837338A (en) | Storage index processing method and device | |
| CN114816247A (en) | Logic data acquisition method and device | |
| CN113296693A (en) | Data storage method and device | |
| GB2612727A (en) | Direct data loading of middleware-generated records | |
| CN108776578B (en) | Method and system for rapidly merging objects | |
| CN113468105A (en) | Data structure of data snapshot, related data processing method, device and system | |
| CN112559457A (en) | Data access method and device | |
| WO2022166071A1 (en) | Stream data access method and apparatus in stream data storage system | |
| WO2024001413A1 (en) | Data reading method, data loading apparatus, and communication system | |
| CN116302178B (en) | Acceleration processing method and device for column storage data | |
| CN114925020B (en) | Snapshot version data migration method based on data increment writing mode | |
| WO2023274197A1 (en) | Operation request processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211112 |