CN112540731A - Data additional writing method, device, equipment, medium and program product - Google Patents
Data additional writing method, device, equipment, medium and program product Download PDFInfo
- Publication number
- CN112540731A CN112540731A CN202011528388.4A CN202011528388A CN112540731A CN 112540731 A CN112540731 A CN 112540731A CN 202011528388 A CN202011528388 A CN 202011528388A CN 112540731 A CN112540731 A CN 112540731A
- Authority
- CN
- China
- Prior art keywords
- index information
- data
- memory
- disk
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The application discloses a method, a device, equipment, a medium and a program product for additionally writing data, relates to the field of cloud computing, and particularly relates to a cloud storage technology. The specific implementation scheme is as follows: additionally writing the current data to a disk; additionally writing the first-class index information with the preset number in the memory to the disk; writing second type index information of the first type index information written into the disk into the memory; and writing the first type index information of the current data into the memory. According to the method and the device, after the current data is additionally written into the disk in an index and data sharing flow mode, the first type index information with the preset number in the memory is also additionally written into the disk, and then the first type index information of the current data is written into the memory, so that the problem of insufficient memory is avoided, meanwhile, random writing can be completely eliminated, the hardware performance is fully utilized, and the writing speed of user data is greatly improved.
Description
Technical Field
The present application relates to the field of cloud computing, and in particular, to a cloud storage technology, and more particularly, to a method, an apparatus, a device, a medium, and a program product for additionally writing data.
Background
At present, the additional write performance of the mainstream nonvolatile storage medium (SSD/HDD) is much higher than that of random write, and the additionally written storage system can fully exert the hardware performance. Therefore, many storage systems adopt a method of additionally writing a disk to improve the writing efficiency of the system.
Since the additional writing changes the original data location, a mapping relationship between a logical address and a Physical address (LBA [ local block address ] — PBA [ Physical block address ]), that is, index information needs to be maintained. Therefore, the access performance of the system is seriously affected by the maintenance mode of the index information.
Disclosure of Invention
The application provides a method, a device, equipment, a medium and a program product for additionally writing data, which are used for fully utilizing hardware performance and improving the writing speed of user data.
In a first aspect, the present application provides a method for additionally writing data, including:
additionally writing the current data to a disk;
additionally writing the first-class index information with the preset number in the memory to the disk;
writing second type index information of the first type index information written into the disk into the memory;
and writing the first type index information of the current data into the memory.
In a second aspect, the present application further provides an additional writing apparatus for data, including:
the data writing module is used for additionally writing the current data to a magnetic disk;
the first-class index information writing module is used for additionally writing the first-class index information with the preset number in the memory to the disk;
a second type index information writing module, configured to write second type index information of the first type index information written to the disk into the memory;
the first-class index information writing module is further configured to write the first-class index information of the current data in the memory.
In a third aspect, the present application further provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of appending write data as described in any embodiment of the present application.
In a fourth aspect, the present application further provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method of additionally writing data according to any of the embodiments of the present application.
In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method of appending writing of data according to any embodiment of the present application.
It should be understood that the statements herein do not intend to identify key or critical features of the present application, nor to limit the scope of the present application. Other features of the present application will become readily apparent from the following description, and other effects of the above alternatives will be described hereinafter in conjunction with specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a flow chart illustrating a method for appending writing data according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating a method for appending writing data according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of reading data in a method for appending and writing data according to an embodiment of the present application;
FIG. 4 is a schematic diagram of an apparatus for additionally writing data according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device for implementing the additional data writing method according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic flow chart of a data additional writing method according to an embodiment of the present application, which is applicable to a case where user data is written into a disk, and relates to the field of cloud computing, in particular to a cloud storage technology, for example, a case where log data generated by a user behavior is written into a disk. The method can be executed by an additional data writing device, which is implemented by software and/or hardware, and is preferably configured in an electronic device, such as a terminal, a computer device or a server. As shown in fig. 1, the method specifically includes the following steps:
s101, additionally writing the current data to a magnetic disk.
The current data may be, for example, log data generated in real time by a user behavior, and the currently generated log data is written into a log stream in a disk by an additional writing method.
And S102, additionally writing the first-class index information with the preset number in the memory to a disk.
Since the original data position is changed by the additional writing, a mapping relationship from a logical address to a physical address, i.e. index information, needs to be maintained. In the prior art, index information is usually written into a memory, but the memory space of the memory is limited, and the requirement for writing a large amount of user data cannot be met. In the embodiment of the present application, a part of the index information written into the memory may also be additionally written to the disk, and share the data stream with the user data in the disk, so that a space may be made for the memory, and the memory capacity limitation is avoided.
In addition, some prior arts also adopt a method of writing index information and data into a disk, however, these prior arts specifically write data and index information into a partition, that is, the index information is stored in an index area separately. When new user data is inserted, the storage area of the data is updated first, and then the index area is updated. Thus, there are more than two data streams written sequentially, and when two areas are written sequentially, it appears to the disk as a random writing process, especially the HDD needs to switch tracks, resulting in increased latency. In the technical scheme of the embodiment of the application, the data and the index information are shared data streams, and are written into the same data stream in an additional writing mode without random writing, so that the performance of a magnetic disk is utilized to the maximum extent, and the increase of time delay is avoided.
The preset number of the first-type index information written into the disk can be configured according to actual conditions, and the embodiment of the present application is not limited at all. In implementation, a preset number of first-class index information can be packaged and written into the memory.
S103, writing the second type index information of the first type index information written into the disk into the memory.
After the first-type index information in the memory is written into the disk, the index information of the first-type index information needs to be established, which is referred to as second-type index information in this embodiment of the present application. That is, the first type of index information is a mapping relationship between a logical address of data and a real physical address of the data in the disk, and the second type of index information is a mapping relationship between a logical address of data and the first type of index information of data. Therefore, when data is read, the corresponding first-class index information is found in the disk through the second-class index information, and then the data is read from the disk through the first-class index information, so that the correctness of data reading and writing is ensured, and the condition that the disk data corresponding to the first-class index information cannot be read is avoided.
And S104, writing the first type index information of the current data in the memory.
After the second type index information of the preset number of first type index information written into the disk is written into the memory, the memory space can be vacated, and then the first type index information of the current data can be written into the memory. When the memory is full again, the preset number of the first-class index information in the memory can be packed again and written into the memory according to the above operation.
According to the technical scheme of the embodiment of the application, after the current data is additionally written in the disk, the first type index information with the preset number in the memory is also additionally written in the disk, and then the first type index information of the current data is written in the memory, so that the problem of insufficient memory is avoided in a mode of sharing stream by index and data, random writing can be completely eliminated, hardware performance is fully utilized, and the writing speed of user data is greatly improved. Meanwhile, the second type of index information of the first type of index information written into the disk is written into the memory, so that when data is read, the data loss of the disk corresponding to the first type of index information is avoided, and the integrity of the data and the correctness of reading and writing are ensured.
Fig. 2 is a schematic flow chart of a method for additionally writing data according to an embodiment of the present application, and the present embodiment is further optimized based on the above embodiments. As shown in fig. 2, the method specifically includes the following steps:
s201, additionally writing the current data to a magnetic disk.
S202, judging whether the total number of the first type index information in the memory exceeds a first preset threshold value, if not, executing S203, and if so, executing S204.
S203, in response to that the total number of the first-class index information in the memory does not exceed a first preset threshold, writing the first-class index information of the current data in the memory.
That is, if there is enough space in the memory to write the first-class index information, and the total number of the first-class index information does not exceed the threshold, the first-class index information of the current data can be continuously written into the memory, otherwise, the preset number of the first-class index information is packaged and additionally written into the disk.
And S204, in response to the fact that the total number of the first-class index information in the memory exceeds a first preset threshold value, additionally writing the first-class index information with the preset number in the memory to a disk.
Specifically, the first-class index information of the preset number in the history write-in area in the memory can be additionally written to the disk; or additionally writing the first-class index information of the preset number adjacent to the logical address in the memory to the disk.
In one embodiment, the history writing area refers to an older area in the memory, that is, an area in which the index information is written before, and since the longer the index writing time is, the less the possibility that the user data corresponding to the index is accessed in a short time is, the first type index information of the preset number of the older areas written in the history can be additionally written to the disk according to the time, thereby improving the data access efficiency.
As another embodiment, because the probability that data adjacent to the logical address is accessed at the same time is relatively greater, the preset number of pieces of first-class index information adjacent to the logical address can be additionally written to the disk, so as to improve the data access efficiency.
It should be further noted that the above two embodiments are only examples, and the examples of the present application do not limit which first-type index information is selected to be additionally written to the disk. Further, the determination process in S202 is an optional operation.
S205, writing the second type index information of the first type index information written into the disk into the memory.
And S206, writing the first type index information of the current data in the memory.
In one implementation, the method for additionally writing data according to the embodiment of the present application may further include:
writing the first type index information and the second type index information in the current period in the memory into a disk in a check point mode according to a set period; the checkpoint is a mark used for reloading index information from a disk into a memory when the system is restarted.
Specifically, the index in the memory, including the first-class index information and the second-class index information, may be written into the disk periodically in the form of a checkpoint (checkpoint), so that, when the system is restarted, the index information may be loaded into the memory from the checkpoint that was written into the disk last time, and a quick restart function is performed. In addition, if the system is between two checkpoint writings during the restart, the corresponding user data after the checkpoint written into the disk for the last time is determined from the disk, the index information is reconstructed for the user data from the checkpoint written into the disk for the last time, and then the index information is written into the memory, so that the efficiency of data recovery is improved.
In another embodiment, the method for additionally writing data according to the embodiment of the present application may further include:
segmenting the additionally written data stream in the disk to obtain at least two data segments, wherein the data stream comprises data and first-class index information; and if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first-class index information in the data segment to the head of the data stream, and emptying the disk space of the data segment.
Specifically, operations such as data deletion or overwrite all cause some holes in the data stream, and the existence of these holes in the data stream affects the space utilization rate of the disk. Therefore, in the embodiment of the present application, the data stream is divided into a plurality of data segments, when there are more holes in any data segment, for example, exceeding the second preset threshold, the valid data and/or the first type index information in the data segment are migrated to the head of the data stream, and then the whole data segment is emptied, so that space cleaning can be completed, thereby improving the space utilization rate of the disk.
According to the technical scheme of the embodiment of the application, after the current data is additionally written in the disk in an index and data sharing flow mode, the first type index information of the preset number in the memory is also additionally written in the disk, and then the first type index information of the current data is written in the memory, so that the problem of insufficient memory is avoided, random writing can be completely eliminated, the hardware performance is fully utilized, and the writing speed of user data is greatly improved. In addition, the quick restart can be realized through a checkpoint mode, and the space utilization rate of the disk is improved through a data segment migration mode.
Fig. 3 is a schematic flow chart of reading data in a method for additionally writing data according to an embodiment of the present application, and the present embodiment is further optimized based on the foregoing embodiments. As shown in fig. 3, the method specifically includes the following steps:
s301, acquiring corresponding target index information in the memory according to the logical address in the data reading request.
S302, if the target index information is the first-class index information, reading data in the disk according to the first-class index information.
S303, if the target index information is the second-class index information, acquiring the first-class index information corresponding to the logical address in the disk according to the second-class index information, and reading data in the disk according to the first-class index information.
Specifically, in order to increase the storage space of the memory, the embodiment of the present application additionally writes a part of the first type index information stored in the memory to the disk, and writes the first type index information of the current data into the memory after the space is vacated, and writes the second type index information of the first type index information written into the disk into the memory, where the second type index information is actually the first type index information pointing to the disk and belongs to the index information of the index information. Therefore, when reading data, the index information of the data to be read may be stored in the memory or may have been written to the disk, in which case the index information of the data, i.e., the second type of index information, is stored in the memory. Therefore, if the target index information acquired in the memory according to the logical address is the first-class index information, the data can be directly read from the disk according to the first-class index information; if the target index information is the second-type index information, the first-type index information corresponding to the logical address needs to be acquired in the disk according to the second-type index information, and then data is read in the disk according to the first-type index information.
According to the technical scheme of the embodiment of the application, the problem of insufficient memory is avoided through the mode of index and data sharing stream, random writing can be completely eliminated, hardware performance is fully utilized, the writing speed of user data is greatly improved, and the efficiency and the correctness of data reading are also ensured.
Fig. 4 is a schematic structural diagram of an additional data writing device according to an embodiment of the present application, where the embodiment is applicable to a case where user data is written in a disk, and relates to the field of cloud computing, in particular to a cloud storage technology, for example, a case where log data generated by a user behavior is written in a disk. The device can realize the additional writing method of the data according to any embodiment of the application. As shown in fig. 4, the apparatus 400 specifically includes:
a data writing module 401, configured to write current data to a disk in an appending manner;
a first-class index information writing module 402, configured to write a preset number of first-class index information in the memory to the disk additionally;
a second-type index information writing module 403, configured to write second-type index information of the first-type index information written in the disk into the memory;
the first-class index information writing module 401 is further configured to write the first-class index information of the current data in the memory.
Optionally, the apparatus further includes a determining module, configured to execute the following operations before the first-type index information writing module 402 additionally writes a preset number of first-type index information in the memory to the disk:
in response to that the total number of the first-class index information in the memory exceeds a first preset threshold, the first-class index information writing module 402 performs the operation of additionally writing a preset number of first-class index information in the memory to the disk;
in response to that the total number of the first-class index information in the memory does not exceed a first preset threshold, the first-class index information writing module 402 executes the operation of writing the first-class index information of the current data in the memory.
Optionally, the first-type index information writing module 402 is specifically configured to:
and additionally writing the first type index information of the preset number in the history writing area in the memory to the disk.
Optionally, the first-type index information writing module 402 is specifically configured to:
and additionally writing the first-class index information of the preset number adjacent to the logic address in the memory to the disk.
Optionally, the apparatus further includes a data reading module, configured to perform the following operations:
acquiring corresponding target index information in the memory according to the logic address in the data reading request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second-class index information, acquiring first-class index information corresponding to the logical address in the disk according to the second-class index information, and reading data in the disk according to the first-class index information.
Optionally, the apparatus further comprises:
a checkpoint writing module, configured to write the first type index information and the second type index information in the current period in the memory into the disk in a checkpoint form according to a set period; the checkpoint is a mark used for reloading index information from a disk to a memory when the system is restarted.
Optionally, the apparatus further comprises:
the hole processing module is used for segmenting a data stream additionally written in the disk to obtain at least two data segments, wherein the data stream comprises the data and the first type of index information; and if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first-class index information in the data segment to the head of the data stream, and clearing the disk space of the data segment.
The data additional writing device 400 provided by the embodiment of the present application can execute the data additional writing method provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. Reference may be made to the description of any method embodiment of the present application for details not explicitly described in this embodiment.
There is also provided, in accordance with an embodiment of the present application, an electronic device, a readable storage medium, and a computer program product.
FIG. 5 illustrates a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 5, the apparatus 500 comprises a computing unit 501 which may perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM)502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The calculation unit 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
A number of components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, or the like; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The computing unit 501 may be a variety of general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 501 executes the respective methods and processes described above, such as an additional writing method of data. For example, in some embodiments, the method of appending write data may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the above-described method of appending writing of data may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured in any other suitable manner (e.g., by means of firmware) to perform an append write method of data.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (17)
1. A method of appending write data, comprising:
additionally writing the current data to a disk;
additionally writing the first-class index information with the preset number in the memory to the disk;
writing second type index information of the first type index information written into the disk into the memory;
and writing the first type index information of the current data into the memory.
2. The method according to claim 1, before the additionally writing a preset number of first-class index information in the memory to the disk, the method further comprising:
responding to the fact that the total number of the first-class index information in the memory exceeds a first preset threshold value, and executing the operation of additionally writing the first-class index information with the preset number in the memory to the disk;
and in response to that the total number of the first-class index information in the memory does not exceed a first preset threshold, executing the operation of writing the first-class index information of the current data in the memory.
3. The method of claim 1, wherein additionally writing a predetermined number of first-class index information in the memory to the disk comprises:
and additionally writing the first type index information of the preset number in the history writing area in the memory to the disk.
4. The method of claim 1, wherein additionally writing a predetermined number of first-class index information in the memory to the disk comprises:
and additionally writing the first-class index information of the preset number adjacent to the logic address in the memory to the disk.
5. The method of claim 1, further comprising:
acquiring corresponding target index information in the memory according to the logic address in the data reading request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second-class index information, acquiring first-class index information corresponding to the logical address in the disk according to the second-class index information, and reading data in the disk according to the first-class index information.
6. The method of claim 1, further comprising:
writing the first type index information and the second type index information in the current period in the memory into the disk in a check point mode according to a set period;
the checkpoint is a mark used for reloading index information from a disk to a memory when the system is restarted.
7. The method of claim 1, further comprising:
segmenting the additionally written data stream in the disk to obtain at least two data segments, wherein the data stream comprises the data and the first type of index information;
and if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first-class index information in the data segment to the head of the data stream, and clearing the disk space of the data segment.
8. An apparatus for appending write data, comprising:
the data writing module is used for additionally writing the current data to a magnetic disk;
the first-class index information writing module is used for additionally writing the first-class index information with the preset number in the memory to the disk;
a second type index information writing module, configured to write second type index information of the first type index information written to the disk into the memory;
the first-class index information writing module is further configured to write the first-class index information of the current data in the memory.
9. The apparatus according to claim 8, further comprising a determining module, configured to perform the following operations before the first-class index information writing module additionally writes a preset number of first-class index information in the memory to the disk:
in response to that the total number of the first-class index information in the memory exceeds a first preset threshold value, the first-class index information writing module executes the operation of additionally writing the first-class index information with the preset number in the memory to the disk;
and in response to that the total number of the first-class index information in the memory does not exceed a first preset threshold value, the first-class index information writing module executes the operation of writing the first-class index information of the current data in the memory.
10. The apparatus according to claim 8, wherein the first-type index information writing module is specifically configured to:
and additionally writing the first type index information of the preset number in the history writing area in the memory to the disk.
11. The apparatus according to claim 8, wherein the first-type index information writing module is specifically configured to:
and additionally writing the first-class index information of the preset number adjacent to the logic address in the memory to the disk.
12. The apparatus of claim 8, further comprising a data reading module to:
acquiring corresponding target index information in the memory according to the logic address in the data reading request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second-class index information, acquiring first-class index information corresponding to the logical address in the disk according to the second-class index information, and reading data in the disk according to the first-class index information.
13. The apparatus of claim 8, further comprising:
a checkpoint writing module, configured to write the first type index information and the second type index information in the current period in the memory into the disk in a checkpoint form according to a set period; the checkpoint is a mark used for reloading index information from a disk to a memory when the system is restarted.
14. The apparatus of claim 8, further comprising:
the hole processing module is used for segmenting a data stream additionally written in the disk to obtain at least two data segments, wherein the data stream comprises the data and the first type of index information; and if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first-class index information in the data segment to the head of the data stream, and clearing the disk space of the data segment.
15. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of append writing of data as recited in any one of claims 1-7.
16. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method of appending a write of data according to any one of claims 1 to 7.
17. A computer program product comprising a computer program which, when executed by a processor, implements a method of append writing of data according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528388.4A CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528388.4A CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112540731A true CN112540731A (en) | 2021-03-23 |
| CN112540731B CN112540731B (en) | 2023-08-11 |
Family
ID=75019631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011528388.4A Active CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112540731B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127382A (en) * | 2021-04-25 | 2021-07-16 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and medium for additional writing |
| CN113868251A (en) * | 2021-09-24 | 2021-12-31 | 北京百度网讯科技有限公司 | Global secondary indexing method and device for distributed database |
| CN114327278A (en) * | 2021-12-30 | 2022-04-12 | 北京百度网讯科技有限公司 | Data additional writing method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11316654A (en) * | 1999-03-04 | 1999-11-16 | Hitachi Ltd | Data access control method, computer system and disk array system |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN105260136A (en) * | 2015-09-24 | 2016-01-20 | 北京百度网讯科技有限公司 | Data read-write method and distributed storage system |
| CN106547490A (en) * | 2016-11-01 | 2017-03-29 | 上海优刻得信息科技有限公司 | Data read-write method and data read-write system |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
-
2020
- 2020-12-22 CN CN202011528388.4A patent/CN112540731B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11316654A (en) * | 1999-03-04 | 1999-11-16 | Hitachi Ltd | Data access control method, computer system and disk array system |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN105260136A (en) * | 2015-09-24 | 2016-01-20 | 北京百度网讯科技有限公司 | Data read-write method and distributed storage system |
| CN106547490A (en) * | 2016-11-01 | 2017-03-29 | 上海优刻得信息科技有限公司 | Data read-write method and data read-write system |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127382A (en) * | 2021-04-25 | 2021-07-16 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and medium for additional writing |
| CN113868251A (en) * | 2021-09-24 | 2021-12-31 | 北京百度网讯科技有限公司 | Global secondary indexing method and device for distributed database |
| CN114327278A (en) * | 2021-12-30 | 2022-04-12 | 北京百度网讯科技有限公司 | Data additional writing method, device, equipment and storage medium |
| EP4141641A3 (en) * | 2021-12-30 | 2023-06-28 | Beijing Baidu Netcom Science And Technology Co. Ltd. | Method and apparatus for writing data in append mode, device and storage medium |
| US11947842B2 (en) | 2021-12-30 | 2024-04-02 | Beijing Baidu Netcom Science Technology, Co., Ltd. | Method for writing data in append mode, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112540731B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112540731B (en) | Data append writing method, device, equipment, medium and program product | |
| CN103019888B (en) | Backup method and device | |
| CN114327278A (en) | Data additional writing method, device, equipment and storage medium | |
| CN113127382A (en) | Data reading method, device, equipment and medium for additional writing | |
| CN112346647B (en) | Data storage method, device, equipment and medium | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN109918352B (en) | Memory system and method of storing data | |
| CN114996173B (en) | Method and device for managing write operation of storage equipment | |
| CN113220650B (en) | Data storage method, apparatus, device, storage medium, and program | |
| CN117236236A (en) | Chip design data management method and device, electronic equipment and storage medium | |
| CN104899158A (en) | Memory access optimization method and memory access optimization device | |
| CN112631517B (en) | Data storage method and device, electronic equipment and storage medium | |
| CN113868254B (en) | Method, device and storage medium for removing duplication of entity node in graph database | |
| CN115712388A (en) | Data storage method, device and equipment of solid-state disk and storage medium | |
| CN115617802A (en) | Method and device for quickly generating full snapshot, electronic equipment and storage medium | |
| CN114528229A (en) | Cache data access method and device and electronic equipment | |
| CN114518848A (en) | Hierarchical storage system, and method, apparatus, device, and medium for processing storage data | |
| CN113138718A (en) | Storage method, apparatus, system, and medium for distributed block storage system | |
| CN115657940A (en) | Data compression method, device and system | |
| CN113986134B (en) | Method for storing data, method and device for reading data | |
| CN113220233A (en) | Data reading method, device and system | |
| CN113553216B (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN113553216A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN117539394A (en) | Flash memory array deduplication storage method, device, equipment and medium based on logical volume | |
| CN116126249A (en) | Data reading method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |