CN112540731B - Data append writing method, device, equipment, medium and program product - Google Patents
Data append writing method, device, equipment, medium and program product Download PDFInfo
- Publication number
- CN112540731B CN112540731B CN202011528388.4A CN202011528388A CN112540731B CN 112540731 B CN112540731 B CN 112540731B CN 202011528388 A CN202011528388 A CN 202011528388A CN 112540731 B CN112540731 B CN 112540731B
- Authority
- CN
- China
- Prior art keywords
- index information
- type index
- memory
- writing
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Abstract
The application discloses an additional writing method, device, equipment, medium and program product of data, relates to the field of cloud computing, and particularly relates to a cloud storage technology. The specific implementation scheme is as follows: writing the current data to a disk in an additional way; adding and writing the first type index information of the preset number in the memory to the disk; writing second-type index information of the first-type index information written into the disk into the memory; and writing first type index information of the current data into the memory. The application additionally writes the first type index information of the preset number in the memory to the disk after additionally writing the current data in the disk in the mode of the index and data sharing flow, and then writes the first type index information of the current data in the memory, thereby avoiding the problem of insufficient memory, completely eliminating random writing, fully utilizing the hardware performance and greatly improving the writing speed of the user data.
Description
Technical Field
The present application relates to the field of cloud computing, and in particular, to a cloud storage technology, and in particular, to a method, an apparatus, a device, a medium, and a program product for additionally writing data.
Background
Currently, the main stream nonvolatile storage medium (SSD/HDD) has far higher write-in performance than random write, and the write-in storage system often can fully exhibit hardware performance. Therefore, many storage systems use an additional write disk to improve the write efficiency of the system.
Since the append write changes the original data location, a logical address to physical address mapping (LBA [ logical block address ] -PBA [ Physical block address ]) is maintained, i.e., index information. Therefore, the maintenance mode of the index information will seriously affect the access performance of the system.
Disclosure of Invention
The application provides a data additional writing method, device, equipment, medium and program product, which fully utilize hardware performance and improve writing speed of user data.
In a first aspect, the present application provides a method for writing data additionally, including:
writing the current data to a disk in an additional way;
adding and writing the first type index information of the preset number in the memory to the disk;
writing second-type index information of the first-type index information written into the disk into the memory;
and writing first type index information of the current data into the memory.
In a second aspect, the present application also provides an append writing device for data, including:
the data writing module is used for additionally writing the current data to the magnetic disk;
the first type index information writing module is used for additionally writing the preset number of first type index information in the memory to the disk;
the second type index information writing module is used for writing second type index information of the first type index information written into the disk into the memory;
the first type index information writing module is further configured to write first type index information of the current data in the memory.
In a third aspect, the present application also provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the append write method of data according to any embodiment of the present application.
In a fourth aspect, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of append writing data according to any of the embodiments of the present application.
In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the method for append writing data according to any embodiment of the present application.
It should be understood that the description in this section is not intended to identify key or critical features of the application or to delineate the scope of the application. Other features of the present application will become apparent from the following description, and additional effects of the alternative described above will be apparent from the following description of the specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present application and are not to be construed as limiting the application. Wherein:
FIG. 1 is a flow chart of a method of append writing of data according to an embodiment of the application;
FIG. 2 is a flow chart of a method of append writing of data according to an embodiment of the application;
FIG. 3 is a flow chart of reading data in a method for append writing of data according to an embodiment of the application;
FIG. 4 is a schematic diagram of a data append write apparatus according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device for implementing the method of append writing of data according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a flow chart of an append writing method of data according to an embodiment of the present application, where the embodiment is applicable to a case of writing user data into a disk, and relates to the field of cloud computing, and in particular to a cloud storage technology, for example, a case of writing log data generated by user behavior into a disk. The method may be performed by an add-on writing means of data, which means are implemented in software and/or hardware, preferably arranged in an electronic device, such as a terminal, a computer device or a server. As shown in fig. 1, the method specifically includes the following steps:
s101, additionally writing current data to a magnetic disk.
The current data may be, for example, log data generated in real time by a user action, and the current generated log data is written into a log stream in a disk in an additional writing manner.
S102, adding and writing the first type index information of the preset number in the memory to the disk.
Since the original data location is changed by the additional writing, it is necessary to maintain a mapping relationship of logical addresses to physical addresses, i.e., index information. In the prior art, index information is generally written into a memory, however, the storage space of the memory is limited, and a large amount of user data writing requirements cannot be met. In the embodiment of the application, part of the index information written into the memory can be additionally written into the disk and shared with the user data in the disk, so that space can be saved for the memory, and the limitation of memory capacity is avoided.
In addition, some prior art techniques have adopted a method of writing index information and data all to a disk, however, these prior art techniques specifically write data and index information in areas, that is, index information is stored separately in an index area. When new user data is inserted, the data storage area is updated first, and then the index area is updated. Thus, there are more than two data streams written sequentially, and when two areas are written sequentially, respectively, it appears to the disk that it is a random writing process, especially the HDD needs to switch tracks, resulting in an increase in latency. In the technical scheme of the embodiment of the application, the data and the index information are shared data streams and are written into the same data stream in an additional writing mode, and random writing does not exist, so that the disk performance is utilized to the maximum extent, and delay increase is avoided.
The preset number of the first type index information written into the disk can be configured according to actual conditions, and the embodiment of the application is not limited. In implementation, the first type index information with the preset number can be packaged and written into the memory.
S103, writing second type index information of the first type index information written into the disk into the memory.
When the first type index information in the memory is written into the disk, the index information of the first type index information needs to be established. That is, the first type index information is a mapping relationship between a logical address of data and a real physical address of data in a disk, and the second type index information is a mapping relationship between a logical address of data and the first type index information of data. When reading data, the corresponding first type index information is found in the disk through the second type index information, and then the data is read from the disk through the first type index information, so that the accuracy of data reading and writing is ensured, and the situation that the disk data corresponding to the first type index information cannot be read is avoided.
S104, writing first type index information of the current data in the memory.
After the second type index information of the first type index information written into the preset number of the magnetic disk is written into the memory, the memory space can be vacated in the memory, and then the first type index information of the current data can be written into the memory. When the memory is fully written again, the first type index information of the preset number in the memory can be packed again and written into the memory according to the operation.
According to the technical scheme, after the current data is additionally written into the disk, the first type index information of the preset number in the memory is additionally written into the disk, and then the first type index information of the current data is written into the memory, so that the problem of insufficient memory is avoided through the index and data sharing flow mode, meanwhile, random writing can be completely eliminated, hardware performance is fully utilized, and the writing speed of user data is greatly improved. And simultaneously, writing second-type index information of the first-type index information written into the disk into the memory so as to avoid the loss of disk data corresponding to the first-type index information and ensure the integrity of the data and the correctness of reading and writing when the data is read.
Fig. 2 is a flow chart of a method for append writing of data according to an embodiment of the present application, which is further optimized based on the above embodiment. As shown in fig. 2, the method specifically includes the following steps:
s201, adding and writing current data to a magnetic disk.
S202, judging whether the total number of the first type index information in the memory exceeds a first preset threshold, if not, executing S203, and if so, executing S204.
And S203, writing the first type index information of the current data in the memory in response to the total number of the first type index information in the memory not exceeding a first preset threshold.
That is, if there is enough space in the memory to write the first type index information, and the total number of the first type index information does not exceed the threshold value, the first type index information of the current data can be continuously written into the memory, otherwise, the first type index information with the preset number is packaged and additionally written into the disk.
S204, responding to the total number of the first type index information in the memory exceeding a first preset threshold value, and additionally writing the preset number of the first type index information in the memory to the disk.
Specifically, the first type index information of the preset number of the history writing areas in the memory can be additionally written to the disk; the first type index information with the preset number adjacent to the logic address in the memory can be additionally written to the disk.
In one embodiment, the history writing area refers to an older area in the memory, that is, an area where the index information is written before, and the longer the time of writing the index is, the less the possibility that the user data corresponding to the index is accessed in a short time is, so that the preset number of first type index information of the older area written in the history can be additionally written to the disk according to time, thereby improving the data access efficiency.
As another implementation mode, the probability that the data with adjacent logic addresses are accessed simultaneously is relatively larger, so that the first type index information with the adjacent logic addresses and the preset number can be additionally written to the disk, and the data access efficiency is improved.
It should be noted that the above two embodiments are only examples, and the embodiment of the present application does not limit what first type index information is selected to be additionally written to the disk. Further, the judgment process in S202 is an optional operation.
S205, writing second type index information of the first type index information written into the disk into the memory.
S206, writing the first type index information of the current data in the memory.
In one embodiment, the method for writing data additionally according to the embodiment of the present application may further include:
according to the set period, writing first type index information and second type index information in the current period in the memory into the disk in the form of check points; wherein, the check point is a mark for reloading index information from the disk to the memory when the system is restarted.
Specifically, the index in the memory, including the first type index information and the second type index information, is written into the disk in the form of a checkpoint (checkpoint), so that when the system is restarted, the index information can be loaded into the memory from the checkpoint which is written into the disk last time, and a fast restart effect is achieved. In addition, if the system is between two times of check point writing when restarting, user data corresponding to the last check point written into the disk is determined from the disk, index information is reconstructed for the user data from the last check point written into the disk, and then the index information is written into the memory, so that the data recovery efficiency is improved.
In another embodiment, the method for writing data additionally according to the embodiment of the present application may further include:
segmenting a data stream additionally written in a magnetic disk to obtain at least two data segments, wherein the data stream comprises data and first type index information; if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first index information in the data segment to the head of the data stream, and emptying the disk space of the data segment.
Specifically, operations such as data deletion or overwriting may cause some holes in the data stream, and the space utilization of the disk may be affected by the existence of these holes in the data stream. Therefore, in the embodiment of the application, the data stream is divided into a plurality of data segments, when the holes of any one data segment are more, for example, the second preset threshold value is exceeded, the effective data and/or the first index information in the data segment are migrated to the head of the data stream, and then the whole data segment is emptied, so that space cleaning can be completed, and the space utilization rate of the magnetic disk is improved.
According to the technical scheme, after the current data is additionally written into the disk in the mode of index and data sharing flow, the first type index information of the preset number in the memory is additionally written into the disk, and then the first type index information of the current data is written into the memory, so that the problem of insufficient memory is avoided, meanwhile, random writing can be completely eliminated, the hardware performance is fully utilized, and the writing speed of user data is greatly improved. In addition, the method can realize quick restarting through a check point mode, and improves the space utilization rate of the disk through a data segment migration mode.
Fig. 3 is a schematic flow chart of data reading in the method for append writing of data according to an embodiment of the present application, and the embodiment further optimizes based on the above embodiment. As shown in fig. 3, the method specifically includes the following steps:
s301, according to the logic address in the read data request, corresponding target index information is obtained in the memory.
S302, if the target index information is the first type index information, reading data in the disk according to the first type index information.
S303, if the target index information is the second type index information, acquiring the first type index information corresponding to the logical address from the disk according to the second type index information, and reading data from the disk according to the first type index information.
Specifically, in order to improve the storage space of the memory, the embodiment of the application additionally writes a part of first-type index information stored in the memory to the disk, writes the first-type index information of the current data into the memory after the space is vacated, and writes the second-type index information of the first-type index information written into the disk into the memory, wherein the second-type index information is actually the index information pointing to the first-type index information in the disk and belongs to the index information of the index information. Therefore, when reading data, the index information of the data to be read may be stored in the memory or may have been written to the disk, in which case the index information of the data, that is, the second type index information, is stored in the memory. Therefore, if the target index information obtained in the memory according to the logical address is the first type index information, the data can be read in the disk directly according to the first type index information; if the target index information is the second type index information, the first type index information corresponding to the logical address is required to be acquired in the disk according to the second type index information, and then the data is read in the disk according to the first type index information.
According to the technical scheme, the problem of insufficient memory is avoided by means of index and data sharing flow, random writing can be completely eliminated, hardware performance is fully utilized, writing speed of user data is greatly improved, and data reading efficiency and accuracy are guaranteed.
Fig. 4 is a schematic structural diagram of an additional writing device for data according to an embodiment of the present application, where the embodiment is applicable to a case of writing user data into a disk, and relates to the field of cloud computing, and in particular to a cloud storage technology, for example, a case of writing log data generated by user behavior into a disk. The device can realize the additional writing method of the data according to any embodiment of the application. As shown in fig. 4, the apparatus 400 specifically includes:
a data writing module 401, configured to additionally write current data to a disk;
a first type index information writing module 402, configured to additionally write a preset number of first type index information in the memory to the disk;
a second type index information writing module 403, configured to write second type index information of the first type index information written into the disk into the memory;
the first type index information writing module 401 is further configured to write first type index information of the current data in the memory.
Optionally, the apparatus further includes a determining module, configured to perform, before the first type index information writing module 402 additionally writes, to the disk, a preset number of first type index information in the memory, the following operations:
in response to the total number of the first type index information in the memory exceeding a first preset threshold, the first type index information writing module 402 performs the operation of additionally writing the preset number of the first type index information in the memory to the disk;
in response to the total number of the first type index information in the memory not exceeding the first preset threshold, the first type index information writing module 402 performs the operation of writing the first type index information of the current data in the memory.
Optionally, the first type index information writing module 402 is specifically configured to:
and additionally writing the first type index information with the preset number of the history writing areas in the memory to the magnetic disk.
Optionally, the first type index information writing module 402 is specifically configured to:
and additionally writing the first type index information with the preset number adjacent to the logic address in the memory to the disk.
Optionally, the apparatus further includes a data reading module, configured to perform the following operations:
acquiring corresponding target index information from the memory according to the logic address in the read data request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second type index information, acquiring first type index information corresponding to the logical address from the disk according to the second type index information, and reading data from the disk according to the first type index information.
Optionally, the apparatus further includes:
the check point writing module is used for writing the first type index information and the second type index information in the current period in the memory into the disk in a check point mode according to a set period; wherein the checkpoint is a flag used to reload index information from disk to memory upon a system restart.
Optionally, the apparatus further includes:
the hole processing module is used for segmenting the additionally written data stream in the magnetic disk to obtain at least two data segments, wherein the data stream comprises the data and the first type index information; if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first type index information in the data segment to the head of the data stream, and emptying the disk space of the data segment.
The data append writing device 400 provided by the embodiment of the application can execute the append writing method of the data provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. Reference is made to the description of any method embodiment of the application for details not described in this embodiment.
According to embodiments of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
Fig. 5 illustrates a schematic block diagram of an example electronic device 500 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 5, the apparatus 500 includes a computing unit 501 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, ROM 502, and RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
Various components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, etc.; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508 such as a magnetic disk, an optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 501 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 501 performs the respective methods and processes described above, for example, an append write method of data. For example, in some embodiments, the append write method of data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the append writing method of data described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the append write method of data in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (14)
1. A method of append writing of data, comprising:
writing the current data to a disk in an additional way;
adding and writing the first type index information of the preset number in the memory to the disk;
writing second-type index information of the first-type index information written into the disk into the memory;
writing first type index information of the current data into the memory;
wherein the method further comprises:
segmenting the additionally written data stream in the magnetic disk to obtain at least two data segments, wherein the data stream comprises the data and the first type index information;
if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first type index information in the data segment to the head of the data stream, and emptying the disk space of the data segment.
2. The method of claim 1, further comprising, prior to additionally writing a predetermined number of first type index information in the memory to the disk:
executing the operation of additionally writing the preset number of first type index information in the memory to the disk in response to the total number of the first type index information in the memory exceeding a first preset threshold;
and executing the operation of writing the first type index information of the current data in the memory in response to the total number of the first type index information in the memory not exceeding a first preset threshold.
3. The method of claim 1, wherein additionally writing a preset number of first type index information in the memory to the disk comprises:
and additionally writing the first type index information with the preset number of the history writing areas in the memory to the magnetic disk.
4. The method of claim 1, wherein additionally writing a preset number of first type index information in the memory to the disk comprises:
and additionally writing the first type index information with the preset number adjacent to the logic address in the memory to the disk.
5. The method of claim 1, further comprising:
acquiring corresponding target index information from the memory according to the logic address in the read data request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second type index information, acquiring first type index information corresponding to the logical address from the disk according to the second type index information, and reading data from the disk according to the first type index information.
6. The method of claim 1, further comprising:
according to a set period, writing first type index information and second type index information in the current period in the memory into the disk in the form of check points;
wherein the checkpoint is a flag used to reload index information from disk to memory upon a system restart.
7. An append write apparatus for data, comprising:
the data writing module is used for additionally writing the current data to the magnetic disk;
the first type index information writing module is used for additionally writing the preset number of first type index information in the memory to the disk;
the second type index information writing module is used for writing second type index information of the first type index information written into the disk into the memory;
the first type index information writing module is further configured to write first type index information of the current data in the memory;
wherein the apparatus further comprises:
the hole processing module is used for segmenting the additionally written data stream in the magnetic disk to obtain at least two data segments, wherein the data stream comprises the data and the first type index information; if the number of the holes in any data segment exceeds a second preset threshold, migrating the data and/or the first type index information in the data segment to the head of the data stream, and emptying the disk space of the data segment.
8. The apparatus of claim 7, further comprising a determining module configured to perform the following operations before the first type index information writing module additionally writes a preset number of first type index information in the memory to the disk:
responding to the total number of the first type index information in the memory exceeding a first preset threshold value, and executing the operation of additionally writing the preset number of the first type index information in the memory to the magnetic disk by the first type index information writing module;
and responding to that the total number of the first type index information in the memory does not exceed a first preset threshold value, and executing the operation of writing the first type index information of the current data in the memory by the first type index information writing module.
9. The apparatus of claim 7, wherein the first type index information writing module is specifically configured to:
and additionally writing the first type index information with the preset number of the history writing areas in the memory to the magnetic disk.
10. The apparatus of claim 7, wherein the first type index information writing module is specifically configured to:
and additionally writing the first type index information with the preset number adjacent to the logic address in the memory to the disk.
11. The apparatus of claim 7, further comprising a data reading module to perform operations comprising:
acquiring corresponding target index information from the memory according to the logic address in the read data request;
if the target index information is the first type index information, reading data in the disk according to the first type index information;
and if the target index information is the second type index information, acquiring first type index information corresponding to the logical address from the disk according to the second type index information, and reading data from the disk according to the first type index information.
12. The apparatus of claim 7, further comprising:
the check point writing module is used for writing the first type index information and the second type index information in the current period in the memory into the disk in a check point mode according to a set period; wherein the checkpoint is a flag used to reload index information from disk to memory upon a system restart.
13. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the append write method of data according to any one of claims 1-6.
14. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the append write method of data of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528388.4A CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011528388.4A CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112540731A CN112540731A (en) | 2021-03-23 |
| CN112540731B true CN112540731B (en) | 2023-08-11 |
Family
ID=75019631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011528388.4A Active CN112540731B (en) | 2020-12-22 | 2020-12-22 | Data append writing method, device, equipment, medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112540731B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113127382A (en) * | 2021-04-25 | 2021-07-16 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and medium for additional writing |
| CN113868251B (en) * | 2021-09-24 | 2022-10-18 | 北京百度网讯科技有限公司 | Global secondary indexing method and device for distributed database |
| CN114327278A (en) | 2021-12-30 | 2022-04-12 | 北京百度网讯科技有限公司 | Data additional writing method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11316654A (en) * | 1999-03-04 | 1999-11-16 | Hitachi Ltd | Data access control method, computer system and disk array system |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN105260136A (en) * | 2015-09-24 | 2016-01-20 | 北京百度网讯科技有限公司 | Data read-write method and distributed storage system |
| CN106547490A (en) * | 2016-11-01 | 2017-03-29 | 上海优刻得信息科技有限公司 | Data read-write method and data read-write system |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
-
2020
- 2020-12-22 CN CN202011528388.4A patent/CN112540731B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11316654A (en) * | 1999-03-04 | 1999-11-16 | Hitachi Ltd | Data access control method, computer system and disk array system |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN105260136A (en) * | 2015-09-24 | 2016-01-20 | 北京百度网讯科技有限公司 | Data read-write method and distributed storage system |
| CN106547490A (en) * | 2016-11-01 | 2017-03-29 | 上海优刻得信息科技有限公司 | Data read-write method and data read-write system |
| CN106886375A (en) * | 2017-03-27 | 2017-06-23 | 百度在线网络技术（北京）有限公司 | The method and apparatus of data storage |
| CN111190537A (en) * | 2019-12-10 | 2020-05-22 | 优刻得科技股份有限公司 | Method and system for managing sequential storage disks in write-addition scene |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111625191A (en) * | 2020-05-21 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112540731A (en) | 2021-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112540731B (en) | Data append writing method, device, equipment, medium and program product | |
| CN113127382A (en) | Data reading method, device, equipment and medium for additional writing | |
| US11947842B2 (en) | Method for writing data in append mode, device and storage medium | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN111625546B (en) | Data writing method, device, equipment and medium | |
| CN114996173B (en) | Method and device for managing write operation of storage equipment | |
| CN113364877A (en) | Data processing method, device, electronic equipment and medium | |
| CN113220650B (en) | Data storage method, apparatus, device, storage medium, and program | |
| CN117236236B (en) | Chip design data management method and device, electronic equipment and storage medium | |
| CN113010535B (en) | Cache data updating method, device, equipment and storage medium | |
| CN114035750A (en) | File processing method, device, equipment, medium and product | |
| CN112631517B (en) | Data storage method and device, electronic equipment and storage medium | |
| CN113553216A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN113138718A (en) | Storage method, apparatus, system, and medium for distributed block storage system | |
| CN113986134B (en) | Method for storing data, method and device for reading data | |
| CN115586871B (en) | Cloud computing scene-oriented data additional writing method, device, equipment and medium | |
| CN116126249A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN117539394A (en) | Flash memory array deduplication storage method, device, equipment and medium based on logical volume | |
| CN113722389B (en) | Data management method, device, electronic equipment and computer readable storage medium | |
| CN112069794B (en) | Text prediction method, device, equipment and storage medium | |
| CN113220233A (en) | Data reading method, device and system | |
| EP3907648A2 (en) | Text prediction method and apparatus, device and storage medium | |
| CN117420943A (en) | Disk wear balancing method and device in distributed cluster, electronic equipment and medium | |
| CN117931689A (en) | Data processing method, device, chip, equipment and storage medium | |
| CN116737453A (en) | Data dump method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |