WO2023015866A1 - 一种数据写入方法、装置、系统、电子设备及存储介质 - Google Patents

一种数据写入方法、装置、系统、电子设备及存储介质 Download PDF

Info

Publication number
WO2023015866A1
WO2023015866A1 PCT/CN2022/078185 CN2022078185W WO2023015866A1 WO 2023015866 A1 WO2023015866 A1 WO 2023015866A1 CN 2022078185 W CN2022078185 W CN 2022078185W WO 2023015866 A1 WO2023015866 A1 WO 2023015866A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
written
ring buffer
persistent memory
node device
Prior art date
Application number
PCT/CN2022/078185
Other languages
English (en)
French (fr)
Inventor
秦朝阳
Original Assignee
苏州浪潮智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州浪潮智能科技有限公司 filed Critical 苏州浪潮智能科技有限公司
Publication of WO2023015866A1 publication Critical patent/WO2023015866A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present application relates to the technical field of data storage, and in particular to a data writing method, device, system, electronic equipment, and computer-readable storage medium.
  • Persistent Memory Persistent Memory
  • PMem Persistent Memory
  • DRAM Dynamic Random Memory Access memory
  • non-volatile devices such as traditional mechanical hard drives and solid-state drives
  • persistent memory devices are more expensive and costly to use as storage media on a large scale.
  • users still widely use mechanical hard disks and NAS (Network Attached Storage, network storage) as the main storage, so the data writing speed is limited by the consumption of small data blocks and low bandwidth; among them,
  • the implementation of cross-host data partitioning uses slow network copying, which further reduces the data writing speed and may cause more data loss in the event of computer room failures (such as power outages and network fluctuations, etc.). Therefore, how to increase the writing speed of data, reduce the possibility of data loss during failure, and improve user experience is an urgent problem to be solved today.
  • the purpose of this application is to provide a data writing method, device, system, electronic equipment, and computer-readable storage medium, so as to use persistent memory devices to increase the writing speed of data, reduce the possibility of data loss during failure, and improve user experience.
  • this application provides a data writing method, including:
  • write the data to be written to the ring buffer of the persistent memory device wherein, the working mode of the persistent memory device is an application direct access mode;
  • the data to be written is written to the ring buffer of the persistent memory device according to byte addressing, including:
  • the current data unit is any data unit in the circular double-linked list
  • next data unit is the data pointed to by the back pointer in the current data unit unit, the back pointer in the last data unit in the circular doubly linked list points to the first data unit.
  • the acquiring the data to be written includes:
  • the master node device obtains the data to be written corresponding to the data write request, and sends the data to be written to the synchronization node device, wherein the persistent memory device is set in the master node device.
  • the acquiring the data to be written includes:
  • the synchronization node device copies the data to be written received by the master node device through the RDMA network, wherein the synchronization node device is provided with the persistent memory device.
  • the synchronization node device copies the data to be written received by the master node device through the RDMA network, including:
  • the synchronization node device uses an RDMA network card to receive the data to be written sent by the master node device through the RDMA network.
  • storing the data to be written in the ring buffer into a data storage includes:
  • the aggregated file is stored in a data store.
  • storing the data to be written in the ring buffer into a data storage includes:
  • the The stored data is the data to be written stored in the data memory.
  • the data to be written in the ring buffer is stored in the data memory, and the data in the ring buffer is cleared.
  • store data including:
  • the present application also provides a data writing device, including:
  • the data receiving module is used to obtain the data to be written; according to byte addressing, write the data to be written into the ring buffer of the persistent memory device; wherein, the working mode of the persistent memory device is direct access by the application model;
  • a data storage module configured to store the data to be written in the ring buffer into a data storage.
  • the present application also provides an electronic device, comprising:
  • a processor configured to implement the steps of the above-mentioned data writing method when executing the computer program.
  • the present application also provides a data writing system, including: a master node device and a synchronization node device connected to the master node device; wherein, the master node device and the synchronization node device are both electronic equipment.
  • the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned data writing method are realized.
  • a data writing method provided by the present application includes: acquiring data to be written; writing the data to be written to the ring buffer of the persistent memory device according to byte addressing; wherein, the working mode of the persistent memory device To apply the direct access mode; store the data to be written in the ring buffer into the data memory;
  • the present application writes the data to be written into the ring buffer of the persistent memory device according to byte addressing, and utilizes the writing speed and bandwidth provided by the persistent memory device to be several times higher than that of the traditional data storage, thereby improving the data storage efficiency.
  • the writing speed reduces the possibility of data loss in the event of a failure; and the ring buffer in the persistent memory device is used for transit storage of the data to be written, which can continuously write the data to be written and improve the speed of data persistence
  • the cost of technology use is reduced, and the technical effect of "shaving peaks and filling valleys" is realized.
  • the present application also provides a data writing device, system, electronic equipment, and computer-readable storage medium, which also have the above beneficial effects.
  • FIG. 1 is a flowchart of a data writing method provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of remote synchronization of another data writing method provided by the embodiment of the present application.
  • FIG. 3 is a schematic flow chart of a data writing method in the prior art
  • FIG. 4 is a schematic structural diagram of a data writing system provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a ring buffer of another data writing method provided by an embodiment of the present application.
  • FIG. 6 is a structural block diagram of a data writing device provided by an embodiment of the present application.
  • FIG. 1 is a flow chart of a data writing method provided by an embodiment of the present application.
  • the method can include:
  • Step 101 Obtain data to be written.
  • the data to be written in this step may be data that needs to be written and stored in the electronic device.
  • the specific content of the data to be written in this embodiment can be set by the designer according to the usage scenario and user needs.
  • the data to be written can be written to the master node device for storage data;
  • the data to be written may be data that needs to be synchronously written and stored in the synchronous node device.
  • the data writing method provided in this embodiment can be applied to the master node device, that is, the processor in the electronic device serving as the master node device can execute the method provided in this embodiment to write The data in the master node device (that is, the data to be written) is written and stored; the data writing method provided in this embodiment can also be applied to the synchronization node device (that is, the slave node device) corresponding to the master node device, that is, That is to say, the processor in the electronic device as a synchronous node device can execute the method provided by this embodiment to synchronously store the data written in the master node device (that is, the data to be written), so as to realize the storage in the master node device Content data synchronization and disaster recovery functions.
  • This embodiment does not impose any limitation on this.
  • the specific method for the processor of the electronic device to obtain the data to be written in this step can be set by the designer according to the usage scenario and user needs.
  • the processor of the master node device can Obtain the data to be written corresponding to the data write request, that is, the master node device can receive the data write request, and obtain the data to be written (that is, the data to be written); further, the master node device can obtain the data to be written Finally, the data to be written is sent to the synchronization node device, so that the synchronization node device can store the data written to the master node device, and realize cross-host data synchronization and disaster recovery backup.
  • the processor of the synchronous node device can obtain the data to be written received by the master node device, so as to store the data to be written written into the master node device, and realize data synchronization and disaster recovery backup of the host .
  • This embodiment does not impose any limitation on this.
  • the processor of the synchronous node device in this step can obtain the data to be written received by the master node device through the RDMA (Remote Direct Memory Access, remote direct data access) network, that is, the synchronous node
  • the device can be in the RDMA network with the corresponding master node device, so that the sync node device can use the RDMA network card to quickly copy the data that needs to be written (that is, the data to be written) received by the master node device through the RDMA network, and the data in the master node device
  • the stored data is synchronized to the synchronization node device almost without delay, such as another server in the same computer room as the server serving as the master node device.
  • persistent memory devices can be inserted into both the master node device and the sync node device, and an ultra-low-latency network (RDMA network) is configured between the master node device and the sync node device to use the persistent memory device as a storage medium and key value
  • RDMA network ultra-low-latency network
  • the storage engine using efficient RDMA-driven network transport as the network layer, enables the persistent memory devices on the primary node device and the sync node device to act as a replica pool set.
  • the persistent memory device of host 1 can form a copy pool set, which is used as the persistent memory 1 of host 1 of the master node device
  • host 1 can use the remote replication service (such as using the remote persistent memory background service rpmemd tool) to immediately transmit the data at high speed to host 2 as a synchronization node device through the RDMA network
  • the host 2 can use the remote copy service to immediately call the persistent memory driver to write the data received through the RDMA network into the persistent memory 2 to complete the data copy, based on the high bandwidth of the persistent memory device and the low latency of the RDMA network in a very short time Finish.
  • Step 102 According to byte addressing, write the data to be written to the ring buffer of the persistent memory device; wherein, the working mode of the persistent memory device is the application direct access mode.
  • the persistent memory device in this step may be a device using persistent memory (PMem) for data storage.
  • the ring buffer of the persistent memory device in this step may be a data structure of a ring buffer created in the persistent memory device, such as a ring double-linked list.
  • the AD mode App Direct Mode, that is, the application direct access mode
  • Persistent memory because the write speed of persistent memory is several to dozens of times faster than the file I/O speed of traditional storage devices, it can greatly improve the efficiency of data writing. As shown in Table 1, the write speed of persistent memory devices The input efficiency can reach hundreds of times that of mechanical hard drives; among them, IOPS can be the number of write operations completed per second.
  • the buffer for storing the data to be written is circular, the earliest written data will be overwritten when the data is full, which can reduce the storage space of the persistent memory device occupied by the data writing speed and reduce the cost.
  • the specific method of writing the data to be written to the ring buffer of the persistent memory device can be set by the designer according to the practical scenario and user needs, such as the processor
  • the data to be written can be written to a ring buffer of the persistent memory device, that is, only one ring buffer for storing the data to be written can be set in this embodiment; the processor can also write the data to be written into Multiple ring buffers to the persistent memory device, that is, multiple ring buffers for storing data to be written can be set in this embodiment.
  • This embodiment does not impose any limitation on this.
  • the traditional data writing scheme uses the file system standard API (Application Programming Interface) to write data, which cannot use the direct access (DAX) feature of the kernel, the application cannot directly address, and the writing process involves To multiple memory buffer copy operations, the speed is relatively slow. As shown in Figure 3, it generally needs to go through three steps: memory buffer writing, file system I/O (input/output) transfer, and data storage device storage. Guarantee data persistence; and in this embodiment, the persistent memory device is used as a character device, or the persistent memory device is mounted as a block device of the file system.
  • DAX direct access
  • the direct access mode is applied, and the processor running the application program It can be addressed by byte, and the data persistence of the data to be written can be completed by writing the data to be written into the ring buffer, and the data writing is declared successful. For example, if an application wants to update 64 bytes of content, according to the traditional data writing scheme, the processor needs to read the entire storage block containing these 64 bytes into the memory buffer, update the 64 bytes, and then store Devices (such as mechanical hard disks) write entire blocks of storage to make them persistent.
  • Devices such as mechanical hard disks
  • the persistent memory device is used to update the content of 64 bytes, and only need to write 64 bytes directly to the persistent memory device.
  • the master node device can use the ring buffer (persistent memory ring buffer 1) of its own persistent memory device to store the received data to be written; the electronic device is synchronous
  • each synchronous node device (synchronous node device 1, 2 or 3) in the RDMA network with the master node device can copy the data to be written received by the master node device through the RDMA network, and use its own persistent memory
  • the device's ring buffer (persistent memory ring buffer 2, 3, or 4) for intermediate storage.
  • the specific structure of the ring buffer in the persistent memory device is not limited.
  • the ring buffer can adopt a double-linked list structure, that is, the ring buffer can be a ring double-linked list; for example, the ring buffer can be Including data units for storing data, each data unit can include a front pointer pointing to the previous data unit and a back pointer pointing to the next data unit, that is, the back pointer in the last data unit in the circular double-linked list can point to the first data units, the front pointer in the first data unit in the circular doubly linked list can point to the last data unit.
  • the ring buffer can also include the root node at the beginning of the linked list, and the root node can include a head pointer pointing to the first data unit and a tail pointer pointing to the last data unit;
  • the data can be stored from the first data unit pointed to by the head pointer of the root node, then the second data unit, and then the third one, and so on.
  • Each data unit records the data itself and the adjacent data before and after The location of the unit; when the last data unit is written, it can return to the first data unit to continue writing.
  • the root node in the ring buffer can also include a synchronization lock when writing data to avoid writing conflicts; the root node in the ring buffer can also include the total data size of all current data units;
  • the data unit in the ring buffer may also include metadata recording the data size and pointer of the data unit, and a key value used as a unique name for the actually stored data, so as to facilitate subsequent data reading.
  • the processor can write the data to be written into the current data unit; after the current data unit is full, according to the back pointer in the current data unit, the The write data is written into the next data unit; wherein, the current data unit is any data unit in the circular double-linked list, and the next data unit is the data unit pointed to by the rear pointer in the current data unit (i.e. the latter data unit ), the back pointer in the last data unit in the circular doubly linked list points to the first data unit.
  • the current data unit may be the data unit that currently needs to be written into the data to be written;
  • the next data unit may be the next data unit that needs to be written into the data to be written after the current data unit is full, that is, in the current data unit The data unit pointed to by the back pointer. That is to say, after the processor writes the data to be written into the current data unit, if the current data unit is full and there are remaining data to be written to the ring buffer, the remaining data to be written can be The data continues to be written into the next data unit until the data to be written is completely written into the ring buffer.
  • this embodiment does not limit the specific method of determining the current data unit when starting to write the current data unit; for example, the processor can determine the current data unit according to the recorded written position; for example, the processor can record the ring buffer The written location and the emptied location in , for circular writing of the ring buffer.
  • Step 103 Store the data to be written in the ring buffer into the data memory.
  • the data memory in this step can be a large-capacity storage device used to store the data in the ring buffer (that is, the data to be written) in the electronic device, such as traditional mechanical hard disk and solid-state hard disk, etc. device and/or network storage. That is to say, in this embodiment, the persistent memory device with higher writing speed and bandwidth is used for data transfer, the data writing speed is increased, and the data storage in the persistent memory device is used for data archiving with lower price, reducing the configuration cost.
  • the processor of the electronic device can copy the data in the ring buffer of the persistent memory device (that is, the data to be written) to the data storage, complete the data archiving, and make the ring buffer continuously write
  • the data to be written can be archived and stored in the data storage in time.
  • the specific method for the processor to store the data to be written in the ring buffer into the data memory can be set by the designer according to practical scenarios and user needs.
  • the processor can directly press the preset time
  • the data to be written in each ring buffer is stored in batches in the data memory at intervals.
  • the processor can also batch store the data to be written in the ring buffer into the data memory according to the amount of data written in the ring buffer; for example, the processor can monitor the amount of data written in all the ring buffers, When the amount reaches the preset data size, the data in all ring buffers is archived; the processor can also monitor the amount of data written in each ring buffer (that is, the amount of stored data), and when the amount of written data reaches the threshold The data in the ring buffer (that is, the data to be written) is archived.
  • the processor judges whether the amount of stored data in the current ring buffer reaches the threshold; if so, the data to be written in the current ring buffer is copied to the data memory for storage; if not, then continue to monitor each The amount of data stored in the ring buffer.
  • the threshold can be set according to the preset percentage of the capacity of each ring buffer, such as the amount of data corresponding to 50% of the capacity, so that the processor can store the amount of data in the current ring buffer to reach the current ring buffer When the preset percentage (such as 50%) of the capacity is reached, the data to be written in the current ring buffer is copied to the data memory.
  • the processor can clear the stored data in the ring buffer during or after copying the data in the ring buffer to the data memory; wherein , the stored data is the data stored in the data memory to be written, that is, the data in the ring buffer that has been copied to the data memory.
  • the processor when the processor stores the data to be written in the ring buffer into the data memory, the small files in the data to be written can be combined and stored in the memory to improve the data storage efficiency.
  • Distributed storage such as distributed system infrastructure
  • the processor can aggregate the small files in the ring buffer to obtain the aggregated file during the process of storing the data to be written in the ring buffer into the data memory; store the aggregated file in the data memory ;
  • the small file may be a file whose file size in the data to be written is smaller than the threshold.
  • the application program can use the Java (an object-oriented programming language) local interface to associate and call the programming function of the underlying persistent memory device, and then use the transactional persistent memory allocator to Provides persistent memory in the form of memory-mapped files.
  • the processor can use the functions in the programming library of libpmemobj (a programming library) to allocate and manage the space of the persistent memory device, write or clear data; for example, use the pmemobj_list_insert_new function to allocate a new space, use the pmemobj_list_insert function Insert data at the specified location, and use the pmemobj_list_remove function to delete the specified data.
  • the processor can also use a more friendly key-value API programming library pmemkv (a programming library) to implement similar functions, which is not limited in this embodiment.
  • the embodiment of the present application writes the data to be written to the ring buffer of the persistent memory device according to byte addressing, and utilizes the writing speed and Bandwidth improves the writing speed of data and reduces the possibility of data loss in the event of a failure; and uses the ring buffer in the persistent memory device to store the data to be written in transit, which can continuously write the data to be written. While improving the speed of data persistence, the cost of technology use is reduced, and the technical effect of "shaving peaks and filling valleys" is realized.
  • the embodiment of the present application also provides a data writing device, and the data writing device described below and the data writing method described above can be referred to in correspondence with each other.
  • FIG. 6 is a structural block diagram of a data writing device provided by an embodiment of the present application.
  • the device can include:
  • the data receiving module 10 is used to obtain the data to be written; according to byte addressing, write the data to be written into the ring buffer of the persistent memory device; wherein, the working mode of the persistent memory device is that the application directly access mode;
  • the data storage module 20 is configured to store the data to be written in the ring buffer into the data storage.
  • the data receiving module 10 may include:
  • the persistent write submodule is used to write the data to be written into the current data unit; where the current data unit is any data unit in the circular double-linked list;
  • the data receiving module 10 may include:
  • the request receiving sub-module is used to obtain the data to be written corresponding to the data write request, and send the data to be written to the synchronization node device.
  • the data receiving module 10 may include:
  • the copy receiving sub-module is used to obtain the data to be written received by the master node device through the RDMA network.
  • the copy receiving submodule may be specifically configured to use an RDMA network card to receive the data to be written sent by the master node device through the RDMA network.
  • the data storage module 20 may include:
  • the aggregation submodule is used to aggregate the small files in the ring buffer to obtain the aggregated file; store the aggregated file in the data storage; wherein, the small file is the size of the file in the data to be written Files smaller than the threshold.
  • the data storage module 20 may be specifically configured to store the data to be written in the ring buffer into the data memory according to the amount of data written into the ring buffer.
  • the data storage module 20 may include:
  • a judging sub-module is used to judge whether the amount of stored data in the current ring buffer reaches a threshold; wherein, the current ring buffer is any ring buffer;
  • the copy sub-module is used to copy the data to be written in the current ring buffer to the data memory if the threshold is reached, and clear the copied data to be written in the current ring buffer.
  • the embodiment of the application uses the data receiving module 10 according to byte addressing, writes the data to be written into the ring buffer of the persistent memory device, and uses the persistent memory device to provide several times the traditional data memory.
  • the writing speed and bandwidth improve the data writing speed and reduce the possibility of data loss during failure; and use the ring buffer in the persistent memory device to transfer and store the data to be written, which can continuously write to be written It improves the speed of data persistence while reducing the cost of technology use, and realizes the technical effect of "shaving peaks and filling valleys".
  • the embodiment of the present application also provides an electronic device, and the electronic device described below and the data writing method described above can be referred to in correspondence.
  • the embodiment of the present application also provides an electronic device, including:
  • the processor is configured to implement the steps of the data writing method provided in the foregoing embodiments when executing the computer program.
  • the electronic device provided in this embodiment may specifically be a server, such as a server serving as a master node device or a server serving as a synchronization node device.
  • the embodiment of the present application further provides a data writing system, and the data writing system described below and the data writing method described above can be referred to in correspondence.
  • the embodiment of the present application also provides a data writing system, including: a master node device and a synchronization node device connected to the master node device; wherein, both the master node device and the synchronization node device are electronic devices as provided in the above embodiments .
  • both the master node device and the synchronization node device in the system provided by this embodiment can use the ring buffers in their respective persistent memory devices to transfer data and complete data writing; and the synchronization node device can write to the master node device Efficient data synchronization of incoming data, improving the writing efficiency of distributed storage, further reducing the possibility of data loss when the server or computer room fails, and has a certain disaster recovery effect.
  • the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data writing method provided by the above-mentioned embodiments are implemented.
  • each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.
  • the description is relatively simple, and for the related information, please refer to the description of the method part.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Transfer Systems (AREA)

Abstract

本申请公开了一种数据写入方法、装置、系统、电子设备及计算机可读存储介质，该方法包括：获取待写入数据；按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区；其中，持久内存设备的工作模式为应用直接访问模式；将环形缓冲区中的待写入数据存储到数据存储器中。本申请通过按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区，利用持久内存设备提供的数倍于传统的数据存储器的写入速度和带宽，提高了数据的写入速度，降低了故障时数据丢失的可能性；并且利用持久内存设备中的环形缓冲区对待写入数据进行中转存储，能够源源不断的写入待写入数据，在提升数据持久化速度的同时降低了技术使用成本。

Description

一种数据写入方法、装置、系统、电子设备及存储介质
本申请要求在2021年8月9日提交中国专利局、申请号为202110905582.8、发明名称为“一种数据写入方法、装置、系统、电子设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据存储技术领域，特别涉及一种数据写入方法、装置、系统、电子设备及计算机可读存储介质。
背景技术
持久内存(Persistent Memory，PMem)是新一代的存储介质，除了字节可寻址和高速读写性能以外，具有掉电非易失、高存储密度和低静态功耗等这些DRAM(动态随机存取存储器)不具备的优点，为构建更高效的存储计算系统带来了新的机遇。
然而，相较于传统的机械硬盘和固态硬盘等非易失性设备，持久内存设备的价格较高，在大规模用作存储介质时成本高昂。在大数据的存储场景中，用户仍然广泛使用机械硬盘及NAS(Network Attached Storage，网络存储器)等作为主要存储，这样数据的写入速度受制于小数据块的消耗和较低的带宽；其中，跨主机的数据分区实现更是采用缓慢的网络拷贝，使得数据写入速度进一步降低，并可能导致在机房故障(如断电和网络波动等)时丢失更多的数据。因此，如何能够提高数据的写入速度，降低故障时数据丢失的可能性，提升用户体验，是现今急需解决的问题。
发明内容
本申请的目的是提供一种数据写入方法、装置、系统、电子设备及计算机可读存储介质，以利用持久内存设备，提高数据的写入速度，降低故障时数据丢失的可能性，提升用户体验。
为解决上述技术问题，本申请提供一种数据写入方法，包括：
获取待写入数据；
按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区；其中，所述持久内存设备的工作模式为应用直接访问模式；
将所述环形缓冲区中的所述待写入数据存储到数据存储器中。
可选的，所述环形缓冲区为环形双链表时，按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区，包括：
将所述待写入数据写入到当前数据单元；其中，当前数据单元为所述环形双链表中的任一数据单元；
在当前数据单元写满后，根据当前数据单元中的后指针，将所述待写入数据写入到下一数据单元；其中，下一数据单元为当前数据单元中的后指针所指向的数据单元，所述环形双链表中最后一个数据单元中的后指针指向第一个数据单元。
可选的，所述获取待写入数据，包括：
主节点设备获取数据写入请求对应的待写入数据，并向同步节点设备发送所述待写入数据，其中，所述主节点设备中设置有所述持久内存设备。
可选的，所述获取待写入数据，包括：
同步节点设备通过RDMA网络拷贝主节点设备接收的所述待写入数据，其中，所述同步节点设备中设置有所述持久内存设备。
可选的，所述同步节点设备通过RDMA网络拷贝主节点设备接收的所述待写入数据，包括：
所述同步节点设备利用RDMA网卡接收所述主节点设备通过RDMA网络发送的所述待写入数据。
可选的，所述将所述环形缓冲区中的所述待写入数据存储到数据存储器中，包括：
对所述环形缓冲区中的小文件进行聚合，获取聚合文件；其中，所述小文件为所述待写入数据中文件大小小于阈值的文件；
将所述聚合文件存储到数据存储器中。
可选的，所述将所述环形缓冲区中的所述待写入数据存储到数据存储器中，包括：
根据所述环形缓冲区的数据写入量，将所述环形缓冲区中的所述待写入数据存储到所述数据存储器中，并清空所述环形缓冲区中的已存储数据；其中，所述已存储数据为存储到所述数据存储器中的所述待写入数据。
可选的，所述根据所述环形缓冲区的数据写入量，将所述环形缓冲区中的所述待写入数据存储到所述数据存储器中，并清空所述环形缓冲区中的已存储数据，包括：
判断当前环形缓冲区的存储数据量是否达到阈值；其中，当前环形缓冲区为任一所述环形缓冲区；
若是，则将当前环形缓冲区中的所述待写入数据拷贝到所述数据存储器中存储，并清空当前环形缓冲区中已拷贝的所述待写入数据。
本申请还提供了一种数据写入装置，包括：
数据接收模块，用于获取待写入数据；按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区；其中，所述持久内存设备的工作模式为应用直接访问模式；
数据存储模块，用于将所述环形缓冲区中的所述待写入数据存储到数据存储器中。
本申请还提供了一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上述所述的数据写入方法的步骤。
本申请还提供了一种数据写入系统，包括：主节点设备和与所述主节点设备连接的同步节点设备；其中，所述主节点设备和所述同步节点设备均如上述所述的电子设备。
此外，本申请还提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如上述所述的数据写入方法的步骤。
本申请所提供的一种数据写入方法，包括：获取待写入数据；按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区；其中，持久内存设备的工作模式为应用直接访问模式；将环形缓冲区中的待写入数据存储到数据存储器中；
可见，本申请通过按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区，利用持久内存设备提供的数倍于传统的数据存储器的写入速度和带宽，提高了数据的写入速度，降低了故障时数据丢失的可能性；并且利用持久内存设备中的环形缓冲区对待写入数据进行中转存储，能够源源不断的写入待写入数据，在提升数据持久化速度的同时降低了技术使用成本，实现了“削峰填谷”的技术效果。此外，本申请还提供了一种数据写入装置、系统、电子设备及计算机可读存储介质，同样具有上述有益效果。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。
图1为本申请实施例所提供的一种数据写入方法的流程图；
图2为本申请实施例所提供的另一种数据写入方法的远程同步示意图；
图3为现有技术中的数据写入方法的流程示意图；
图4为本申请实施例所提供的一种数据写入系统的结构示意图；
图5为本申请实施例所提供的另一种数据写入方法的环形缓冲区的结构示意图；
图6为本申请实施例所提供的一种数据写入装置的结构框图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
请参考图1，图1为本申请实施例所提供的一种数据写入方法的流程图。该方法可以包括：
步骤101：获取待写入数据。
其中，本步骤中的待写入数据可以为需要写入存储到电子设备中的数据。对于本实施例中待写入数据的具体内容，可以由设计人员根据使用场景和用户需求自行设置，如电子设备为主节点设备时，待写入数据可以为请求写入到主节点设备中存储的数据；电子设备为同步节点设备时，待写入数据可以为需要同步写入到同步节点设备中存储的数据。
可以理解的是，本实施例所提供的数据写入方法可以应用于主节点设备，也就是说，作为主节点设备的电子设备中的处理器可以执行本实施例所提供的方法对需要写入到主节点设备中的数据(即待写入数据)进行写入存储；本实施例所提供的数据写入方法也可以应用于主节点设备对应的同步节点设备(即从节点设备)，也就是说，作为同步节点设备的电子设备中的处理器可以执行本实施例所提供的方法对写入到主节点设备中的数据(即待写入数据)进行同步存储，实现对主节点设备中存储内容的数据同步和容灾功能。本实施例对此不做任何限制。
具体的，对于本步骤中电子设备的处理器获取待写入数据的具体方式，可以由设计人员根据使用场景和用户需求自行设置，如电子设备为主节点设备时，主节点设备的处理器可以获取数据写入请求对应的待写入数据，即主节点设备可以接收数据写入请求，获取请求写入的数据(即待写入数据)；进一步的，主节点设备可以在获取待写入数据后，将待写入数据发送到同步节点设备，以使同步节点设备能够对写入到主节点设备的数据进行存储，实现跨主机的数据同步以及容灾备份。电子设备为同步节点设备时，同步节点设备的处理器可以获取主节点设备接收的待写入数据，从而对写入主节点设备的待写入数据进行存储，实现主机的数据同步以及容灾备份。本实施例对此不做任何限制。
进一步的，电子设备为同步节点设备时，本步骤中同步节点设备的处理器可以通过RDMA(Remote Direct Memory Access，远程直接数据存取)网络获取主节点设备接收的待写入数据，即同步节点设备可以与相应的主节点设备处于RDMA网络内，使得同步节点设备能够利用RDMA网卡通过RDMA网络快速拷贝主节点设备接收的需要写入的数据(即待写入数据)，将主节点设备中的存储的数据几乎无延迟地同步到同步节点设备，如与作为主节点设 备的服务器在同一机房的另一服务器。
对应的，主节点设备和同步节点设备中均可以插入持久内存设备，并且主节点设备和同步节点设备之间配置超低延迟的网络(RDMA网络)，以利用持久内存设备作为存储介质和键值存储引擎，利用高效的RDMA驱动的网络传输作为网络层，使得主节点设备和同步节点设备上的持久内存设备能够作为一个副本池集。如图2所示，主机1的持久内存设备(持久内存1)和主机2的持久内存设备(持久内存2)可以构成一个副本池集，当作为主节点设备的主机1中的持久内存1中被写入一份数据(即待写入数据)后，主机1可以利用远程复制服务(如使用远程持久内存后台服务rpmemd工具)通过RDMA网络立即将数据高速传输到作为同步节点设备的主机2，主机2可以利用远程复制服务立即调用持久内存驱动将通过RDMA网络接收的数据写入到持久内存2中完成数据拷贝，以基于持久内存设备的高带宽和RDMA网络的低延迟在极短的时间内完成。
步骤102：按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区；其中，持久内存设备的工作模式为应用直接访问模式。
其中，本步骤中的持久内存设备可以为利用持久内存(PMem)进行数据存储的设备。本步骤中持久内存设备的环形缓冲区可以为持久内存设备中创建的一种环形缓冲区的数据结构，如环形双链表。本实施例中利用采用AD模式(App Direct Mode，即应用直接访问模式)持久内存设备对待写入数据进行中转存储，使处理器可以按照字节寻址将待写入数据写入到采用AD模式的持久内存，由于持久内存的写入速度是传统存储设备的文件I/O速度几倍到几十倍，能够极大地提升了数据写入的效率，如表1所示，持久内存设备的写入效率可以达到机械硬盘的上百倍；其中，IOPS可以为每秒完成的写入操作的次数。
表1持久内存设备与传统存储设备的写入效率比较表
测试项目 持久内存-IOPS 机械硬盘-IOPS
4k顺序写 2145000 518
4k随机写 778000 351
512k顺序写 2040 19
512k随机写 1829 8
并且，由于存储待写入数据的缓冲区是环形的，数据写满后会覆盖最早写入的数据，能够在数据写入速度的同时降低所占用的持久内存设备的存储空间，降低成本。
具体的，对于电子设备的处理器按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区的具体方式，可以由设计人员根据实用场景和用户需求自行设置，如处理器可以将待写入数据写入到持久内存设备的一个环形缓冲区，即本实施例中可以仅设置一个用于存放待写入数据的环形缓冲区；处理器也可以将待写入数据写入到持久内存设备的多个环形缓冲区，即本实施例中可以设置多个用于存放待写入数据的环形缓冲区。本实施例对此不做任何限制。
可以理解的是，传统的数据写入方案采用文件系统标准API(应用程序接口)的方式写入数据，不能利用内核的直接访问(DAX)特性，应用程序无法直接寻址，并且写入流程涉及到多次内存缓冲区拷贝操作，速度相对较慢，如图3所示，一般需要经过内存缓冲区写入、文件系统I/O(输入/输出)中转和数据存储设备存储这三个步骤才能保证数据持久化；而本实施例中将持久内存设备作为字符设备，或将持久内存设备挂载为文件系统的块设备，由于持久内存设备的工作模式应用直接访问模式，运行应用程序的处理器可以按字节寻址，将待写入数据写入到环形缓冲区便可完成待写入数据的数据持久化，宣告数据写入成功。例如，应用程序如果要更新64字节的内容，按照传统的数据写入方案，处理器需要将包含这些64字节的整个存储块读取到内存缓冲区，更新64字节，然后在数据存储设备(如机械硬盘)写入整个存储块，以对其进行持久化。这是因为传统的数据写入方案中数据存储只能使用块IO完成，通常一次访问4k字节，所以必须先读取4k再写入4k，才能完成64字节内容的更新任务；而本实施例中利用持久内存设备，同样是更新64字节的内容，只需直接在持久内存设备写入64字节即可。
如图4所示，电子设备为主节点设备时，主节点设备可以利用自身的持久内存设备的环形缓冲区(持久内存环形缓冲区1)对接收的待写入数据进行存储；电子设备为同步节点设备时，与主节点设备处于RDMA网络内的每个同步节点设备(同步节点设备1、2或3)均可以通过RDMA网络拷贝主节点设备接收的待写入数据，并利用自身的持久内存设备的环形缓冲区(持久内存环 形缓冲区2、3或4)进行中转存储。
需要说明的是，本实施例中并不限定持久内存设备中的环形缓冲区的具体结构，如环形缓冲区可以采用双链表结构，即环形缓冲区可以为环形双链表；例如，环形缓冲区可以包括用于存储数据的数据单元，每个数据单元可以包括指向前一个数据单元的前指针和指向后一个数据单元的后指针，即环形双链表中最后一个数据单元中的后指针可以指向第一个数据单元，环形双链表中第一个数据单元中的前指针可以指向最后一个数据单元。如图5所示，环形缓冲区还可以包括链表开头的根节点，根节点可以包括指向第一个数据单元的头指针和指向最后一个数据单元的尾指针；也就是说，写入环形缓冲区的数据可以从根节点的头指针指向的第一数据单元开始存放数据，然后存放第二个数据单元，然后是第三个依次类推，每个数据单元记录着数据本身、以及相邻的前后数据单元的位置；当写入最后一个数据单元后，又可以回到第一个数据单元继续写入。如图5所示，环形缓冲区中的根节点还可以包括写入数据时的同步锁，以避免写入冲突；环形缓冲区中的根节点还可以包括当前所有数据单元总的数据量大小；环形缓冲区中的数据单元还可以包括记录本数据单元的数据大小和指针等的元数据，以及用于作为实际存储的数据一个唯一名称的键值，以方便后续的数据读取。
对应的，环形缓冲区具体为环形双链表时，本步骤中处理器可以将待写入数据写入到当前数据单元；在当前数据单元写满后，根据当前数据单元中的后指针，将待写入数据写入到下一数据单元；其中，当前数据单元为环形双链表中的任一数据单元，下一数据单元为当前数据单元中的后指针所指向的数据单元(即后一数据单元)，环形双链表中最后一个数据单元中的后指针指向第一个数据单元。
具体的，当前数据单元可以为当前需要写入待写入数据的数据单元；下一数据单元可以为当前数据单元写满后下一个需要写入待写入数据的数据单元，即当前数据单元中的后指针所指向的数据单元。也就是说，处理器将待写入数据写入到当前数据单元后，若当前数据单元写满后还有剩余的待写入数据要写入到环形缓冲区，则可以将剩余的待写入数据继续写入到下一数据单元，直至待写入数据完全写入到环形缓冲区。相应的，本实施例并不限定开始写入当前数据单元时，当前数据单元的具体确定方式；如处理器可以根 据记录的已写入位置，确定当前数据单元；例如处理器可以记录环形缓冲区中的已写入位置和已清空位置，以便环形缓冲区的循环写入。
步骤103：将环形缓冲区中的待写入数据存储到数据存储器中。
其中，本步骤中的数据存储器可以为电子设备中用于对环形缓冲区中的数据(即待写入数据)进行存储的大容量的存储设备，如传统的机械硬盘和固态硬盘等非易失性设备和/或网络存储器。也就是说，本实施例中利用写入速度和带宽更高的持久内存设备进行数据中转，提高数据写入速度，利用价格更低的数据存储器对持久内存设备中的数据进行数据归档，降低配置成本。
可以理解的是，本步骤中电子设备的处理器可以将持久内存设备的环形缓冲区中的数据(即待写入数据)拷贝到数据存储器中，完成数据归档，使环形缓冲区源源不断的写入待写入数据能够及时归档到数据存储器中存储。
具体的，对于本步骤中处理器将环形缓冲区中的待写入数据存储到数据存储器中的具体方式，可以由设计人员根据实用场景和用户需求自行设置，如处理器可以直接按预设时间间隔将各环形缓冲区中的待写入数据批量存储到数据存储器中。处理器也可以根据环形缓冲区中的数据写入量，将环形缓冲区中的待写入数据批量存储到数据存储器中；例如处理器可以监测全部环形缓冲区中写入的数据量，在数据量达到预设数据大小时，对全部环形缓冲区中的数据进行归档；处理器也可以监测各环形缓冲区各自写入的数据量(即存储数据量)，对写入的数据量达到阈值的环形缓冲区中的数据(即待写入数据)进行归档。
也就是说，本步骤中处理器判断当前环形缓冲区的存储数据量是否达到阈值；若是，则将当前环形缓冲区中的待写入数据拷贝到数据存储器中存储；若否，则继续监测各环形缓冲区的存储数据量。例如本实施例中可以按照每个环形缓冲区的容量的预设百分比来设置阈值，如容量的50％对应的数据量，使得处理器可以在当前环形缓冲区的存储数据量达到当前环形缓冲区容量的预设百分比(如50％)时，将当前环形缓冲区中的待写入数据拷贝到数据存储器。
进一步的，为了提高环形缓冲区的数据写入速度，本实施例中处理器在将环形缓冲区中的数据拷贝到数据存储器的过程中或之后，可以清空环形缓 冲区中的已存储数据；其中，已存储数据为存储到数据存储器中的待写入数据，即环形缓冲区中已拷贝到数据存储器的数据。
进一步的，处理器在将环形缓冲区中的待写入数据存储到数据存储器中的过程中，可以对待写入数据中的小文件合并存储到存储器，以提升数据存储效率，对Hadoop(一种分布式系统基础架构)等分布式存储增益明显。也就是说，处理器可以在将环形缓冲区中的待写入数据存储到数据存储器中的过程中，对环形缓冲区中的小文件进行聚合，获取聚合文件；将聚合文件存储到数据存储器中；其中，小文件可以为待写入数据中文件大小小于阈值的文件。
具体的，本实施例中应用程序可以使用Java(一门面向对象的编程语言)本地接口的方式来关联调用底层的持久内存设备的编程函数，然后使用事务性持久内存分配器，在文件系统层面以内存映射文件的形式提供持久内存。其中，处理器可以利用libpmemobj(一种编程库)的编程库中的函数来分配管理持久内存设备的空间、写入或者清除数据；例如，利用pmemobj_list_insert_new函数用来分配一个新的空间，利用pmemobj_list_insert函数在指定位置插入数据，利用pmemobj_list_remove函数可以删除指定的数据。优选地，处理器也可以利用更为友好的键值API编程库pmemkv(一种编程库)来实现类似的功能，本实施例对此不做任何限制。
本实施例中，本申请实施例通过按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区，利用持久内存设备提供的数倍于传统的数据存储器的写入速度和带宽，提高了数据的写入速度，降低了故障时数据丢失的可能性；并且利用持久内存设备中的环形缓冲区对待写入数据进行中转存储，能够源源不断的写入待写入数据，在提升数据持久化速度的同时降低了技术使用成本，实现了“削峰填谷”的技术效果。
相应于上面的方法实施例，本申请实施例还提供了一种数据写入装置，下文描述的数据写入装置与上文描述的数据写入方法可相互对应参照。
请参考图6，图6为本申请实施例所提供的一种数据写入装置的结构框图。该装置可以包括：
数据接收模块10，用于获取待写入数据；按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区；其中，所述持久内存设备的工作模式为应用直接访问模式；
数据存储模块20，用于将环形缓冲区中的待写入数据存储到数据存储器中。
可选的，所述环形缓冲区为环形双链表时，数据接收模块10可以包括：
持久写入子模块，用于将所述待写入数据写入到当前数据单元；其中，当前数据单元为所述环形双链表中的任一数据单元；
更换写入子模块，用于在当前数据单元写满后，根据当前数据单元中的后指针，将所述待写入数据写入到下一数据单元；其中，下一数据单元为当前数据单元中的后指针所指向的数据单元，所述环形双链表中最后一个数据单元中的后指针指向第一个数据单元。
可选的，数据接收模块10可以包括：
请求接收子模块，用于获取数据写入请求对应的待写入数据，并向同步节点设备发送待写入数据。
可选的，数据接收模块10可以包括：
拷贝接收子模块，用于通过RDMA网络获取主节点设备接收的待写入数据。
可选的，拷贝接收子模块可以具体用于利用RDMA网卡接收所述主节点设备通过RDMA网络发送的所述待写入数据。
可选的，数据存储模块20可以包括：
聚合子模块，用于对所述环形缓冲区中的小文件进行聚合，获取聚合文件；将所述聚合文件存储到数据存储器中；其中，所述小文件为所述待写入数据中文件大小小于阈值的文件。
可选的，数据存储模块20可以具体用于根据环形缓冲区的数据写入量，将环形缓冲区中的待写入数据存储到数据存储器中。
可选的，数据存储模块20可以包括：
判断子模块，用于判断当前环形缓冲区的存储数据量是否达到阈值；其中，当前环形缓冲区为任一环形缓冲区；
拷贝子模块，用于若达到阈值，则将当前环形缓冲区中的待写入数据拷贝到数据存储器，并清空当前环形缓冲区中已拷贝的待写入数据。
本实施例中，本申请实施例通过数据接收模块10按照字节寻址，将待写入数据写入到持久内存设备的环形缓冲区，利用持久内存设备提供的数倍于传统的数据存储器的写入速度和带宽，提高了数据的写入速度，降低了故障时数据丢失的可能性；并且利用持久内存设备中的环形缓冲区对待写入数据进行中转存储，能够源源不断的写入待写入数据，在提升数据持久化速度的同时降低了技术使用成本，实现了“削峰填谷”的技术效果。
相应于上面的方法实施例，本申请实施例还提供了一种电子设备，下文描述的电子设备与上文描述的数据写入方法可相互对应参照。
本申请实施例还提供了一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行计算机程序时实现如上述实施例所提供的数据写入方法的步骤。
其中，本实施例所提供的电子设备可以具体为服务器，如作为主节点设备的服务器或作为同步节点设备的服务器。
相应于上面的方法实施例，本申请实施例还提供了一种数据写入系统，下文描述的数据写入系统与上文描述的数据写入方法可相互对应参照。
本申请实施例还提供了一种数据写入系统，包括：主节点设备和与主节点设备连接的同步节点设备；其中，主节点设备和同步节点设备均为如上述实施例所提供的电子设备。
也就是说，本实施例所提供的系统中主节点设备和同步节点设备均可以利用各自持久内存设备中的环形缓冲区进行数据中转，完成数据写入；并且同步节点设备能够对主节点设备写入的数据进行高效的数据同步，提升分布式存储的写入效率，进一步降低服务器或机房故障时的数据丢失可能性，具有一定的容灾作用。
此外，本申请还提供了一种计算机可读存储介质，计算机可读存储介质 上存储有计算机程序，计算机程序被处理器执行时实现如上述实施例所提供的数据写入方法的步骤。
说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。
以上对本申请所提供的一种数据写入方法、装置、系统、电子设备及计算机可读存储介质进行了详细介绍。本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请权利要求的保护范围内。

Claims (10)

  1. 一种数据写入方法，其特征在于，包括：
    获取待写入数据；
    按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区；其中，所述持久内存设备的工作模式为应用直接访问模式；
    将所述环形缓冲区中的所述待写入数据存储到数据存储器中。
  2. 根据权利要求1所述的数据写入方法，其特征在于，所述环形缓冲区为环形双链表时，按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区，包括：
    将所述待写入数据写入到当前数据单元；其中，当前数据单元为所述环形双链表中的任一数据单元；
    在当前数据单元写满后，根据当前数据单元中的后指针，将所述待写入数据写入到下一数据单元；其中，下一数据单元为当前数据单元中的后指针所指向的数据单元，所述环形双链表中最后一个数据单元中的后指针指向第一个数据单元。
  3. 根据权利要求1所述的数据写入方法，其特征在于，所述获取待写入数据，包括：
    同步节点设备通过RDMA网络拷贝主节点设备接收的所述待写入数据，其中，所述同步节点设备中设置有所述持久内存设备。
  4. 根据权利要求1所述的数据写入方法，其特征在于，所述将所述环形缓冲区中的所述待写入数据存储到数据存储器中，包括：
    对所述环形缓冲区中的小文件进行聚合，获取聚合文件；其中，所述小文件为所述待写入数据中文件大小小于阈值的文件；
    将所述聚合文件存储到数据存储器中。
  5. 根据权利要求1至4任一项所述的数据写入方法，其特征在于，所述将所述环形缓冲区中的所述待写入数据存储到数据存储器中，包括：
    根据所述环形缓冲区的数据写入量，将所述环形缓冲区中的所述待写入数据存储到所述数据存储器中，并清空所述环形缓冲区中的已存储数据；其中，所述已存储数据为存储到所述数据存储器中的所述待写入数据。
  6. 根据权利要求5所述的数据写入方法，其特征在于，所述根据所述环形缓冲区的数据写入量，将所述环形缓冲区中的所述待写入数据存储到所述数据存储器中，并清空所述环形缓冲区中的已存储数据，包括：
    判断当前环形缓冲区的存储数据量是否达到阈值；其中，当前环形缓冲区为任一所述环形缓冲区；
    若是，则将当前环形缓冲区中的所述待写入数据拷贝到所述数据存储器中存储，并清空当前环形缓冲区中已拷贝的所述待写入数据。
  7. 一种数据写入装置，其特征在于，包括：
    数据接收模块，用于获取待写入数据；按照字节寻址，将所述待写入数据写入到持久内存设备的环形缓冲区；其中，所述持久内存设备的工作模式为应用直接访问模式；
    数据存储模块，用于将所述环形缓冲区中的所述待写入数据存储到数据存储器中。
  8. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述计算机程序时实现如权利要求1至6任一项所述的数据写入方法的步骤。
  9. 一种数据写入系统，其特征在于，包括：主节点设备和与所述主节点设备连接的同步节点设备；其中，所述主节点设备和所述同步节点设备均为如权利要求8所述的电子设备。
  10. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至6任一项所述的数据写入方法的步骤。
PCT/CN2022/078185 2021-08-09 2022-02-28 一种数据写入方法、装置、系统、电子设备及存储介质 WO2023015866A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110905582.8 2021-08-09
CN202110905582.8A CN113360098A (zh) 2021-08-09 2021-08-09 一种数据写入方法、装置、系统、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023015866A1 true WO2023015866A1 (zh) 2023-02-16

Family

ID=77540728

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/078185 WO2023015866A1 (zh) 2021-08-09 2022-02-28 一种数据写入方法、装置、系统、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN113360098A (zh)
WO (1) WO2023015866A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117407338A (zh) * 2023-12-15 2024-01-16 北京壁仞科技开发有限公司 用于数据传输同步的系统、方法及计算设备

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113360098A (zh) * 2021-08-09 2021-09-07 苏州浪潮智能科技有限公司 一种数据写入方法、装置、系统、电子设备及存储介质
CN115933997B (zh) * 2023-01-30 2023-06-13 南京芯驰半导体科技有限公司 数据访问方法、相关设备及存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050273649A1 (en) * 2004-05-31 2005-12-08 Electronics And Telecommunications Research Institute Apparatus for high-speed streaming data transmission using PMEM controller and method thereof
CN105677258A (zh) * 2016-02-23 2016-06-15 浪潮(北京)电子信息产业有限公司 一种日志数据管理方法及系统
CN110691062A (zh) * 2018-07-06 2020-01-14 浙江大学 一种数据写入方法、装置及其设备
CN111221922A (zh) * 2019-12-31 2020-06-02 苏州浪潮智能科技有限公司 一种RocksDB数据库数据写入方法及RocksDB数据库
CN111352589A (zh) * 2020-02-24 2020-06-30 苏州浪潮智能科技有限公司 一种分布式存储的方法、装置、设备及可读介质
CN111475114A (zh) * 2020-04-03 2020-07-31 苏州浪潮智能科技有限公司 一种数据下刷系统、方法及计算机可读存储介质
CN112083881A (zh) * 2020-08-24 2020-12-15 云南财经大学 一种基于持久化内存的一体化天文数据采集与存储系统
CN113360098A (zh) * 2021-08-09 2021-09-07 苏州浪潮智能科技有限公司 一种数据写入方法、装置、系统、电子设备及存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111400268B (zh) * 2020-03-13 2022-06-17 清华大学 一种分布式持久性内存事务系统的日志管理方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050273649A1 (en) * 2004-05-31 2005-12-08 Electronics And Telecommunications Research Institute Apparatus for high-speed streaming data transmission using PMEM controller and method thereof
CN105677258A (zh) * 2016-02-23 2016-06-15 浪潮(北京)电子信息产业有限公司 一种日志数据管理方法及系统
CN110691062A (zh) * 2018-07-06 2020-01-14 浙江大学 一种数据写入方法、装置及其设备
CN111221922A (zh) * 2019-12-31 2020-06-02 苏州浪潮智能科技有限公司 一种RocksDB数据库数据写入方法及RocksDB数据库
CN111352589A (zh) * 2020-02-24 2020-06-30 苏州浪潮智能科技有限公司 一种分布式存储的方法、装置、设备及可读介质
CN111475114A (zh) * 2020-04-03 2020-07-31 苏州浪潮智能科技有限公司 一种数据下刷系统、方法及计算机可读存储介质
CN112083881A (zh) * 2020-08-24 2020-12-15 云南财经大学 一种基于持久化内存的一体化天文数据采集与存储系统
CN113360098A (zh) * 2021-08-09 2021-09-07 苏州浪潮智能科技有限公司 一种数据写入方法、装置、系统、电子设备及存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117407338A (zh) * 2023-12-15 2024-01-16 北京壁仞科技开发有限公司 用于数据传输同步的系统、方法及计算设备
CN117407338B (zh) * 2023-12-15 2024-03-19 北京壁仞科技开发有限公司 用于数据传输同步的系统、方法及计算设备

Also Published As

Publication number Publication date
CN113360098A (zh) 2021-09-07

Similar Documents

Publication Publication Date Title
WO2023015866A1 (zh) 一种数据写入方法、装置、系统、电子设备及存储介质
EP3726364B1 (en) Data write-in method and solid-state drive array
CN105549905A (zh) 一种多虚拟机访问分布式对象存储系统的方法
US20090077311A1 (en) Storage control method and system for performing backup and/or restoration
WO2024051109A1 (zh) 一种数据存储方法、装置、系统、设备和介质
CN112632069B (zh) 哈希表数据存储管理方法、装置、介质和电子设备
CN108089825B (zh) 一种基于分布式集群的存储系统
AU2013403132A1 (en) Data storage method, data storage apparatus, and storage device
CN113553346B (zh) 大规模实时数据流一体化处理、转发和存储方法及系统
CN113568582B (zh) 数据管理方法、装置和存储设备
CN109918352B (zh) 存储器系统和存储数据的方法
CN112988680B (zh) 数据加速方法、缓存单元、电子设备及存储介质
US20210303213A1 (en) Semiconductor storage device and control method thereof
CN115794669A (zh) 一种扩展内存的方法、装置及相关设备
WO2014153931A1 (zh) 文件存储方法、装置、访问客户端及元数据服务器系统
US20210263668A1 (en) Information processing device and computer-readable recording medium recording storage control program
US20210223957A1 (en) Storage apparatus and storage control method
WO2023241528A1 (zh) 数据处理方法以及装置
CN112463073A (zh) 一种对象存储分布式配额方法、系统、设备和存储介质
CN114490540B (zh) 数据存储方法、介质、装置和计算设备
WO2022033269A1 (zh) 数据处理的方法、设备及系统
US20220253238A1 (en) Method and apparatus for accessing solid state disk
CN113377288B (zh) 硬件队列管理系统、方法、固态硬盘控制器及固态硬盘
CN109542356B (zh) 面向容错的nvm持久化过程冗余信息的压缩方法和装置
CN116893789B (zh) 一种数据管理方法、系统、装置、设备及计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22854871

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE