CN113157497A - Data storage method, device, equipment and medium - Google Patents
Data storage method, device, equipment and medium Download PDFInfo
- Publication number
- CN113157497A CN113157497A CN202110482490.3A CN202110482490A CN113157497A CN 113157497 A CN113157497 A CN 113157497A CN 202110482490 A CN202110482490 A CN 202110482490A CN 113157497 A CN113157497 A CN 113157497A
- Authority
- CN
- China
- Prior art keywords
- data
- written
- storage
- metadata
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
In the data storage method, device, equipment and medium provided by the application, after the data writing equipment acquires the data to be written, if the data management equipment fails to acquire the application target storage file, the data writing equipment selects the target storage file from the preset storage file, and writes the metadata recording the corresponding relation between the data to be written and the target storage file into the target storage node together with the data to be written. Therefore, when the data management node is abnormal, the data writing node can also continue to write data into the storage node, and the data loss is avoided.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a data storage method, apparatus, device, and medium.
Background
To meet the increasing data storage needs, distributed storage schemes have been proposed. In the distributed storage scheme, to achieve high availability of the storage system, the metadata management node generally adopts a master-backup mode. That is, when the master node in the cluster fails, other standby nodes can become a new master node to continue providing services.
However, the inventor researches and discovers that the metadata management node can be unavailable for a short time in the process of switching the main and standby nodes. During this period of unavailability, data loss may occur because the data cannot be written to the storage node any further.
Disclosure of Invention
In order to overcome at least one of the deficiencies in the prior art, in a first aspect, an embodiment of the present application provides a data storage method applied to a data writing device, where the method includes:
acquiring data to be written;
when detecting that a storage file applied from a data management device is abnormal, selecting a target file from a pre-distributed storage file, wherein the pre-distributed storage file is applied from the data management device;
and writing the data to be written and the metadata of the data to be written into a target storage node, wherein the metadata records the corresponding relation between the data to be written and the target file.
In a second aspect, an embodiment of the present application provides a data storage method, which is applied to a target storage node, and the method includes:
receiving data to be written and metadata of the data to be written, wherein the metadata records a corresponding relation between the data to be written and a target file, the target file is selected from pre-allocated storage files when the writing device detects that a storage file applied from a data management device is abnormal, and the pre-allocated storage file is applied from the data management device;
writing the data to be written and the metadata of the data to be written into a storage medium;
and when detecting that the data management equipment is recovered to be normal, sending the metadata of the data to be written to the data management equipment, wherein the metadata is used for enabling the data management equipment to establish the corresponding relation between the data to be written and the target file.
In a third aspect, an embodiment of the present application provides a data storage system, where the data storage system includes a data writing device, a target storage node, and a data management device;
the data writing equipment acquires data to be written; when detecting that the application of the storage file from the data management equipment is abnormal, selecting a target file from the pre-allocated storage file; writing the data to be written and the metadata of the data to be written into a target storage node; the pre-distributed storage file is applied from the data management equipment, and the metadata records the corresponding relation between the data to be written and the target file;
the target storage node receives the data to be written and the metadata of the data to be written; when detecting that the data management equipment is recovered to be normal, sending the metadata of the data to be written to the data management equipment;
the data management equipment receives metadata of data to be written, which is sent by the target storage node; and establishing a corresponding relation between the data to be written and the target file according to the metadata.
In a fourth aspect, an embodiment of the present application provides a data storage device, which is applied to a data writing apparatus, where the data storage device includes:
the data acquisition module is used for acquiring data to be written;
the file allocation module is used for selecting a target file from pre-allocated storage files when detecting that the storage files applied from the data management equipment are abnormal, wherein the pre-allocated storage files are applied from the data management equipment;
and the data sending module is used for writing the data to be written and the metadata of the data to be written into a target storage node, wherein the metadata records the corresponding relation between the data to be written and the target file.
In a fifth aspect, an embodiment of the present application provides a data storage apparatus, which is applied to a target storage node, and the data storage apparatus includes:
the data receiving module is used for receiving data to be written and metadata of the data to be written, wherein the metadata records the corresponding relation between the data to be written and a target file;
the data writing module writes the data to be written and the metadata of the data to be written into a storage medium;
and the data synchronization module is used for sending metadata of the data to be written to the data management equipment when the data management equipment is detected to be recovered to normal, wherein the metadata is used for enabling the data management equipment to establish the corresponding relation between the data to be written and the target file.
In a sixth aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores a computer program, and the computer program, when executed by the processor, implements the data storage method or the data storage method.
In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the data storage method or the data storage method.
Compared with the prior art, the method has the following beneficial effects:
in the data storage method, the data storage device, the data storage apparatus, and the medium provided in the embodiments of the present application, after acquiring data to be written, if acquiring an application target storage file from the data management apparatus fails, the data writing apparatus selects the target storage file from the preset storage file, and writes metadata, which records a correspondence between the data to be written and the target storage file, and the data to be written into the target storage node together. Therefore, when the data management node is abnormal, the data writing node can also continue to write data into the storage node, and the data loss is avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic structural diagram of a distributed storage system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of data writing according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating steps of a data storage method according to an embodiment of the present disclosure;
FIG. 4 is a second flowchart illustrating steps of a data storage method according to an embodiment of the present application;
FIG. 5 is a diagram illustrating metadata synchronization provided by an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present application;
FIG. 7 is a second schematic structural diagram of a data storage device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Icon: 101-a data acquisition module; 102-a file allocation module; 103-a data transmission module; 201-a data receiving module; 202-a data write module; 203-data synchronization module; 304-a communication device; 303-a processor; 302-memory.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
It should be understood that the operations of the flow diagrams may be performed out of order, and steps without logical context may be performed in reverse order or simultaneously. One skilled in the art, under the guidance of this application, may add one or more other operations to, or remove one or more operations from, the flowchart.
The embodiment of the present application relates to a distributed storage scenario in the field of computers, and in order to facilitate the explanation of the technical solution provided by the embodiment of the present application, the following first introduces devices and the role of each device involved in the distributed storage scenario.
An embodiment of the present application provides a schematic diagram of a distributed storage scenario, as shown in fig. 1, in the distributed storage scenario, the distributed storage scenario includes a data writing device, a data management device, and a data storage node. The data writing device, the data management device and the data storage node can establish communication connection through a network.
The data writing device is configured to obtain data to be written, where the data to be written may be, but is not limited to, a picture, audio, and video. And, the data writing device may be a different device in a different scenario. For example, in a video surveillance scenario, the data writing device may be a webcam; in a live scenario, the data writing device may be a mobile terminal for live broadcasting.
And the data storage node is used for destaging the data to be written into a storage medium configured by the data storage node. The number of the data storage nodes may be multiple (node 1, node 2, and node N), and the storage medium configured for the data storage nodes may be a magnetic disk or a solid state disk, which is not specifically limited in this embodiment of the present application.
And the data management equipment is used for integrally managing the data which needs to be written into the data storage nodes and all the data which is written into the data storage nodes. For example, the data management node records index information, file size, and the like of data written in the storage node.
In order to improve system reliability in a distributed storage scenario, a master device and a slave device are generally provided for a data management device, and when an abnormality occurs in the master device, the master device is switched to the slave device, and the slave device is used as a new master device to continue providing services.
As shown in fig. 2, before storing data to be written in a data storage node, a data writing device needs to initiate an application to a data management device, so that the data management device allocates a target storage file for the data to be written; and determining a target storage node from the data storage nodes serving as candidate storage nodes, wherein the target storage node is used for storing the data to be written.
It is worth to be noted that, in a distributed storage scenario, a complete data file may be split into different data segments, and stored in distributed data storage nodes; the user is not aware of this process. After the data is written, the file identifier of the data file is provided for the user to represent the data file.
However, in the process of switching between the master device and the slave device, the data management node cannot provide a service for the data writing node, and therefore, if the data writing device acquires a large amount of data to be written and the data cache area of the data writing device is not enough, data loss occurs.
In view of this, an embodiment of the present application provides a data storage method applied to a data writing device, which overcomes a drawback that the data writing device cannot write data during a primary/standby switching period in a pre-allocation storage file manner.
The steps are described in detail below with reference to the schematic flow chart of steps of the data storage method shown in fig. 3. As shown in fig. 3, the data storage method includes:
in step S101A, data to be written is acquired.
The data to be written may be, but is not limited to, pictures, audio, and video. The embodiments of the present application are not particularly limited.
Step S102A, when detecting that the application for the storage file from the data management device is abnormal, selecting the target file from the pre-allocated storage file.
The pre-allocation storage file is applied from the data management device, namely the data management device pre-allocates a specified number of storage files for the data writing device when no abnormality occurs. In the embodiment of the application, the number of the pre-allocated storage files can be adaptively adjusted according to the speed of acquiring the data to be written by the data writing equipment and the writing performance of the data storage nodes.
For example, assuming that the writing performance of the data storage node is 300MB/s, and the speed of the data writing device acquiring the data to be written is 2MB/s, the number of the preset storage files may be set to 4000.
In addition, when the data management device detects that the number of the remaining storage files of the data writing device is less than 4000, the data management device allocates a new storage file to the data writing device again, so that the total number of the storage files reaches 4000.
In addition, it should be noted that, if the target storage file to be written with data is a storage file used before the data management device is abnormal, the data writing device does not need to apply for a new target storage file from the data management device, and does not need to select the target storage file from the pre-allocated storage files. It can be understood that the application to the data management device is only required when the data to be written needs to be written into a new target file.
Step S103A, writing the data to be written and the metadata of the data to be written into the target storage node.
Wherein, the metadata records the corresponding relation between the data to be written and the target file. In the embodiment of the application, if the data management device is not abnormal, the data management node allocates the target storage node and the target storage file to the data to be written, that is, the data management device knows the target file in which the data to be written is written.
Therefore, after the data management node is recovered to be normal, the target storage file to which the data to be written belongs is known; the data writing device sends the metadata of the data to be written to the target storage node together with the data to be written.
In addition, compared with the normal state of the data management equipment, the data to be written is received by the target storage node; when the data management node is abnormal, the data received by the target storage node comprises data to be written and metadata of the data to be written, and the target storage node configures a state identifier for the data to be written according to the characteristic and is used for indicating that the metadata of the data to be written needs to be synchronized to the data management node.
Therefore, in the embodiment of the application, after the data to be written is acquired, if it is difficult to acquire the application from the data management device to the target storage file, the data writing device selects the target storage file from the preset storage file, and writes the metadata recording the corresponding relationship between the data to be written and the target storage file into the target storage node together with the data to be written. Therefore, when the data management node is abnormal, the data writing node can also continue to write data into the storage node, and the data loss is avoided.
In addition, the metadata of the data to be written records the index information of the data to be written and the index information of the target storage file. The index information of the data to be written comprises a node identifier of a target storage node in which the data to be written is written, and the like; the index information of the target storage file comprises a file identifier, a file path, a file size and the like of the target storage file.
It is considered that the data management apparatus cannot allocate a target storage node to data to be written after the occurrence of an abnormality as well. Therefore, the embodiments of the present application provide various ways for determining a target storage node for data to be written after an exception occurs in the data management device.
In one embodiment, the data writing device determines a preset storage node from candidate storage nodes; taking a preset storage node as a target storage node; and writing the data to be written and the metadata of the data to be written into the target storage node.
It is understood that some specific storage nodes are prepared in advance for being used as target storage nodes after the abnormality occurs in the data management device.
In another embodiment, the data writing device selects a history storage node used in a preset time period from the candidate storage nodes; taking the history storage node as a target storage node; and writing the data to be written and the metadata of the data to be written into the target storage node.
Illustratively, the data writing device maintains a mapping table for recording a mapping relationship between the root directory and an Erasure Code (EC). In the embodiment of the application, the mapping table is called a RootEc-Hash, and one directory corresponds to one storage file.
Wherein, the root directory records root directory information, EC information and data fragment information; the data fragment information represents the corresponding relationship between the data fragments and the storage nodes after the complete data file is divided into fragments.
Therefore, the data writing device may use the directory name as a key, obtain the target storage node allocated for the last time before the data management device is abnormal from the root directory, and use the target storage node as the target storage node to which data is to be written this time.
Similarly, the data writing device maintains a mapping table of the thread and data fragment information, and takes the thread ID as the key. In the embodiment, the data fragment is called object, and the mapping table is called threadObj-Hash.
When the data writing device fails to apply for the target storage file from the metadata management device, the last written data fragment information is obtained through the thread ID, and if the root directory written in this time is the same as the last time, the last allocated target storage file is directly multiplexed, and the pre-allocated storage file is not used.
Based on the schematic diagram of the distributed storage scenario shown in fig. 1, an embodiment of the present application further provides a data storage method applied to a target storage node. The following describes each step in detail with reference to a step flowchart shown in fig. 4, and as shown in fig. 4, the data storage method includes:
in step S101B, data to be written and metadata of the data to be written are received.
The metadata records the corresponding relation between the data to be written and the target file, and the target file is selected from the pre-allocated storage file when the writing equipment detects that the storage file applied from the data management equipment is abnormal, and the pre-allocated storage file is applied from the data management equipment.
Step S102B, writing the data to be written and the metadata of the data to be written into the storage medium.
In step S103B, when it is detected that the data management apparatus is recovered to normal, the metadata of the data to be written is sent to the data management apparatus.
The metadata is used for enabling the data management equipment to establish a corresponding relation between the data to be written and the target file.
Just as when the data storage method executed by the write-in device is described, it is mentioned that after the data management node is recovered to normal, the target storage file to which the data to be written belongs is known; the data writing device sends the metadata of the data to be written to the target storage node together with the data to be written.
The target storage node may send the metadata corresponding to the data configured with the state identifier to the data management device at preset time intervals or after the target storage node is restarted, so that the data management device performs synchronization.
For example, as shown in fig. 5, the data storage node sends metadata of data to be written to the data management device, so that the data management device establishes a correspondence between the data to be written and the target file. For example, the data management device may record the correspondence between the data to be written and the target file in a redis database.
Based on the above description of the data writing device and the target storage device, an embodiment of the present application further provides a data storage system, where the data storage system includes the data writing device, the target storage node, and the data management device.
The data writing equipment acquires data to be written; when detecting that the application of the storage file from the data management equipment is abnormal, selecting a target file from the pre-allocated storage file; and writing the data to be written and the metadata of the data to be written into the target storage node.
The pre-distributed storage file is applied from data management equipment, and the metadata records the corresponding relation between the data to be written and the target file.
The target storage node receives data to be written and metadata of the data to be written; and when detecting that the data management equipment is recovered to be normal, sending the metadata of the data to be written to the data management equipment.
The data management equipment receives metadata of data to be written, which is sent by a target storage node; and establishing a corresponding relation between the data to be written and the target file according to the metadata.
Based on the same inventive concept as the data storage method executed by the data writing device, the embodiment of the present application also provides a data storage device, which is applied to the data writing device. The data storage means comprises at least one functional module which can be stored in the memory in the form of software. As shown in fig. 6, the data storage device is functionally divided into:
the data obtaining module 101 is configured to obtain data to be written.
In the embodiment of the present application, the data obtaining module 101 is configured to implement step S101A in fig. 3, and for a detailed description of the data obtaining module 101, refer to a detailed description of step S101A.
The file allocation module 102 selects a target file from the pre-allocated storage file when detecting that the storage file applied from the data management device is abnormal, wherein the pre-allocated storage file is applied from the data management device.
In the embodiment of the present application, the file allocation module 102 is configured to implement step S102A in fig. 3, and for a detailed description of the file allocation module 102, refer to a detailed description of step S102A.
The data sending module 103 writes the data to be written and the metadata of the data to be written into the target storage node, where the metadata records a correspondence between the data to be written and the target file.
In the embodiment of the present application, the data sending module 103 is configured to implement step S103A in fig. 3, and for a detailed description of the data sending module 103, refer to a detailed description of step S103A.
In addition, it should be noted that the data storage device applied to the data writing apparatus may further include other functional modules for implementing other steps or sub-steps of the corresponding data storage method. The data acquisition module 101, the file allocation module 102 and the data transmission module 103 may also be used to implement other steps or sub-steps of the corresponding data storage method. This is not specifically limited in the practice of the present application.
Based on the same inventive concept as the data storage method executed by the target storage device, the embodiment of the present application further provides a data storage apparatus, which is applied to the target storage node. The data storage means comprises at least one functional module which can be stored in the memory in the form of software. As shown in fig. 7, the data storage device includes:
the data receiving module 201 receives data to be written and metadata of the data to be written, where the metadata records a correspondence between the data to be written and a target file.
In the embodiment of the present application, the data receiving module 201 is configured to implement step S101B in fig. 4, and for a detailed description of the data receiving module 201, refer to a detailed description of step S101B.
The data writing module 202 writes the data to be written and the metadata of the data to be written into the storage medium.
In the embodiment of the present application, the data writing module 202 is configured to implement step S102B in fig. 4, and for a detailed description of the data writing module 202, refer to a detailed description of step S102B.
The data synchronization module 203 is configured to send metadata of the data to be written to the data management device when it is detected that the data management device is recovered to normal, where the metadata is used to enable the data management device to establish a corresponding relationship between the data to be written and the target file.
In the embodiment of the present application, the data synchronization module 203 is configured to implement step S103B in fig. 4, and for a detailed description of the data synchronization module 203, refer to a detailed description of step S103B.
In addition, it should be noted that the data storage apparatus applied to the target storage device may further include other functional modules for implementing other steps or sub-steps of the corresponding data storage method. The data receiving module 201, the data writing module 202 and the data synchronization module 203 may also be used to implement other steps, or sub-steps, of the corresponding data storage method. This is not specifically limited in the practice of the present application.
The application also provides an electronic device, which comprises a processor and a memory, wherein the memory stores a computer program.
As shown in fig. 8, the electronic device includes a memory 302, a communication device 304, and a processor 303.
The memory 302, processor 303 and communication device 304 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
The Memory 302 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like. The memory 302 is used for storing a program, and the communication device 304 executes the program after receiving an execution instruction. The communication device 304 is used for sending data through a network.
The processor 303 may be an integrated circuit chip having signal processing capabilities. The Processor 303 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. The general purpose processor 303 may be a microprocessor or the processor 303 may be any conventional processor or the like.
When the electronic device is a data writing device, the computer program realizes a data storage method executed by the data writing device when being executed by the processor 303; when the electronic device is a target storage device, the computer program realizes, when executed by the processor 303, a data storage method executed by the target storage device.
The embodiment of the application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program realizes a data storage method executed by data writing equipment or a data storage method executed by target storage equipment.
In summary, in the data storage method, the data storage device, the data storage apparatus, and the data storage medium provided in the embodiments of the present application, after acquiring data to be written, if acquiring an application target storage file from the data management apparatus fails, a target storage file is selected from the preset storage files, and metadata recording a correspondence between the data to be written and the target storage file is written into a target storage node together with the data to be written. Therefore, when the data management node is abnormal, the data writing node can also continue to write data into the storage node, and the data loss is avoided.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only for various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and all such changes or substitutions are included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data storage method, applied to a data writing apparatus, the method comprising:
acquiring data to be written;
when detecting that a storage file applied from a data management device is abnormal, selecting a target file from a pre-distributed storage file, wherein the pre-distributed storage file is applied from the data management device;
and writing the data to be written and the metadata of the data to be written into a target storage node, wherein the metadata records the corresponding relation between the data to be written and the target file.
2. The data storage method according to claim 1, wherein the writing the data to be written and the metadata of the data to be written into the target storage node comprises:
determining preset storage nodes from the candidate storage nodes;
taking the preset storage node as the target storage node;
and writing the data to be written and the metadata of the data to be written into the target storage node.
3. The data storage method according to claim 1, wherein the writing the data to be written and the metadata of the data to be written into the target storage node comprises:
selecting used historical storage nodes in a preset time period from the candidate storage nodes;
taking the history storage node as the target storage node;
and writing the data to be written and the metadata of the data to be written into the target storage node.
4. The data storage method according to claim 1, wherein the metadata further records index information of the target file and index information of the data to be written.
5. A data storage method applied to a target storage node, the method comprising:
receiving data to be written and metadata of the data to be written, wherein the metadata records a corresponding relation between the data to be written and a target file, the target file is selected from pre-allocated storage files when a data writing device detects that a storage file applied from a data management device is abnormal, and the pre-allocated storage file is applied from the data management device;
writing the data to be written and the metadata of the data to be written into a storage medium;
and when detecting that the data management equipment is recovered to be normal, sending the metadata of the data to be written to the data management equipment, wherein the metadata is used for enabling the data management equipment to establish the corresponding relation between the data to be written and the target file.
6. A data storage system, characterized in that the data storage system comprises a data writing device, a target storage node and a data management device;
the data writing equipment acquires data to be written; when detecting that the application of the storage file from the data management equipment is abnormal, selecting a target file from the pre-allocated storage file; writing the data to be written and the metadata of the data to be written into a target storage node; the pre-distributed storage file is applied from the data management equipment, and the metadata records the corresponding relation between the data to be written and the target file;
the target storage node receives the data to be written and the metadata of the data to be written; when detecting that the data management equipment is recovered to be normal, sending the metadata of the data to be written to the data management equipment;
the data management equipment receives metadata of data to be written, which is sent by the target storage node; and establishing a corresponding relation between the data to be written and the target file according to the metadata.
7. A data storage device, for use in a data writing apparatus, the data storage device comprising:
the data acquisition module is used for acquiring data to be written;
the file allocation module is used for selecting a target file from pre-allocated storage files when detecting that the storage files applied from the data management equipment are abnormal, wherein the pre-allocated storage files are applied from the data management equipment;
and the data sending module is used for writing the data to be written and the metadata of the data to be written into a target storage node, wherein the metadata records the corresponding relation between the data to be written and the target file.
8. A data storage device, applied to a target storage node, the data storage device comprising:
the data receiving module is used for receiving data to be written and metadata of the data to be written, wherein the metadata records the corresponding relation between the data to be written and a target file;
the data writing module writes the data to be written and the metadata of the data to be written into a storage medium;
and the data synchronization module is used for sending metadata of the data to be written to the data management equipment when the data management equipment is detected to be recovered to normal, wherein the metadata is used for enabling the data management equipment to establish the corresponding relation between the data to be written and the target file.
9. An electronic device, characterized in that the electronic device comprises a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the data storage method of any one of claims 1-4 or the data storage method of claim 5.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the data storage method of any one of claims 1-4 or the data storage method of claim 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110482490.3A CN113157497B (en) | 2021-04-30 | 2021-04-30 | Data storage method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110482490.3A CN113157497B (en) | 2021-04-30 | 2021-04-30 | Data storage method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113157497A true CN113157497A (en) | 2021-07-23 |
| CN113157497B CN113157497B (en) | 2023-04-07 |
Family
ID=76873212
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110482490.3A Active CN113157497B (en) | 2021-04-30 | 2021-04-30 | Data storage method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113157497B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114415936A (en) * | 2021-12-03 | 2022-04-29 | 北京汽车研究总院有限公司 | Data storage method, data storage device, vehicle device and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107153671A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for realizing the read-write of multifile copy in a distributed system |
| US20190073283A1 (en) * | 2017-09-05 | 2019-03-07 | International Business Machines Corporation | Method for the implementation of a high performance, high resiliency and high availability dual controller storage system |
| CN109491837A (en) * | 2018-11-01 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of the log fault-tolerance processing method and device of Nonvolatile memory reservoir |
| US20200065198A1 (en) * | 2018-08-27 | 2020-02-27 | International Business Machines Corporation | Deleted data set preservation and recovery |
| CN110908838A (en) * | 2019-11-19 | 2020-03-24 | 杭州安恒信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111104063A (en) * | 2019-12-06 | 2020-05-05 | 浪潮电子信息产业股份有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN111104448A (en) * | 2019-12-17 | 2020-05-05 | 深圳前海环融联易信息科技服务有限公司 | Method and device for exporting large-data-volume Excel file, computer equipment and storage medium |
| CN112650621A (en) * | 2020-12-24 | 2021-04-13 | 浪潮云信息技术股份公司 | Backup implementation method based on file storage |
-
2021
- 2021-04-30 CN CN202110482490.3A patent/CN113157497B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107153671A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for realizing the read-write of multifile copy in a distributed system |
| US20190073283A1 (en) * | 2017-09-05 | 2019-03-07 | International Business Machines Corporation | Method for the implementation of a high performance, high resiliency and high availability dual controller storage system |
| US20200065198A1 (en) * | 2018-08-27 | 2020-02-27 | International Business Machines Corporation | Deleted data set preservation and recovery |
| CN109491837A (en) * | 2018-11-01 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of the log fault-tolerance processing method and device of Nonvolatile memory reservoir |
| CN110908838A (en) * | 2019-11-19 | 2020-03-24 | 杭州安恒信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111104063A (en) * | 2019-12-06 | 2020-05-05 | 浪潮电子信息产业股份有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN111104448A (en) * | 2019-12-17 | 2020-05-05 | 深圳前海环融联易信息科技服务有限公司 | Method and device for exporting large-data-volume Excel file, computer equipment and storage medium |
| CN112650621A (en) * | 2020-12-24 | 2021-04-13 | 浪潮云信息技术股份公司 | Backup implementation method based on file storage |
Non-Patent Citations (3)
| Title |
|---|
| PAUL MARTIN: ""Mapping heterogeneous research infrastructure metadata into a unified catalogue for use in a generic virtual research environment"", 《FUTURE GENERATION COMPUTER SYSTEMS》 * |
| 云栖社区: ""Linux存储入门：简易数据恢复方案--分区和LVM实战"", 《HTTPS://WWW.51CTO.COM/ARTICLE/571444.HTML》 * |
| 陈起: ""高性能计算中并行文件系统可扩展元数据服务的研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114415936A (en) * | 2021-12-03 | 2022-04-29 | 北京汽车研究总院有限公司 | Data storage method, data storage device, vehicle device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113157497B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113220236B (en) | Data management method, system and equipment | |
| EP3468216B1 (en) | Video data storage system, operation method thereof and retrieval server | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN107229425B (en) | Data storage method and device | |
| CN111176888B (en) | Disaster recovery method, device and system for cloud storage | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN114979158B (en) | Resource monitoring method, system, equipment and computer readable storage medium | |
| CN112243044A (en) | Container address allocation method and device | |
| CN113946291A (en) | Data access method, device, storage node and readable storage medium | |
| CN113157497B (en) | Data storage method, device, equipment and medium | |
| CN111355600A (en) | Method and device for determining main node | |
| CN111399761B (en) | Storage resource allocation method, device and equipment, and storage medium | |
| CN113064553B (en) | Data storage method, device, equipment and medium | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| CN107547605B (en) | message reading and writing method based on node queue and node equipment | |
| CN107528703B (en) | Method and equipment for managing node equipment in distributed system | |
| CN111147226B (en) | Data storage method, device and storage medium | |
| CN109542841B (en) | Method for creating data snapshot in cluster and terminal equipment | |
| US20170149893A1 (en) | Metadata server, network device and automatic resource management method | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN111400327B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN112860494A (en) | Data center switching method and related equipment thereof | |
| CN112035413A (en) | Metadata information query method and device and storage medium | |
| CN111625502A (en) | Data reading method and device, storage medium and electronic device | |
| CN107203559B (en) | Method and device for dividing data strips |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |