CN111831618A - Data writing method, data reading method, device, equipment and storage medium - Google Patents
Data writing method, data reading method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111831618A CN111831618A CN202010706269.7A CN202010706269A CN111831618A CN 111831618 A CN111831618 A CN 111831618A CN 202010706269 A CN202010706269 A CN 202010706269A CN 111831618 A CN111831618 A CN 111831618A
- Authority
- CN
- China
- Prior art keywords
- storage
- writing
- data
- storage object
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004590 computer program Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data writing method, a data reading method, a data writing device, a data reading device, equipment and a storage medium. The data writing method comprises the following steps: acquiring a data writing request sent by a client, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content; acquiring metadata information according to the names of the storage buckets and the names of the storage objects; searching a target file according to the metadata information; and additionally writing the content of the storage object into the target file by taking the writing position as a starting point, and directly additionally writing the content generated in real time into the existing storage object in an additional writing mode through the technical scheme of the invention.
Description
Technical Field
Embodiments of the present invention relate to computer technologies, and in particular, to a data writing method, a data reading method, an apparatus, a device, and a storage medium.
Background
With the rapid development of the internet, the generated data grows exponentially, and most of the data is unstructured data, so how to safely and effectively store the massive data becomes a hot spot of current distributed storage research. Compared with file storage, object storage has high read-write speed, is beneficial to sharing compared with block storage, and has good expandability, safety and reliability, so that the object storage is concerned by a large number of users and developers at present. In an object storage system, there is no concept of directory, and all objects adopt flat management. A complete object usually includes an index portion and a data portion, the index portion mainly records basic information of the object, such as size and type of the object, and provides a storage location for the object data, and the data portion stores specific content of the object.
For some scenes such as video monitoring, live video and the like, data is generally generated continuously in real time, and if the data is stored by adopting a traditional object storage method, a data stream generated in real time needs to be cut into a plurality of storage objects for uploading.
Disclosure of Invention
The embodiment of the invention provides a data writing method, a data reading method, a device, equipment and a storage medium, which directly additionally write the content generated in real time into an existing storage object in an additional writing mode.
In a first aspect, an embodiment of the present invention provides a data writing method, including:
acquiring a data writing request sent by a client, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content;
acquiring metadata information according to the names of the storage buckets and the names of the storage objects;
searching a target file according to the metadata information;
and additionally writing the storage object content into the target file by taking the writing position as a starting point.
In a second aspect, an embodiment of the present invention provides a data reading method, including:
acquiring a data reading request sent by a client, wherein the data reading request comprises: a bucket name and a storage object name;
searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and reading the content of the storage object according to the metadata information.
In a third aspect, an embodiment of the present invention further provides a data writing apparatus, where the apparatus includes:
a first obtaining module, configured to obtain a data write request sent by a client, where the data write request includes: the storage bucket name, the storage object name, the writing position and the storage object content;
the second acquisition module is used for acquiring metadata information according to the storage bucket name and the storage object name;
the first searching module is used for searching a target file according to the metadata information;
and the writing module is used for additionally writing the storage object content into the target file by taking the writing position as a starting point.
In a fourth aspect, an embodiment of the present invention further provides a data reading apparatus, where the apparatus includes:
a third obtaining module, configured to obtain a data reading request sent by a client, where the data reading request includes: a bucket name and a storage object name;
the second searching module is used for searching the metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and the reading module is used for reading the content of the storage object according to the metadata information.
In a fifth aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method according to any one of the embodiments of the present invention.
In a sixth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the method according to any one of the embodiments of the present invention.
The embodiment of the invention obtains a data writing request sent by a client, wherein the data writing request comprises the following steps: the storage bucket name, the storage object name, the writing position and the storage object content; acquiring metadata information according to the names of the storage buckets and the names of the storage objects; searching a target file according to the metadata information; and additionally writing the content of the storage object into the target file by taking the writing position as a starting point, and directly additionally writing the content generated in real time into the existing storage object in an additional writing mode.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a flow chart of a data writing method according to a first embodiment of the present invention;
FIG. 1a is a diagram of a system architecture according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a data reading method according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data writing apparatus according to a third embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a data reading apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example one
Fig. 1 is a flowchart of a data writing method according to an embodiment of the present invention, where the present embodiment is applicable to a data writing situation, and the method may be executed by a data writing apparatus according to an embodiment of the present invention, where the apparatus may be implemented in a software and/or hardware manner, as shown in fig. 1, the method specifically includes the following steps:
s110, acquiring a data writing request sent by a client, wherein the data writing request comprises: bucket name, storage object name, write location, and storage object content.
And the data writing request is an http request.
Wherein the name of the storage bucket is the name of bucket; the name of the object is stored. A bucket may store a plurality of storage objects.
Specifically, a data write request sent by a client is obtained, where the data write request includes: the bucket name, the storage object name, the write location, and the storage object content may be, for example, a bucket, an object, a write location, and a storage object content specified in a data write request.
And S120, acquiring metadata information according to the bucket name and the storage object name.
Specifically, metadata information is obtained according to the bucket name and the storage object name. For example, metadata information may be obtained according to a bucket and an object specified in the write request.
S130, searching a target file according to the metadata information.
Specifically, the target file is searched according to the position information in the metadata information.
And S140, additionally writing the storage object content into the target file by taking the writing position as a starting point.
Specifically, according to the writing position carried in the data writing request, the storage object content is additionally written into the target file with the writing position as a starting point.
Optionally, after additionally writing the content of the storage object into the target file with the writing position as a starting point, the method further includes:
and updating metadata information in an index database according to the bucket names and the storage object names.
Optionally, after additionally writing the content of the storage object into the target file with the writing position as a starting point, the method further includes:
and sending a return header to the client so that the client determines a writing position according to the return header.
Optionally, additionally writing the content of the storage object into the target file with the writing position as a starting point, including:
when the space used by the storage node where the target file is located exceeds a preset threshold, determining the target storage node according to the working state of the storage node and creating a file as the target file;
and additionally writing the storage object content into the target file of the target storage node by taking the writing position as a starting point.
Optionally, the data write request further includes: storing type information;
correspondingly, the obtaining the working states of all the storage nodes in the storage cluster includes:
if the storage type information is standard storage, acquiring the working states of all storage nodes in a standard storage cluster;
and if the storage type information is low-frequency storage, acquiring the working states of all storage nodes in the low-frequency storage cluster.
In a specific example, as shown in fig. 1a, the system architecture mainly includes an access module, an index module and a storage module. The access module is mainly responsible for analyzing the client request and forwarding the request to the gateway service; the index module is mainly used for recording related metadata information; the storage module is mainly responsible for data storage, and comprises a plurality of storage clusters, which mainly comprise standard storage and low-frequency storage at present. Each module can dynamically expand along with the increase of the used data volume, and the independent index module and the independent storage module can avoid unnecessary resource waste during expansion. The flow of writing general object data is as follows: a client sends a request; analyzing the request packet and forwarding the request to the gateway; writing data into the storage module; and updating the metadata information in the index module. For some scenes such as video monitoring, live video and the like, data is generally generated continuously in real time, and if the data is stored by adopting a traditional object storage method, a data stream generated in real time needs to be cut into a plurality of objects for uploading. For these scene requirements, in order to reduce development cost and facilitate management, based on the architecture of fig. 1a, the content generated in real time is directly additionally written in the existing object by means of additional writing, and the processing flow is as follows: the client sends a request, and a bucket, an object and a writing position are specified; analyzing the request packet and forwarding the request to the gateway; additionally writing data to the designated position to the selected storage cluster; and updating the metadata information in the index module. The client may initiate an http request via the console, sdk, etc., and the request may specify a bucket, an object, and a write location. The access module analyzes and forwards the request to the gateway service, the whole back-end storage service comprises a plurality of gateway nodes, the request is preferentially forwarded to the gateway service of the node closest to the network when the request is forwarded, and meanwhile, the request can be automatically forwarded to the gateway services of other nodes when the gateway service of the node is wrong, so that the high availability of the whole back-end service is ensured. The data is stored in a segmented storage mode, the segmentation logic is that when the size of a single segment exceeds a set threshold value or a currently used storage cluster exceeds a threshold value, a new segment is started, and all data has multiple copies, so that the reliability of the data is ensured. The metadata managed by the index module mainly comprises bucket metadata, object segmentation metadata, statistical charging data and the like, and the metadata are stored by adopting a plurality of copies, so that the reliability of the data is further ensured.
The embodiment of the invention is based on HTTP streaming write data, and supports writing various unstructured data into a specified position in an additional mode for storage. Storage is provided for a large number of data streams generated in real time, such as monitoring data, video live data, etc. And uploading data to the designated position, writing the acquired data in an additional mode, storing the data, and recording corresponding index information. The stored object content can be modified and can be read at any time, the storage and reading of data streams generated in real time under various scenes can be simplified, the development cost is reduced, and the data management is facilitated.
According to the technical scheme of the embodiment, a data writing request sent by a client is obtained, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content; acquiring metadata information according to the names of the storage buckets and the names of the storage objects; searching a target file according to the metadata information; and additionally writing the content of the storage object into the target file by taking the writing position as a starting point, and directly additionally writing the content generated in real time into the existing storage object in an additional writing mode.
Example two
Fig. 2 is a flowchart of a data reading method according to a second embodiment of the present invention, where this embodiment is applicable to a data reading situation, and the method may be executed by a data reading apparatus according to the second embodiment of the present invention, and the apparatus may be implemented in a software and/or hardware manner, as shown in fig. 2, the method specifically includes the following steps:
s210, acquiring a data reading request sent by a client, where the data reading request includes: bucket names and storage object names.
The read request includes a bucket name and a storage object name, for example, the read request may include a bucket name a and a storage object name B.
Specifically, a data reading request sent by a client is obtained, where the data reading request carries a bucket name and a storage object name.
S220, searching metadata information corresponding to the reading request from an index database according to the bucket name and the storage object name.
The index database is stored with the corresponding relation between the names of the storage buckets and the names of the storage objects and the metadata information in advance.
Specifically, the metadata information corresponding to the read request is searched from an index database according to the bucket name and the storage object name carried in the data read request.
And S230, reading the content of the storage object according to the metadata information.
Specifically, the storage object content is read according to the metadata information.
It should be noted that the stored object is readable at any time, after a data reading request sent by the client is obtained, metadata information corresponding to the reading request is searched from the index database according to a bucket name and a storage object name carried in the data reading request, and the content of the storage object is read according to the metadata information.
According to the technical scheme of the embodiment, a data reading request sent by a client is obtained, wherein the data reading request comprises: a bucket name and a storage object name; searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name; and reading the content to be stored according to the metadata information, thereby realizing reading of the content to be stored.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data writing device according to a third embodiment of the present invention. The present embodiment may be applicable to the case of data writing, the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be integrated in any device providing a data writing function, as shown in fig. 3, where the data writing apparatus specifically includes: a first obtaining module 310, a second obtaining module 320, a first searching module 330, and a writing module 340.
The first obtaining module is configured to obtain a data writing request sent by a client, where the data writing request includes: the storage bucket name, the storage object name, the writing position and the storage object content;
the second acquisition module is used for acquiring metadata information according to the storage bucket name and the storage object name;
the first searching module is used for searching a target file according to the metadata information;
and the writing module is used for additionally writing the storage object content into the target file by taking the writing position as a starting point.
The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
According to the technical scheme of the embodiment, a data writing request sent by a client is obtained, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content; acquiring metadata information according to the names of the storage buckets and the names of the storage objects; searching a target file according to the metadata information; and additionally writing the content of the storage object into the target file by taking the writing position as a starting point, and directly additionally writing the content generated in real time into the existing storage object in an additional writing mode.
Example four
Fig. 4 is a schematic structural diagram of a data reading apparatus according to a fourth embodiment of the present invention. The present embodiment may be applicable to the case of data reading, the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be integrated in any device providing a data reading function, as shown in fig. 4, where the data reading apparatus specifically includes: a third acquisition module 410, a second lookup module 420, and a read module 430.
The third obtaining module is configured to obtain a data reading request sent by a client, where the data reading request includes: a bucket name and a storage object name;
the second searching module is used for searching the metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and the reading module is used for reading the content of the storage object according to the metadata information.
The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
According to the technical scheme of the embodiment, a data reading request sent by a client is obtained, wherein the data reading request comprises: a bucket name and a storage object name; searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name; and reading the content to be stored according to the metadata information, thereby realizing reading of the content to be stored.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a computer device in the fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing a data writing method provided by an embodiment of the present invention:
acquiring a data writing request sent by a client, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content;
acquiring metadata information according to the names of the storage buckets and the names of the storage objects;
searching a target file according to the metadata information;
and additionally writing the storage object content into the target file by taking the writing position as a starting point.
Or the data reading method provided by the embodiment of the invention is realized as follows:
acquiring a data reading request sent by a client, wherein the data reading request comprises: a bucket name and a storage object name;
searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and reading the content of the storage object according to the metadata information.
EXAMPLE six
A sixth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data writing method provided in all the embodiments of the present invention:
acquiring a data writing request sent by a client, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content;
acquiring metadata information according to the names of the storage buckets and the names of the storage objects;
searching a target file according to the metadata information;
and additionally writing the storage object content into the target file by taking the writing position as a starting point.
Or the data reading method provided by the embodiment of the invention is realized as follows:
acquiring a data reading request sent by a client, wherein the data reading request comprises: a bucket name and a storage object name;
searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and reading the content of the storage object according to the metadata information.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of writing data, the method comprising:
acquiring a data writing request sent by a client, wherein the data writing request comprises: the storage bucket name, the storage object name, the writing position and the storage object content;
acquiring metadata information according to the names of the storage buckets and the names of the storage objects;
searching a target file according to the metadata information;
and additionally writing the storage object content into the target file by taking the writing position as a starting point.
2. The method according to claim 1, wherein after additionally writing the storage object content into the target file with the writing location as a starting point, further comprising:
and updating metadata information in an index database according to the bucket names and the storage object names.
3. The method according to claim 1, wherein after additionally writing the storage object content into the target file with the writing location as a starting point, further comprising:
and sending a return header to the client so that the client determines a writing position according to the return header.
4. The method according to claim 1, wherein additionally writing the storage object content into the target file with the writing position as a starting point comprises:
when the space used by the storage node where the target file is located exceeds a preset threshold, determining the target storage node according to the working state of the storage node and creating a file as the target file;
and additionally writing the storage object content into the target file of the target storage node by taking the writing position as a starting point.
5. The method of claim 4, wherein the data write request further comprises: storing type information;
correspondingly, the obtaining the working states of all the storage nodes in the storage cluster includes:
if the storage type information is standard storage, acquiring the working states of all storage nodes in a standard storage cluster;
and if the storage type information is low-frequency storage, acquiring the working states of all storage nodes in the low-frequency storage cluster.
6. A data reading method, comprising:
acquiring a data reading request sent by a client, wherein the data reading request comprises: a bucket name and a storage object name;
searching metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and reading the content of the storage object according to the metadata information.
7. A data writing apparatus, characterized in that the apparatus comprises:
a first obtaining module, configured to obtain a data write request sent by a client, where the data write request includes: the storage bucket name, the storage object name, the writing position and the storage object content;
the second acquisition module is used for acquiring metadata information according to the storage bucket name and the storage object name;
the first searching module is used for searching a target file according to the metadata information;
and the writing module is used for additionally writing the storage object content into the target file by taking the writing position as a starting point.
8. A data reading apparatus, comprising:
a third obtaining module, configured to obtain a data reading request sent by a client, where the data reading request includes: a bucket name and a storage object name;
the second searching module is used for searching the metadata information corresponding to the reading request from an index database according to the storage bucket name and the storage object name;
and the reading module is used for reading the content of the storage object according to the metadata information.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-6 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010706269.7A CN111831618A (en) | 2020-07-21 | 2020-07-21 | Data writing method, data reading method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010706269.7A CN111831618A (en) | 2020-07-21 | 2020-07-21 | Data writing method, data reading method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111831618A true CN111831618A (en) | 2020-10-27 |
Family
ID=72923844
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010706269.7A Pending CN111831618A (en) | 2020-07-21 | 2020-07-21 | Data writing method, data reading method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111831618A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112684985A (en) * | 2021-01-04 | 2021-04-20 | 北京金山云网络技术有限公司 | Data writing method and device |
| CN112965950A (en) * | 2021-03-09 | 2021-06-15 | 浪潮云信息技术股份公司 | Method for realizing storage of stream data write-in object |
| CN112988681A (en) * | 2021-04-07 | 2021-06-18 | 拉卡拉支付股份有限公司 | File processing method, file processing device, electronic equipment, storage medium and program product |
| CN113126920A (en) * | 2021-04-02 | 2021-07-16 | 深圳市杉岩数据技术有限公司 | Object storage batch reading and writing method, client, storage gateway and medium |
| CN113301095A (en) * | 2020-12-08 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method and device for providing data of cloud object |
| CN113625952A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Object storage method, device, equipment and storage medium |
| CN113886347A (en) * | 2021-09-23 | 2022-01-04 | 北京金山云网络技术有限公司 | Node data compression method and device, storage medium and electronic equipment |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103731489A (en) * | 2013-12-27 | 2014-04-16 | 华为技术有限公司 | Data storage method, system and equipment |
| US20140122429A1 (en) * | 2012-10-31 | 2014-05-01 | International Business Machines Corporation | Data processing method and apparatus for distributed systems |
| CN105100149A (en) * | 2014-05-13 | 2015-11-25 | 中国电信股份有限公司 | Method and system for file management |
| US20160147479A1 (en) * | 2014-11-26 | 2016-05-26 | International Business Machines Corporation | Metadata storing technique |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| CN106294870A (en) * | 2016-08-25 | 2017-01-04 | 苏州酷伴软件科技有限公司 | Object-based distributed cloud storage method |
| US20170091262A1 (en) * | 2015-09-25 | 2017-03-30 | Netapp, Inc. | Object storage backed file system cache |
| CN108234594A (en) * | 2017-11-28 | 2018-06-29 | 北京市商汤科技开发有限公司 | File memory method and device, electronic equipment, program and medium |
| CN108664208A (en) * | 2017-03-27 | 2018-10-16 | 纳思达股份有限公司 | File appending write operation method based on FLASH memory and device |
| CN109271364A (en) * | 2018-09-18 | 2019-01-25 | 图普科技（广州）有限公司 | Storage method and device |
| CN109669632A (en) * | 2018-12-10 | 2019-04-23 | 浪潮电子信息产业股份有限公司 | Metadata wiring method, device and medium based on distributed objects storage system |
| CN109831540A (en) * | 2019-04-12 | 2019-05-31 | 成都四方伟业软件股份有限公司 | Distributed storage method, device, electronic equipment and storage medium |
| CN110557964A (en) * | 2018-03-30 | 2019-12-10 | 华为技术有限公司 | Data writing method, client server and system |
| CN110580127A (en) * | 2018-06-07 | 2019-12-17 | 华中科技大学 | Resource management method and resource management system based on multi-tenant cloud storage |
| CN110703990A (en) * | 2019-09-16 | 2020-01-17 | 北京百度网讯科技有限公司 | Data archiving and storing method and device, electronic equipment and storage medium |
| CN110879800A (en) * | 2018-09-05 | 2020-03-13 | 阿里巴巴集团控股有限公司 | Data writing, compressing and reading method, data processing method and device |
| CN110928497A (en) * | 2019-11-15 | 2020-03-27 | 浪潮电子信息产业股份有限公司 | Metadata processing method, device and equipment and readable storage medium |
| CN111008185A (en) * | 2019-10-29 | 2020-04-14 | 厦门网宿有限公司 | Data sharing method, system and equipment |
| CN111078653A (en) * | 2019-10-29 | 2020-04-28 | 厦门网宿有限公司 | Data storage method, system and equipment |
| CN111125002A (en) * | 2019-11-15 | 2020-05-08 | 卢万龙 | Data backup filing method and system based on distributed storage |
| CN111124280A (en) * | 2019-11-29 | 2020-05-08 | 浪潮电子信息产业股份有限公司 | Data additional writing method and device, electronic equipment and storage medium |
| CN111427841A (en) * | 2020-02-26 | 2020-07-17 | 平安科技（深圳）有限公司 | Data management method and device, computer equipment and storage medium |
-
2020
- 2020-07-21 CN CN202010706269.7A patent/CN111831618A/en active Pending
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140122429A1 (en) * | 2012-10-31 | 2014-05-01 | International Business Machines Corporation | Data processing method and apparatus for distributed systems |
| CN103731489A (en) * | 2013-12-27 | 2014-04-16 | 华为技术有限公司 | Data storage method, system and equipment |
| CN105100149A (en) * | 2014-05-13 | 2015-11-25 | 中国电信股份有限公司 | Method and system for file management |
| US20160147479A1 (en) * | 2014-11-26 | 2016-05-26 | International Business Machines Corporation | Metadata storing technique |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| US20170091262A1 (en) * | 2015-09-25 | 2017-03-30 | Netapp, Inc. | Object storage backed file system cache |
| CN106294870A (en) * | 2016-08-25 | 2017-01-04 | 苏州酷伴软件科技有限公司 | Object-based distributed cloud storage method |
| CN108664208A (en) * | 2017-03-27 | 2018-10-16 | 纳思达股份有限公司 | File appending write operation method based on FLASH memory and device |
| CN108234594A (en) * | 2017-11-28 | 2018-06-29 | 北京市商汤科技开发有限公司 | File memory method and device, electronic equipment, program and medium |
| CN110557964A (en) * | 2018-03-30 | 2019-12-10 | 华为技术有限公司 | Data writing method, client server and system |
| CN110580127A (en) * | 2018-06-07 | 2019-12-17 | 华中科技大学 | Resource management method and resource management system based on multi-tenant cloud storage |
| CN110879800A (en) * | 2018-09-05 | 2020-03-13 | 阿里巴巴集团控股有限公司 | Data writing, compressing and reading method, data processing method and device |
| CN109271364A (en) * | 2018-09-18 | 2019-01-25 | 图普科技（广州）有限公司 | Storage method and device |
| CN109669632A (en) * | 2018-12-10 | 2019-04-23 | 浪潮电子信息产业股份有限公司 | Metadata wiring method, device and medium based on distributed objects storage system |
| CN109831540A (en) * | 2019-04-12 | 2019-05-31 | 成都四方伟业软件股份有限公司 | Distributed storage method, device, electronic equipment and storage medium |
| CN110703990A (en) * | 2019-09-16 | 2020-01-17 | 北京百度网讯科技有限公司 | Data archiving and storing method and device, electronic equipment and storage medium |
| CN111008185A (en) * | 2019-10-29 | 2020-04-14 | 厦门网宿有限公司 | Data sharing method, system and equipment |
| CN111078653A (en) * | 2019-10-29 | 2020-04-28 | 厦门网宿有限公司 | Data storage method, system and equipment |
| CN110928497A (en) * | 2019-11-15 | 2020-03-27 | 浪潮电子信息产业股份有限公司 | Metadata processing method, device and equipment and readable storage medium |
| CN111125002A (en) * | 2019-11-15 | 2020-05-08 | 卢万龙 | Data backup filing method and system based on distributed storage |
| CN111124280A (en) * | 2019-11-29 | 2020-05-08 | 浪潮电子信息产业股份有限公司 | Data additional writing method and device, electronic equipment and storage medium |
| CN111427841A (en) * | 2020-02-26 | 2020-07-17 | 平安科技（深圳）有限公司 | Data management method and device, computer equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| 徐强: "第五章：对象存储OSS笔记", Retrieved from the Internet <URL:https://zhuanlan.zhihu.com/p/60539835> * |
| 浪潮云: "追加上传 · 帮助文档", Retrieved from the Internet <URL:https://console1.cloud.inspur.com/document/oss/6-SDK-and-tool/SDK/java-sdk/6.2.10-javasdk-appendObject.html> * |
| 阿里云: "对象存储OSS开发人员指南", Retrieved from the Internet <URL:https://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/pdf/oss-user-guide-intl-zh-2016-07-27.pdf> * |
| 阿里云: "视频直播实时产生数据怎么存储", Retrieved from the Internet <URL:https://developer.aliyun.com/ask/2154#> * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113301095A (en) * | 2020-12-08 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method and device for providing data of cloud object |
| CN113301095B (en) * | 2020-12-08 | 2024-05-10 | 阿里巴巴集团控股有限公司 | Method and device for providing data of cloud object |
| CN112684985A (en) * | 2021-01-04 | 2021-04-20 | 北京金山云网络技术有限公司 | Data writing method and device |
| CN112684985B (en) * | 2021-01-04 | 2024-04-05 | 北京金山云网络技术有限公司 | Data writing method and device |
| CN112965950A (en) * | 2021-03-09 | 2021-06-15 | 浪潮云信息技术股份公司 | Method for realizing storage of stream data write-in object |
| CN113126920A (en) * | 2021-04-02 | 2021-07-16 | 深圳市杉岩数据技术有限公司 | Object storage batch reading and writing method, client, storage gateway and medium |
| CN112988681A (en) * | 2021-04-07 | 2021-06-18 | 拉卡拉支付股份有限公司 | File processing method, file processing device, electronic equipment, storage medium and program product |
| CN113625952A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Object storage method, device, equipment and storage medium |
| CN113886347A (en) * | 2021-09-23 | 2022-01-04 | 北京金山云网络技术有限公司 | Node data compression method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111831618A (en) | Data writing method, data reading method, device, equipment and storage medium | |
| CN109254733B (en) | Method, device and system for storing data | |
| CN111309732B (en) | Data processing method, device, medium and computing equipment | |
| CN110046133B (en) | Metadata management method, device and system for storage file system | |
| JP5886447B2 (en) | Location independent files | |
| KR20120102664A (en) | Allocating storage memory based on future use estimates | |
| CN111339186A (en) | Workflow engine data synchronization method, device, medium and electronic equipment | |
| CN114385091B (en) | Method and device for realizing network disk drive character, network disk and storage medium | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| US20170153909A1 (en) | Methods and Devices for Acquiring Data Using Virtual Machine and Host Machine | |
| CN107733894B (en) | Comparison method, system, equipment and storage medium of logical interface message | |
| CN110119386B (en) | Data processing method, data processing apparatus, medium, and computing device | |
| CN112433920A (en) | Buried point data reporting method, device, equipment and medium | |
| WO2020199659A1 (en) | Method and apparatus for determining push priority information | |
| CN109960554A (en) | Show method, equipment and the computer storage medium of reading content | |
| CN110781159B (en) | Ceph directory file information reading method and device, server and storage medium | |
| CN110706148A (en) | Face image processing method, device, equipment and storage medium | |
| CN114416670A (en) | Index creating method and device suitable for network disk document, network disk and storage medium | |
| CN112395141B (en) | Data page management method and device, electronic equipment and storage medium | |
| CN113805864A (en) | Project engineering generation method and device, electronic equipment and storage medium | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| US20050091266A1 (en) | Data file system, data access server and data access program storage medium | |
| CN115826828B (en) | Network disk file operation method, device, terminal and storage medium | |
| CN110515803B (en) | Processing method and device for log message and electronic equipment | |
| US20210216507A1 (en) | Method, device and computer program product for storage management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100000 Unit 501A, Floor 5, 101, Building 1, Yard 5, Laiguangying West Road, Chaoyang District, Beijing Applicant after: Beijing Qingyun Science and Technology Co.,Ltd. Address before: 100020 601, 6 floor, 16 building, No. 36 Chuang Yuan Road, Chaoyang District, Beijing. Applicant before: Beijing Qingyun Science and Technology Co.,Ltd. |