CN113342837B - Data transmission method, device, electronic equipment and computer readable medium - Google Patents
Data transmission method, device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN113342837B CN113342837B CN202110899346.XA CN202110899346A CN113342837B CN 113342837 B CN113342837 B CN 113342837B CN 202110899346 A CN202110899346 A CN 202110899346A CN 113342837 B CN113342837 B CN 113342837B
- Authority
- CN
- China
- Prior art keywords
- target
- data
- metadata
- target data
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure discloses a data transmission method, a data transmission device, an electronic device and a computer readable medium. One embodiment of the method comprises: in response to receiving a processing request for a target object, determining whether metadata corresponding to target data exists in a metadata set corresponding to each object, wherein the target object comprises: target data and metadata corresponding to the target data; responding to the fact that metadata corresponding to the target data exist, and acquiring first storage information representing whether the target data exist in the data cache or not; and responding to the first storage information representation data cache without target data, acquiring the target objects in the object set stored in the object storage, and sending the target data in the target objects to the data cache to store the target data. The implementation method can quickly and efficiently acquire the target object. In addition, the target data is sent to a data cache to enable efficient subsequent invocation of the target data.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a data transmission method, an apparatus, an electronic device, and a computer-readable medium.
Background
Currently, in the object store, each object corresponds to data. There is no hierarchical relationship between the objects in a hierarchy. That is, each object is stored in the same level of the flat address space of the storage pool. For the acquisition of the target object, the following methods are generally adopted: first, according to a target metadata interface of an object store, whether a target object exists in the object store is determined. Then, in response to determining that there is, a target object is retrieved from the object store.
However, when the target object is acquired in the above manner, there are often the following technical problems:
and the target metadata interface is excessively depended, so that the response speed is low, and the target object acquisition efficiency is low. In addition, the target data in the target object is acquired from the object storage, which has the problems of time and labor consumption, and affects the acquisition efficiency of the target object to a certain extent.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose data transmission methods, apparatuses, electronic devices and computer readable media to solve one or more of the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a data sending method, including: in response to receiving a processing request for a target object, determining whether metadata corresponding to target data exists in a metadata set corresponding to each object, wherein the target object comprises: the target data and metadata corresponding to the target data; responding to the metadata corresponding to the target data, and acquiring first storage information representing whether the target data exists in a data cache; responding to the first storage information representing that the target data does not exist in the data cache, acquiring the target objects in an object set stored in an object storage, and sending the target data in the target objects to the data cache to store the target data, wherein the object set comprises the objects.
Optionally, the method further includes: in response to determining that no metadata corresponding to the target data exists, calling a target metadata interface to acquire second storage information representing whether the target object exists in the object set; responding to the second storage information to represent that the target object exists in the object set, and acquiring metadata corresponding to the target object in the object storage; and responding to the acquired metadata corresponding to the target object, storing the metadata corresponding to the target object, and sending the target data to the data cache to store the target data.
Optionally, the method further includes: responding to the first storage information representing that the target data exists in the data cache, and acquiring the target data in the data cache; and generating the target object according to the target data and the metadata corresponding to the target data.
Optionally, the method further includes: and in response to receiving object transformation information representing object transformation in the object set, adjusting the metadata in the metadata set according to the object transformation information.
Optionally, the method further includes: and periodically updating the metadata sets corresponding to the objects according to the metadata sets corresponding to the object sets.
Optionally, the metadata set is displayed in a directory tree form, and the metadata set is stored in a distributed storage form.
In a second aspect, some embodiments of the present disclosure provide a data transmission apparatus, including: a determination unit configured to determine, in response to receiving a processing request for a target object including: the target data and metadata corresponding to the target data; a first obtaining unit configured to obtain, in response to determining that metadata corresponding to the target data exists, first storage information representing whether the target data exists in a data cache; a second obtaining unit, configured to respond to the first storage information representing that the target data does not exist in the data cache, obtain the target objects in an object set stored in an object storage, and send the target data in the target objects to the data cache to store the target data, wherein the object set includes the objects.
Optionally, the apparatus further comprises: in response to determining that no metadata corresponding to the target data exists, calling a target metadata interface to acquire second storage information representing whether the target object exists in the object set; responding to the second storage information to represent that the target object exists in the object set, and acquiring metadata corresponding to the target object in the object storage; and responding to the acquired metadata corresponding to the target object, storing the metadata corresponding to the target object, and sending the target data to the data cache to store the target data.
Optionally, the apparatus further comprises: responding to the first storage information representing that the target data exists in the data cache, and acquiring the target data in the data cache; and generating the target object according to the target data and the metadata corresponding to the target data.
Optionally, the apparatus further comprises: and in response to receiving object transformation information representing object transformation in the object set, adjusting the metadata in the metadata set according to the object transformation information.
Optionally, the apparatus further comprises: and periodically updating the metadata sets corresponding to the objects according to the metadata sets corresponding to the object sets.
Optionally, the apparatus further comprises: the metadata set is displayed in a directory tree form, and the storage form of the metadata set is a distributed storage form.
In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, where the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
The above embodiments of the present disclosure have the following beneficial effects: the data sending method of some embodiments of the disclosure can quickly and efficiently obtain the target object. In addition, the target data is sent to a data cache to enable efficient subsequent invocation of the target data. In particular, the metadata interface is excessively relied on, so that the response speed is slow, and the target object acquisition efficiency is low. In addition, the target data in the target object is acquired from the object storage, which has the problems of time and labor consumption, and affects the acquisition efficiency of the target object to a certain extent. Based on this, the data transmission method of some embodiments of the present disclosure may first determine, in response to receiving a processing request for a target object, whether metadata corresponding to the target data exists in a metadata set corresponding to each object that is stored in advance. Here, the target object does not need to be directly acquired from the object store through the target metadata interface, and the dependence on the target metadata interface can be alleviated to some extent. Wherein the target object includes: the target data and the metadata corresponding to the target data. Then, in response to determining that the metadata corresponding to the target data exists, first storage information representing whether the target data exists in the data cache is acquired. Here, the data cache stores partial data, so that the data acquisition efficiency can be effectively improved. The data cache can cache part of data with higher utilization rate. By setting the data cache, when the target data is acquired, the data is not required to be searched and acquired from the object storage storing a large amount of data in a complicated manner. It may be determined whether the target data exists in the data set stored by the data cache. Wherein the data number of the data set is smaller than the data number of the data set stored in the object store. And finally, responding to the first storage information representing that the target data does not exist in the data cache, acquiring the target objects in an object set stored in an object storage, and sending the target data in the target objects to the data cache to store the target data. Wherein the object set includes the respective objects. By sending the data of the target object to the data cache to store the target data, the target data can be reused efficiently and conveniently.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
Fig. 1 is a schematic diagram of one application scenario of a data transmission method according to some embodiments of the present disclosure;
fig. 2 is a flow diagram of some embodiments of a data transmission method according to the present disclosure;
FIG. 3 is a flow chart of further embodiments of a data transmission method according to the present disclosure;
fig. 4 is a flow chart of still further embodiments of a data transmission method according to the present disclosure;
fig. 5 is a schematic structural diagram of some embodiments of a data transmission apparatus according to the present disclosure;
FIG. 6 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic diagram of an application scenario of a data transmission method according to some embodiments of the present disclosure.
In the application scenario of fig. 1, in response to receiving a processing request 103 for a target object 102, the metadata processing end 101 may first determine whether metadata 1022 corresponding to target data exists in a metadata set 105 corresponding to each object 104 stored in advance. Wherein, the target object 102 includes: the target data 1021 and metadata 1022 corresponding to the target data. In this application scenario, the metadata 1022 corresponding to the target data may be: "image person sex: female, image resolution: 640 × 480, age of image person: age 15, image capture time: day 12 and 6 months 2019, image capturing location: beijing ″. The above-mentioned respective objects 104 may include: object 1041, object 1042, object 1043. The set of metadata 105 may include: metadata 1051 in object 1041, metadata 1052 in object 1042, and metadata 1053 in object 1043. Then, in response to determining that the metadata 1022 corresponding to the target data exists, the metadata processing end 101 may obtain the first storage information 107 representing whether the target data 1021 exists in the data cache 106. In this application scenario, the metadata 1051 is the same as the metadata 1022 corresponding to the target data. Finally, in response to the first storage information 107 indicating that the target data 1021 does not exist in the data cache 106, the target object 1094 in the object set 109 stored in the object storage 108 is obtained, and the target data 1021 in the target object 1094 is sent to the data cache 106 to store the target data 1021. Wherein the object set 109 includes the objects 104. In the present application scenario, the object set 109 includes: an object 1091 identical to the object 1041, an object 1092 identical to the object 1042, an object 1093 identical to the object 1043, an object 1094 identical to the target object 102, and an object 1095.
The metadata processing side 101 may be hardware or software. When the metadata processing end is hardware, the metadata processing end can be implemented as a distributed cluster formed by a plurality of servers or terminal devices, or can be implemented as a single server or a single terminal device. When the metadata processing end is embodied as software, the metadata processing end can be installed in the hardware devices listed above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.
Further, the metadata sets corresponding to the respective objects may be stored in the metadata processing side. That is, the metadata processing end completes various processing of the metadata set corresponding to each object.
It should be understood that the number of metadata processing sides in fig. 1 is merely illustrative. There may be any number of metadata processing ports, as desired for implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a data transmission method according to the present disclosure is shown. The data transmission method comprises the following steps:
In some embodiments, in response to receiving a processing request for a target object, an executing body (e.g., the electronic device shown in fig. 1) of the above-described data transmission method may determine whether metadata corresponding to the target data exists in a metadata set corresponding to each object, which is stored in advance. Wherein the target object includes: the target data and the metadata corresponding to the target data. An object is the basic unit of operation of the object store. The Storage protocol of the object in the object Storage may be a Simple Storage Service (S3 Simple Storage Service, S3 Service) protocol. For the simple storage service protocol, the main interface commands include: a write interface command, a get interface command, and a delete interface command. It should be noted that there is often no hierarchical relationship between the objects in the object store, that is, each file or directory is an independent object. For example, for/a/b.jpg and/a/c.jpg, the file systems b.jpg and c.jpg are both files under directory a, but they are independent objects in the object storage system, with no necessary relationship between each other. The processing request of the target object may be a request for processing the target object. The processing steps corresponding to a specific processing request are not limited herein. The metadata may be descriptive information of the data. For example, the data is a photograph of a person. The metadata may include: the name of the person in the person photograph, the sex of the person in the person photograph, the nationality of the person in the person photograph, the age of the person in the person photograph, the location where the person photograph was taken, and the time at which the person photograph was taken. In conventional file storage, metadata and file data are stored encapsulated together. For object storage, metadata is separate and not encapsulated with the corresponding data. Here, the metadata may greatly speed the ordering, sorting, and finding of objects in the object store.
As an example, in response to receiving a processing request for a target object, the execution subject may determine whether metadata corresponding to target data exists in a metadata set corresponding to each object, which is stored in advance, in various target object query manners.
In some embodiments, in response to determining that the metadata corresponding to the target data exists, the execution subject may obtain first storage information characterizing whether the target data exists in the data cache. Wherein, the data cache can be used for storing the hot spot data. The hot spot data may be data with a user usage rate greater than a target threshold. The data cache may be hardware or software. When the data cache is hardware, the data cache can be implemented as a distributed cluster consisting of a plurality of servers or terminal devices, or can be implemented as a single server or a single terminal device.
In the past, the target data is often acquired by querying from the object store. The target data acquisition efficiency is low, and time and labor are consumed due to the fact that the data amount stored in the object storage is large. The data cache is arranged to cache part of hot spot data. When the target data is acquired subsequently, whether the target data exists in part of the hot spot data cached in the data cache is determined firstly. In response to determining that the target data exists, the executing entity need not retrieve the target data from the object store. Therefore, the efficiency of data acquisition can be greatly improved, the network overhead and IO (Input/Output) requests are reduced on the side surface, and the service throughput is improved.
As an example, in response to determining that the metadata corresponding to the target data exists, the execution subject may first send a request for whether the target data exists to the data cache to receive the first storage information sent by the data cache.
In some embodiments, in response to the first storage information indicating that the target data does not exist in the data cache, the execution subject may obtain the target object in an object set stored in an object storage, and send the target data in the target object to the data cache to store the target data.
As an example, in response to the first storage information indicating that the target data does not exist in the data cache, the execution main body may first send a request for obtaining a target object to an object storage. Then, the execution body receives the target object in the stored object set transmitted by the object storage, and transmits the target data in the target object to the data cache to store the target data.
In some optional implementations of some embodiments, the foregoing step further includes:
the first step is that the target data in the data cache is acquired in response to the first storage information representing that the target data exists in the data cache.
As an example, in response to the first storage information indicating that the target data exists in the data cache, the execution main body may obtain the target data in the data cache by sending a target data obtaining request to the data cache.
And secondly, generating the target object according to the target data and the metadata corresponding to the target data.
As an example, the execution subject may combine the target data and the metadata corresponding to the target data according to a predetermined format to generate the target object.
In some optional implementations of some embodiments, in response to receiving object transformation information characterizing an object transformation in the object set, the execution subject may adjust the metadata in the metadata set according to the object transformation information. Wherein the object transformation information may be one of: deletion information of an object, modification information of an object, addition information of an object.
As an example, in response to receiving object transformation information characterizing transformation of an object in the object set and deletion information of the object transformation information as at least one object in the object set, the execution subject may delete at least one metadata corresponding to the at least one object in the metadata set.
As another example, in response to receiving object transformation information characterizing transformation of an object in the object set and modification information indicating that the object transformation information is at least one object in the object set, the execution subject may modify at least one metadata in the metadata set corresponding to the at least one object.
In some alternative implementations of some embodiments, the metadata set is presented in a directory tree, and the metadata set is stored in a distributed storage form. Further, the execution subject can adopt a distributed storage mode to realize data multi-copy and high availability, so as to guarantee the performance and data integrity of the system.
Optionally, the target user may perform various processing on the metadata in the metadata set through a pre-constructed software package and a pre-built metadata processing interface.
The above embodiments of the present disclosure have the following beneficial effects: the data sending method of some embodiments of the disclosure can quickly and efficiently obtain the target object. In addition, the target data is sent to a data cache to enable efficient subsequent invocation of the target data. In particular, the metadata interface is excessively relied on, so that the response speed is slow, and the target object acquisition efficiency is low. In addition, the target data in the target object is acquired from the object storage, which has the problems of time and labor consumption, and affects the acquisition efficiency of the target object to a certain extent. Based on this, the data transmission method of some embodiments of the present disclosure may first determine, in response to receiving a processing request for a target object, whether metadata corresponding to the target data exists in a metadata set corresponding to each object that is stored in advance. Here, the target object does not need to be directly acquired from the object store through the target metadata interface, and the dependence on the target metadata interface can be alleviated to some extent. Wherein the target object includes: the target data and the metadata corresponding to the target data. Then, in response to determining that the metadata corresponding to the target data exists, first storage information representing whether the target data exists in the data cache is acquired. Here, the data cache stores partial data, so that the data acquisition efficiency can be effectively improved. The data cache can cache part of data with higher utilization rate. By setting the data cache, when the target data is acquired, the data is not required to be searched and acquired from the object storage storing a large amount of data in a complicated manner. It may be determined whether the target data exists in the data set stored by the data cache. Wherein the data number of the data set is smaller than the data number of the data set stored in the object store. And finally, responding to the first storage information representing that the target data does not exist in the data cache, acquiring the target objects in an object set stored in an object storage, and sending the target data in the target objects to the data cache to store the target data. Wherein the object set includes the respective objects. By sending the data of the target object to the data cache to store the target data, the target data can be reused efficiently and conveniently.
With further reference to fig. 3, a flow 300 of further embodiments of a data transmission method according to the present disclosure is shown. The data transmission method comprises the following steps:
In some embodiments, the specific implementation of steps 301 and 303 and the technical effect thereof can refer to steps 201 and 203 in the embodiment corresponding to fig. 2, which are not described herein again.
In some embodiments, in response to determining that there is no metadata corresponding to the target data, the execution principal (e.g., the metadata processing side shown in fig. 1) may call a target metadata interface to obtain second storage information characterizing whether there is the target object in the object set.
Step 305, in response to the second storage information indicating that the target object exists in the object set, obtaining metadata corresponding to the target object in the object storage.
In some embodiments, in response to the second storage information indicating that the target object exists in the object set, the execution principal may obtain metadata corresponding to the target object in the object storage.
As an example, in response to the second storage information indicating that the target object exists in the object set, the execution main body may send a metadata request corresponding to the target data to an object storage to obtain metadata corresponding to the target object in the object storage.
In some embodiments, in response to obtaining the metadata corresponding to the target object, the execution subject may add the metadata corresponding to the target object to a metadata set corresponding to each object, so as to store the metadata corresponding to the target object. In addition, the execution main body sends the target data to the data cache so that the data cache stores the target data.
As can be seen from fig. 3, compared with the description of some embodiments corresponding to fig. 2, the flow 300 of the data sending method in some embodiments corresponding to fig. 3 highlights a specific step of storing the metadata corresponding to the target data. Therefore, in the solutions described in the embodiments, by storing the metadata corresponding to the target object and sending the target data to the data cache to store the target data, the acquisition efficiency of subsequent target data can be improved, and the network overhead and the IO request can be reduced.
With further reference to fig. 4, a flow 400 of still further embodiments of a data transmission method according to the present disclosure is shown. The data transmission method comprises the following steps:
in step 401, in response to receiving a processing request for a target object, it is determined whether metadata corresponding to target data exists in a metadata set corresponding to each object, which is stored in advance.
In some embodiments, the specific implementation of steps 401 and 403 and the technical effects thereof can refer to steps 201 and 203 in the embodiment corresponding to fig. 2, which are not described herein again.
And step 404, periodically updating the metadata sets corresponding to the objects according to the metadata sets corresponding to the object sets.
In some embodiments, an execution subject (e.g., the metadata processing end shown in fig. 1) may periodically update the metadata sets corresponding to the respective objects according to the metadata sets corresponding to the object sets. Wherein the cycle time may be preset.
As an example, the execution subject may periodically determine whether at least one metadata that is not present in the metadata set corresponding to each object is present in the metadata set corresponding to the object set. And in response to determining that the metadata exists, adding the at least one piece of metadata to the metadata set corresponding to each object to realize the updating of the metadata set.
As can be seen from fig. 4, compared with the description of some embodiments corresponding to fig. 2, the specific steps of periodically updating the metadata sets corresponding to the above objects are highlighted in the flow 400 of the data sending method in some embodiments corresponding to fig. 4. Therefore, in the solutions described in the embodiments, the metadata sets corresponding to the objects are periodically updated, so that the metadata sets corresponding to the objects can be further improved, and the subsequent data acquisition is more efficient. In addition, the dependence on the target metadata interface can be relieved to a certain extent, and the network overhead and IO requests are reduced.
With further reference to fig. 5, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data transmission apparatus, which correspond to those shown in fig. 2, and which may be applied in various electronic devices in particular.
As shown in fig. 5, a data transmission apparatus 500 includes: a determination unit 501, a first acquisition unit 502 and a second acquisition unit 503. The determining unit 501 is configured to determine, in response to receiving a processing request for a target object, whether metadata corresponding to target data exists in a metadata set corresponding to each object, where the target object includes: the target data and metadata corresponding to the target data; a first obtaining unit 502 configured to obtain, in response to determining that metadata corresponding to the target data exists, first storage information representing whether the target data exists in a data cache; a second obtaining unit 503, configured to, in response to the first storage information indicating that the target data does not exist in the data cache, obtain the target objects in an object set stored in an object storage, and send the target data in the target objects to the data cache to store the target data, wherein the object set includes the respective objects.
In some optional implementations of some embodiments, the apparatus 500 further includes: a calling unit, a third acquiring unit and a storage sending unit (not shown in the figure). Wherein the calling unit may be configured to: and in response to determining that the metadata corresponding to the target data does not exist, calling a target metadata interface to acquire second storage information representing whether the target object exists in the object set. The third acquisition unit may be configured to: and acquiring metadata corresponding to the target object in the object storage in response to the second storage information representing that the target object exists in the object set. The storage transmitting unit may be configured to: and responding to the acquired metadata corresponding to the target object, storing the metadata corresponding to the target object, and sending the target data to the data cache to store the target data.
In some optional implementations of some embodiments, the apparatus 500 further includes: a fourth acquisition unit and a generation unit (not shown in the figure). Wherein the fourth obtaining unit may be configured to: and acquiring the target data in the data cache in response to the first storage information representing that the target data exists in the data cache. The generation unit may be configured to: and generating the target object according to the target data and the metadata corresponding to the target data.
In some optional implementations of some embodiments, the apparatus 500 further includes: an adjustment unit (not shown). Wherein the adjusting unit may be configured to: and in response to receiving object transformation information representing object transformation in the object set, adjusting the metadata in the metadata set according to the object transformation information.
In some optional implementations of some embodiments, the apparatus 500 further includes: an update unit (not shown). Wherein the update unit may be configured to: and periodically updating the metadata sets corresponding to the objects according to the metadata sets corresponding to the object sets.
It will be understood that the elements described in the apparatus 500 correspond to various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 500 and the units included therein, and are not described herein again.
Referring now to fig. 6, a schematic diagram of an electronic device (e.g., the metadata processing side of fig. 1) 600 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 6 may represent one device or may represent multiple devices as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described above in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: in response to receiving a processing request for a target object, determining whether metadata corresponding to target data exists in a metadata set corresponding to each object, wherein the target object comprises: the target data and metadata corresponding to the target data; responding to the metadata corresponding to the target data, and acquiring first storage information representing whether the target data exists in a data cache; responding to the first storage information representing that the target data does not exist in the data cache, acquiring the target objects in an object set stored in an object storage, and sending the target data in the target objects to the data cache to store the target data, wherein the object set comprises the objects.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor comprising: the device comprises a determining unit, a first acquiring unit and a second acquiring unit. Where the names of these units do not constitute a limitation on the units themselves in some cases, for example, the determination unit may also be described as "a unit that determines whether metadata corresponding to target data exists in a metadata set corresponding to each object stored in advance in response to receiving a processing request for the target object".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (10)
1. A method of data transmission, comprising:
in response to receiving a processing request for a target object, determining whether metadata corresponding to target data exists in a metadata set corresponding to each object, the target object including: the target data and metadata corresponding to the target data;
responding to the fact that metadata corresponding to the target data exist, and acquiring first storage information representing whether the target data exist in a data cache or not;
responding to the first storage information representing that the target data does not exist in the data cache, acquiring the target objects in an object set stored in an object storage, and sending the target data in the target objects to the data cache to store the target data, wherein the object set comprises the objects.
2. The method of claim 1, wherein the method further comprises:
in response to determining that no metadata corresponding to the target data exists, calling a target metadata interface to obtain second storage information representing whether the target object exists in the object set;
responding to the second storage information representing that the target object exists in the object set, and acquiring metadata corresponding to the target object in the object storage;
and responding to the acquired metadata corresponding to the target object, storing the metadata corresponding to the target object, and sending the target data to the data cache to store the target data.
3. The method of claim 1, wherein the method further comprises:
responding to the first storage information representing that the target data exists in the data cache, and acquiring the target data in the data cache;
and generating the target object according to the target data and the metadata corresponding to the target data.
4. The method of claim 1, wherein the method further comprises:
in response to receiving object transformation information characterizing an object transformation in the object set, adjusting metadata in the metadata set in accordance with the object transformation information.
5. The method of claim 1, wherein the method further comprises:
and periodically updating the metadata set corresponding to each object according to the metadata set corresponding to the object set.
6. The method of claim 1, wherein the set of metadata is presented in the form of a directory tree and the stored form of the set of metadata is a distributed stored form.
7. A data transmission apparatus comprising:
a determination unit configured to determine, in response to receiving a processing request for a target object including: the target data and metadata corresponding to the target data;
a first obtaining unit configured to obtain, in response to determining that metadata corresponding to the target data exists, first storage information representing whether the target data exists in a data cache;
a second obtaining unit configured to, in response to the first storage information representing that the target data does not exist in the data cache, obtain the target objects in an object set stored in an object storage, and send the target data in the target objects to the data cache to store the target data, wherein the object set includes the respective objects.
8. The apparatus of claim 7, wherein the apparatus further comprises:
in response to determining that no metadata corresponding to the target data exists, calling a target metadata interface to obtain second storage information representing whether the target object exists in the object set;
responding to the second storage information representing that the target object exists in the object set, and acquiring metadata corresponding to the target object in the object storage;
and responding to the acquired metadata corresponding to the target object, storing the metadata corresponding to the target object, and sending the target data to the data cache to store the target data.
9. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110899346.XA CN113342837B (en) | 2021-08-06 | 2021-08-06 | Data transmission method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110899346.XA CN113342837B (en) | 2021-08-06 | 2021-08-06 | Data transmission method, device, electronic equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113342837A CN113342837A (en) | 2021-09-03 |
| CN113342837B true CN113342837B (en) | 2021-11-09 |
Family
ID=77480888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110899346.XA Active CN113342837B (en) | 2021-08-06 | 2021-08-06 | Data transmission method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342837B (en) |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102523285B (en) * | 2011-12-15 | 2014-04-02 | 杭州电子科技大学 | Storage caching method of object-based distributed file system |
| CN103108047A (en) * | 2013-02-06 | 2013-05-15 | 浪潮电子信息产业股份有限公司 | Optimization method for object storage system metadata cache |
| CN103595797B (en) * | 2013-11-18 | 2017-01-18 | 上海爱数信息技术股份有限公司 | Caching method for distributed storage system |
| US10021212B1 (en) * | 2014-12-05 | 2018-07-10 | EMC IP Holding Company LLC | Distributed file systems on content delivery networks |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN107689876A (en) * | 2016-08-04 | 2018-02-13 | 上海德拓信息技术股份有限公司 | The distribution management method of metadata in distributed objects storage system |
| CN107634853A (en) * | 2017-08-28 | 2018-01-26 | 深圳市云舒网络技术有限公司 | A kind of customizable distributed caching method based on object storage |
| US11178246B2 (en) * | 2018-08-25 | 2021-11-16 | Panzura, Llc | Managing cloud-based storage using a time-series database |
| CN110365745B (en) * | 2019-06-21 | 2022-08-05 | 北京奇艺世纪科技有限公司 | Data processing system, method, apparatus, server and computer readable medium |
| CN110764708A (en) * | 2019-10-25 | 2020-02-07 | 北京浪潮数据技术有限公司 | Data reading method, device, equipment and storage medium |
| CN112269534A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Data reading method, device and equipment and computer readable storage medium |
| CN112163176A (en) * | 2020-11-02 | 2021-01-01 | 北京城市网邻信息技术有限公司 | Data storage method and device, electronic equipment and computer readable medium |
| CN112346659B (en) * | 2020-11-05 | 2022-07-29 | 苏州浪潮智能科技有限公司 | Storage method, equipment and storage medium for distributed object storage metadata |
-
2021
- 2021-08-06 CN CN202110899346.XA patent/CN113342837B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113342837A (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11146502B2 (en) | Method and apparatus for allocating resource | |
| CN111581563B (en) | Page response method and device, storage medium and electronic equipment | |
| CN112256733A (en) | Data caching method and device, electronic equipment and computer readable storage medium | |
| CN113760536A (en) | Data caching method and device, electronic equipment and computer readable medium | |
| CN112015816A (en) | Data synchronization method, device, medium and electronic equipment | |
| CN111309366B (en) | Method, device, medium and electronic equipment for managing registration core | |
| CN111262907B (en) | Service instance access method and device and electronic equipment | |
| CN113553206B (en) | Data event execution method and device, electronic equipment and computer readable medium | |
| CN113342837B (en) | Data transmission method, device, electronic equipment and computer readable medium | |
| CN114785770A (en) | Mirror layer file sending method and device, electronic equipment and computer readable medium | |
| CN110740138A (en) | Data transmission method and device | |
| CN112732457B (en) | Image transmission method, image transmission device, electronic equipment and computer readable medium | |
| CN112507676B (en) | Method and device for generating energy report, electronic equipment and computer readable medium | |
| CN112163176A (en) | Data storage method and device, electronic equipment and computer readable medium | |
| CN113760929A (en) | Data synchronization method and device, electronic equipment and computer readable medium | |
| CN114490718A (en) | Data output method, data output device, electronic equipment and computer readable medium | |
| CN112346661A (en) | Data processing method and device and electronic equipment | |
| CN111580890A (en) | Method, apparatus, electronic device, and computer-readable medium for processing features | |
| CN115993942B (en) | Data caching method, device, electronic equipment and computer readable medium | |
| CN112311840A (en) | Multi-terminal data synchronization method, device, equipment and medium | |
| CN117170986B (en) | Chip consistency processing system, method, device, equipment and medium thereof | |
| CN116820354B (en) | Data storage method, data storage device and data storage system | |
| CN116361254B (en) | Image storage method, apparatus, electronic device, and computer-readable medium | |
| CN111258670B (en) | Method and device for managing component data, electronic equipment and storage medium | |
| CN110633324B (en) | Method, apparatus, electronic device and computer readable medium for synchronizing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |