CN112612791A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112612791A CN112612791A CN202011531125.9A CN202011531125A CN112612791A CN 112612791 A CN112612791 A CN 112612791A CN 202011531125 A CN202011531125 A CN 202011531125A CN 112612791 A CN112612791 A CN 112612791A
- Authority
- CN
- China
- Prior art keywords
- target
- sub
- data
- bitmap
- rbd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2237—Vectors, bitmaps or matrices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data processing method and a device, wherein the method is applied to an RBD client, and the method comprises the following steps: when a write request for requesting to write data in a target data object in target RBD block equipment from user equipment is received, if the opened target RBD block equipment starts an object bitmap function, judging whether a target sub-object bitmap corresponding to the target data object needs to be updated or not according to an object bitmap corresponding to the target RBD block equipment stored in a memory; if so, constructing a corresponding updating request, and caching the updating request into a cache region corresponding to the target sub-object bitmap; merging the update requests to be sent in the cache region into a new update request, and sending the new update request to a target OSD for storing a target sub-object bitmap; and when receiving a notification message sent by the target OSD, writing data carried in a write request corresponding to the update request before combination into a corresponding data object. The method and the device can improve the performance of writing data.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a data processing method and apparatus.
Background
Ceph (distributed storage system) is a distributed storage system with excellent performance, high reliability, and high scalability.
An rbd (rados Block device) Block device is one of three storage service components (Block storage, file storage and object storage) externally provided by Ceph.
Currently, for an RBD client, if an object-map function is turned on by an RBD block device to be newly created, the RBD client generates an object-map (which may be denoted as RBD _ object _ map) for all data objects corresponding to the newly created RBD block device; the rbd _ Object _ map is saved to an Object Storage Device (OSD).
Here, the RBD _ object _ map is used to record a status of each data object corresponding to the RBD block device, and the status of each data object can be represented by two bits, for example, 00 indicates that no data object exists, 01 indicates that a data object exists, and the like.
When a plurality of user equipments in Ceph need to write data in the RBD block device, the user equipments all send corresponding write requests to the RBD client, and after the RBD client receives the write requests, if it is determined that the RBD _ object _ map needs to be updated, the RBD client generates an update request for updating the RBD _ object _ map and sends the update request to a corresponding OSD, so that more update requests are accumulated on the OSD, and the RBD client needs to wait for the OSD to process the operation of the related write data after the update request is processed, thereby seriously affecting the performance of the write data.
Disclosure of Invention
In order to overcome the problems in the related art, the application provides a data processing method and a data processing device.
According to a first aspect of embodiments of the present application, there is provided a data processing method, where the method is applied to an RBD client, and the method includes:
when a write request for requesting to write data in a target data object in target RBD block equipment from user equipment is received, if the opened target RBD block equipment starts an object bitmap function, judging whether a target sub-object bitmap corresponding to the target data object needs to be updated or not according to an object bitmap corresponding to the target RBD block equipment stored in a memory, wherein the content of the object bitmap comprises the content of a plurality of sub-object bitmaps, the content of all sub-object bitmaps comprises the state of all data objects in the target RBD block equipment, the plurality of data objects corresponding to different sub-object bitmaps are different, and OSD (on screen display) corresponding to all sub-object bitmaps are not completely the same;
if so, constructing an update request for updating the state of the target data object in the target sub-object bitmap, and caching the update request into a cache region corresponding to the target sub-object bitmap;
merging the update requests to be sent in the cache region into a new update request, and sending the new update request to a target OSD for storing the target sub-object bitmap;
and when receiving a notification message which is sent by the target OSD and used for notifying that the state which is indicated by the new updating request and needs to be updated is updated, writing data carried in the writing request corresponding to the updating request before combination into a corresponding data object.
According to a second aspect of embodiments of the present application, there is provided a data processing apparatus, which is applied to an RBD client, the apparatus including:
a determining module, configured to, when a write request for requesting to write data in a target data object in a target RBD block device is received from a user equipment, if an object bitmap function is turned on by the turned-on target RBD block device, determine whether a target sub-object bitmap corresponding to the target data object needs to be updated according to an object bitmap corresponding to the target RBD block device stored in a memory, where content of the object bitmap includes content of a plurality of sub-object bitmaps, content of all sub-object bitmaps includes states of all data objects in the target RBD block device, where a plurality of data objects corresponding to different sub-object bitmaps are different, and OSDs corresponding to all sub-object bitmaps are not completely the same;
the construction module is used for constructing an updating request for updating the state of the target data object in the target sub-object bitmap and caching the updating request into a cache region corresponding to the target sub-object bitmap when the judgment result of the judgment module is yes;
a sending module, configured to combine the update requests to be sent in the cache region into a new update request, and send the new update request to a target OSD storing the target sub-object bitmap;
and the data writing module is used for writing the data carried in the writing request corresponding to the updating request before combination into the corresponding data object when receiving a notification message which is sent by the target OSD and used for notifying that the state which is indicated by the new updating request and needs to be updated is updated.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
in the embodiment of the application, when receiving a write request of user equipment for requesting to write data in a target data object in target RBD block equipment, and under the condition that the opened target RBD block equipment starts an object bitmap function, if a target sub-object bitmap corresponding to the target data object needs to be updated, an RBD client side can construct a corresponding update request and firstly caches the update request in a cache region corresponding to the target sub-object bitmap; then, the RBD client side combines the update requests to be sent in the cache region into a new update request, sends the new update request to a target OSD storing a target sub-object bitmap, and updates the state of a corresponding data object by the target OSD according to the new update request; finally, when receiving a notification message sent by the target OSD for notifying that the state required to be updated indicated by the new update request is updated, the RBD client writes the data carried in the write request corresponding to the update request before merging into the corresponding data object.
In the data processing flow, the object bitmap corresponding to the RBD block device with the object bitmap function being turned on is split into a plurality of sub-object bitmaps, each sub-object bitmap is used to record the state of a part of data objects in the RBD block device, and the OSD in which the sub-object bitmaps are stored is not completely the same, so that when receiving write requests from different user devices for writing data in a certain data object in the RBD block device, the RBD client can disperse the relevant update requests to different OSDs for processing, and can also perform merge processing on the update requests for updating the same sub-object bitmap, so as to reduce the queuing processing time on the corresponding OSD side, thereby greatly improving the performance of writing data.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is an interaction diagram of a ceph user device writing data with an RBD client according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
Next, examples of the present application will be described in detail.
An embodiment of the present application provides a data processing method, which is applied to an RBD client, and as shown in fig. 1, the method may include the following steps:
s11, when a write request for requesting to write data in a target data object in a target RBD block device is received from user equipment, if the opened target RBD block device starts an object bitmap function, judging whether a target sub-object bitmap corresponding to the target data object needs to be updated according to an object bitmap corresponding to the target RBD block device stored in a memory; if the determination result is no, step S12 is executed; if the determination result is yes, step S13 is executed.
The content of the object bitmap comprises the content of a plurality of sub-object bitmaps, the content of all the sub-object bitmaps comprises the states of all data objects in the target RBD block device, a plurality of data objects corresponding to different sub-object bitmaps are different, and OSD corresponding to all the sub-object bitmaps are not completely the same.
Specifically, in this step, the RBD client may determine whether to update the sub-object bitmap corresponding to the target data object by:
judging whether the state corresponding to the target data object in the object bitmap is a target state, wherein the target state is that the data object does not exist or the data object exists and a clone data object is not created;
if so, determining a sub-object bitmap corresponding to the target data object to be updated;
and when the judgment result is negative, determining that the sub-object bitmap corresponding to the target data object does not need to be updated.
It should be noted that the target data object in this step is determined by the RBD client according to the address offset information carried in the write request.
And S12, writing the data carried in the writing request into the target data object.
In this step, the specific process of writing data is prior art and will not be described in detail here.
S13, constructing an update request for updating the state of the target data object in the target sub-object bitmap, and caching the update request into a cache region corresponding to the target sub-object bitmap.
And S14, merging the update requests to be sent in the buffer into a new update request, and sending the update request before merging to the target OSD storing the target sub-object bitmap.
It should be noted that after the RBD client caches the update request in the buffer corresponding to the target sub-object bitmap, if it is just the time to send the update request in the buffer, and when there is only one update request in the buffer, the RBD client sends the update request as a new update request to the target OSD storing the target sub-object bitmap; when there are multiple update requests (one of which is the update request) in the buffer, the RBD client merges the update requests into a new update request, and sends the new update request to the target OSD storing the target sub-object bitmap.
If the update request in the cache region is not sent, the RBD client can continue to cache until the related processing flow is executed when the update request in the cache region needs to be sent.
In addition, after receiving a new update request, the target OSD updates the state of the data object that needs to be updated and is indicated by the new update request according to the new update request. For example, if the new update request indicates that the status of data object 1 and data object 2 in the target sub-object bitmap are both updated from data object absent to data object present, then the target OSD would update the status of data object 1 and data object 2 in the target sub-object bitmap both from data object absent to data object present.
And S15, when receiving a notification message sent by the target OSD for notifying that the status indicated by the new update request needs to be updated, writing the data carried in the write request corresponding to the update request before combination into the corresponding data object.
In this step, for the write request corresponding to each update request before merging, the OSDs where the data objects corresponding to the carried data are located may all be the same, may all be different, or may be partially the same. The RBD client writes the data carried in the write request corresponding to each update request before combination into the corresponding data object on the OSD where the data object corresponding to the data is stored.
Specifically, in this embodiment of the present application, before opening the target RBD block device, the RBD client may further perform the following operations:
according to a new building instruction input by an administrator and used for building a target RBD block device, building the target RBD block device;
if the target RBD block equipment starts the object bitmap function, grouping each data object in the target RBD block equipment according to a preset grouping rule;
aiming at each group of data objects, generating an identifier of a sub-object bitmap for the group of data objects, constructing a creation instruction for creating the sub-object bitmap corresponding to the identifier, and sending the creation instruction to the determined OSD corresponding to the identifier, so that the OSD corresponding to the identifier creates the sub-object bitmap corresponding to the identifier according to the creation instruction, wherein the sub-object bitmap corresponding to the identifier is used for recording the state of each data object in the group of data objects;
if an opening instruction for opening the target RBD block equipment input by an administrator is received, acquiring the identifications of all sub-object bitmaps corresponding to the target RBD block equipment;
aiming at the identifier of each sub-object bitmap, constructing a reading instruction for reading the content of the sub-object bitmap corresponding to the identifier, and sending the reading instruction to the OSD corresponding to the identifier;
when the content of the corresponding sub-object bitmap sent by the OSD corresponding to the identification of all the sub-object bitmaps is received, merging the received content, wherein the content of each sub-object bitmap is sent by the corresponding OSD after receiving a reading instruction sent by the RBD client;
and taking the merged content as the content of the object bitmap corresponding to the target RBD block equipment, and storing the content in the memory.
In this operation flow, the RBD client may group the respective data objects in the target RBD block device by:
and grouping the data objects in the target RBD block equipment according to a grouping rule that a set number of data objects are continuously arranged.
For example, each data object in the target RBD block device has a size of 4MB, and the RBD client can group the data objects in the target RBD block device into a group of 65536 consecutive data objects. Of course, the RBD clients may also be grouped in other grouping manners, which are not listed here.
In addition, in this operation flow, for any group of data objects, the identifier of the sub-object bitmap generated by the RBD client for the group of data objects may also be referred to as the name of the sub-object bitmap, for example, written as RBD _ object _ map.
After the RBD client generates an identifier of a sub-object bitmap for the group of data objects, an OSD is determined according to an existing manner, for example, by using a preset hash algorithm, and the OSD is responsible for creating the sub-object bitmap corresponding to the identifier and managing the sub-object bitmap. Here, the process of creating the sub-object bitmap corresponding to the identifier by the OSD is the prior art and is not described in detail herein.
It should be noted that the OSDs determined by the RBD client for all the sub-object bitmaps are not exactly the same, so that the update requests created according to the received write request can be dispersed to different OSDs for processing in the following process, so as to improve the performance of writing data.
Further, in this embodiment of the present application, the RBD client may further perform the following operations:
and after receiving a notification message which is sent by the OSD and used for notifying that the state which is indicated by the new updating request and needs to be updated is updated, updating the state of the data object corresponding to the state which needs to be updated in the object bitmap.
It should be noted that, in this embodiment of the present application, the content of each sub-object bitmap corresponding to the newly-created target RBD block device is empty, the content of each sub-object bitmap is also empty when the target RBD block device is initially opened, and as data is continuously written into the data object in the target RBD block device, the content of each sub-object bitmap corresponding to the opened target RBD block device and the content of an object bitmap corresponding to the target RBD block device in the memory change, and when the content of the sub-object bitmap corresponding to the target RBD block device is updated, the RBD client will update the object bitmap corresponding to the target RBD block device synchronously, so that the content of each sub-object bitmap corresponding to the opened target RBD block device and the content of the object bitmap corresponding to the target RBD block device in the memory are identical.
The data processing method will be described in detail with reference to specific embodiments.
As shown in fig. 2, assume that RBD client 1 in Ceph receives write request 1, write request 2, write request 3, and write request 4. Wherein, the write request 1 is a write request sent by the user equipment 1 for writing data in the data object 1 in the RBD block device 1; the write request 2 is a write request sent by the user equipment 2 for writing data in the data object 2 in the RBD block device 1; the write request 3 is a write request sent by the user equipment 3 for writing data in the data object 3 in the RBD block device 1; the write request 4 is a write request sent by the user equipment 4 for writing data in the data object 4 in the RBD block device 1.
Assume that the opened RBD block device 1 has the object bitmap function turned on, and assume that the states of the data object 1, the data object 2, the data object 3, and the data object 4 are all object-nonexistent.
For the write request 1, because the RBD block device 1 opens the object bitmap function, and the state corresponding to the data object 1 in the memory is that the object does not exist, the sub-object bitmap (e.g., the sub-object bitmap 1) corresponding to the data object 1 needs to be updated, at this time, the RBD client 1 constructs the update request 1, and caches the update request 1 in the cache region (e.g., the cache region 1) corresponding to the sub-object bitmap 1, and assuming that it is not time to send the update request in the cache region 1 at this time, the RBD client 1 does not perform any operation on the cache region 1.
For the write request 2, since the RBD block device 1 opens the object bitmap function, and the state corresponding to the data object 2 in the memory is that the object does not exist, the sub-object bitmap (e.g., the sub-object bitmap 1) corresponding to the data object 2 needs to be updated, at this time, the RBD client 1 constructs the update request 2, and caches the update request 2 in the buffer area (i.e., the buffer area 1) corresponding to the sub-object bitmap 1, and assuming that it is right at this time to send the update request in the buffer area 1, the RBD client 1 merges the update request 1 and the update request 2 in the buffer area 1 to obtain the update request 3, and sends the update request 3 to the OSD (e.g., the OSD1 shown in fig. 2) corresponding to the sub-object bitmap 1.
After receiving the update request 3, the OSD1 updates the states of the data object 1 and the data object 2 in the sub-object bitmap 1 from object absence to object presence according to the update request 3, and sends a notification message indicating that the update request 3 has been updated and indicating that the update is required to the RBD client 1 after the update is successful.
After receiving the notification message, the RBD client 1 updates the states of the data object 1 and the data object 2 in the object bitmap corresponding to the RBD block device 1 from object absence to object presence, writes the data carried in the write request 1 corresponding to the update request 1 into the newly created data object 1, and writes the data carried in the write request 2 corresponding to the update request 2 into the newly created data object 2.
Suppose that data object 3 and data object 4 correspond to the same sub-object bitmap, and the OSD corresponding to the sub-object bitmap is OSD2, and suppose that for write request 3, the situation when RBD client 1 caches update request 4 constructed is the same as the situation when update request 1 is cached; for the write request 4, the situation when the RBD client 1 caches the constructed update request 4 is the same as the situation when the update request 2 is cached, and then the subsequent processing flow is similar to the related processing flow described above, and detailed description thereof is omitted.
According to the technical scheme, in the embodiment of the application, when receiving a write request of user equipment for requesting to write data in a target data object in target RBD block equipment, and under the condition that the opened target RBD block equipment starts an object bitmap function, if a target sub-object bitmap corresponding to the target data object needs to be updated, an RBD client builds a corresponding update request and caches the corresponding update request in a cache region corresponding to the target sub-object bitmap; then, the RBD client side combines the update requests to be sent in the cache region into a new update request, sends the new update request to a target OSD storing a target sub-object bitmap, and updates the state of a corresponding data object by the target OSD according to the new update request; finally, when receiving a notification message sent by the target OSD for notifying that the state required to be updated indicated by the new update request is updated, the RBD client writes the data carried in the write request corresponding to the update request before merging into the corresponding data object.
In the data processing flow, the object bitmap corresponding to the RBD block device with the object bitmap function being turned on is split into a plurality of sub-object bitmaps, each sub-object bitmap is used to record the state of a part of data objects in the RBD block device, and the OSD in which the sub-object bitmaps are stored is not completely the same, so that when receiving write requests from different user devices for writing data in a certain data object in the RBD block device, the RBD client can disperse the relevant update requests to different OSDs for processing, and can also perform merge processing on the update requests for updating the same sub-object bitmap, so as to reduce the queuing processing time on the corresponding OSD side, thereby greatly improving the performance of writing data.
Based on the same inventive concept, the present application further provides a data processing apparatus, where the apparatus is applied to an RBD client, and a schematic structural diagram of the apparatus is shown in fig. 3, and specifically includes:
a determining module 31, configured to, when a write request for requesting to write data in a target data object in a target RBD block device is received from a user equipment, if an object bitmap function is turned on by the turned-on target RBD block device, determine whether a target sub-object bitmap corresponding to the target data object needs to be updated according to an object bitmap corresponding to the target RBD block device stored in a memory, where content of the object bitmap includes content of multiple sub-object bitmaps, content of all sub-object bitmaps includes states of all data objects in the target RBD block device, where multiple data objects corresponding to different sub-object bitmaps are different, and OSDs corresponding to all sub-object bitmaps are not completely the same;
a constructing module 32, configured to construct, when the determination result of the determining module 31 is yes, an update request for updating the state of the target data object in the target sub-object bitmap, and cache the update request in a cache region corresponding to the target sub-object bitmap;
a sending module 33, configured to combine the update requests to be sent in the cache region into a new update request, and send the new update request to the target OSD storing the target sub-object bitmap;
and a data writing module 34, configured to write, when receiving a notification message sent by the target OSD and used to notify that the state required to be updated indicated by the new update request is updated, data carried in a write request corresponding to the update request before being combined into a corresponding data object.
Preferably, the determining module 31 is specifically configured to:
judging whether the state corresponding to the target data object in the object bitmap is a target state, wherein the target state is that the data object does not exist or the data object exists and a clone data object is not created;
if so, determining that the sub-object bitmap corresponding to the target data object needs to be updated;
and if the judgment result is negative, determining that the sub-object bitmap corresponding to the target data object does not need to be updated.
Preferably, the apparatus further comprises a new creation module (not shown in fig. 3) and an opening module (not shown in fig. 3);
the new building module is used for executing the following operations:
before the RBD client opens the target RBD block device, according to a new-establishing instruction input by an administrator for establishing the target RBD block device, establishing the target RBD block device;
if the target RBD block equipment starts an object bitmap function, grouping each data object in the target RBD block equipment according to a preset grouping rule;
aiming at each group of data objects, generating an identifier of a sub-object bitmap for the group of data objects, constructing a creation instruction for creating the sub-object bitmap corresponding to the identifier, and sending the creation instruction to the determined OSD corresponding to the identifier, so that the OSD corresponding to the identifier creates the sub-object bitmap corresponding to the identifier according to the creation instruction;
the opening module is used for executing the following operations:
if an opening instruction for opening the target RBD block equipment input by an administrator is received and the target RBD block equipment opens an object bitmap function, acquiring identifiers of all sub-object bitmaps corresponding to the target RBD block equipment;
aiming at the identifier of each sub-object bitmap, constructing a reading instruction for reading the content of the sub-object bitmap corresponding to the identifier, and sending the reading instruction to the determined OSD corresponding to the identifier;
when the content of the corresponding sub-object bitmap sent by the OSD corresponding to the identification of all the sub-object bitmaps is received, merging the received content, wherein the content of each sub-object bitmap is sent by the corresponding OSD after receiving a reading instruction sent by the RBD client;
and taking the merged content as the content of the object bitmap corresponding to the target RBD block equipment, and storing the content in the memory.
Preferably, the newly-built module is specifically configured to:
and grouping the data objects in the target RBD block equipment according to a grouping rule that a set number of data objects are continuously arranged.
Preferably, the apparatus further comprises:
an updating module (not shown in fig. 3) configured to update the state of the data object corresponding to the state that needs to be updated in the object bitmap after receiving a notification message sent by the target OSD and used to notify that the state that needs to be updated indicated by the new update request has been updated.
According to the technical scheme, in the embodiment of the application, when receiving a write request of user equipment for requesting to write data in a target data object in target RBD block equipment, and under the condition that the opened target RBD block equipment starts an object bitmap function, if a target sub-object bitmap corresponding to the target data object needs to be updated, an RBD client builds a corresponding update request and caches the corresponding update request in a cache region corresponding to the target sub-object bitmap; then, the RBD client side combines the update requests to be sent in the cache region into a new update request, sends the new update request to a target OSD storing a target sub-object bitmap, and updates the state of a corresponding data object by the target OSD according to the new update request; finally, when receiving a notification message sent by the target OSD for notifying that the state required to be updated indicated by the new update request is updated, the RBD client writes the data carried in the write request corresponding to the update request before merging into the corresponding data object.
In the data processing flow, the object bitmap corresponding to the RBD block device with the object bitmap function being turned on is split into a plurality of sub-object bitmaps, each sub-object bitmap is used to record the state of a part of data objects in the RBD block device, and the OSD in which the sub-object bitmaps are stored is not completely the same, so that when the RBD client receives a write request from different user devices at the same time for writing data in a certain data object in the RBD block device, the related update requests can be distributed to different OSDs for processing, and the update requests that need to update the same sub-object bitmap can be combined for reducing the queuing time at the corresponding OSD side, thereby greatly improving the performance of writing data.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (10)
1. A data processing method is applied to an RBD client, and comprises the following steps:
when a write request for requesting to write data in a target data object in target RBD block equipment from user equipment is received, if the opened target RBD block equipment starts an object bitmap function, judging whether a target sub-object bitmap corresponding to the target data object needs to be updated or not according to an object bitmap corresponding to the target RBD block equipment stored in a memory, wherein the content of the object bitmap comprises the content of a plurality of sub-object bitmaps, the content of all sub-object bitmaps comprises the state of all data objects in the target RBD block equipment, the plurality of data objects corresponding to different sub-object bitmaps are different, and OSD (on screen display) of object storage equipment corresponding to all sub-object bitmaps are not completely the same;
if so, constructing an update request for updating the state of the target data object in the target sub-object bitmap, and caching the update request into a cache region corresponding to the target sub-object bitmap;
merging the update requests to be sent in the cache region into a new update request, and sending the new update request to a target OSD for storing the target sub-object bitmap;
and when receiving a notification message which is sent by the target OSD and used for notifying that the state which is indicated by the new updating request and needs to be updated is updated, writing data carried in the writing request corresponding to the updating request before combination into a corresponding data object.
2. The method of claim 1, wherein whether the sub-object bitmap corresponding to the target data object needs to be updated is determined by:
judging whether the state corresponding to the target data object in the object bitmap is a target state, wherein the target state is that the data object does not exist or the data object exists and a clone data object is not created;
if so, determining that the sub-object bitmap corresponding to the target data object needs to be updated;
and if the judgment result is negative, determining that the sub-object bitmap corresponding to the target data object does not need to be updated.
3. The method of claim 1, wherein prior to opening the target RBD block device, the method further comprises:
according to a new building instruction input by an administrator and used for building the target RBD block device, building the target RBD block device;
if the target RBD block equipment starts an object bitmap function, grouping each data object in the target RBD block equipment according to a preset grouping rule;
aiming at each group of data objects, generating an identifier of a sub-object bitmap for the group of data objects, constructing a creation instruction for creating the sub-object bitmap corresponding to the identifier, and sending the creation instruction to the determined OSD corresponding to the identifier, so that the OSD corresponding to the identifier creates the sub-object bitmap corresponding to the identifier according to the creation instruction;
if an opening instruction for opening the target RBD block device and an object bitmap function of the target RBD block device are started, which are input by the administrator, the identifiers of all sub-object bitmaps corresponding to the target RBD block device are obtained;
aiming at the identifier of each sub-object bitmap, constructing a reading instruction for reading the content of the sub-object bitmap corresponding to the identifier, and sending the reading instruction to the determined OSD corresponding to the identifier;
when the content of the corresponding sub-object bitmap sent by the OSD corresponding to the identification of all the sub-object bitmaps is received, merging the received content, wherein the content of each sub-object bitmap is sent by the corresponding OSD after receiving a reading instruction sent by the RBD client;
and taking the merged content as the content of the object bitmap corresponding to the target RBD block equipment, and storing the content in the memory.
4. The method according to claim 3, wherein grouping each data object in the target RBD block device according to a preset grouping rule specifically comprises:
and grouping the data objects in the target RBD block equipment according to a grouping rule that a set number of data objects are continuously arranged.
5. The method of claim 1, further comprising:
and after receiving a notification message sent by the target OSD and used for notifying that the state required to be updated indicated by the new update request is updated, updating the state of the data object corresponding to the state required to be updated in the object bitmap.
6. A data processing apparatus, wherein the apparatus is applied to an RBD client, and the apparatus comprises:
a determining module, configured to, when a write request for requesting to write data in a target data object in a target RBD block device is received from a user equipment, if an object bitmap function is turned on by the turned-on target RBD block device, determine whether a target sub-object bitmap corresponding to the target data object needs to be updated according to an object bitmap corresponding to the target RBD block device stored in a memory, where content of the object bitmap includes content of a plurality of sub-object bitmaps, content of all sub-object bitmaps includes states of all data objects in the target RBD block device, where a plurality of data objects corresponding to different sub-object bitmaps are different, and object storage devices OSD corresponding to all sub-object bitmaps are not completely the same;
the construction module is used for constructing an updating request for updating the state of the target data object in the target sub-object bitmap and caching the updating request into a cache region corresponding to the target sub-object bitmap when the judgment result of the judgment module is yes;
a sending module, configured to combine the update requests to be sent in the cache region into a new update request, and send the new update request to a target OSD storing the target sub-object bitmap;
and the data writing module is used for writing the data carried in the writing request corresponding to the updating request before combination into the corresponding data object when receiving a notification message which is sent by the target OSD and used for notifying that the state which is indicated by the new updating request and needs to be updated is updated.
7. The apparatus of claim 6, wherein the determining module is specifically configured to:
judging whether the state corresponding to the target data object in the object bitmap is a target state, wherein the target state is that the data object does not exist or the data object exists and a clone data object is not created;
if so, determining that the sub-object bitmap corresponding to the target data object needs to be updated;
and if the judgment result is negative, determining that the sub-object bitmap corresponding to the target data object does not need to be updated.
8. The apparatus of claim 6, further comprising a new module and an open module;
the new building module is used for executing the following operations:
before the RBD client opens the target RBD block device, according to a new-establishing instruction input by an administrator for establishing the target RBD block device, establishing the target RBD block device;
if the target RBD block equipment starts an object bitmap function, grouping each data object in the target RBD block equipment according to a preset grouping rule;
aiming at each group of data objects, generating an identifier of a sub-object bitmap for the group of data objects, constructing a creation instruction for creating the sub-object bitmap corresponding to the identifier, and sending the creation instruction to the determined OSD corresponding to the identifier, so that the OSD corresponding to the identifier creates the sub-object bitmap corresponding to the identifier according to the creation instruction;
the opening module is used for executing the following operations:
if an opening instruction for opening the target RBD block equipment, which is input by an administrator, is received, acquiring the identifications of all sub-object bitmaps corresponding to the target RBD block equipment;
aiming at the identifier of each sub-object bitmap, constructing a reading instruction for reading the content of the sub-object bitmap corresponding to the identifier, and sending the reading instruction to the determined OSD corresponding to the identifier;
when the content of the corresponding sub-object bitmap sent by the OSD corresponding to the identification of all the sub-object bitmaps is received, merging the received content, wherein the content of each sub-object bitmap is sent by the corresponding OSD after receiving a reading instruction sent by the RBD client;
and taking the merged content as the content of the object bitmap corresponding to the target RBD block equipment, and storing the content in the memory.
9. The apparatus of claim 8, wherein the newly created module is specifically configured to:
and grouping the data objects in the target RBD block equipment according to a grouping rule that a set number of data objects are continuously arranged.
10. The apparatus of claim 6, further comprising:
and the updating module is used for updating the state of the data object corresponding to the state needing to be updated in the object bitmap after receiving a notification message which is sent by the target OSD and used for notifying that the state needing to be updated and indicated by the new updating request is updated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011531125.9A CN112612791B (en) | 2020-12-22 | 2020-12-22 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011531125.9A CN112612791B (en) | 2020-12-22 | 2020-12-22 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112612791A true CN112612791A (en) | 2021-04-06 |
| CN112612791B CN112612791B (en) | 2022-05-27 |
Family
ID=75245408
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011531125.9A Active CN112612791B (en) | 2020-12-22 | 2020-12-22 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112612791B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113253925A (en) * | 2021-04-30 | 2021-08-13 | 新华三大数据技术有限公司 | Method and device for optimizing read-write performance |
| CN113467721A (en) * | 2021-07-22 | 2021-10-01 | 杭州海康威视数字技术股份有限公司 | Data deleting system, method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11103429A (en) * | 1997-09-29 | 1999-04-13 | Sony Corp | Image data decoding device and on-screen display data updating method |
| US6462746B1 (en) * | 1998-04-23 | 2002-10-08 | Le Electronics Inc. | On screen display memory structure in a digital video display unit and method therefor |
| US20100094847A1 (en) * | 2008-10-10 | 2010-04-15 | Malan Steven J | Method and apparatus for multiple-protocol access to object-based storage |
| CN105302668A (en) * | 2015-10-21 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Method for optimizing block device data access performance in object-based storage system |
| US10515063B1 (en) * | 2016-12-19 | 2019-12-24 | Cboe Exchange, Inc. | System and method for real-time data acquisition and display |
| CN111580932A (en) * | 2020-05-12 | 2020-08-25 | 西安交通大学 | Virtual machine disk online migration redundancy removing method |
| CN111858402A (en) * | 2020-06-12 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Read-write data processing method and system based on cache |
| CN112052218A (en) * | 2020-09-25 | 2020-12-08 | 杭州宏杉科技股份有限公司 | Snapshot implementation method and distributed storage cluster |
-
2020
- 2020-12-22 CN CN202011531125.9A patent/CN112612791B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH11103429A (en) * | 1997-09-29 | 1999-04-13 | Sony Corp | Image data decoding device and on-screen display data updating method |
| US6462746B1 (en) * | 1998-04-23 | 2002-10-08 | Le Electronics Inc. | On screen display memory structure in a digital video display unit and method therefor |
| US20100094847A1 (en) * | 2008-10-10 | 2010-04-15 | Malan Steven J | Method and apparatus for multiple-protocol access to object-based storage |
| CN105302668A (en) * | 2015-10-21 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Method for optimizing block device data access performance in object-based storage system |
| US10515063B1 (en) * | 2016-12-19 | 2019-12-24 | Cboe Exchange, Inc. | System and method for real-time data acquisition and display |
| CN111580932A (en) * | 2020-05-12 | 2020-08-25 | 西安交通大学 | Virtual machine disk online migration redundancy removing method |
| CN111858402A (en) * | 2020-06-12 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Read-write data processing method and system based on cache |
| CN112052218A (en) * | 2020-09-25 | 2020-12-08 | 杭州宏杉科技股份有限公司 | Snapshot implementation method and distributed storage cluster |
Non-Patent Citations (2)
| Title |
|---|
| YINLIANG YUE ET AL: "DS-MOSD: A Dynamic Selectable Master OSD Model for Large-Scale Object-Based Storage System", 《IEEE》 * |
| 刘榴: "分布式存储系统容错机制的研究与实现", 《中国优秀硕士学位论文全文数据库（电子期刊）（信息科技辑）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113253925A (en) * | 2021-04-30 | 2021-08-13 | 新华三大数据技术有限公司 | Method and device for optimizing read-write performance |
| CN113467721A (en) * | 2021-07-22 | 2021-10-01 | 杭州海康威视数字技术股份有限公司 | Data deleting system, method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112612791B (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109842651B (en) | Uninterrupted service load balancing method and system | |
| CN112612791B (en) | Data processing method and device | |
| US10592464B2 (en) | Methods for enabling direct memory access (DMA) capable devices for remote DMA (RDMA) usage and devices thereof | |
| US9025608B2 (en) | Route switching device and data cashing method thereof | |
| US7406577B2 (en) | Data migration method | |
| CN106790629A (en) | Data synchronization unit and its realize the method for data syn-chronization, client access system | |
| CN107066570A (en) | Data managing method and device | |
| CN104572845B (en) | Document distribution method, device, equipment and system | |
| KR100793349B1 (en) | Multicast forwarding apparatus and control method in system for using PPP multi-link | |
| CN105338078A (en) | Data storage method and device used for storing system | |
| CN111159233B (en) | Distributed caching method, system, computer equipment and storage medium | |
| JPH1078943A (en) | Group event management method and device in distributed computer environment | |
| CN106302827A (en) | Communication means based on distributed system, equipment and distributed system | |
| CN103036948A (en) | Network file processing method and execution node and software as a service (SaaS) platform | |
| CN107979629B (en) | Distributed cache system and data cache method and device thereof | |
| CN106599323B (en) | Method and device for realizing distributed pipeline in distributed file system | |
| US7136969B1 (en) | Using the message fabric to maintain cache coherency of local caches of global memory | |
| CN111857979B (en) | Information management method, system, storage medium and equipment of distributed system | |
| CN110413679B (en) | Database information processing method, device, equipment and readable storage medium | |
| CN109845199B (en) | Merging read requests in a network device architecture | |
| CN113946376B (en) | Load adjustment method and device, electronic equipment and storage medium | |
| CN111831403A (en) | Service processing method and device | |
| CN109814979A (en) | Data load method, device, computer equipment and readable storage medium storing program for executing | |
| JP2012053796A (en) | Information processing system | |
| US10728165B2 (en) | Buffer control apparatus, communication node, and relay apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |