CN112925482B - Data processing method, device, system, electronic equipment and computer storage medium - Google Patents
Data processing method, device, system, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN112925482B CN112925482B CN202110277769.8A CN202110277769A CN112925482B CN 112925482 B CN112925482 B CN 112925482B CN 202110277769 A CN202110277769 A CN 202110277769A CN 112925482 B CN112925482 B CN 112925482B
- Authority
- CN
- China
- Prior art keywords
- target object
- image
- storage node
- target
- candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Processing Or Creating Images (AREA)
- Stored Programmes (AREA)
Abstract
The application discloses a data processing method, a device and a system, electronic equipment and a computer storage medium. The specific implementation scheme comprises the following steps: receiving a storage request sent by a first storage node for updating a base image of a target object; determining a source image of the target object in the second storage node based on the storage request; determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request; determining difference data between a source image of the target object and a base image of the target object; and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
Description
Technical Field
The present application relates to the field of information processing, and in particular, to a data processing method, apparatus, system, electronic device, and computer storage medium.
Background
The cloud image is an image file which is provided by a cloud service manufacturer to a client and used for creating a server, and the file contains cloud products provided by the cloud service manufacturer and customized services required by the client. In order to maintain various cloud-going requirements of different regions and different customers, cloud service manufacturers need to copy and transmit cloud mirror image files among a plurality of physical storage nodes, and the cloud mirror image files often occupy large storage space, so that the copy and transmission efficiency is low, and the daily use requirements cannot be met. Therefore, how to improve the data transmission efficiency of the cloud mirror image file between different physical storage nodes becomes a problem to be solved.
Disclosure of Invention
In order to solve at least one of the above problems in the prior art, embodiments of the present application provide a data processing method, apparatus, system, electronic device, and computer storage medium.
In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a server, and the method includes:
receiving a storage request sent by a first storage node for updating a base image of a target object;
determining a source image of the target object in a second storage node based on the storage request;
determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request;
determining difference data between a source image of the target object and a base image of the target object;
and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a first storage node, and the method includes:
sending a storage request to a server;
receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node;
and merging the basic mirror image of the target object with the difference data to generate the target mirror image of the target object.
In a third aspect, this embodiment provides a data processing method applied to a second storage node, where the method includes:
receiving a transmission instruction sent by a server;
and sending difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node in response to the transmission instruction, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated.
In a fourth aspect, the present embodiment provides a server, including:
a first receiving module, configured to receive a storage request sent by a first storage node for updating a base image of a target object;
a first determining module for determining a source image of the target object in a second storage node based on the storage request;
a second determining module for determining a base image of a target object corresponding to a source image of the target object in the first storage node based on the storage request;
a third determination module to determine difference data between a source image of the target object and a base image of the target object;
the first sending module is used for sending a transmission instruction to the second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
In a fifth aspect, the present embodiment provides a first storage node, including:
the second sending module is used for sending a storage request to the server;
a second receiving module, configured to receive difference data between the source image of the target object and the base image of the target object sent by the second storage node;
and the merging module is used for merging the basic mirror image of the target object with the difference data to generate a target mirror image of the target object.
In a sixth aspect, the present embodiment provides a second storage node, including:
the third receiving module is used for receiving a transmission instruction sent by the server;
and the third sending module is used for responding to the transmission instruction and sending the difference data between the source mirror image of the target object and the base mirror image of the target object to the first storage node, so that the base mirror image of the target object in the first storage node is merged with the difference data, and the target mirror image of the target object is generated.
In a seventh aspect, this embodiment provides a data processing system, including:
a server for receiving a storage request sent by a first storage node for updating a base image of a target object; determining a source image of the target object in a second storage node based on the storage request; determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request; determining difference data between a source image of the target object and a base image of the target object; and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
The first storage node is used for sending a storage request to the server; receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node; and merging the basic mirror image of the target object with the difference data to generate the target mirror image of the target object.
The second storage node is used for receiving a transmission instruction sent by the server; and sending difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node in response to the transmission instruction, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated.
In an eighth aspect, the present embodiment provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform a method provided by any one of the embodiments of the present application.
In a ninth aspect, embodiments of the present application provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method provided by any one of the embodiments of the present application.
One embodiment in the above application has the following advantages or benefits: after receiving a storage request sent by a first storage node, a server determines a source mirror image of a target object and a base mirror image of the target object corresponding to the source mirror image of the target object in a second storage node and the first storage node respectively based on the storage request, and determines difference data between the source mirror image of the target object and the base mirror image of the target object; and then sending a transmission instruction to a second storage node by the server, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object. Therefore, the original full-volume transmission is replaced by incremental transmission, the data transmission quantity among different physical storage nodes is greatly reduced, and the transmission efficiency of the cloud mirror image file among the different physical storage nodes is improved.
Other effects of the above alternatives will be described below with reference to specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a first flowchart illustrating a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a server and storage node relationship according to an embodiment of the present application;
FIG. 3 is a second flowchart illustrating a data processing method according to an embodiment of the present application;
FIG. 4 is a third flowchart illustrating a data processing method according to an embodiment of the present application;
FIG. 5 is a first block diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 6 is a block diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 7 is a block diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 8 is a fourth block diagram of an object processing apparatus according to an embodiment of the present application;
FIG. 9 is a schematic block diagram of an object processing system according to an embodiment of the present application;
fig. 10 is a schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In a first aspect, an embodiment of the present application provides a data processing method applied to a server, and as shown in fig. 1, the method includes:
s101: receiving a storage request sent by a first storage node for updating a base image of a target object;
s102: determining a source image of the target object in the second storage node based on the storage request;
s103: determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request;
s104: determining difference data between a source image of the target object and a base image of the target object;
s105: and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base mirror image of the target object with the difference data to generate the target mirror image of the target object.
In this embodiment, the server is a core computing device required by a cloud service provider to provide cloud services, and the server may perform communication interaction with a plurality of physical storage nodes. The first storage node and the second storage node are both the cloud mirror image rear end, namely physical storage for storing the cloud mirror image. After the cloud mirror image is uploaded, the cloud mirror image is stored in a storage back end connected with the mirror image service in a mirror image file mode, and when a cloud server is created by using the mirror image and a hard disk is started by using a cloud, the computing service and the storage service read the mirror image file in the storage connected with the mirror image service to complete an application action of the cloud service. Different physical storages may be located in different areas, respectively serving the cloud-up demand of users of the corresponding areas.
When the user applies for the cloud service in the area corresponding to the first storage node, step S101 is executed, and the server receives a storage request sent by the first storage node for updating the base image of the target object.
The first storage node may be a target site of a target area corresponding to a user applying for cloud service, the base image of the target object is one of a plurality of existing image files in the first storage node, and the base image of the target object and the target image of the target object have high file similarity. Correspondingly, the target image of the target object refers to an image file which is applied for calling in the first storage node by the user.
For example, when a user applies for a cloud service in a beijing area, a corresponding target image of a target object needs to be called, the server receives a storage request sent by a first storage node located in beijing, and the storage request is used for updating an existing basic image of the target object in the first storage node to the target image of the target object applied for calling by the user.
After receiving the storage request sent by the first storage node, the server executes step S102 to determine the source image of the target object in the second storage node based on the storage request.
The source mirror image of the target object and the target mirror image of the target object which the user applies to call are the same mirror image file, but are stored in different physical storage nodes. And the source image of the target object is stored in the second storage node.
And determining the source mirror image of the target object, wherein mirror image file information such as mirror image ID, mirror image name, operating system type, operating system version, mirror image state and the like of the source mirror image of the target object is determined. Wherein, the image ID may be a character string representing image file information, such as: 0009521a-7738-41cd-8e65-be88df7c 2161; the image name also includes corresponding information of the image file, such as the image name CDS-OS-Windows2016data center-64bit, which indicates that the operating system corresponding to the image file is a Windows system, the version of the operating system is a 2016data center version, the bit number of the operating system is 64 bits, and the like. Other image file information is not explained one by one, and in addition, the base image of the target object and the target image of the target object also contain the image file information.
After determining the source image of the target object in the second storage node based on the storage request, step S103 is performed, and determining a base image of the target object corresponding to the source image of the target object in the first storage node based on the storage request.
Specifically, the first storage node includes a plurality of image files, wherein one of the image files may be the same as or similar to a source image of a target object in the second storage node, and is determined as a base image of the target object, so that when a user applies for calling the target image of the target object at the first storage node, only difference data between the source image and the base image of the target object needs to be transmitted from the second storage node to the first storage node, thereby greatly reducing the transmission amount of data and improving the transmission efficiency.
In one embodiment, the storage request includes a target path for a target image of the target object in the first storage node.
Determining, in the first storage node based on the storage request, a base image of a target object corresponding to a source image of the target object, including:
and taking the candidate mirror image of the target object pointed by the target path as the basic mirror image of the target object under the condition that the target path exists in the first storage node.
Specifically, when the user applies for the cloud service under the target path in the first storage node, the storage request sent by the first storage node and received by the server may include the target path of the target image of the target object in the first storage node. At this time, the server preferentially judges whether a target path exists in the first storage node, when the target path already exists in the first storage node, it is described that the user has applied for invoking the same cloud service in the first storage node before this, and correspondingly, the candidate mirror image pointed by the target path is just a mirror image file generated by the user applying for the same cloud service last time, and since most of user data in files formed in time sequence is the same, the candidate mirror image pointed by the target path is taken as a basic mirror image of the target object.
In one embodiment, in the case that the target path does not exist in the first storage node, calculating a file similarity between a source image of the target object and an ith candidate image of the N candidate images in the first storage node;
and taking the candidate mirror image corresponding to the maximum value of the file similarity as the basic mirror image of the target object.
Specifically, when the target path does not exist in the first storage node, it indicates that the user has not applied for invoking the same cloud service in the first storage node before that, at this time, one of the N candidate images in the first storage node needs to be selected as the base image of the target object. Further, the server may construct an image file management system to perform similarity calculation between image files according to the storage request sent by the first storage node and the data information associated with the image files in the storage request. The image file management system may be in the form of a database or a data table, which is not limited herein.
When the image file management system takes the form of a database, it may include a plurality of fields relating to the image file base information, such as image ID, image name, operating system type, operating system version, operating system number, image size, user ID, image file status, virtualization platform type, etc.
The mirror image file management system may further include a plurality of fields related to mirror image site relationship information, such as storage node ID, a region to which the storage node belongs, information related to the original storage node, and the like, which are not exhaustive here.
Through the mirror image file management system, the server can perform data summarization, data processing, data calculation and the like on the stored mirror image file data information. Specifically, the mirror image file management system may perform similarity calculation on data information of N candidate mirror images stored in the first storage node and a source mirror image of a target object in the second storage node, respectively, to obtain N file similarity values, then perform descending order on the N file similarity values, and take the candidate mirror image corresponding to the maximum value of the file similarity as a base mirror image of the target object. The N candidate images may be all image files in the first storage node, or may be partial image files under a specific path in the first storage node, which is not limited herein. The value of N may be any positive integer, such as 1,5,10, etc., and is not limited herein.
The server can generate corresponding similarity data cache information according to the similarity calculation result, and can calculate the file similarity between the source mirror image and the target object of the corresponding target object and correspondingly update the file similarity according to the need on the basis of the cache information without performing repeated calculation each time a mirror image transmission instruction is received.
In one embodiment, the storage request further includes a name of a source image of the target object;
the calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node comprises:
calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images based on the name of the source image of the target object and the name of the ith candidate image of the N candidate images in the first storage node.
Specifically, the server receives a storage request sent by the first storage node, and may include, in addition to the target path, a file name of a source image of the target object. Because the name of the mirror image file contains information such as the type of the operating system, the version of the operating system, the number of bits of the operating system and the like, after the server receives the storage request, the mirror image file management system can perform similarity calculation based on the file name of the source image of the target object and the file names of the N candidate images in the first storage node, and when the types of the operating system, the version of the system, the number of bits of the system and the like of the operating system of the two mirror image files are closer, the similarity between the two mirror image files is higher. For example, the name of the source image of the target object included in the storage request is CDS-OS-Windows2016data center-64bit-WE9LCDBO, which indicates that the operating system corresponding to the image file is a 64-bit Windows2016data center version, 3 candidate images are selected from the first storage node to perform similarity calculation, and the file information of the 3 candidate images is specifically shown in table 1, where the file name of the candidate image 01 is CDS-OS-Windows2016data center-64bit-7c518, which indicates that the operating system corresponding to the candidate image 01 is also a 64-bit Windows2016data center version, and the file similarity between the operating system corresponding to the candidate image 01 and the source image of the target object is 70%; the file name of the candidate image 02 is CDS-OS-Windows2012-Standard-64bit-1f497b, which indicates that the operating system corresponding to the candidate image 02 is 64-bit Windows2012 Standard version, and the file similarity between the candidate image 02 and the source image of the target object is 30%; the file name of the candidate image 03 is CDS-OS-CentOS7.2-32bit-V543d6b8, which indicates that the operating system corresponding to the candidate image 03 is 32-bit CentOS version 7.2, and the file similarity between the candidate image and the source image of the target object is 0. And taking the candidate mirror image 01 as a basic mirror image of the target object according to the similarity values of the 3 candidate mirror images.
TABLE 1
| Candidate mirror ID | Candidate mirror name | Degree of similarity |
| 01 | CDS-OS-Windows2016Datacenter-64bit-7c518 | 70％ |
| 02 | CDS-OS-Windows2012-Standard-64bit-1f497b | 30％ |
| 03 | CDS-OS-CentOS7.2-32bit-V543d6b8 | 0 |
In one embodiment, the calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node includes:
calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images based on the data information of the source image of the target object and the data information of the ith candidate image of the N candidate images in the first storage node.
In one embodiment, the data information includes system data information and user data information.
Specifically, the similarity of the files between the two images is calculated based on the data information, and the similarity calculation may be performed based on the system data information or based on the user data information.
The file similarity between the two images is calculated based on the system data information, and the file similarity can be obtained by further performing logic calculation after the system information of the source image and the basic image of the target object is acquired by the image file management system. The content of the system information is referred to the above example, and is not described herein again.
The file similarity between the two mirror images is calculated based on the user data information, and the similarity calculation can be performed on the mirror image files of the same target object customized by the user in sequence through a mirror image file management system. For the image files of the same target object customized by the user in sequence, except for the user data change generated between two moments, the other data contents are consistent, wherein the closer the time interval between the two moments is, the higher the similarity between the two corresponding image files is. For example, the creation time of the source image of the target object included in the storage request sent by the first storage node is "2020/11/0723: 04: 19", the image file management system locates candidate image files of 3 identical objects in the second storage node according to the association information of the image file of the target object, and the related data information is shown in table 2, wherein the modification time of the candidate image 04 is closest to the creation time of the source image of the target object, the similarity between the candidate image 04 and the source image of the target object is calculated to be 92% (maximum), and the candidate image 04 is determined to be the base image of the target object. In this way, the data volume of the difference data transmitted from the first storage node to the second storage node can be reduced to the maximum extent.
TABLE 2
And calculating the file similarity between the two mirror images based on the data information, and further performing weighted summation to obtain the final file similarity between the two mirror image files after obtaining the system data similarity and the user data similarity. The setting of the weight may refer to the respective corresponding proportions of the system data and the user data in the image file of the target object.
In one embodiment, when the file similarity between the source image of the target object and the ith candidate image of the N candidate images is smaller than a first threshold value, a new image is established in the first storage node as a base image of the target object;
alternatively, the first and second liquid crystal display panels may be,
and under the condition that the file similarity between the source mirror image of the target object and the ith candidate mirror image in the N candidate mirror images is smaller than a first threshold value, taking the specified mirror image in the first storage node as the basic mirror image of the target object.
Specifically, when the similarity calculated based on the name of the candidate mirror image and based on the data information of the candidate mirror image is smaller than a first threshold, it means that none of the candidate mirror images in the first storage node determined by the mirror image file management system meets the requirement of the basic mirror image of the target object. At this time, a new image file may be created in the first storage node as the base image of the target object. And the data content of the new mirror image file is null. An image file may also be designated in the first storage node as the base image of the target object.
The first threshold may be preset as needed, and the higher the preset first threshold is, the smaller the number of the basic images of the target object obtained by screening is, but the higher the accuracy of the screening is, specifically, the first threshold may be set to 30%, 20%, or 10%, and is not limited specifically.
After determining the base image of the target object in the first storage node according to the method, the server executes step S104: determining difference data between a source image of the target object and a base image of the target object.
As shown in fig. 2, a first storage node stores a base image of a target object, a second storage node stores a source image of the target object, and difference data between the source image of the target object and the base image of the target object, that is, data content to be transmitted from the second storage node to the first storage node.
Based on the base image of the target object and the difference data, the server performs step S105: and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
Specifically, the base image of the target object may be locally copied to a specified target path, and at the same time, the difference data is also transmitted from the second storage node to the target path in the first storage node, and the base image and the difference data of the target object are merged under the target path to generate the target image of the target object.
In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a first storage node, and as shown in fig. 3, the method includes:
s301: sending a storage request to a server;
s302: receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node;
s303: and merging the basic mirror image of the target object with the difference data to generate the target mirror image of the target object.
In a third aspect, an embodiment of the present application provides a data processing method, which is applied to a second storage node, and as shown in fig. 4, the method includes:
s401: receiving a transmission instruction sent by a server;
s402: and sending difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node in response to the transmission instruction, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated.
In a fourth aspect, an embodiment of the present application provides a server 500, as shown in fig. 5, including:
a first receiving module 501, configured to receive a storage request sent by a first storage node for updating a base image of a target object;
a first determining module 502, configured to determine a source image of the target object in the second storage node based on the storage request;
a second determining module 503, configured to determine, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request;
a third determining module 504 for determining difference data between the source image of the target object and the base image of the target object;
a first sending module 505, configured to send a transmission instruction to the second storage node, where the transmission instruction is used to instruct the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object and the difference data to generate a target image of the target object.
In one embodiment, the storage request comprises: a target path of a target image of the target object in the first storage node;
and, as shown in fig. 6, the second determining module 503 comprises a judging sub-module 601 and a first determining sub-module 602, wherein,
the judging submodule 601 is configured to judge whether the first storage node has the target path;
the first determining sub-module 602 is configured to, if the target path exists in the first storage node, take a candidate mirror of a target object pointed by the target path as a base mirror of the target object.
In one embodiment, as shown in fig. 6, the second determining module 503 further includes a calculating submodule 603 and a second determining submodule 604;
the calculating sub-module 603 is configured to calculate a file similarity between the source image of the target object and an ith candidate image of the N candidate images in the first storage node when the target path does not exist in the first storage node;
the second determining sub-module 604 is configured to use the candidate image corresponding to the maximum value of the file similarity as the base image of the target object.
In one embodiment, the storage request further comprises: a name of a source image of the target object;
the calculating sub-module 603 is configured to calculate a file similarity between the source image of the target object and an ith candidate image of the N candidate images based on the name of the source image of the target object and the name of the ith candidate image of the N candidate images in the first storage node.
In an embodiment, the calculating sub-module 603 is configured to calculate a file similarity between the source image of the target object and an ith candidate image of the N candidate images based on the data information of the source image of the target object and the data information of the ith candidate image of the N candidate images in the first storage node.
In one embodiment, the data information includes system data information and user data information.
In one embodiment, as shown in fig. 6, the second determining module 503 further includes a new sub-module 605 or a specific sub-module 606;
the new creation sub-module 605 is configured to, when the file similarity between the source image of the target object and the ith candidate image of the N candidate images is smaller than a first threshold, create a new image in the first storage node as a base image of the target object;
the designating sub-module 606 is configured to, when the file similarity between the source image of the target object and the ith candidate image of the N candidate images is smaller than a first threshold value, use the designated image in the first storage node as the base image of the target object.
In a fifth aspect, an embodiment of the present application provides a first storage node 700, as shown in fig. 7, including:
a second sending module 701, configured to send a storage request to a server;
a second receiving module 702, configured to receive difference data between a source image of a target object and a base image of the target object sent by a second storage node;
a merging module 703, configured to merge the basic mirror image of the target object with the difference data to generate a target mirror image of the target object.
In a sixth aspect, an embodiment of the present application provides a second storage node 800, as shown in fig. 8, including:
a third receiving module 801, configured to receive a transmission instruction sent by a server;
a third sending module 802, configured to send, in response to the transmission instruction, difference data between a source mirror of a target object and a base mirror of the target object to the first storage node, so that the base mirror of the target object in the first storage node is merged with the difference data, and a target mirror of the target object is generated.
In a seventh aspect, an embodiment of the present application provides a data processing system 900, as shown in fig. 9, including:
a server 901, configured to receive a storage request sent by a first storage node for updating a base image of a target object; determining a source image of the target object in the second storage node based on the storage request; determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request; determining difference data between a source image of the target object and a base image of the target object; and sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate the target image of the target object.
A first storage node 902, configured to send a storage request to a server 901; receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node; and merging the basic mirror image of the target object with the difference data to generate the target mirror image of the target object.
A second storage node 903, configured to receive a transmission instruction sent by the server 901; and sending difference data between the source image of the target object and the base image of the target object to the first storage node 902 in response to the transmission instruction, so that the base image of the target object in the first storage node 902 is merged with the difference data to generate the target image of the target object.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 10, it is a block diagram of an electronic device according to an embodiment of the present application. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 10, the electronic apparatus includes: one or more processors 1001, memory 1002, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, if desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 10 exemplifies one processor 1001.
The memory 1002 is a non-transitory computer readable storage medium provided herein. Wherein the memory stores instructions executable by at least one processor to cause the at least one processor to perform the image-based problem determination method provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the image-based question deciding method provided by the present application.
The memory 1002, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the image-based question making method in the embodiments of the present application. The processor 1001 executes various functional applications of the server and data processing, i.e., implements the image-based problem determination method in the above method embodiment, by running non-transitory software programs, instructions, and modules stored in the memory 1002.
The memory 1002 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory 1002 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 1002 may optionally include memory located remotely from the processor 1001, which may be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003, and the output device 1004 may be connected by a bus or other means, and the bus connection is exemplified in fig. 10.
The input device 1003 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick or other input device. The output devices 1004 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (19)
1. A data processing method is applied to a server, and the method comprises the following steps:
receiving a storage request sent by a first storage node for updating a base image of a target object; the storage request comprises a target path of a target image of the target object in the first storage node;
determining a source image of the target object in a second storage node based on the storage request;
determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request;
determining difference data between a source image of the target object and a base image of the target object;
sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base mirror image of the target object with the difference data to generate a target mirror image of the target object;
wherein the determining, in the first storage node based on the storage request, a base image of a target object corresponding to a source image of the target object comprises:
under the condition that the first storage node does not have the target path, calculating the file similarity between the source image of the target object and the ith candidate image in the N candidate images in the first storage node;
and taking the candidate mirror image corresponding to the maximum value of the file similarity as the basic mirror image of the target object.
2. The method of claim 1, the determining, in the first storage node based on the storage request, a base image of a target object corresponding to a source image of the target object, comprising:
and taking the candidate mirror image of the target object pointed by the target path as the basic mirror image of the target object under the condition that the target path exists in the first storage node.
3. The method of claim 2, the storage request further comprising, a name of a source image of a target object;
the calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node comprises:
calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images based on the name of the source image of the target object and the name of the ith candidate image of the N candidate images in the first storage node.
4. The method of claim 2, the calculating a file similarity between the source image of the target object and an ith candidate image of the N candidate images in the first storage node, comprising:
calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images based on the data information of the source image of the target object and the data information of the ith candidate image of the N candidate images in the first storage node.
5. The method of claim 4, the data information comprising system data information and user data information.
6. The method of any of claims 2-5, further comprising:
under the condition that the file similarity between the source mirror image of the target object and the ith candidate mirror image in the N candidate mirror images is smaller than a first threshold value, establishing a new mirror image in the first storage node as a basic mirror image of the target object;
alternatively, the first and second electrodes may be,
and under the condition that the file similarity between the source mirror image of the target object and the ith candidate mirror image in the N candidate mirror images is smaller than a first threshold value, taking the specified mirror image in the first storage node as the basic mirror image of the target object.
7. A data processing method is applied to a first storage node, and comprises the following steps:
sending a storage request to a server; the storage request comprises a target path of a target image of a target object in the first storage node;
receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node;
merging the basic mirror image of the target object with the difference data to generate a target mirror image of the target object;
the base image of the target object is determined by the server according to the candidate image corresponding to the maximum value of the file similarity by calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node by the server under the condition that the first storage node does not have the target path.
8. A data processing method is applied to a second storage node, and comprises the following steps:
receiving a transmission instruction sent by a server; the server receives a storage request sent by a first storage node, wherein the storage request comprises a target path of a target image of a target object in the first storage node;
sending difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node in response to the transmission instruction, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated;
the base image of the target object is determined by the server according to the candidate image corresponding to the maximum value of the file similarity by calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node by the server under the condition that the first storage node does not have the target path.
9. A server, comprising:
a first receiving module, configured to receive a storage request sent by a first storage node for updating a base image of a target object;
a first determining module for determining a source image of the target object in a second storage node based on the storage request;
a second determining module for determining a base image of a target object corresponding to a source image of the target object in the first storage node based on the storage request;
a third determination module to determine difference data between a source image of the target object and a base image of the target object;
a first sending module, configured to send a transmission instruction to a second storage node, where the transmission instruction is used to instruct the second storage node to send the difference data to the first storage node, so that the first storage node merges the base image of the target object with the difference data to generate a target image of the target object;
the second determining module comprises a judging submodule, a calculating submodule and a second determining submodule, wherein the judging submodule is used for judging whether the first storage node has a target path or not;
the calculation sub-module is configured to calculate a file similarity between a source image of the target object and an ith candidate image of the N candidate images in the first storage node when the target path does not exist in the first storage node;
and the second determining submodule is used for taking the candidate mirror image corresponding to the maximum value of the file similarity as the basic mirror image of the target object.
10. The server of claim 9, wherein the second determination module comprises a determination submodule and a first determination submodule, wherein,
the first determining submodule is configured to, when the target path exists in the first storage node, use a candidate mirror of a target object pointed to by the target path as a base mirror of the target object.
11. The server of claim 10, wherein the storage request further comprises: a name of a source image of the target object;
the calculation sub-module is configured to calculate a file similarity between the source image of the target object and an ith candidate image of the N candidate images based on the name of the source image of the target object and the name of the ith candidate image of the N candidate images in the first storage node.
12. The server according to claim 10, wherein the computing sub-module is configured to compute the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node based on the data information of the source image of the target object and the data information of the ith candidate image of the N candidate images in the first storage node.
13. The server of claim 12, wherein the data information comprises system data information and user data information.
14. The server according to any one of claims 10-12, wherein the second determining module further comprises a new submodule or a designated submodule;
the new creation sub-module is configured to create a new mirror image in the first storage node as a basic mirror image of the target object when the file similarity between the source mirror image of the target object and an ith candidate mirror image of the N candidate mirror images is smaller than a first threshold value;
the designating sub-module is configured to, when the file similarity between the source image of the target object and the ith candidate image of the N candidate images is smaller than a first threshold value, use the designated image in the first storage node as the base image of the target object.
15. A first storage node, comprising:
the second sending module is used for sending a storage request to the server; the storage request comprises a target path of a target mirror of a target object in the first storage node;
a second receiving module, configured to receive difference data between a source image of a target object and a base image of the target object sent by a second storage node;
the merging module is used for merging the basic mirror image of the target object with the difference data to generate a target mirror image of the target object;
the base image of the target object is determined by the server according to the candidate image corresponding to the maximum value of the file similarity by calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node by the server under the condition that the first storage node does not have the target path.
16. A second storage node, comprising:
the third receiving module is used for receiving a transmission instruction sent by the server; the server receiving a storage request sent by a first storage node, the storage request including a target path of a target image of a target object in the first storage node;
a third sending module, configured to send, in response to the transmission instruction, difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated;
the base image of the target object is determined by the server according to the candidate image corresponding to the maximum value of the file similarity by calculating the file similarity between the source image of the target object and the ith candidate image of the N candidate images in the first storage node under the condition that the target path does not exist in the first storage node.
17. A data processing system comprising:
a server for receiving a storage request sent by a first storage node for updating a base image of a target object; the storage request comprises a target path of a target image of the target object in the first storage node; determining a source image of the target object in a second storage node based on the storage request; determining, in the first storage node, a base image of a target object corresponding to a source image of the target object based on the storage request; determining difference data between a source image of the target object and a base image of the target object; sending a transmission instruction to a second storage node, wherein the transmission instruction is used for instructing the second storage node to send the difference data to the first storage node, so that the first storage node merges the base mirror image of the target object with the difference data to generate a target mirror image of the target object;
wherein the determining, in the first storage node based on the storage request, a base image of a target object corresponding to a source image of the target object comprises: under the condition that the first storage node does not have the target path, calculating the file similarity between the source image of the target object and the ith candidate image in the N candidate images in the first storage node; taking the candidate mirror image corresponding to the maximum value of the file similarity as a basic mirror image of the target object;
the first storage node is used for sending a storage request to the server; receiving difference data between a source image of a target object and a base image of the target object sent by a second storage node; merging the basic mirror image of the target object with the difference data to generate a target mirror image of the target object;
the second storage node is used for receiving a transmission instruction sent by the server; and sending difference data between a source mirror image of a target object and a base mirror image of the target object to a first storage node in response to the transmission instruction, so that the base mirror image of the target object in the first storage node is merged with the difference data, and a target mirror image of the target object is generated.
18. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
19. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110277769.8A CN112925482B (en) | 2021-03-15 | 2021-03-15 | Data processing method, device, system, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110277769.8A CN112925482B (en) | 2021-03-15 | 2021-03-15 | Data processing method, device, system, electronic equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112925482A CN112925482A (en) | 2021-06-08 |
| CN112925482B true CN112925482B (en) | 2022-08-23 |
Family
ID=76175111
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110277769.8A Active CN112925482B (en) | 2021-03-15 | 2021-03-15 | Data processing method, device, system, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112925482B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103116618A (en) * | 2013-01-28 | 2013-05-22 | 南开大学 | Telefile system mirror image method and system based on lasting caching of client-side |
| CN110188801A (en) * | 2019-05-10 | 2019-08-30 | 深圳灵图慧视科技有限公司 | Image-recognizing method, device, electronic equipment and storage medium |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8914338B1 (en) * | 2011-12-22 | 2014-12-16 | Emc Corporation | Out-of-core similarity matching |
| US9274818B2 (en) * | 2013-02-06 | 2016-03-01 | International Business Machines Corporation | Reliable and scalable image transfer for data centers with low connectivity using redundancy detection |
| US20150269032A1 (en) * | 2014-03-18 | 2015-09-24 | Netapp, Inc. | Backing up data to cloud data storage while maintaining storage efficiency |
| CN106202070A (en) * | 2015-04-29 | 2016-12-07 | 中国电信股份有限公司 | File storage processing method and system |
| US9933970B2 (en) * | 2015-10-30 | 2018-04-03 | Netapp, Inc. | Deduplicating data for a data storage system using similarity determinations |
| US9954946B2 (en) * | 2015-11-24 | 2018-04-24 | Netapp, Inc. | Directory level incremental replication |
| US10282115B2 (en) * | 2017-04-13 | 2019-05-07 | International Business Machines Corporation | Object synchronization in a clustered system |
| CN109375872B (en) * | 2018-09-27 | 2020-07-24 | 腾讯科技（深圳）有限公司 | Data access request processing method, device and equipment and storage medium |
| CN110780819A (en) * | 2019-10-25 | 2020-02-11 | 浪潮电子信息产业股份有限公司 | Data read-write method of distributed storage system |
| CN112235427B (en) * | 2020-12-14 | 2021-03-30 | 广东睿江云计算股份有限公司 | Merging method and system for mirror image files |
-
2021
- 2021-03-15 CN CN202110277769.8A patent/CN112925482B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103116618A (en) * | 2013-01-28 | 2013-05-22 | 南开大学 | Telefile system mirror image method and system based on lasting caching of client-side |
| CN110188801A (en) * | 2019-05-10 | 2019-08-30 | 深圳灵图慧视科技有限公司 | Image-recognizing method, device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 一种从马尔可夫聚类簇发现潜在WEB社区特征的方法;杨楠等;《计算机学报》;20070715(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112925482A (en) | 2021-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765024B (en) | Simulation test method, simulation test device, electronic equipment and computer readable storage medium | |
| CN111524166B (en) | Video frame processing method and device | |
| US20210209417A1 (en) | Method and apparatus for generating shared encoder | |
| CN111752843A (en) | Method, device, electronic equipment and readable storage medium for determining influence surface | |
| CN112565356B (en) | Data storage method and device and electronic equipment | |
| CN112015468B (en) | Interface document processing method and device, electronic equipment and storage medium | |
| CN110619002A (en) | Data processing method, device and storage medium | |
| EP3859575A2 (en) | Method and apparatus for providing applet service capability, electronic device, and storage medium | |
| CN111752960B (en) | Data processing method and device | |
| CN111475164A (en) | Component dependency relationship detection method and device and electronic equipment | |
| CN112269706A (en) | Interface parameter checking method and device, electronic equipment and computer readable medium | |
| EP3872704A2 (en) | Header model for instance segmentation, instance segmentation model, image segmentation method and apparatus | |
| CN110933227A (en) | Assistance method, device, equipment and medium for intelligent terminal | |
| CN110752968A (en) | Performance benchmark test method and device, electronic equipment and storage medium | |
| CN112579897B (en) | Information searching method and device | |
| CN112069137B (en) | Method, device, electronic equipment and computer readable storage medium for generating information | |
| CN111177479B (en) | Method and device for acquiring feature vector of node in relational network graph | |
| CN112925482B (en) | Data processing method, device, system, electronic equipment and computer storage medium | |
| CN111339187A (en) | Data processing method, device, equipment and storage medium based on intelligent contract | |
| US11632552B2 (en) | Method and apparatus for coding video, device and medium | |
| CN113010811B (en) | Webpage acquisition method and device, electronic equipment and computer readable storage medium | |
| KR102485109B1 (en) | Session Recommendation Methods, Devices and Electronic Devices | |
| CN111639116B (en) | Data access connection session protection method and device | |
| CN112052347A (en) | Image storage method and device and electronic equipment | |
| CN111582480A (en) | Method and device for pruning a model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |