CN112688905B - Data transmission method, device, client, server and storage medium - Google Patents
Data transmission method, device, client, server and storage medium Download PDFInfo
- Publication number
- CN112688905B CN112688905B CN201910995678.0A CN201910995678A CN112688905B CN 112688905 B CN112688905 B CN 112688905B CN 201910995678 A CN201910995678 A CN 201910995678A CN 112688905 B CN112688905 B CN 112688905B
- Authority
- CN
- China
- Prior art keywords
- data
- data fragment
- check value
- fragment
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The disclosure provides a data transmission method, a data transmission device, a client, a server and a storage medium, and belongs to the field of data communication. The method comprises the following steps: a client receives a first data fragment of a target object and a first check value of the first data fragment; the client generates a check value of the first data fragment; the client compares the check value of the first data fragment with the first check value to determine whether the first data fragment is accurate; and generating a check value of the first data fragment by using the same algorithm as the algorithm for generating the first check value. In the process, the check values of the data fragments are compared, so that the stored data fragments are consistent with the data fragments of the object to be stored, which are sent to the server by the terminal, and the reliability in the data transmission process is improved to a certain extent.
Description
Technical Field
The present disclosure relates to the field of data communications, and in particular, to a data transmission method, apparatus, client, server, and storage medium.
Background
The transmission technology is an important component of computer network technology, and currently, common transmission protocols include hypertext transfer protocol (HTTP), distributed transmission system (HDFS), network File System (NFS), common Internet File Service (CIFS), and the like. In the transmission process, data is lost or damaged sometimes, and in order to solve the problem, the data needs to be checked to ensure the integrity of transmission.
In the current data verification method, when a client uploads data, a message-digest algorithm (md5) value of the data is carried in a header field of a transmission protocol, a server calculates an MD5 value according to the received data after receiving the data, when two MD5 values are consistent, the data is complete, the MD5 value is recorded in metadata, and the data and the metadata of the data are stored in a data persistence layer. When a client sends a request for downloading the data, the server reads the data and the metadata of the data from the data persistence layer, calculates an MD5 value for the read data, compares the MD5 value with the MD5 value recorded in the read metadata, if the two MD5 values are consistent, the server sends the data and the metadata of the data to the client, after the client receives the data, the client calculates an MD5 value for the received data, compares the MD5 value with the MD5 value recorded in the metadata, and if the two MD5 values are inconsistent, the client indicates that the data may have data loss or damage in the downloading process.
When any kind of modification is performed on the data through any protocol interface in the multi-protocol, the content of the data changes, and the MD5 value of the corresponding data also changes, but the change is not synchronized into the metadata stored in the data persistence layer, so that the MD5 value of the data downloaded by the subsequent client is inconsistent with the MD5 value recorded in the metadata, which causes a user to easily generate an error when verifying the integrity of the data, and reduces the reliability of the data to a certain extent.
Disclosure of Invention
The embodiment of the disclosure provides a data transmission method, a data transmission device, a client, a server and a storage medium. The technical problem that data reliability is not high in the related technology can be solved, and the technical scheme is as follows:
in one aspect, a data transmission method is provided, where the method includes:
a client receives a first data fragment of a target object and a first check value of the first data fragment;
the client generates a check value of the first data fragment;
the client compares the check value of the first data fragment with the first check value to determine whether the first data fragment is accurate; and generating a check value of the first data fragment by using the same algorithm as the algorithm for generating the first check value.
In the process, the check values of the data fragments are compared, so that the stored data fragments are consistent with the data fragments of the object to be stored, which are sent to the server by the terminal, and the reliability in the data transmission process is improved to a certain extent.
In one possible implementation, the method further includes:
and when the client determines that the first data fragment is inaccurate, generating error indication information for indicating that the target object is erroneous.
In the implementation mode, when any data fragment is inaccurate, the transmission error is indicated, and the transmission error can be judged after all data fragments are not required to be downloaded, so that the transmission data volume is greatly reduced, and the timeliness of error judgment is improved.
In one possible implementation, the method further comprises:
the client receives a second data fragment of the target object and a second check value of the second data fragment;
the client generates a check value of the second data fragment;
the client compares the check value of the second data fragment with the second check value to determine whether the second data fragment is accurate; and generating the check value of the second data fragment by the same algorithm as the algorithm for generating the second check value.
In the implementation manner, under the condition of transmitting a plurality of data fragments, whether transmission errors occur or not can be determined based on the data fragments received before and after, so that the accuracy of verification is ensured.
In a possible implementation manner, the generating a check value of the second data fragment specifically includes:
and the client generates the check value of the second data fragment according to the check value of the first data fragment and the second data fragment.
In the above implementation manner, since the first data fragment is a previous fragment of the second data fragment, and the check value is obtained by iterative computation of the server, it is possible to perform checking through iterative computation in the same manner, and ensure accuracy of the transmission sequence of the data fragments, and once the check value of the second data fragment obtained based on this manner does not coincide with the received second check value of the second data fragment, an error in the transmission sequence is likely to occur.
In a possible implementation manner, the generating, by the client, the check value of the first data fragment specifically includes:
and generating a check value of the first data fragment according to the first data fragment and the key of the client.
The key of the client is shared by the client and the server, is unique to the client and is not shared by other clients, so that the verification value is generated based on the key, the safety of data can be greatly improved, and the verification cannot be performed even if other clients obtain the data fragments.
In one aspect, a data transmission method is provided, where the method includes:
the server acquires a first data fragment of a target object;
the server generates a first check value of the first data fragment;
and sending a first data fragment of a target object and a first check value of the first data fragment to a client.
In one possible implementation manner, the server acquires a second data fragment of the target object;
the server generates a second check value for the second data slice.
In one possible implementation, the method further includes:
after the first data fragment of the target object and the first check value of the first data fragment are sent to the client, the method further includes:
and if the correct indication information of the client aiming at the first data fragment is received, sending a second data fragment and a second check value of the second data fragment to the client.
In a possible implementation manner, the generating a second check value of the second data fragment specifically includes:
and the server generates a second check value of the second data fragment according to the first check value of the first data fragment and the second data fragment.
In a possible implementation manner, the generating, by the server, the first check value of the first data fragment specifically includes:
and the server generates a check value of the first data fragment according to the first data fragment and the key of the client.
In one possible implementation, the method further comprises:
when a modification request for any data fragment is received, reading the stored data fragment, and modifying the data fragment in response to the modification request to obtain a new data fragment;
and replacing the data slice and the corresponding characteristic value with the new data slice and the corresponding characteristic value.
In a third aspect, a data transmission apparatus is provided for executing the data transmission method. Specifically, the data transmission apparatus includes a functional module configured to execute the data transmission method provided in the first aspect or any one of the optional manners of the first aspect.
In a fourth aspect, a data transmission apparatus is provided for executing the data transmission method. Specifically, the data transmission device includes a functional module for executing the data transmission method provided by the second aspect or any one of the alternatives of the second aspect.
In a fifth aspect, there is provided a client comprising an interface and one or more processors configured to perform the method of the first aspect or any of the alternatives of the first aspect.
In a sixth aspect, there is provided a server comprising an interface and one or more processors to perform the method of the second aspect above or any alternative of the second aspect above.
In a seventh aspect, there is provided a computer readable storage medium having stored therein at least one instruction which is loaded and executed by a processor to implement the method as in any one of the above alternatives.
Drawings
Fig. 1 is a schematic diagram of an implementation environment of a data transmission method provided in an embodiment of the present disclosure;
FIG. 2 is a block diagram illustrating a server 200 in accordance with an exemplary embodiment;
FIG. 3 is a schematic diagram of an object storage service system shown in accordance with an exemplary embodiment;
FIG. 4 is an interaction flow diagram illustrating a method of data transmission in accordance with an exemplary embodiment;
FIG. 5 is a data flow diagram illustrating a method of data transmission according to an exemplary embodiment;
FIG. 6 is an interaction flow diagram illustrating a method of data transmission in accordance with an exemplary embodiment;
FIG. 7 is a data flow diagram illustrating a data transmission method in accordance with an exemplary embodiment;
FIG. 8 is a schematic diagram illustrating a modification of a 3 rd data slice in object data in accordance with an illustrative embodiment;
FIG. 9 is a storage diagram illustrating a plurality of data slices of object data and feature values of the plurality of data slices, according to an example embodiment;
FIG. 10 is a schematic diagram illustrating modified storage for the 3 rd data slice of a plurality of data slices in accordance with an illustrative embodiment;
FIG. 11 is a block diagram of a data transfer device according to an exemplary embodiment;
fig. 12 is a schematic structural diagram illustrating a data transmission apparatus according to an exemplary embodiment.
Detailed Description
To make the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of a specific implementation environment of a data transmission method provided in an embodiment of the present disclosure, and referring to fig. 1, the implementation environment includes: a server 101 and a client 102.
A server 101, which may include at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 101 is used to provide background services for applications that support data transfer. For example, the server 101 may be configured to perform fragmentation processing on data of an object to obtain a plurality of data fragments, and perform calculation of a first check value on the plurality of data fragments. When a request of a client is received, the plurality of data fragments stored in the data persistence layer are sent to the client 102, so that the client 102 performs integrity check on the plurality of data fragments.
The client 102 is connected to the server 101 through a wireless network or a wired network. The client 102 may be an application server, a mobile terminal, or the like. The client 102 may be used as an uploader of object data to send data of an object to be stored to the server, and the client may upload the data through an interface of any one of the following protocols, for example: HTTP transport protocol, HDFS transport protocol, NFS transport protocol, CIFS transport protocol, and the like.
The client 102 may refer to one of a plurality of clients, and the embodiment is only illustrated by the client 102. Those skilled in the art will appreciate that the number of client devices described above may be greater or fewer. The number of clients and the type of the device are not limited by the embodiment of the disclosure.
Fig. 2 is a block diagram illustrating a server 200 according to an example embodiment. Referring to fig. 2, a server 200 includes an interface 201 and one or more processors 202. Further, the one or more processors 202 perform the above-described data transmission method. The interface may be a network interface card or a host bus adapter, which is not limited in this embodiment of the present invention. The server 200 may also include a power component configured to perform power management of the server 200. It should be noted that the basic structure of the client and the server 200 shown in fig. 2 may be the same, and is not described herein again. In addition, the interface may be one or multiple interfaces, and may be configured according to performance requirements of the device and a network environment, which is not limited in this embodiment of the present application.
Referring to fig. 3, the object storage service system is shown in fig. 3, and all application layers involved in the embodiment of the present disclosure are briefly described around fig. 3, and the object storage service system operates in a server. As shown in fig. 3, the object storage system includes an object protocol access layer, a portable operating system interface of unix (Posix), a distributed transmission system transport protocol (HDFS) access layer, and a data persistence layer.
The object protocol access layer is used for carrying out fragment processing on object data when the client uploads the object data through a network protocol, calculating the characteristic value of each data fragment, and storing the characteristic values of a plurality of data fragments and a plurality of data fragments in the data persistence layer.
The client may operate the plurality of data fragments stored in the data persistence layer through a Posix protocol access layer and an HDFS protocol access layer, and the client may be a client that uploads object data or other clients, which is not limited in this disclosure. The operation includes any operation of reading data content of the data fragment, writing data content of the data fragment, adding data content of the data fragment, deleting data content of the data fragment, moving data content of the data fragment, modifying data content of the data fragment, and the like.
The data persistence layer is configured to store a plurality of data fragments and metadata of each data fragment, where the metadata includes a storage location of the data fragment in the data persistence layer and a characteristic value of the data fragment, and the characteristic values of the data fragments and the data fragments may be stored in one hard disk in the data persistence layer or may be stored in a plurality of hard disks in the data persistence layer, which is not limited in this disclosure.
The characteristic value refers to a value calculated by using any check algorithm according to the data content of the data slice, and may be an MD5 value, for example.
Referring to fig. 4 and fig. 5, fig. 4 is an interaction flowchart of a data transmission method according to an exemplary embodiment, and fig. 5 is a data flow diagram of a data transmission method according to an exemplary embodiment, which specifically includes the following steps:
401. and the client acquires the check value of the data of the target object.
The target object may be any type of object such as audio, video, document, and picture, which is not limited in this disclosure. In step 401, how the client acquires the data of the target object is not limited, and when the client needs to upload the data of the target object to the target object storage service system, an upload process may be triggered by operating on the client of the target object storage service system provided by the client. When the client detects the uploading operation, the characteristic value of the data of the target object can be calculated based on a target characteristic value algorithm and is used as a check value of the data.
The target characteristic value algorithm may be any one of MD5, secure Hash Algorithm (SHA), cyclic redundancy check algorithm (CRC), and the like, which is not limited in this disclosure.
402. And the client sends an uploading request to the server, wherein the uploading request carries the data of the target object and the check value of the target object.
In step 402, after acquiring the data of the target object and calculating the check value of the data of the target object, the client may automatically send the data and the check value of the target object to the server, or send the data and the check value of the target object to the server after detecting the confirmation instruction of the user, which is not limited in this disclosure. The upload request in the embodiment of the present invention may specifically be a write request sent by a client, and is used to write a target object to a server.
403. The server receives the data and the check value of the target object, obtains a feature value of the data of the target object, determines whether the feature value is consistent with the check value, and executes step 404 when the feature value is consistent with the check value.
In the embodiment of the disclosure, after receiving the data and the check value of the target object sent by the client, the server calculates the characteristic value of the received data of the target object by using a target characteristic value algorithm, compares the characteristic value with the check value sent by the client, and when the comparison is consistent, indicates that the data of the target object is not damaged in the uploading process. When the comparison result is inconsistent, the data of the target object is damaged in the uploading process, the server sends error indication information to the client to inform the client that the data of the target object is damaged, and the client can upload the data of the target object again after receiving the error indication information.
404. The server carries out fragmentation processing on the data to obtain a plurality of data fragments.
The step of slicing the data is to slice the data of the target object according to a fixed size to obtain a plurality of data slices with the size within the fixed size. For example, the size of a data slice may be controlled to be within 32kb, that is, the fixed size may be 32kb. Of course, the above-mentioned segmentation may also be according to a fixed number, which is not limited in this disclosure.
It should be noted that the server may allocate a security key and an access key to the data of the target object, where the security key is a key shared by the server and the client and cannot be shared by a third party, the access key is a public key, and all the clients may access the target object to be stored.
405. And the server acquires the characteristic value of each data fragment.
In step 405, the server may perform feature value calculation on the plurality of data segments respectively based on a target feature value algorithm.
406. The server stores each data fragment and the characteristic value of each data fragment in the data persistence layer.
In step 406, the server stores the feature values of the data fragments in the data persistence layer, which may be implemented in the following two ways:
in a possible implementation manner, the data fragment may include a data header and a data content segment, and the server may add the characteristic value of each data fragment to the data header of the data fragment. In a possible implementation manner, the server adds the feature value of each data fragment to the tail of the data fragment, and the embodiment of the present disclosure does not limit to which part the feature value is specifically added.
It should be noted that, in the process of storing the plurality of data fragments in the data persistence layer, the server may generate metadata of the data based on the storage address of each data fragment and the data itself, where the metadata may be used to indicate a storage location of the data, so that data operations can be performed based on the metadata in subsequent query or modification processes.
407. When the data storage is completed, the server returns a storage success message to the client.
It should be noted that the returning of the storage success message may also be performed when the upload request is received, and is not necessarily completed at the time of the step 407, so as to achieve the purpose of timely notification.
According to the method provided by the embodiment of the disclosure, a client acquires a check value of an object to be stored, the client sends data of the object to be stored and the check value of the object to a server, the server receives the data and the check value of the object to be stored and verifies a characteristic value of the object to be stored to confirm that the object to be stored is uploaded completely, the server performs fragmentation processing on the data of the object to be stored to obtain a plurality of data fragments, the characteristic values of the data fragments are acquired according to a target characteristic value algorithm, and the data fragments and the characteristic values of the data fragments are stored. In the process, the stored data fragments are ensured to be consistent with the data of the object to be stored, which is sent to the server by the client, through comparison of the check values for multiple times, so that the reliability in the data transmission process is improved to a certain extent.
Referring to fig. 6 and fig. 7, fig. 6 is an interaction flowchart of a data transmission method according to an exemplary embodiment, and fig. 7 is a data flow diagram of a data transmission method according to an exemplary embodiment, which specifically includes the following steps:
601. the client sends a download request to the server, wherein the download request is used for downloading the data of the target object.
In step 601, the download request carries a blocking transmission code identifier, which may be a blocking transmission format of a hypertext transfer protocol, i.e., HTTP chunk.
For example, in response to the download request, an object data is divided into a plurality of data fragments in the download message body for transmission, and each data fragment is preceded by check information for identifying the length of the data fragment and the data content of the data fragment.
602. And the server responds to a downloading request of the client and acquires a plurality of data fragments of the target object and characteristic values of the data fragments from the data persistence layer.
603. And the server respectively calculates the characteristic value of each data fragment according to the plurality of data fragments and compares the characteristic value with the characteristic value extracted from the data persistence layer.
In step 603, after the server extracts the multiple data fragments to be downloaded from the data persistence layer, a target characteristic value algorithm is applied to calculate characteristic values of the multiple data fragments, so as to avoid damage to data contents of the data fragments stored in the hard disk due to damage to the hard disk and the like. And comparing the calculated characteristic values with the characteristic values extracted by the server, when the comparison result is consistent, indicating that the data fragment is not damaged in the data persistence layer, continuing to execute the step 604, when any pair of characteristic values is inconsistent, indicating that the data fragment is damaged in the data persistence layer, and stopping sending the object data to the client by the server.
604. And when the characteristic values of the data fragments pass the comparison, the server respectively generates the check values of the plurality of data fragments.
In step 604, the server generates the check value of each data fragment in any one of the following implementations:
for a first data slice, the characteristic value of the first data slice may be used as a check value of the first data slice.
For the data fragments after the first data fragment, in order to ensure that the received fragments will not generate errors in the subsequent process, iterative computation may be performed on the feature values of the data fragments obtained by the computation in step 603, taking the second data fragment as an example, the first data fragment may be used as the first data fragment of the second data fragment, the computation may be performed by applying a target feature value algorithm according to the first check value of the first data fragment and the second data fragment, and the computation result is used as the second check value of the second data fragment.
For example, the process of obtaining the check value by performing iterative computation on N data fragments of the data to be downloaded is as follows:
crc-1＝crc64(chunk-data-1)；
in the formula, crc-1 is a first check value of the first data fragment, chunk-data-1 represents the first data fragment, and the formula indicates that the check value of the first data fragment is generated according to the first data fragment. The following formula can be referred to for the meaning of coincidence, and is not described again.
And the subsequent iterative computation process is as follows: crc-2= crc64 (crc-1 + chunk-data-2);
crc-3＝crc64(crc-2+chunk-data-3)；
crc-4＝crc64(crc-3+chunk-data-4)；
…
crc-N＝crc64(crc-N-1+chunk-data-N)；
wherein, crc-1 is a first parity of the first data fragment, crc-2 is a second parity of the second data fragment, and so on.
In a possible implementation manner, in order to ensure that the data to be downloaded is safer and more reliable, when the first check value of the first data fragment is calculated, the first check value of the first data fragment may be generated according to the first data fragment and a key of the client, where the key is a key shared by the client and the server that upload data, and a third-party user may download or modify the object data only if the third-party user needs to hold the key. The process of adding the key in the calculation of the first check value of the first data slice is as follows:
crc-1＝crc64(key+chunk-data-1)；
in the formula, crc-1 is a first check value of the first data fragment, key is a key shared by the client and the server for uploading data, chunk-data-1 represents the first data fragment, and the formula indicates that the check value of the first data fragment is generated according to the key and the first data fragment. The following formula can be referred to for the meaning of coincidence, and is not described again.
And the subsequent iterative computation process is as follows:
crc-2＝crc64(crc-1+chunk-data-2)；
crc-3＝crc64(crc-2+chunk-data-3)；
crc-4＝crc64(crc-3+chunk-data-4)；
…
crc-n＝crc64(crc-n-1+chunk-data-n)；
it should be noted that any implementation manner described above may be adopted to calculate the check value of each data slice, which is not limited in this disclosure.
605. And the server sends the plurality of data fragments and the check value of each data fragment to the client.
In step 605, after sending the nth data fragment and the check value of the nth data fragment, if receiving the correct indication information of the client for the nth data fragment, sending the check values of the (N + 1) th data fragment and the (N + 1) th data fragment to the client, where N is an integer greater than or equal to 1 and less than N, and N is an integer greater than 1. And if receiving error indication information of the client to the nth data fragment, wherein the error indication information is used for indicating that the target object is wrong, stopping sending or resending the nth data fragment.
For example, after the server sends the first data fragment and the first check value of the first data fragment to the client, and receives correct indication information for the first data fragment sent by the client to the server, the server sends a second data fragment with a fragment sequence after the first data fragment and a second check value of the second data fragment to the client.
606. The client receives a plurality of data fragments and the check value of each data fragment sent by the server.
607. And the client performs integrity check on the plurality of data fragments according to the check values of the plurality of data fragments, and determines that the target object is downloaded completely when the check is passed.
In step 607, the client performs integrity check on the multiple data fragments according to the check values of the multiple data fragments, and the following implementation manners may be adopted: after receiving the first data fragment and the first check value of the first data fragment, the client generates the check value of the first data fragment; the client compares the check value of the first data fragment with the first check value to determine whether the first data fragment is accurate; and generating a check value of the first data fragment by using the same algorithm as the algorithm for generating the first check value.
In an optional implementation manner, after receiving the first data fragment and the first check value of the first data fragment, the client may generate the check value of the first data fragment according to the first data fragment and the key of the client, and then perform a subsequent check process.
The method provided by the embodiment of the disclosure includes that a client sends a download request to a server, the server responds to the download request of the client, a plurality of data fragments and characteristic values corresponding to the download request are extracted from a data persistence layer, the server calculates the characteristic value of each extracted data fragment, compares the characteristic value with the extracted characteristic value, calculates a first check value of each data fragment after the comparison is consistent, sends the data fragments and the corresponding first check values to the client, the client receives the data fragments and the corresponding first check values sent by the server, performs integrity check on the data fragments, and when the check is passed, it is determined that the download of the data fragments is completed. In the process of downloading the object data, the integrity of each data fragment in the transmission process is ensured by calculating the check value of each data fragment for multiple times, and the reliability of the data in the transmission process is improved to a certain extent.
In the embodiment of the present disclosure, since the server employs the method of storing the fragment and storing the characteristic value of the corresponding data fragment during data storage, reading and writing of the whole data can be avoided, and I/O when modification operation occurs is greatly reduced, that is, when the server receives a modification request for any data fragment, the stored data fragment is read, and the data fragment is modified in response to the modification request to obtain a new data fragment; the data slice and corresponding characteristic value are replaced with the new data slice and corresponding characteristic value.
Fig. 8 is a schematic diagram illustrating a 3 rd data slice in modified object data according to an embodiment of the present disclosure. The client user modifies the 3 rd data fragment stored in the data persistence layer through protocol interfaces such as NFS protocol and CIFS protocol, the server responds to a modification request of the client and receives modification, the server calculates the characteristic value of the modified 3 rd data fragment, and stores the modified 3 rd data fragment and the recalculated characteristic value of the 3 rd data fragment in the storage positions of the characteristic values of the original 3 rd data fragment and the original 3 rd data fragment in the data persistence layer. Fig. 9 is a schematic view illustrating storage of a plurality of data fragments of object data and characteristic values of the plurality of data fragments, and fig. 10 is a schematic view illustrating storage after modification of a 3 rd data fragment of the plurality of data fragments.
All the above optional technical solutions may be combined arbitrarily to form the optional embodiments of the present disclosure, and are not described herein again.
Fig. 11 is a schematic structural diagram of a data transmission device according to an embodiment of the present disclosure. Referring to fig. 11, the apparatus includes:
a receiving module 1101, configured to receive a first data fragment of a target object and a first check value of the first data fragment;
a generating module 1102, configured to generate a check value of the first data fragment;
a checking module 1103, configured to compare a check value of the first data fragment with the first check value to determine whether the first data fragment is accurate; and generating a check value of the first data fragment by using the same algorithm as the algorithm for generating the first check value.
In one possible implementation, the apparatus further includes:
and the information generation module is used for generating error indication information for indicating that the target object is in error when the first data fragment is determined to be inaccurate.
In one possible implementation, the receiving module is further configured to receive a second data fragment of the target object and a second check value of the second data fragment;
the generating module is further used for generating a check value of the second data fragment;
the check module is further configured to compare the check value of the second data fragment with the second check value to determine whether the second data fragment is accurate; and generating the check value of the second data fragment by using the same algorithm as the algorithm for generating the second check value.
In a possible implementation manner, the generating module is configured to generate the check value of the second data fragment according to the check value of the first data fragment and the check value of the second data fragment.
In a possible implementation manner, the generating module is configured to generate a check value of the first data fragment according to the first data fragment and the key of the client.
The division of the functional modules is exemplified, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the data transmission device and the data transmission method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.
Fig. 12 is a schematic structural diagram of a data transmission device according to an embodiment of the present disclosure. Referring to fig. 12, the apparatus includes:
a fragment obtaining module 1201, configured to obtain a first data fragment of a target object;
a generating module 1202, configured to generate a first check value of the first data fragment;
the sending module 1203 is configured to send, to the client, the first data fragment of the target object and the first check value of the first data fragment.
In a possible implementation manner, the fragment obtaining module is further configured to obtain a second data fragment of the target object;
the generating module is further configured to generate a second check value of the second data segment.
In a possible implementation manner, the sending module is further configured to send, to the client, a second data fragment and a second check value of the second data fragment if the correct indication information of the client for the first data fragment is received.
In one possible implementation, the generating module is configured to generate the second check value of the second data fragment according to the first check value of the first data fragment and the second check value of the second data fragment.
In a possible implementation manner, the generating module is configured to generate a check value of the first data fragment according to the first data fragment and a key of the client.
In one possible implementation, the apparatus further includes:
the modification module is used for reading the stored data fragments when receiving a modification request for any data fragment, and modifying the data fragments in response to the modification request to obtain new data fragments;
and the storage module is used for replacing the data fragment and the corresponding characteristic value with the new data fragment and the corresponding characteristic value.
It should be noted that: in the data transmission device provided in the above embodiment, only the division of the functional modules is illustrated in the data transmission, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data transmission device and the data transmission method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.
In an exemplary embodiment, a computer-readable storage medium, such as a memory, including instructions executable by a processor to perform the data transfer method of the above embodiments is also provided. For example, the computer-readable storage medium may be a read-only memory (ROM), a Random Access Memory (RAM), a compact disc-read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.
The embodiment of the present invention is not only applicable to object storage, but also applicable to block storage, file storage, and the like, and the embodiment of the present invention is not limited thereto. The embodiment of the invention is also suitable for data storage in an object storage system, a block storage system and a file storage system.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Claims (22)
1. A method of data transmission, the method comprising:
a client sends a download request to a server, wherein the download request is used for the server to obtain a plurality of data fragments of a target object and the characteristic value of each data fragment, the characteristic value of each data fragment is calculated according to each data fragment, the characteristic value calculated based on each data fragment is the same as the obtained characteristic value, and the check value of each data fragment is generated;
the client receives a first data fragment of the target object and a first check value of the first data fragment, which are sent by the server;
the client generates a check value of the first data fragment;
the client compares the check value of the first data fragment with the first check value to determine whether the first data fragment is accurate; the algorithm for generating the check value of the first data fragment is the same as the algorithm for generating the first check value;
when the client determines that the first data fragment is accurate, the client sends correct indication information to the server, where the correct indication information is used to indicate the server to send a second data fragment and a second check value of the second data fragment to the client, and the second data fragment is a data fragment that is adjacent to and subsequent to the first data fragment among multiple data fragments of the target object.
2. The method of claim 1, further comprising:
and when the client determines that the first data fragment is inaccurate, generating error indication information for indicating that the target object is wrong.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
the client receives a second data fragment of the target object and a second check value of the second data fragment;
the client generates a check value of the second data fragment;
the client compares the check value of the second data fragment with the second check value to determine whether the second data fragment is accurate; and generating the check value of the second data fragment by the same algorithm as the algorithm for generating the second check value.
4. The method according to claim 3, wherein the generating the check value of the second data fragment specifically includes:
and the client generates the check value of the second data fragment according to the check value of the first data fragment and the second data fragment.
5. The method according to claim 1, wherein the generating, by the client, the check value of the first data fragment specifically includes:
and generating a check value of the first data fragment according to the first data fragment and the key of the client.
6. A method of data transmission, the method comprising:
a server receives a downloading request sent by a client;
the server acquires a plurality of data fragments of a target object and characteristic values of the data fragments according to the downloading request;
the server respectively calculates the characteristic value of each data fragment according to each data fragment;
based on the fact that the characteristic value calculated by each data fragment is the same as the obtained characteristic value, the server generates a first check value of a first data fragment;
the server sends a first data fragment of the target object and a first check value of the first data fragment to the client;
and the server receives correct indication information sent by the client, and sends a second data fragment of the target object and a second check value of the second data fragment to the client, wherein the second data fragment is a data fragment which is adjacent to the first data fragment and is behind the first data fragment in a plurality of data fragments of the target object.
7. The method of claim 6, further comprising:
the server acquires a second data fragment of the target object;
the server generates a second check value for the second data slice.
8. The method according to claim 7, wherein the generating the second check value of the second data slice specifically includes:
and the server generates a second check value of the second data fragment according to the first check value of the first data fragment and the second data fragment.
9. The method according to claim 6, wherein the generating, by the server, the first check value of the first data segment specifically includes:
and the server generates a first check value of the first data fragment according to the first data fragment and the key of the client.
10. The method of claim 6, further comprising:
when a modification request for any data fragment is received, reading the stored data fragment, and modifying the data fragment in response to the modification request to obtain a new data fragment;
and replacing the data slice and the corresponding characteristic value with the new data slice and the corresponding characteristic value.
11. A data transmission apparatus, applied to a client, the apparatus comprising:
the system comprises a receiving module and a verification module, wherein the receiving module is used for sending a downloading request to a server, the downloading request is used for the server to obtain a plurality of data fragments of a target object and the characteristic value of each data fragment, the characteristic value of each data fragment is calculated according to each data fragment, the characteristic value calculated based on each data fragment is the same as the obtained characteristic value, and the verification value of each data fragment is generated; receiving a first data fragment of the target object and a first check value of the first data fragment, which are sent by the server;
the generating module is used for generating a check value of the first data fragment;
the verification module is used for comparing the verification value of the first data fragment with the first verification value to determine whether the first data fragment is accurate or not; the algorithm for generating the check value of the first data fragment is the same as the algorithm for generating the first check value; when the client determines that the first data fragment is accurate, sending correct indication information to the server, where the correct indication information is used to indicate the server to send a second data fragment and a second check value of the second data fragment to the client, and the second data fragment is a data fragment that is adjacent to and subsequent to the first data fragment in the multiple data fragments of the target object.
12. The apparatus of claim 11, further comprising:
and the information generation module is used for generating error indication information for indicating that the target object is in error when the first data fragment is determined to be inaccurate.
13. The apparatus of claim 11 or 12, wherein the receiving module is further configured to receive a second data slice of the target object and a second parity value of the second data slice;
the generating module is further configured to generate a check value of the second data fragment;
the check module is further configured to compare a check value of the second data fragment with the second check value to determine whether the second data fragment is accurate; wherein an algorithm for generating the check value of the second data slice is the same as an algorithm for generating the second check value.
14. The apparatus of claim 13, wherein the generating module is configured to generate the check value of the second data slice according to the check value of the first data slice and the check value of the second data slice.
15. The apparatus of claim 13, wherein the generating module is configured to generate the check value of the first data fragment according to the first data fragment and a key of the client.
16. A data transmission apparatus, characterized in that the apparatus comprises:
the fragment acquisition module is used for receiving a downloading request sent by a client; acquiring a plurality of data fragments of a target object and the characteristic value of each data fragment according to the downloading request; respectively calculating the characteristic value of each data fragment according to each data fragment;
the generating module is used for generating a first check value of the first data fragment based on the feature value calculated by each data fragment being the same as the obtained feature value;
a sending module, configured to send a first data fragment of the target object and a first check value of the first data fragment to the client;
the sending module is further configured to receive correct indication information sent by the client, and send a second data fragment of the target object and a second check value of the second data fragment to the client, where the second data fragment is a data fragment that is adjacent to and subsequent to the first data fragment in the multiple data fragments of the target object.
17. The apparatus of claim 16,
the fragment acquisition module is further used for acquiring a second data fragment of the target object;
the generating module is further configured to generate a second check value of the second data segment.
18. The apparatus of claim 17, wherein the generating module is configured to generate the second parity value for the second data segment based on the first parity value for the first data segment and the second data segment.
19. The apparatus of claim 16, wherein the generating module is configured to generate the check value of the first data segment according to the first data segment and a key of the client.
20. The apparatus of claim 16, further comprising:
the modification module is used for reading the stored data fragments when a modification request for any data fragment is received, and modifying the data fragments in response to the modification request to obtain a new data fragment;
and the storage module is used for replacing the data fragments and the corresponding characteristic values with the new data fragments and the corresponding characteristic values.
21. A client, characterized in that the client comprises an interface and one or more processors for performing the method of any one of claims 1 to 5.
22. A server, characterized in that the server comprises an interface and one or more processors for performing the method of any one of claims 6 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910995678.0A CN112688905B (en) | 2019-10-18 | 2019-10-18 | Data transmission method, device, client, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910995678.0A CN112688905B (en) | 2019-10-18 | 2019-10-18 | Data transmission method, device, client, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112688905A CN112688905A (en) | 2021-04-20 |
| CN112688905B true CN112688905B (en) | 2023-04-18 |
Family
ID=75445016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910995678.0A Active CN112688905B (en) | 2019-10-18 | 2019-10-18 | Data transmission method, device, client, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112688905B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115150133A (en) * | 2022-06-20 | 2022-10-04 | 浙江大华技术股份有限公司 | Data transmission method and device and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102025444A (en) * | 2009-09-09 | 2011-04-20 | 腾讯科技（深圳）有限公司 | Resource publishing method, resource downloading method and data verification system |
| CN103294569A (en) * | 2013-04-27 | 2013-09-11 | 深圳市雄帝科技股份有限公司 | COS version check method and check device of intelligent card |
| CN103957080A (en) * | 2014-04-30 | 2014-07-30 | 浙江中控技术股份有限公司 | Data integrity checking method and system |
| CN105550276A (en) * | 2015-12-10 | 2016-05-04 | 广东欧珀移动通信有限公司 | Method and apparatus for judging completeness of audio upgrading file in network |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| CN107807792A (en) * | 2017-10-27 | 2018-03-16 | 郑州云海信息技术有限公司 | A kind of data processing method and relevant apparatus based on copy storage system |
| CN109492436A (en) * | 2018-11-29 | 2019-03-19 | 深圳市网心科技有限公司 | A kind of file verification method and relevant apparatus |
-
2019
- 2019-10-18 CN CN201910995678.0A patent/CN112688905B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102025444A (en) * | 2009-09-09 | 2011-04-20 | 腾讯科技（深圳）有限公司 | Resource publishing method, resource downloading method and data verification system |
| CN103294569A (en) * | 2013-04-27 | 2013-09-11 | 深圳市雄帝科技股份有限公司 | COS version check method and check device of intelligent card |
| CN103957080A (en) * | 2014-04-30 | 2014-07-30 | 浙江中控技术股份有限公司 | Data integrity checking method and system |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| CN105550276A (en) * | 2015-12-10 | 2016-05-04 | 广东欧珀移动通信有限公司 | Method and apparatus for judging completeness of audio upgrading file in network |
| CN107807792A (en) * | 2017-10-27 | 2018-03-16 | 郑州云海信息技术有限公司 | A kind of data processing method and relevant apparatus based on copy storage system |
| CN109492436A (en) * | 2018-11-29 | 2019-03-19 | 深圳市网心科技有限公司 | A kind of file verification method and relevant apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112688905A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111414334B (en) | File fragment uploading method, device, equipment and storage medium based on cloud technology | |
| CN112866310B (en) | CDN back-to-source verification method and verification server, and CDN cluster | |
| US20200128075A1 (en) | System and method for service level agreement based data verification | |
| CN109597717B (en) | Data backup and recovery method and device, electronic equipment and storage medium | |
| CN104410692B (en) | A kind of method and system uploaded for duplicate file | |
| CN110716895A (en) | Target data archiving method and device, computer equipment and medium | |
| CN112565393B (en) | File uploading method, downloading method, device, computer equipment and storage medium | |
| WO2017215646A1 (en) | Data transmission method and apparatus | |
| US9883006B2 (en) | Techniques to transfer large collection containers | |
| WO2020147403A1 (en) | Cloud storage based file processing method, system and computer device | |
| CN113273163A (en) | File uploading method, file downloading method and file management device | |
| WO2017097106A1 (en) | Method and apparatus for transmitting file difference | |
| CN109905479B (en) | File transmission method and device | |
| WO2019057023A1 (en) | Data recovery method, sending/receiving apparatus, and computer-readable storage medium | |
| US20110055179A1 (en) | System for and method of partial file hashing | |
| CN112688905B (en) | Data transmission method, device, client, server and storage medium | |
| CN110889143A (en) | File verification method and device | |
| CN106203179B (en) | A kind of completeness check system and method for pair of file | |
| CN114020522A (en) | Data backup method and device, electronic equipment and system | |
| CN111464258B (en) | Data verification method, device, computing equipment and medium | |
| CN111245949A (en) | File filing and transmission method, device and equipment | |
| CN115297104B (en) | File uploading method and device, electronic equipment and storage medium | |
| CN113806815B (en) | File signing method and system | |
| CN113900990A (en) | File fragment storage method, device, equipment and storage medium | |
| CN113986835A (en) | Management method, device, equipment and storage medium for FastDFS distributed files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220211 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: No.1899 Xiyuan Avenue, high tech Zone (West District), Chengdu, Sichuan, 611730 Applicant before: Chengdu Huawei Technologies Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |