CN112968763A - Method, device and system for transmitting data - Google Patents
Method, device and system for transmitting data Download PDFInfo
- Publication number
- CN112968763A CN112968763A CN201911279982.1A CN201911279982A CN112968763A CN 112968763 A CN112968763 A CN 112968763A CN 201911279982 A CN201911279982 A CN 201911279982A CN 112968763 A CN112968763 A CN 112968763A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud
- storage server
- source data
- check code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
- H04L63/0442—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload wherein the sending and receiving network entities apply asymmetric encryption, i.e. different keys for encryption and decryption
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0618—Block ciphers, i.e. encrypting groups of characters of a plain text message using fixed encryption transformation
- H04L9/0631—Substitution permutation network [SPN], i.e. cipher composed of a number of stages or rounds each involving linear and nonlinear transformations, e.g. AES algorithms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
Abstract
The embodiment of the disclosure discloses a method, a device and a system for transmitting data. One embodiment of the system comprises: the mobile storage server is configured to respond to a copy task which comprises a source data path and is input by a user, execute the copy task in parallel through at least one USB device and/or NFS file system and carry out stream encryption on copied data, and after copying is finished, the mobile storage server is transported to a machine room where the cloud-end controller is located; the cloud-end controller is configured to decrypt data in the mobile storage server to obtain source data, generate a check code according to the source data, and upload the source data with the check code to the cloud storage server through the intranet; and the cloud storage server is configured to perform data consistency verification on the received source data, and if the verification fails, a retransmission instruction is sent to the cloud-end controller. The bandwidth cost of the embodiment is reduced, the transmission speed is increased, and the data safety is guaranteed.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method, an apparatus, and a system for transmitting data.
Background
With the development of the internet era, the data scale of enterprises continuously shows the leap of magnitude order, the data scale is developed from the original GB level to the TB, PB or even EB level, and the explosive growth of the data volume brings data storage pressure to the enterprises. Meanwhile, due to the popularization of the cloud computing technology, cloud end of data storage becomes a mainstream choice, and the cloud storage can provide safe and reliable storage service with high mass expansion and data disaster tolerance for enterprises and reduce cost.
The PB level data and even the EB level data are transmitted from the local data center to the cloud storage system, so that high cost and long-time consumption project is achieved, new challenges are brought to mass data transmission and cloud data migration due to data explosion, and the traditional data migration scheme is difficult to deal with the transmission scene of mass data
At present, there are several ways for mass Data to be transmitted from a local IDC (Internet Data Center) to a cloud Data Center in the industry:
the first is to set up a special line of a point-to-point network, data is transmitted by an external network, a migration cluster needs to be deployed, resources such as bandwidth are consumed in the processes of reading source data and writing destination data, the overall migration period is long, the consumed time is very long, and the bandwidth cost and the special line setting cost are high. The cost of extranet bandwidth is very expensive relative to the cost of logistics transportation.
The second method is that data is copied to a hard disk in a local data center, a plurality of hard disks are sent to an enterprise logistics at one time, the cloud end receives the hard disks, then the data is read and uploaded to a cloud storage server, and the mode that the number of the hard disks sent at one time is limited, and the data security is low. The method is lower in transmission cost than a private network, but the data is transported in a clear text, and the safety problems of data damage, data leakage and the like exist in the logistics process; and the method does not support various data sources, only can copy data from a local file system to a hard disk, cannot manage the task state in the copying process, is easy to cause data omission and data repetition, and wastes storage space.
The third is data mirroring back to the source, a passive data migration triggering mode, when a user accesses cloud storage data, the user accesses local data center services in real time, pulls the data to cloud storage, and is suitable for incremental data and a small amount of data to be transmitted to the cloud. And the method cannot be applied to the cloud of mass data transmission.
Disclosure of Invention
The embodiment of the disclosure provides a method, a device and a system for transmitting data.
In a first aspect, an embodiment of the present disclosure provides a system for transmitting data, including: the mobile storage server is configured to respond to a copy task which comprises a source data path and is input by a user, execute the copy task in parallel through at least one USB device and/or an NFS file system and perform streaming encryption on copied data, and after the copy is completed, the mobile storage server is transported to a machine room where the cloud-end controller is located; the cloud-end controller is configured to decrypt data in the mobile storage server to obtain source data, generate a check code according to the source data, and upload the source data with the check code to the cloud storage server through the intranet; and the cloud storage server is configured to perform data consistency verification on the received source data, and if the verification fails, a retransmission instruction is sent to the cloud controller.
In some embodiments, in response to receiving a user input copy job including a source data path, the copying method includes: responding to and receiving an instruction which is input by a user through a web end and used for copying data from a source data path, encrypting the instruction in an asymmetric mode and then sending the encrypted instruction to a server end; and the server decrypts the data and analyzes the copy task.
In some embodiments, the mobile storage server and the web-side respectively require password login.
In some embodiments, generating the check code from the source data comprises: if the source data is larger than a preset first threshold value, blocking the source data into at least one small file according to a preset format; and for each small file in the at least one small file, if the small file is smaller than a preset second threshold value, generating a check code through an MD5 digest algorithm, otherwise, generating the check code through a CRC algorithm.
In some embodiments, the cloud-side controller is further configured to: and retransmitting the file failed in verification in response to receiving a retransmission instruction sent by the cloud storage server.
In some embodiments, the mobile storage server has at least one network port, at least one USB interface, at least one hard disk, and RAM reaching a predetermined memory threshold, wherein the at least one hard disk guarantees data security through RAID 5.
In a second aspect, an embodiment of the present disclosure provides a method for transmitting data, applied to a mobile storage server, including: receiving a copy task input by a user and comprising a source data path; executing copy tasks in parallel by at least one USB device and/or NFS file system; carrying out stream encryption on the copied data; and sending the data to the cloud-end controller.
In a third aspect, an embodiment of the present disclosure provides a method for transmitting data, applied to a cloud-end controller, including: receiving data from a mobile storage server; decrypting the data to obtain source data; generating a check code according to the source data; and uploading the source data with the check code to a cloud storage server through an intranet.
In a fourth aspect, an embodiment of the present disclosure provides a method for transmitting data, applied to a cloud storage server, including: carrying out data consistency check on the received source data; and if the verification fails, sending a retransmission instruction to the cloud-end controller.
In a fifth aspect, an embodiment of the present disclosure provides an apparatus for transmitting data, applied to a mobile storage server, including: a receiving unit configured to receive a copy task including a source data path input by a user; an execution unit configured to execute a copy task in parallel through at least one USB device and/or NFS file system; an encryption unit configured to stream-encrypt the copied data; a transmitting unit configured to transmit the data to the cloud-side controller.
In a sixth aspect, an embodiment of the present disclosure provides an apparatus for transmitting data, applied to a cloud-end controller, including: the decryption unit is configured to receive the data from the mobile storage server and decrypt the data to obtain source data; a generating unit configured to generate a check code from the source data; and the uploading unit is configured to upload the source data with the check code to the cloud storage server through the intranet.
In a seventh aspect, an embodiment of the present disclosure provides an apparatus for transmitting data, applied to a cloud storage server, including: the verification unit is configured to perform data consistency verification on the received source data; and the retransmission unit is configured to send a retransmission instruction to the cloud-end controller if the verification fails.
In an eighth aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement the method as in any one of the second to fourth aspects.
In a ninth aspect, embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method of any one of the second to fourth aspects.
According to the method, the device and the system for transmitting data, a special data storage device and a data migration software system are designed to provide a high-speed and safe data cloud transmission scheme for customers, and the method, the device and the system are widely applied to data cloud storage in data migration of data centers of small and medium-sized enterprises; compared with the traditional network dedicated line or logistics hard disk mode, the bandwidth cost is reduced, the transmission speed is improved, and the data safety is guaranteed.
The system for transmitting data is composed of a mobile storage server, a cloud end controller and a cloud storage server. The mobile storage server adopts software disk array RAID5 level to ensure data security, and gives consideration to data security and storage cost, and any N-1 hard disk has complete data. The system for transmitting data provides a high-concurrency and multi-task data transmission strategy, and the transmission speed is improved by fully utilizing the characteristics of multiple network cards and multiple disks of equipment; meanwhile, various data sources such as a local storage device and an NFS network file system are supported to be transmitted, and the multi-scene requirements of clients are met; the data transmission system supports task start-stop and state management, provides data AES256 end-to-end encryption of migration task granularity, and ensures safety in data storage and transmission processes.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
FIG. 2 is a flow diagram for one embodiment of a method for transmitting data, according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a method for transmitting data according to the present disclosure;
FIG. 4 is a structural schematic diagram of one embodiment of an apparatus for transmitting data according to the present disclosure;
FIG. 5 is a schematic block diagram of yet another embodiment of an apparatus for transmitting data according to the present disclosure;
FIG. 6 is a schematic block diagram of another embodiment of an apparatus for transmitting data according to the present disclosure;
FIG. 7 is a schematic block diagram of a computer system suitable for use with an electronic device implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and the features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates a system architecture 100 for transmitting data to which the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include a mobile storage server 101, a cloud controller 102, and a cloud storage server 103. The mobile storage server 101 can be transported from the vicinity of a data source to a machine room where the cloud-end controller 102 is located through logistics, and is directly connected with the cloud-end controller 102. The cloud-side controller 102 may upload data to the cloud storage server 103 via the intranet.
The mobile storage server 101 comprises two parts of a storage device and transmission software:
the storage device is a customized portable computing and storing integrated server, seven 10Gbps high-bandwidth network ports and six 3.0USB interfaces can be built in the storage device, and high-speed parallel network and disk transmission speed are provided. The device can also provide 9 × 4TB SATA (Serial ATA, Serial hard Disk) III hard Disk storage, 500GB SSD (Solid State Disk) storage, and up to 256GB RAM (Random Access Memory), the multiple hard disks guarantee the speed of concurrently reading and writing data, and the SSD fast storage device and sufficient RAM guarantee the operating speed of the data transfer software system and the execution speed of the migration task. Meanwhile, the equipment provides high-level safety guarantee, is dustproof, waterproof, anti-seismic and pressure-resistant, and can sufficiently deal with the data damage risk caused by impact, extrusion and the like possibly occurring in the logistics transportation process; the 9 disks adopt RAID (Redundant Arrays of Independent Drives) 5-level software disk Arrays to ensure data security (the soft RAID completes the RAID function by using an operating system and a CPU, and presents a virtual RAID volume configured by a software layer to a user, so that the user can use the RAID volume like using a common disk), complete data on any N-1 disks is ensured, and storage performance, data security and storage cost are considered.
The transmission software provides a multitask state management mechanism and a humanized web operation interface, allows a plurality of migration tasks to be started simultaneously, different tasks run on different threads, and makes full use of the parallel throughput capacity of a plurality of disks; the local database manages the migration multitask state, supports breakpoint continuous transmission and updates and maintains the transmission progress in real time; the migration tasks support various data sources such as a local disk, a USB device and an NFS (Network File System) File System, the migration tasks of different data sources are not affected with each other, the USB device detects mounting in real time, and multi-scene requirements of clients are met.
The transmission system is provided with a multiple encryption mechanism, a random password is generated for startup login when the equipment is initialized, and the transmission software needs to use the password to verify the login again. The two passwords may be the same or different. The migration task provides single task granularity data encryption, and the data of each copy task is encrypted respectively. All the passwords of the user are stored in a password MD5 summary and salt adding mode and then compared, and during login, the summary comparison is calculated in the same mode to verify the login result. The system login adopts the full-memory session to maintain the login state, the login state can be reused only during the successful login, and password login needs to be repeated when the browser is turned on or off or restarted, so that the condition that a hacker invades and other non-self login situations is avoided. The data and instruction transmission between the Web end and the server end can adopt an RSA asymmetric encryption mode, a public key is used for encrypting data at the Web end, and a private key is used for decrypting data at the server end. Data of the migration task are encrypted and decrypted in an AES256 streaming mode, so that the encryption efficiency is improved; and simultaneously, the task encryption key is also encrypted and stored by AES256, so that all forms of plaintext are avoided.
After the mobile storage server is copied, the user can express the mobile storage server to a cloud service provider. And uploading the data stored in the mobile storage server to the cloud storage server through the cloud end controller 102 by the cloud service provider.
The cloud-end controller 102 supports block parallel transmission of large files, improves transmission efficiency of the large files and reduces failure probability. And if the source data is larger than a preset first threshold value, partitioning the source data into at least one small file according to a preset format. And a perfect data consistency checking mechanism is provided, and data error transmission and data missing transmission are avoided. The MD5 digest Check is directly performed on the files smaller than the preset second threshold, and the files larger than or equal to the preset second threshold are checked through CRC (Cyclic Redundancy Check), for example, CRC32 value, so that the calculation efficiency is ensured, and meanwhile, the data consistency is also checked.
The cloud storage server 103 is a storage server of the distributor. The cloud storage server and the cloud controller are connected through an intranet, so that the transmission speed is high. When the cloud storage server receives the uploaded data, consistency verification is firstly carried out, and if the verification fails, a retransmission instruction is sent to the cloud controller. The retransmission instruction can include the identification of the failed data block, and the cloud-end controller retransmits the data block which is failed to be checked.
It should be noted that the mobile storage server 101, the cloud end controller 102, and the cloud storage server 103 execute the method for transmitting data according to the embodiments of the present disclosure, and accordingly, the apparatus for transmitting data is generally disposed in the mobile storage server 101, the cloud end controller 102, and the cloud storage server 103.
It should be understood that the number of mobile storage servers, cloud-side controllers, and cloud storage servers in fig. 1 is merely illustrative. Any number of mobile storage servers, cloud-side controllers, and cloud storage servers may be provided according to implementation needs.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for transmitting data in accordance with the present disclosure is shown. The method for transmitting data comprises the following steps:
in step 201, the mobile storage server receives a copy task including a source data path input by a user.
In this embodiment, the user may enter a copy task including the source data path through a command line or web interface. A plurality of copy tasks can be set simultaneously, and the user can specify the copy mode, such as NFS or USB. The copy mode can also be selected by the mobile storage server according to the file size, so that the completion time of different tasks is approximately the same.
In step 202, the mobile storage server executes the copy task in parallel through at least one USB device and/or NFS file system.
In this embodiment, the file can be copied in parallel by at least one USB device, and can also be copied in parallel by the NFS file system. It is also possible to combine the two approaches together for parallel copying.
And step 203, the mobile storage server performs streaming encryption on the copied data.
In this embodiment, the copied data may be encrypted and decrypted in a streaming manner by using an algorithm such as AES 256.
And step 204, the mobile storage server sends the data to the cloud-end controller.
In this embodiment, after the source data to be copied is copied and encrypted, the mobile storage server can be transported to the machine room where the cloud-end controller is located. And then, quickly copying the source data to the cloud-end controller in parallel in a plurality of modes such as at least one USB device and/or an NFS file system.
And step 205, the cloud end controller receives the data from the mobile storage server, and decrypts the data to obtain the source data.
In this embodiment, the data received by the cloud controller is encrypted and needs to be decrypted to upload to the cloud. After decryption, the large file can be transmitted in parallel in a blocking mode. And if the source data is larger than a preset first threshold value, blocking the source data into at least one small file according to a preset format.
And step 206, the cloud terminal controller generates a check code according to the source data.
In this embodiment, for each of the at least one small file, if the small file is smaller than the predetermined second threshold, the check code is generated through the MD5 digest algorithm, otherwise, the check code is generated through the CRC algorithm. CRC32 may be employed. Other CRC check algorithms may also be selected based on the data length.
And step 207, the cloud end controller uploads the source data with the check code to a cloud storage server through an intranet.
In this embodiment, the cloud-side controller uploads the source data with the check code to the cloud storage server through the switch of the intranet. The speed of the intranet switch can reach giga.
And step 208, the cloud storage server performs data consistency check on the received source data.
In this embodiment, the cloud storage server may determine which check method is adopted according to the size of the received data, calculate the corresponding check code, and compare the calculated check code with the received check code, and if the calculated check code is not consistent with the received check code, it indicates that the data is erroneously transmitted or missed.
Step 209, if the cloud storage server fails to verify, a retransmission instruction is sent to the cloud end controller.
In this embodiment, if the verification fails, a retransmission instruction is sent to the cloud-side controller, and the retransmission instruction may include a failed data identifier. If the verification is successful, an uploading success instruction can be sent to the cloud end controller, and the cloud end controller can delete the successfully uploaded data.
Step 210, the cloud-side controller retransmits the source data with the check code and failed in check
In this embodiment, the cloud-side controller always stores data that is not successfully transmitted, and performs retransmission when receiving a retransmission instruction. Until the data can be deleted after the uploading success command is received.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the method for transmitting data according to the present embodiment. In the application scenario of fig. 3, when a user wants to upload a large amount of data, the user may apply for the mobile storage server from the cloud service provider. The cloud service provider express the mobile storage server with the set boot password and the set system password to the user and additionally inform the user of the two passwords. And after receiving the mobile storage server, the user logs in the system by using the password, and adds a copy task through the web interface to enable the USB equipment and the NFS to be copied at the same time. And the server side of the mobile storage server inserts the copy task into the MySQL database. And then the file is read according to the copying task to be copied. And encrypting the copied file and then storing the encrypted file in a black box. And after all the copying tasks are finished and encrypted, the mobile storage server is then express-delivered back to the cloud service provider. And the cloud service provider directly connects the mobile storage server with the cloud end controller, blocks the data, adds the check code, and uploads the data to the cloud storage server.
The main advantages of the invention are as follows:
1. high transmission speed, low cost, large storage space and high expansion
a. The data volume with the same size, the logistics speed is far higher than the transmission speed of an external network, and the migration speed is greatly improved.
b. The logistics transportation cost is far lower than the bandwidth cost of the external network.
c. The data storage space of a single device is up to 30TB-100TB, and the simultaneous use of a plurality of devices is supported, so that the PB level data migration is more than enough.
2. Data multiple encryption, safety and reliability
a. All data and encryption keys are encrypted end to end by adopting AES256, and RSA asymmetric encryption is adopted between the web side and the service side.
Crc32 and MD5 digests guarantee data consistency.
3. The system has the advantages of perfect multi-task management mechanism, support of diversified data sources, portable hardware equipment and humanized migration software system.
With further reference to fig. 4, as an implementation of the methods shown in the above-mentioned figures, the present disclosure provides an embodiment of an apparatus for transmitting data, which is applied to a mobile storage server, and the apparatus embodiment corresponds to the method embodiment shown in fig. 2, and the apparatus may be applied to various electronic devices in particular.
As shown in fig. 4, the apparatus 400 for transmitting data of the present embodiment includes: a receiving unit 401 configured to receive a copy task including a source data path input by a user; an execution unit 402 configured to execute copy tasks in parallel by at least one USB device and/or NFS file system; an encryption unit 403 configured to stream-encrypt the copied data; a sending unit 404 configured to send the data to the cloud-side controller.
With further reference to fig. 5, as an implementation of the method shown in the above-mentioned figures, the present disclosure provides an embodiment of an apparatus for transmitting data, which is applied to a cloud-side controller, and the apparatus embodiment corresponds to the method embodiment shown in fig. 2, and the apparatus may be applied to various electronic devices in particular.
As shown in fig. 5, the apparatus 500 for transmitting data of the present embodiment includes: the decryption unit 501 is configured to receive data from the mobile storage server and decrypt the data to obtain source data; a generating unit 502 configured to generate a check code according to the source data; an uploading unit 503 configured to upload the source data with the check code to the cloud storage server through the intranet.
With further reference to fig. 6, as an implementation of the method shown in the above-mentioned figures, the present disclosure provides an embodiment of an apparatus for transmitting data, which is applied to a cloud storage server, and the apparatus embodiment corresponds to the method embodiment shown in fig. 2, and the apparatus may be applied to various electronic devices in particular.
As shown in fig. 6, the apparatus 600 for transmitting data of the present embodiment includes: the verification unit 601 is configured to perform data consistency verification on the received source data; the retransmission unit 602 is configured to send a retransmission instruction to the cloud-side controller if the verification fails.
Referring now to fig. 7, a schematic diagram of an electronic device (e.g., the mobile storage server or the cloud-side controller or the cloud storage server of fig. 1) 700 suitable for implementing embodiments of the present disclosure is shown. The mobile storage server or the cloud end controller or the cloud storage server shown in fig. 7 is only one example, and should not bring any limitation to the functions and the use range of the embodiments of the present disclosure.
As shown in fig. 7, the electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage means 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 illustrates an electronic device 700 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 7 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication means 709, or may be installed from the storage means 708, or may be installed from the ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of embodiments of the present disclosure. It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in embodiments of the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a copy task input by a user and comprising a source data path; executing copy tasks in parallel by at least one USB device and/or NFS file system; carrying out stream encryption on the copied data; and sending the data to the cloud-end controller. Or cause the electronic device to: receiving data from a mobile storage server, and decrypting the data to obtain source data; generating a check code according to the source data; and uploading the source data with the check code to a cloud storage server through an intranet. Or cause the electronic device to: carrying out data consistency check on the received source data; and if the verification fails, sending a retransmission instruction to the cloud-end controller.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a receiving unit, an executing unit, an encrypting unit, and a transmitting unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, a receiving unit may also be described as a "unit that receives user input including a copy task of a source data path".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by a person skilled in the art that the scope of the invention as referred to in the present disclosure is not limited to the embodiments with a specific combination of the above-mentioned features, but also covers other embodiments with any combination of the above-mentioned features or their equivalents without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Claims (14)
1. A system for transmitting data, comprising:
the mobile storage server is configured to respond to a copy task which comprises a source data path and is input by a user, execute the copy task in parallel through at least one USB device and/or NFS file system and carry out stream encryption on copied data, and after copying is finished, the mobile storage server is transported to a machine room where the cloud-end controller is located;
the cloud end controller is configured to decrypt data in the mobile storage server to obtain source data, generate a check code according to the source data, and upload the source data with the check code to the cloud storage server through an intranet;
the cloud storage server is configured to perform data consistency verification on the received source data, and if the verification fails, a retransmission instruction is sent to the cloud end controller.
2. The system of claim 1, wherein the copy task including a source data path in response to receiving user input comprises:
responding to and receiving an instruction which is input by a user through a web end and used for copying data from a source data path, encrypting the instruction in an asymmetric mode and then sending the encrypted instruction to a server end;
and the server decrypts the data and analyzes the copy task.
3. The system of claim 2, wherein the mobile storage server and the web-side each require a password login.
4. The system of claim 1, wherein the generating a check code from the source data comprises:
if the source data is larger than a preset first threshold value, blocking the source data into at least one small file according to a preset format;
and for each small file in the at least one small file, if the small file is smaller than a preset second threshold value, generating a check code through an MD5 digest algorithm, otherwise, generating the check code through a CRC algorithm.
5. The system of claim 4, wherein the cloud-side controller is further configured to:
and retransmitting the file failed in verification in response to receiving a retransmission instruction sent by the cloud storage server.
6. The system according to one of claims 1 to 5, wherein the mobile storage server is internally provided with at least one network port, at least one USB interface, at least one hard disk and RAM reaching a predetermined memory threshold, wherein the at least one hard disk guarantees data security through RAID 5.
7. A method for transmitting data is applied to a mobile storage server and comprises the following steps:
receiving a copy task input by a user and comprising a source data path;
executing copy tasks in parallel by at least one USB device and/or NFS file system;
carrying out stream encryption on the copied data;
and sending the data to a cloud-end controller.
8. A method for transmitting data is applied to a cloud-end controller and comprises the following steps:
receiving data from a mobile storage server, and decrypting the data to obtain source data;
generating a check code according to the source data;
and uploading the source data with the check code to a cloud storage server through an intranet.
9. A method for transmitting data is applied to a cloud storage server and comprises the following steps:
carrying out data consistency check on the received source data;
and if the verification fails, sending a retransmission instruction to the cloud-end controller.
10. An apparatus for transmitting data, applied to a mobile storage server, comprising:
a receiving unit configured to receive a copy task including a source data path input by a user;
an execution unit configured to execute a copy task in parallel through at least one USB device and/or NFS file system;
an encryption unit configured to stream-encrypt the copied data;
a transmitting unit configured to transmit the data to a cloud-side controller.
11. The device for transmitting data is applied to a cloud-end controller and comprises the following components:
the decryption unit is configured to receive data from the mobile storage server and decrypt the data to obtain source data;
a generating unit configured to generate a check code from the source data;
and the uploading unit is configured to upload the source data with the check code to the cloud storage server through the intranet.
12. An apparatus for transmitting data, applied to a cloud storage server, comprises:
the verification unit is configured to perform data consistency verification on the received source data;
and the retransmission unit is configured to send a retransmission instruction to the cloud-end controller if the verification fails.
13. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 7-9.
14. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 7-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911279982.1A CN112968763B (en) | 2019-12-13 | 2019-12-13 | Method, device and system for transmitting data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911279982.1A CN112968763B (en) | 2019-12-13 | 2019-12-13 | Method, device and system for transmitting data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112968763A true CN112968763A (en) | 2021-06-15 |
| CN112968763B CN112968763B (en) | 2023-04-18 |
Family
ID=76270772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911279982.1A Active CN112968763B (en) | 2019-12-13 | 2019-12-13 | Method, device and system for transmitting data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112968763B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104166520A (en) * | 2013-05-20 | 2014-11-26 | 深圳先进技术研究院 | Distributed hard disk system and data migration method thereof |
| US20170085372A1 (en) * | 2015-09-17 | 2017-03-23 | Secturion Systems, Inc. | Client(s) to cloud or remote server secure data or file object encryption gateway |
| CN109254734A (en) * | 2018-09-06 | 2019-01-22 | 郑州云海信息技术有限公司 | A kind of date storage method, device, equipment and computer readable storage medium |
| CN109873784A (en) * | 2017-12-01 | 2019-06-11 | 广州明领基因科技有限公司 | Mixed cloud secure storage management system towards big data |
-
2019
- 2019-12-13 CN CN201911279982.1A patent/CN112968763B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104166520A (en) * | 2013-05-20 | 2014-11-26 | 深圳先进技术研究院 | Distributed hard disk system and data migration method thereof |
| US20170085372A1 (en) * | 2015-09-17 | 2017-03-23 | Secturion Systems, Inc. | Client(s) to cloud or remote server secure data or file object encryption gateway |
| CN109873784A (en) * | 2017-12-01 | 2019-06-11 | 广州明领基因科技有限公司 | Mixed cloud secure storage management system towards big data |
| CN109254734A (en) * | 2018-09-06 | 2019-01-22 | 郑州云海信息技术有限公司 | A kind of date storage method, device, equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112968763B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9864655B2 (en) | Methods and apparatus for mobile computing device security in testing facilities | |
| CN111460453B (en) | Machine learning training method, controller, device, server, terminal and medium | |
| US9231923B1 (en) | Secure data destruction in a distributed environment using key protection mechanisms | |
| US10178077B2 (en) | Preventing persistent storage of cryptographic information using signaling | |
| CN103051664B (en) | A kind of file management method of cloud storage system, device and this cloud storage system | |
| US11657171B2 (en) | Large network attached storage encryption | |
| JP2019061694A (en) | Data transfer optimization | |
| US10958416B2 (en) | Encrypted and compressed data transmission with padding | |
| US11119797B2 (en) | Active drive API | |
| JP5975501B2 (en) | Mechanisms that promote storage data encryption-free integrity protection in computing systems | |
| US10530752B2 (en) | Efficient device provision | |
| CN103118089A (en) | Safe storage method based on a plurality of cloud storage systems and system thereof | |
| US10621055B2 (en) | Adaptive data recovery for clustered data devices | |
| US9288042B1 (en) | Securely and redundantly storing encryption credentials system and method | |
| US10924275B1 (en) | Creating multiple encrypted volumes from a single source | |
| WO2021031655A1 (en) | Method and apparatus for upgrading blockchain network, storage medium, and electronic device | |
| TW201440475A (en) | Streaming ZIP | |
| US11347395B2 (en) | Cache management system and method | |
| US9356782B2 (en) | Block encryption | |
| US10223538B1 (en) | Preventing persistent storage of cryptographic information | |
| CN112968763B (en) | Method, device and system for transmitting data | |
| CN114661420B (en) | Application protection method, device and system based on Kubernetes container platform | |
| US10348705B1 (en) | Autonomous communication protocol for large network attached storage | |
| US11558190B2 (en) | Using keys for selectively preventing execution of commands on a device | |
| JP2016151797A (en) | Information processing system, control method thereof, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |