CN112732175B - Data transmission method and system - Google Patents
Data transmission method and system Download PDFInfo
- Publication number
- CN112732175B CN112732175B CN202011566785.0A CN202011566785A CN112732175B CN 112732175 B CN112732175 B CN 112732175B CN 202011566785 A CN202011566785 A CN 202011566785A CN 112732175 B CN112732175 B CN 112732175B
- Authority
- CN
- China
- Prior art keywords
- data
- transmitted
- storage position
- storage
- hot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012986 modification Methods 0.000 claims abstract description 22
- 230000004048 modification Effects 0.000 claims abstract description 22
- 230000015654 memory Effects 0.000 claims description 32
- 241001025261 Neoraja caerulea Species 0.000 claims description 10
- 238000012216 screening Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 abstract description 3
- 238000005457 optimization Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000001152 differential interference contrast microscopy Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data transmission method and a system, wherein the method comprises the following steps: acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects; sequencing the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted; according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted; and transmitting the plurality of objects to be transmitted to the target storage position in parallel. And ordering the data to be transmitted according to the last modification time of each storage object to generate a list to be transmitted. After the list to be transmitted is generated, the transmission data is decomposed into a plurality of objects to be transmitted for transmission, so that the data transmission mode is optimized, and the data transmission efficiency is improved.
Description
Technical Field
The invention relates to the field of big data storage and blue light storage, in particular to a data transmission method and system.
Background
In the existing Amazon S3 distributed system, a special performance optimization strategy is provided for data transmission or storage type conversion of a storage object in a large throughput so as to improve data transmission efficiency. However, the existing performance optimization strategy can only realize the data processing of the storage object size in the MB level, the GB level and even the TB level, and when a large number of objects with smaller content and KB level are stored in the storage space, when a user needs to perform data transmission on the objects, the existing Amazon S3 distributed system can generate concurrent data transmission congestion, so that the data transmission efficiency is affected.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the defect of slower data transmission speed when the object performs data transmission in the prior art, thereby providing a data transmission method and a data transmission system.
In order to achieve the above purpose, the present invention provides the following technical solutions:
In a first aspect, an embodiment of the present invention provides a data transmission method, including: acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects; ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted; according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted; and transmitting a plurality of objects to be transmitted to a target storage position in parallel.
In an embodiment, the acquiring the data to be transmitted of the original storage location includes: acquiring a hot data file of an original storage position; sorting the hot data files based on the storage space occupied by the hot data files, and screening out the hot data files meeting the KB level range; and merging the screened hot data files to generate data to be transmitted.
In an embodiment, the sorting the hot data files based on the storage space occupied by the hot data files includes: acquiring file parameters of each thermal data file; respectively calculating the storage space occupied by each thermal data file according to the file parameters; and ordering the hot data files according to the storage space.
In one embodiment, the predetermined number is 16.
In an embodiment, when the original storage location is a magnetic disk, the target storage location is a blue-ray memory; when the original storage position is a blue light storage, the target storage position is a magnetic disk.
In an embodiment, before the sorting the data to be transmitted according to the last modification time of each storage object in the data to be transmitted and generating the list to be transmitted, the data transmission method further includes: judging whether the original storage position is a blue light storage or not; and when the original storage position is a blue light storage, transmitting the data to be transmitted to a target storage position in parallel.
In a second aspect, an embodiment of the present invention provides a data transmission system, including: the acquisition module is used for acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects; the ordering module is used for ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted; the decomposition module is used for decomposing the data to be transmitted into a preset number of objects to be transmitted according to the list to be transmitted; and the transmission module is used for transmitting the plurality of objects to be transmitted to the target storage position in parallel.
In one embodiment, the predetermined number is 16.
In a third aspect, an embodiment of the present invention provides a computer readable storage medium, where computer instructions are stored, where the computer instructions are configured to cause the computer to perform the data transmission method according to the first aspect of the embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a computer apparatus, including: the data transmission method comprises the steps of storing computer instructions in a memory and a processor, wherein the memory and the processor are in communication connection, the memory stores the computer instructions, and the processor executes the computer instructions, so that the data transmission method according to the first aspect of the embodiment of the invention is executed.
The technical scheme of the invention has the following advantages:
The data transmission method provided by the invention comprises the following steps: acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects; sequencing the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted; according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted; and transmitting the plurality of objects to be transmitted to the target storage position in parallel. And ordering the data to be transmitted according to the last modification time of each storage object to generate a list to be transmitted. After the list to be transmitted is generated, the transmission data is decomposed into a plurality of objects to be transmitted for transmission, so that the data transmission mode is optimized, and the data transmission efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart showing a specific example of a data transmission method in an embodiment of the present invention;
FIG. 2 is a flowchart of a specific example of acquiring data to be transmitted in an original storage location according to an embodiment of the present invention;
FIG. 3 is a flowchart of one specific example of ordering hot data files in an embodiment of the invention;
FIG. 4 is a flowchart showing another specific example of the data transmission method in the embodiment of the present invention;
FIG. 5 is a schematic block diagram of a specific example of a data transmission system in an embodiment of the present invention;
Fig. 6 is a composition diagram of a specific example of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
An embodiment of the present invention provides a data transmission method, as shown in fig. 1, including the following steps:
Step S1: and acquiring data to be transmitted of the original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects.
In a specific embodiment, as shown in fig. 2, the method for acquiring the data to be transmitted in the original storage location includes the following steps:
step S11: and acquiring the hot data file of the original storage position.
Step S12: and sorting the hot data files based on the storage space occupied by the hot data files, and screening out the hot data files meeting the KB level range.
Step S13: and merging the screened hot data files to generate data to be transmitted.
In the embodiment of the invention, after the hot data files with higher access frequency in the original storage position are acquired, the hot data files are ordered according to the storage space occupied by the hot data files (namely the size of the hot data files), and the hot data files with fewer contents and KB level are screened out. And then, the hot data files with smaller content and KB level are combined into data packets by utilizing the characteristic of large IO throughput capacity in the source node, and the combined data packets are used as data to be transmitted for transmission. Specifically, the size and content of the data packet may be calculated according to parameters such as the destination source of the hot data file, the area of the hot data file, the size of the hot data file, the characteristics of the hot data file, and the available bandwidth between the source and the destination source at the time. Because the existing performance optimization strategy can only realize the data processing of the storage object size in MB level, GB level and even TB level, the effective performance optimization strategy is lacking for small files in KB level. Therefore, in the embodiment of the invention, the small files with smaller content and KB level are combined into the data packet with larger occupied storage space, and the combined data packet is used as data to be transmitted for transmission, so that the data processing of the small files with smaller content and KB level can be realized by utilizing the existing performance optimization strategy. In the embodiment of the invention, the original storage position can be a magnetic disk or a blue-ray memory.
Step S2: and ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted.
In a specific embodiment, before ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted and generating the list to be transmitted, it is further required to determine whether the original storage location is a blue-ray memory, and when the original storage location is not the blue-ray memory, that is, when the original storage location is a magnetic disk, the step of ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted is performed to generate the list to be transmitted.
Step S3: according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted.
In a specific embodiment, because the problem that the data transmission efficiency is affected due to the concurrent data transmission congestion caused by the existing performance optimization strategy when the user performs the storage type conversion is solved, in the embodiment of the invention, the last modification time of each storage object is obtained first, and the data to be transmitted is ordered according to the last modification time of each storage object to generate the list to be transmitted. After the list to be transmitted is generated, the data to be transmitted is decomposed into a preset number of objects to be transmitted, so that the data transmission efficiency is improved. Specifically, the preset number is 16, and experiments prove that when the data to be transmitted is decomposed into 16 sections, the transmission effect is optimal and the transmission efficiency is highest.
Step S4: and transmitting the plurality of objects to be transmitted to the target storage position in parallel.
In a specific embodiment, after the data to be transmitted is decomposed into 16 segments of objects to be transmitted, the 16 segments of objects to be transmitted are written in parallel into a target storage position to achieve the purpose of data transmission, wherein the target storage position can be a magnetic disk or a blue-ray memory.
The data transmission method provided by the invention comprises the following steps: acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects; sequencing the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted; according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted; and transmitting the plurality of objects to be transmitted to the target storage position in parallel. And ordering the data to be transmitted according to the last modification time of each storage object to generate a list to be transmitted. After the list to be transmitted is generated, the transmission data is decomposed into a plurality of objects to be transmitted for transmission, so that the data transmission mode is optimized, and the data transmission efficiency is improved.
In one embodiment, the hot data files are ordered based on the storage space occupied by the hot data files, as shown in fig. 3, and the method comprises the following steps:
Step S121: file parameters of each hot data file are obtained.
Step S122: and respectively calculating the storage space occupied by each hot data file according to the file parameters.
Step S123: the hot data files are ordered according to the storage space.
In a specific embodiment, when sorting the hot data files according to the sizes of the hot data files, file parameters of each hot data file are first acquired, and storage space occupied by each hot data file (i.e., the sizes of the hot data files) is calculated according to the file parameters. In the embodiment of the invention, file parameters of the hot data file include: characteristics of the hot data file, storage area of the hot data file, etc.
In an embodiment, before ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted and generating the list to be transmitted, the data transmission method, as shown in fig. 4, further includes the following steps:
Step S14: judging whether the original storage position is a blue light storage;
step S15: and when the original storage position is the blue light storage, transmitting the data to be transmitted to the target storage position in parallel.
In a specific embodiment, the original storage location may be a magnetic disk or a blue-ray memory, and the target storage location may be a magnetic disk or a blue-ray memory. When the original storage position is a magnetic disk, the target storage position is a blue-ray memory; when the original storage position is a blue light storage, the target storage position is a magnetic disk. Thus, the transmission direction of the data to be transmitted is bidirectional. When the transmission direction of the data to be transmitted is that the data to be transmitted is written into the blue-ray memory by the disk, according to the data transmission method, the fewer KB-level hot data files are subjected to data merging to generate the data to be transmitted, and the data to be transmitted is decomposed into 16 segments of objects to be transmitted for transmission. When the transmission direction of the data to be transmitted is that the data to be transmitted is read out from the blue light memory to the magnetic disk, the data transmission process can be simplified, the data to be transmitted is not required to be decomposed, and the storage optimization efficiency can be improved by using the AWS S3 SELECT. In the embodiment of the invention, the storage format of the data to be transmitted in the blue light memory is CSV, JSON or Apache Parquet. When the to-be-transmitted data with the storage format of CSV, JSON or Apache Parquet is read out by using the AWS S3 SELECT in the Amazon S3 distributed system, the data transmission process is simplified, the storage cost is reduced, and the storage optimization efficiency is also improved.
In the data transmission process, besides the data transmission method to improve the data transmission efficiency, the existing performance optimization strategy of the AWS S3 is utilized to carry out transverse expansion and request parallelization so as to realize high throughput. Among them, amazon S3 is a very large distributed system. To take advantage of its size, parallel requests are typically extended laterally to Amazon S3 service end nodes. This type of extension approach helps to distribute the load to multiple paths throughout the network, in addition to distributing the requests in Amazon S3.
For high throughput transmissions, amazon S3 may use an application that utilizes multiple connections in parallel to GET or PUT data. For example, amazon S3 transmission manager in the AWS Java development kit supports the above functions, most other AWS development kits provide similar constructs. For some applications, parallel connection may be implemented, with the specific method: multiple connections are initiated in parallel in different application threads or different application instances. The best approach to take depends on the application and the structure of the accessed object.
Typically, GET and PUT requests can be issued directly using the AWS development kit, rather than utilizing transport management in the AWS development kit. This approach may adjust the workload more directly while still benefiting from the software toolkit supporting retries and handling any HTTP 503 responses that may occur. In general, when a large object in a region is downloaded from Amazon S3 to Amazon EC2, parallel requests are typically issued to byte ranges of the object at a granularity of 8-16 MB. A parallel request is issued for each required network throughput of 85-90 MB/s. To saturate a 10Gb/s Network Interface Card (NIC), approximately 15 parallel requests may be used over a single connection. Parallel requests may also be extended through more connections to saturate faster NICs, such as 25Gb/s or 100Gb/s NICs.
Performance is important when adjusting the number of requests issued in parallel. Typically, only one request is used at a time as a starting point. The implemented network bandwidth and the use of other resources used by the application to process the data are measured. The number of requests that may be useful is then determined based on the bottleneck resource (i.e., the highest utilization resource). For example, if processing one request at a time would result in CPU utilization up to 25%, up to four parallel requests may be accommodated.
If the application uses REST API to make requests directly to Amazon S3, a pool of HTTP connections can be used and each connection reused for a series of requests. By avoiding the need to request connection-by-connection setup, the need to perform TCP slow start and Secure Socket Layer (SSL) handshakes for each connection can be eliminated.
Finally, care must be taken to pay attention to DNS and to scrutinize that requests are being scattered across a broad pool of Amazon S3IP addresses. The DNS query for Amazon S3 loops through a large list of IP end nodes. But caching of parser or application code reusing a single IP address does not benefit from address diversity and the resulting load balancing. A network utility (e.g., netstat command line tool) may display an IP address to use to communicate with Amazon S3.
The embodiment of the invention also provides a data transmission system, as shown in fig. 5, including:
the acquisition module 1 is configured to acquire data to be transmitted in an original storage location, where the data to be transmitted is formed by a plurality of storage objects in KB stages. The details are referred to the related description of step S1 in the above embodiment, and will not be repeated here.
And the ordering module 2 is used for ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted. The details are referred to the related description of step S2 in the above embodiments, and will not be repeated here.
And the decomposition module 3 is used for decomposing the data to be transmitted into a preset number of objects to be transmitted according to the list to be transmitted. The details are referred to the related description of step S3 in the above embodiments, and will not be repeated here.
And the transmission module 4 is used for transmitting the plurality of objects to be transmitted to the target storage position in parallel. The details are referred to the related description of step S4 in the above embodiment, and will not be repeated here.
According to the data transmission system provided by the invention, the data to be transmitted is ordered according to the last modification time of each storage object, and the list to be transmitted is generated. After the list to be transmitted is generated, the transmission data is decomposed into a plurality of objects to be transmitted for transmission, so that the data transmission mode is optimized, and the data transmission efficiency is improved.
In one embodiment, the predetermined number is 16.
In a specific embodiment, when the data to be transmitted is decomposed into 16 segments, the transmission effect is the best and the transmission efficiency is the highest.
The present invention also provides a computer device, as shown in fig. 6, which may include a processor 61 and a memory 62, where the processor 61 and the memory 62 may be connected by a bus or otherwise, fig. 6 being an example of a connection by a bus.
The processor 61 may be a central processing unit (Central Processing Unit, CPU). The Processor 61 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), field-Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 62 serves as a non-transitory computer readable storage medium that may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as corresponding program instructions/modules in embodiments of the present invention. The processor 61 executes various functional applications of the processor and data processing, i.e., implements the data transmission method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 62.
Memory 62 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created by the processor 61, etc. In addition, the memory 62 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 62 may optionally include memory located remotely from processor 61, which may be connected to processor 61 via a network. Examples of such networks include, but are not limited to, the internet, intranets, mobile communication networks, and combinations thereof.
One or more modules are stored in the memory 62 that, when executed by the processor 61, perform the method for data interaction of an electric energy meter provided by an embodiment of the present invention or perform the method for data transmission provided by an embodiment of the present invention.
The details of the computer device may be understood in response to the corresponding relevant descriptions and effects of the embodiments shown in fig. 1-4, and are not described herein.
It will be appreciated by those skilled in the art that a program implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, and when executed, may include the above-described embodiment method flow. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a hard disk (HARD DISK DRIVE, abbreviated as HDD), a Solid state disk (Solid-STATE DRIVE, SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. And obvious variations or modifications thereof are contemplated as falling within the scope of the present invention.
Claims (9)
1. A data transmission method, comprising:
acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects;
The obtaining the data to be transmitted of the original storage position comprises the following steps: acquiring a hot data file of an original storage position; sorting the hot data files based on the storage space occupied by the hot data files, and screening out the hot data files meeting the KB level range; combining the screened hot data files to obtain a data packet, and taking the combined data packet as data to be transmitted;
Ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted;
according to the list to be transmitted, decomposing the data to be transmitted into a preset number of objects to be transmitted;
And transmitting a plurality of objects to be transmitted to a target storage position in parallel.
2. The data transmission method according to claim 1, wherein the sorting the hot data files based on the storage space occupied by the hot data files includes:
Acquiring file parameters of each thermal data file;
Respectively calculating the storage space occupied by each thermal data file according to the file parameters;
And ordering the hot data files according to the storage space.
3. The data transmission method according to claim 1, wherein the preset number is 16.
4. The method for data transmission according to claim 1, wherein,
When the original storage position is a magnetic disk, the target storage position is a blue-ray memory;
when the original storage position is a blue light storage, the target storage position is a magnetic disk.
5. The data transmission method according to claim 4, wherein before the sorting the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, the data transmission method further comprises:
judging whether the original storage position is a blue light storage or not;
and when the original storage position is a blue light storage, transmitting the data to be transmitted to a target storage position in parallel.
6. A data transmission system, comprising:
The acquisition module is used for acquiring data to be transmitted of an original storage position, wherein the data to be transmitted consists of a plurality of KB-level storage objects;
The obtaining the data to be transmitted of the original storage position comprises the following steps: acquiring a hot data file of an original storage position; sorting the hot data files based on the storage space occupied by the hot data files, and screening out the hot data files meeting the KB level range; combining the screened hot data files to obtain a data packet, and taking the combined data packet as data to be transmitted;
The ordering module is used for ordering the data to be transmitted according to the last modification time of each storage object in the data to be transmitted, and generating a list to be transmitted;
The decomposition module is used for decomposing the data to be transmitted into a preset number of objects to be transmitted according to the list to be transmitted;
and the transmission module is used for transmitting the plurality of objects to be transmitted to the target storage position in parallel.
7. The data transmission system of claim 6, wherein the predetermined number is 16.
8. A computer readable storage medium storing computer instructions for causing the computer to perform the data transmission method according to any one of claims 1 to 5.
9. A computer device, comprising: a memory and a processor, said memory and said processor being communicatively coupled to each other, said memory storing computer instructions, said processor executing said computer instructions to perform the data transmission method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011566785.0A CN112732175B (en) | 2020-12-25 | 2020-12-25 | Data transmission method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011566785.0A CN112732175B (en) | 2020-12-25 | 2020-12-25 | Data transmission method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112732175A CN112732175A (en) | 2021-04-30 |
| CN112732175B true CN112732175B (en) | 2024-05-14 |
Family
ID=75616540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011566785.0A Active CN112732175B (en) | 2020-12-25 | 2020-12-25 | Data transmission method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112732175B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109308170A (en) * | 2018-09-11 | 2019-02-05 | 北京北信源信息安全技术有限公司 | A kind of data processing method and device |
| CN109768926A (en) * | 2019-01-10 | 2019-05-17 | 维沃移动通信有限公司 | A kind of data processing method, terminal device and computer readable storage medium |
| CN111142791A (en) * | 2019-12-12 | 2020-05-12 | 江苏苏宁物流有限公司 | Data migration method and device |
-
2020
- 2020-12-25 CN CN202011566785.0A patent/CN112732175B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109308170A (en) * | 2018-09-11 | 2019-02-05 | 北京北信源信息安全技术有限公司 | A kind of data processing method and device |
| CN109768926A (en) * | 2019-01-10 | 2019-05-17 | 维沃移动通信有限公司 | A kind of data processing method, terminal device and computer readable storage medium |
| CN111142791A (en) * | 2019-12-12 | 2020-05-12 | 江苏苏宁物流有限公司 | Data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112732175A (en) | 2021-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110022330B (en) | Processing method and device for network data packet and electronic equipment | |
| US9148496B2 (en) | Dynamic runtime choosing of processing communication methods | |
| CN108353040B (en) | System and method for distributed packet scheduling | |
| US11240155B2 (en) | Technologies for network device load balancers for accelerated functions as a service | |
| US11695628B2 (en) | Technologies for autonomous edge compute instance optimization and auto-healing using local hardware platform QoS services | |
| US11265235B2 (en) | Technologies for capturing processing resource metrics as a function of time | |
| CN110012044B (en) | Dynamic work transfer method and server | |
| CN104836747A (en) | Network outbound load balancing method and system | |
| WO2010129516A1 (en) | Systems and methods for providing a multi-core architecture for an internet protocol acceleration appliance | |
| CN110620802B (en) | Load balancing method and device | |
| US11726938B2 (en) | Communications for field programmable gate array device | |
| CN112261094A (en) | Message processing method and proxy server | |
| US11799827B2 (en) | Intelligently routing a response packet along a same connection as a request packet | |
| WO2017000094A1 (en) | Data storage method, device and system | |
| CN112732175B (en) | Data transmission method and system | |
| US9270537B2 (en) | Connection distribution within a connectional parallelism architecture | |
| JP6375849B2 (en) | FILE SYSTEM, MANAGEMENT DEVICE CONTROL PROGRAM, AND FILE SYSTEM CONTROL METHOD | |
| WO2016197607A1 (en) | Method and apparatus for realizing route lookup | |
| KR20200051196A (en) | Electronic device providing fast packet forwarding with reference to additional network address translation table | |
| CN110971676B (en) | Equipment state detection method and device, electronic equipment and storage medium | |
| CN109660495A (en) | A kind of document transmission method and device | |
| CN117707774A (en) | Request processing method and device, storage medium and electronic equipment | |
| CN113873036A (en) | Communication method, device, server and storage medium | |
| CN115470010A (en) | Network request link multiplexing method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |