CN111901377A - File transmission method, device, equipment and medium based on AI (Artificial Intelligence) training platform - Google Patents
File transmission method, device, equipment and medium based on AI (Artificial Intelligence) training platform Download PDFInfo
- Publication number
- CN111901377A CN111901377A CN202010598653.XA CN202010598653A CN111901377A CN 111901377 A CN111901377 A CN 111901377A CN 202010598653 A CN202010598653 A CN 202010598653A CN 111901377 A CN111901377 A CN 111901377A
- Authority
- CN
- China
- Prior art keywords
- file
- training platform
- file transmission
- configuration information
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 124
- 238000012549 training Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000013473 artificial intelligence Methods 0.000 title description 66
- 238000011161 development Methods 0.000 claims abstract description 40
- 230000004044 response Effects 0.000 claims abstract description 16
- 238000012546 transfer Methods 0.000 claims description 36
- 238000004590 computer program Methods 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 description 21
- 230000006870 function Effects 0.000 description 13
- 238000007726 management method Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 239000000872 buffer Substances 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000109539 Conchita Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Abstract
The invention discloses a file transmission method based on an AI training platform, which comprises the following steps: downloading and configuring a file transmission client for realizing the file transmission based on the DMA through an AI training platform; detecting configuration information of a development environment through a file transmission client in response to receiving a file transmission command; acquiring the number of CPU cores according to the configuration information, and determining the number of threads for file transmission according to the number of the CPU cores; acquiring the size of a development environment memory according to the configuration information, determining a space value of a file block according to the size of the development environment memory, and dividing a transmitted file into file blocks according to the space value; and carrying out file transmission on the file according to the thread number and the file blocks. The invention also discloses a device, equipment and a medium. The invention can ensure the high-efficiency and stable operation of the AI training platform, effectively shorten the time of model training, improve the performance of the AI training platform, solve the problem of fast transmission pain point of file transmission of the AI training platform and enhance the competitiveness of the AI training platform.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a file transfer method, apparatus, device, and medium based on an AI training platform.
Background
With the rapid development of the related industries of artificial intelligence, more and more research institutions and enterprises have higher and higher requirements on computing power, and meanwhile, how to efficiently promote efficient resource use of algorithm researchers in the enterprise institutions and improve working efficiency is also a goal pursued by all current AI training platforms (platforms for managing and scheduling resources such as cpu, gpu and the like, model training and task management). Meanwhile, when the AI is trained, a large number of data set files are used, a large model data file is generated, how to efficiently upload and download the file for quick transmission and provide transmission performance, the prior art is still more traditional, and the optimization and promotion of the performance are urgently needed for a large data set, particularly for a relevant data set of an AI (artificial intelligence) training platform.
An important module, namely a basic module, which relates to an AI training platform at present, belongs to a file management module of the AI training platform. This module is typically a model of the algorithm personnel, scripts and management of files such as data sets. Most of traditional file management modules are based on http network transmission tools such as web and xftp, and the like, and the transmission tools have the common characteristics that the transmission tools particularly depend on network bandwidth, and can cause some resource consumption pressure on an AI training model platform, such as larger occupied network IO, more CPU and internal memory, and the like; meanwhile, file transmission of the currently common web is limited by constraints such as browser caching, network protocols and browser kernels, so that the method is not very suitable for transmission management of large file data, has poor use experience on algorithm personnel, and seriously affects the efficiency of model training. Meanwhile, the xftp tool has the limitations of system environment and speed performance, and the requirements of algorithm personnel on file uploading and downloading rapid transmission on the AI training platform are far from being met.
Meanwhile, clients targeted by the AI training platform are generally used by algorithm personnel with a computer base, most training platforms limit related operations of the algorithm personnel on system commands, and the algorithm personnel can use different system environments such as linux (centros, ubuntu) and windows when transmitting files. How to efficiently promote the algorithm personnel to carry out file transmission is the problem that needs to be solved by the patent.
Disclosure of Invention
In view of this, an embodiment of the present invention provides an efficient file transmission client scheme for file data based on an AI training platform, and provides an interface independent of web file data management based on the AI training platform, and provides a client module component for an algorithm worker to upload and download file transmission. The stability of the AI training platform and the data file transmission efficiency are improved.
Based on the above purpose, in one aspect, the present invention provides a file transmission method based on an AI training platform, where the method includes:
downloading and configuring a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
detecting configuration information of a development environment through a file transmission client in response to receiving a file transmission command;
acquiring the number of CPU cores according to the configuration information, and determining the number of threads for file transmission according to the number of the CPU cores;
acquiring the size of a development environment memory according to the configuration information, determining a space value of a file block according to the size of the development environment memory, and dividing a transmitted file into file blocks according to the space value;
and carrying out file transmission on the file according to the thread number and the file blocks.
In some embodiments of the file transfer method based on the AI training platform of the present invention, the method further comprises:
and acquiring network information according to the configuration information, and monitoring a network environment according to the network information.
In some embodiments of the file transfer method based on the AI training platform according to the present invention, the obtaining a CPU core count according to the configuration information, and determining a thread number for file transfer according to the CPU core count further includes:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
In some embodiments of the file transmission method based on the AI training platform according to the present invention, obtaining a size of a development environment memory according to the configuration information, determining a space value of a file block according to the size of the development environment memory, and dividing a transmitted file into file blocks according to the space value further includes:
and configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the memory of the development environment exceeds the maximum space threshold value.
In some embodiments of the file transfer method based on the AI training platform of the present invention, the method further comprises:
and displaying the transmission information and the abnormal alarm information of the file transmission through the file transmission client.
In another aspect of the embodiments of the present invention, a file transmission device based on an AI training platform is further provided, where the file transmission device includes:
the client downloading module is configured to download and configure a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
the configuration information detection module is configured to respond to the received file transmission command and detect the configuration information of the development environment through the file transmission client;
the thread quantity acquisition module is configured to acquire the CPU core number according to the configuration information and determine the thread quantity of file transmission according to the CPU core number;
the file block dividing module is configured to acquire the size of the development environment memory according to the configuration information, determine the space value of the file block according to the size of the development environment memory, and divide the transmitted file into file blocks according to the space value;
and the file transmission module is configured to transmit the files according to the thread number and the file blocks.
In some embodiments of the file transmission apparatus based on the AI training platform of the present invention, the thread number obtaining module is further configured to:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
In some embodiments of the AI training platform based file transfer device of the present invention, the file block partitioning module is further configured to:
and configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the memory of the development environment exceeds the maximum space threshold value.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including:
at least one processor; and
the memory stores a computer program which can run on the processor, and the processor executes the file transmission method based on the AI training platform when executing the program.
In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where a computer program is stored in the computer-readable storage medium, and is characterized in that when being executed by a processor, the computer program performs the foregoing file transmission method based on the AI training platform.
The invention has at least the following beneficial technical effects:
the consumption and the use of resources are reduced to a certain extent, so that each service module of the AI training platform occupies less resources, and the influence of excessive resource occupation on the use experience of the AI training platform is prevented; on the other hand, some defects of web browser service are avoided, the performance is better than that of xftp, the advantages are better suitable for different system platforms than that of xftp, algorithm personnel of research and development personnel are easier to use, client files of file transmission clients are uploaded and downloaded, the daily operation of the algorithm personnel is better fitted, the use experience of the algorithm personnel is improved, the model training efficiency is improved, the time cost is reduced, and the performance point and the competitiveness of an AI training platform are increased. And the tool method for maintaining the original web and xftp management file provides a method for a file transmission client, solves the problem of related pain points of file management of algorithm personnel, reduces the pressure of a browser for accessing an AI training platform, improves the related performance of file transmission, reduces the resource consumption of IO, CPU, memory and the like of the AI training platform by using the scheme of file transmission client transmission and copying based on DMA performance, and has efficient and stable support for file transmission from different systems, from windows to clusters and from linux to clusters.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic block diagram illustrating an embodiment of a file transfer method based on an AI training platform in accordance with the present invention;
FIG. 2 is a schematic business operation flowchart of an embodiment of a file transfer method based on an AI training platform according to the invention;
fig. 3 is a schematic diagram illustrating a file transfer principle of an embodiment of a file transfer method based on an AI training platform according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it is understood that "first" and "second" are only used for convenience of description and should not be construed as limiting the embodiments of the present invention, and the descriptions thereof in the following embodiments are omitted.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a file transfer method based on an AI training platform. Fig. 1 is a schematic block diagram of an embodiment of a file transfer method based on an AI training platform according to the present invention. In the embodiment shown in fig. 1, the method comprises at least the following steps:
s100, downloading and configuring a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
s200, responding to the received file transmission command, and detecting configuration information of the development environment through a file transmission client;
s300, acquiring the number of CPU cores according to the configuration information, and determining the number of threads for file transmission according to the number of the CPU cores;
s400, acquiring the size of the memory of the development environment according to the configuration information, determining the space value of the file block according to the size of the memory of the development environment, and dividing the transmitted file into file blocks according to the space value;
and S500, carrying out file transmission on the file according to the thread number and the file blocks.
In some embodiments of the present invention, a file management module of an AI training platform provides an algorithm personnel web operation interface and an xftp operation mode, and in this solution, a transmission mode is added, file transmission can be performed by receiving a file transmission command in a client command mode, the AI training platform provides a server interface, and an algorithm personnel uploads, downloads, and transmits a data set or a model file by using a file transmission client command mode.
When a file is transmitted by a file transmission client (which is a transmission medium and is between an AI cluster and a development environment), file transmission is performed based on Direct Memory Access (DMA), that is, Direct Memory Access, without using a Central Processing Unit (CPU) to transmit file data, and the data is transmitted to a Memory by using the DMA and is directly buffered to a kernel buffer. Different from the traditional DMA transmission scheme, the embodiment of the invention optimizes the memory part of the DMA transmission scheme, if the file is large and exceeds the set value, the whole source data file is not read into the kernel cache region, but the file is divided into file blocks to be read one by one, thereby preventing the memory overflow, reasonably adjusting and setting the size of the space value of the file transmission and achieving the optimal effect.
According to the invention, uploading, downloading and transmission of the file data set can be flexible and various through the AI training platform, the file data set is not limited to uploading a local text file to a cluster and downloading a cluster file to the local in the prior art, and under the file directory of an algorithm worker, the functions of researching and developing a server to the cluster and a cluster file to a research and developing server can be realized, the web pressure is reduced through a file transmission command, the file transmission performance is improved, and the use experience of a user is enhanced.
In the process of file transmission by the file transmission client, the progress of file transmission can be visually displayed, the breakpoint resume function and the resume termination function are supported, the advantage visualization function of the web browser is combined, the stability is improved, the transmission performance is enhanced, and the resource consumption of a file management module of the AI training platform is reduced.
In some specific embodiments of the present invention, fig. 2 is a schematic diagram illustrating a business operation flow of an embodiment of a file transmission method based on an AI training platform according to the present invention, and as shown in fig. 2, a file management interface of the AI training platform not only provides interface operations and a conventional functions of adding, deleting, modifying and checking, but also provides a function of downloading a file transmission client, where the file transmission client is used for uploading, downloading, and transmitting big data or big files. Thus, as shown in fig. 2, the AI training platform provides services for file data management web operations, xftp methods, file transfer client downloads. The file transmission client can be used in windows or linux, can also be other media capable of communicating with an AI training platform, can realize file transmission only by using a specific file transmission command by an algorithm worker, and mainly realizes the functions of file uploading, downloading and transmission operation and AI training environment isolation.
Firstly, the file transmission client needs to detect configuration information of the development environment of the algorithm personnel, wherein the configuration information comprises the number of CPU cores, the size of a memory, network information and the like), and the file transmission client acquires the number of CPU cores and the size of the memory of the development environment of the algorithm personnel. And controlling the number of threads for file transmission according to the acquired number of the CPU cores, wherein the optimal number is obtained under the condition that the data ratio of the number of the CPU cores to the number of the threads is 1:2 in multiple experimental processes. For example, in some embodiments, if the number of CPU cores is 2, the number of corresponding threads is 4.
Fig. 3 is a schematic diagram illustrating a file transmission principle of an embodiment of the file transmission method based on the AI training platform according to the present invention, and as shown in step 2 from the hard disk to the kernel buffer shown in fig. 3, since data transmission between kernel buffers of the operating system needs to be avoided as much as possible during the file transmission process, and transmission operation between the system kernel and the user application is avoided; in some embodiments of the present invention, the size of the file block of the file is set to 10% of the memory size of the development environment, and then the file block is gradually subjected to file transfer. In the step 4 from the socket buffer to the network card cache, the file transfer process based on the DMA is also performed.
According to some embodiments of the file transfer method based on the AI training platform of the present invention, the method further comprises:
and acquiring network information according to the configuration information, and monitoring a network environment according to the network information.
In some embodiments of the present invention, network information is obtained according to the configuration information, the network information is monitored, and network environment related information is prompted according to the network information.
According to some embodiments of the file transmission method based on the AI training platform of the present invention, obtaining the number of CPU cores according to the configuration information, and determining the number of threads for file transmission according to the number of CPU cores further includes:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
In some embodiments of the present invention, the number of threads for file transfer is set to an upper threshold, and in some embodiments, the upper threshold is set to 8, that is, the maximum number of threads for file transfer cannot exceed 8.
According to some embodiments of the file transmission method based on the AI training platform of the present invention, obtaining the size of the development environment memory according to the configuration information, determining the space value of the file block according to the size of the development environment memory, and dividing the transmitted file into file blocks according to the space value further includes:
and configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the memory of the development environment exceeds the maximum space threshold value.
In some embodiments of the invention, the maximum upper limit of the size of the space value of a profile block is 1G, i.e. the space value cannot exceed 1G at maximum. Setting the size of the file block of the file to be 10% of the memory size of the development environment, and when the size of the file block of the file calculated according to the memory size of the development environment is larger than 1G, fixedly setting the size of the space value of the file block to be 1G.
According to some embodiments of the file transfer method based on the AI training platform of the present invention, the method further comprises:
and displaying the transmission information and the abnormal alarm information of the file transmission through the file transmission client.
In some embodiments of the present invention, the file transmission client displays transmission information of file transmission in a file transmission process, where the transmission information includes information related to transmission progress, network speed, and time, and abnormality alarm information, where the abnormality alarm information includes error information caused by a network and a disk.
In another aspect of the embodiments of the present invention, an embodiment of a file transfer device based on an AI training platform is provided. The device includes:
the client downloading module is configured to download and configure a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
the configuration information detection module is configured to respond to the received file transmission command and detect the configuration information of the development environment through the file transmission client;
the thread quantity acquisition module is configured to acquire the CPU core number according to the configuration information and determine the thread quantity of file transmission according to the CPU core number;
the file block dividing module is configured to acquire the size of the development environment memory according to the configuration information, determine the space value of the file block according to the size of the development environment memory, and divide the transmitted file into file blocks according to the space value;
and the file transmission module is configured to transmit the files according to the thread number and the file blocks.
According to some embodiments of the file transmission apparatus based on the AI training platform of the present invention, the thread number obtaining module is further configured to:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
According to some embodiments of the AI training platform based file transfer apparatus of the present invention, the file block dividing module is further configured to:
and configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the memory of the development environment exceeds the maximum space threshold value.
In view of the above object, another aspect of the embodiments of the present invention further provides a computer device, including: at least one processor; and the memory is used for storing a computer program which can run on the processor, and the processor executes the file transmission method based on the AI training platform when executing the program.
In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, where a computer program is stored in the computer-readable storage medium, and is characterized in that when being executed by a processor, the computer program performs the foregoing file transmission method based on the AI training platform.
Likewise, those skilled in the art will appreciate that all of the embodiments, features and advantages set forth above with respect to the AI training platform based file transfer method according to the present invention apply equally well to the apparatus, the computer device and the medium according to the present invention. For the sake of brevity of the present disclosure, no repeated explanation is provided herein.
It should be particularly noted that, the steps in the above-mentioned various embodiments of the file transmission method, apparatus, device and medium based on the AI training platform can be mutually intersected, replaced, added and deleted, so that these reasonable permutations and combinations of the file transmission method, apparatus, device and medium based on the AI training platform shall also belong to the scope of the present invention, and shall not limit the scope of the present invention to the embodiments.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program to instruct related hardware, and the program of the file transfer method based on the AI training platform can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A file transmission method based on an AI training platform is characterized by comprising the following steps:
downloading and configuring a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
detecting configuration information of a development environment through the file transfer client in response to receiving a file transfer command;
acquiring the number of CPU cores according to the configuration information, and determining the number of threads of the file transmission according to the number of the CPU cores;
acquiring the size of a development environment memory according to the configuration information, determining a space value of a file block according to the size of the development environment memory, and dividing a transmitted file into the file blocks according to the space value;
and carrying out the file transmission on the file according to the thread number and the file block.
2. The AI training platform-based file transfer method of claim 1, further comprising:
and acquiring network information according to the configuration information, and monitoring a network environment according to the network information.
3. The AI training platform-based file transfer method of claim 1, wherein the obtaining a CPU core count according to the configuration information and determining the number of threads for file transfer according to the CPU core count further comprises:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
4. The AI training platform-based file transfer method of claim 1, wherein the obtaining a size of a development environment memory according to the configuration information, determining a space value of a file block according to the size of the development environment memory, and dividing the transferred file into the file blocks according to the space value further comprises:
configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the development environment memory exceeds the maximum space threshold value.
5. The AI training platform-based file transfer method of claim 1, further comprising:
and displaying the transmission information and the abnormal alarm information of the file transmission through the file transmission client.
6. A file transfer device based on an AI training platform, the device comprising:
the client downloading module is configured to download and configure a file transmission client for realizing the file transmission based on the DMA through an AI training platform;
a configuration information detection module configured to detect configuration information of a development environment through the file transfer client in response to receiving a file transfer command;
the thread quantity acquisition module is configured to acquire the number of CPU cores according to the configuration information and determine the number of threads for file transmission according to the number of the CPU cores;
the file block dividing module is configured to acquire the size of a development environment memory according to the configuration information, determine a space value of a file block according to the size of the development environment memory, and divide a transmitted file into the file blocks according to the space value;
and the file transmission module is configured to transmit the file according to the thread number and the file blocks.
7. The AI training platform-based file transfer device of claim 6, wherein the thread number acquisition module is further configured to:
and configuring an upper limit quantity threshold value for the thread quantity, and configuring the thread quantity according to the upper limit quantity threshold value in response to the fact that the thread quantity acquired according to the CPU core number exceeds the upper limit quantity threshold value.
8. The AI training platform-based file transfer device of claim 6, wherein the file block partitioning module is further configured to:
configuring a maximum space threshold value for the space value, and configuring the space value according to the maximum space threshold value in response to the fact that the space value determined according to the size of the development environment memory exceeds the maximum space threshold value.
9. A computer device, comprising:
at least one processor; and
memory storing a computer program operable on the processor, wherein the processor, when executing the program, performs the method of any of claims 1-5.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010598653.XA CN111901377B (en) | 2020-06-28 | 2020-06-28 | AI training platform-based file transmission method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010598653.XA CN111901377B (en) | 2020-06-28 | 2020-06-28 | AI training platform-based file transmission method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111901377A true CN111901377A (en) | 2020-11-06 |
| CN111901377B CN111901377B (en) | 2022-12-27 |
Family
ID=73207317
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010598653.XA Active CN111901377B (en) | 2020-06-28 | 2020-06-28 | AI training platform-based file transmission method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111901377B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511633A (en) * | 2020-12-03 | 2021-03-16 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN112615937A (en) * | 2020-12-28 | 2021-04-06 | 南方电网深圳数字电网研究院有限公司 | Data transmission method, data training method, server, electronic device and storage medium |
| CN113377720A (en) * | 2021-06-23 | 2021-09-10 | 上海商汤科技开发有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113835856A (en) * | 2021-09-17 | 2021-12-24 | 苏州浪潮智能科技有限公司 | Storage statistical method, device and equipment for AI platform |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412950A (en) * | 2013-08-28 | 2013-11-27 | 浙江大学 | Method for increasing read-write speed of spatial big data files |
| CN105373450A (en) * | 2015-11-30 | 2016-03-02 | 国网北京市电力公司 | Data backup method and apparatus |
| US20180321985A1 (en) * | 2017-05-02 | 2018-11-08 | Intel Corporation | High-performance input-output devices supporting scalable virtualization |
| US20190140913A1 (en) * | 2018-12-28 | 2019-05-09 | Intel Corporation | Techniques for artificial intelligence capabilities at a network switch |
| US20190363988A1 (en) * | 2018-05-25 | 2019-11-28 | Software Ag | System and/or method for predictive resource management in file transfer servers |
| WO2020001488A1 (en) * | 2018-06-28 | 2020-01-02 | 杭州海康威视系统技术有限公司 | File download method and apparatus, client and computer readable storage medium |
| CN110659134A (en) * | 2019-09-04 | 2020-01-07 | 腾讯云计算（北京）有限责任公司 | Data processing method and device applied to artificial intelligence platform |
| CN111104459A (en) * | 2019-08-22 | 2020-05-05 | 华为技术有限公司 | Storage device, distributed storage system, and data processing method |
| CN111339000A (en) * | 2020-02-24 | 2020-06-26 | 深圳震有科技股份有限公司 | AMP system memory file transmission method and device |
-
2020
- 2020-06-28 CN CN202010598653.XA patent/CN111901377B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412950A (en) * | 2013-08-28 | 2013-11-27 | 浙江大学 | Method for increasing read-write speed of spatial big data files |
| CN105373450A (en) * | 2015-11-30 | 2016-03-02 | 国网北京市电力公司 | Data backup method and apparatus |
| US20180321985A1 (en) * | 2017-05-02 | 2018-11-08 | Intel Corporation | High-performance input-output devices supporting scalable virtualization |
| US20190363988A1 (en) * | 2018-05-25 | 2019-11-28 | Software Ag | System and/or method for predictive resource management in file transfer servers |
| WO2020001488A1 (en) * | 2018-06-28 | 2020-01-02 | 杭州海康威视系统技术有限公司 | File download method and apparatus, client and computer readable storage medium |
| US20190140913A1 (en) * | 2018-12-28 | 2019-05-09 | Intel Corporation | Techniques for artificial intelligence capabilities at a network switch |
| CN111104459A (en) * | 2019-08-22 | 2020-05-05 | 华为技术有限公司 | Storage device, distributed storage system, and data processing method |
| CN110659134A (en) * | 2019-09-04 | 2020-01-07 | 腾讯云计算（北京）有限责任公司 | Data processing method and device applied to artificial intelligence platform |
| CN111339000A (en) * | 2020-02-24 | 2020-06-26 | 深圳震有科技股份有限公司 | AMP system memory file transmission method and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511633A (en) * | 2020-12-03 | 2021-03-16 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN112511633B (en) * | 2020-12-03 | 2022-11-29 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN112615937A (en) * | 2020-12-28 | 2021-04-06 | 南方电网深圳数字电网研究院有限公司 | Data transmission method, data training method, server, electronic device and storage medium |
| CN113377720A (en) * | 2021-06-23 | 2021-09-10 | 上海商汤科技开发有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113835856A (en) * | 2021-09-17 | 2021-12-24 | 苏州浪潮智能科技有限公司 | Storage statistical method, device and equipment for AI platform |
| CN113835856B (en) * | 2021-09-17 | 2023-07-14 | 苏州浪潮智能科技有限公司 | Storage statistics method, device and equipment of AI platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111901377B (en) | 2022-12-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111901377B (en) | AI training platform-based file transmission method, device, equipment and medium | |
| JP2020528179A (en) | Optimizing user satisfaction when training a cognitive tiered storage management system | |
| CN112506427A (en) | Method, system, equipment and medium for reducing power consumption of solid state disk | |
| CN111324457A (en) | Method, device, equipment and medium for issuing inference service in GPU cluster | |
| CN109446093B (en) | Method and device for testing expansion platform interface | |
| CN110647460A (en) | Test resource management method and device and test client | |
| CN111078497A (en) | Data storage method, equipment and storage medium of BMC (baseboard management controller) | |
| US20170046071A1 (en) | Selecting a primary storage device | |
| CN112422604B (en) | File uploading method, device and system and computer equipment | |
| US11431669B2 (en) | Server configuration method and apparatus | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| CN111221715A (en) | Method, system, device and medium for dynamically optimizing Caffe performance | |
| CN111651126B (en) | Memory writing operation method, system, electronic device and medium | |
| CN110659035A (en) | Method and device for batch mounting of mirror images based on BMC | |
| CN101383733B (en) | Method and device for monitoring network state when calling | |
| CN110990148A (en) | Method, device and medium for optimizing storage performance | |
| CN114138597B (en) | Operating system performance tuning device, method, equipment and storage medium | |
| CN115951845A (en) | Disk management method, device, equipment and storage medium | |
| CN115878042A (en) | Data storage system, data processing method and storage medium | |
| CN112367384B (en) | Kafka cluster-based dynamic speed limiting method and device and computer equipment | |
| US10268375B2 (en) | Methods for proactive prediction of disk failure in the disk maintenance pipeline and devices thereof | |
| CN114416690A (en) | Data migration device, method and storage medium for file storage to object storage | |
| CN113138772B (en) | Construction method and device of data processing platform, electronic equipment and storage medium | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN104346101A (en) | Dynamic storage space allocation system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |