CN112733892A - Data interaction method and device for model training - Google Patents
Data interaction method and device for model training Download PDFInfo
- Publication number
- CN112733892A CN112733892A CN202011595881.8A CN202011595881A CN112733892A CN 112733892 A CN112733892 A CN 112733892A CN 202011595881 A CN202011595881 A CN 202011595881A CN 112733892 A CN112733892 A CN 112733892A
- Authority
- CN
- China
- Prior art keywords
- gateway
- object storage
- data
- model training
- target server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 93
- 230000003993 interaction Effects 0.000 title claims abstract description 50
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000004044 response Effects 0.000 claims description 47
- 238000012545 processing Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention discloses a data interaction method and device for model training, relates to the technical field of computers, and can effectively improve the data interaction efficiency of model training. The method comprises the following steps: acquiring a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways; scoring each object storage gateway in the gateway list according to a preset rule; and responding to a data request of the model training task, and requesting data from the target server through an object storage gateway with the highest score in the gateway list. The invention can be applied to model training.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data interaction method and device for model training.
Background
In recent years, artificial intelligence technology has become more and more widely used in industry and life. Machine learning is an important branch in the field of artificial intelligence, and an ideal mathematical model can be obtained through training of a large amount of data. Because the data volume required by model training is huge, the operation task is heavy, and in many cases, data storage and operation are required to be carried out by depending on a computer cluster. How to support the rapid data interaction between the huge deep learning training system and the data storage system becomes a problem to be solved urgently in the field.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data interaction method and apparatus for model training, which can effectively improve data interaction efficiency of model training.
In a first aspect, an embodiment of the present invention provides a data interaction method for model training, including: acquiring a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways; scoring each object storage gateway in the gateway list according to a preset rule; and responding to a data request of the model training task, and requesting data from the target server through an object storage gateway with the highest score in the gateway list.
Optionally, the obtaining of the gateway list corresponding to the target server includes: and pulling a gateway list from each object storage gateway corresponding to the target server through a cache manager of the computing node with the model training task.
Optionally, the scoring, according to a preset rule, each object storage gateway in the gateway list respectively includes: acquiring historical operation information of each object storage gateway in the gateway list; and scoring each object storage gateway in the gateway list according to the historical operation information.
Optionally, the historical operating information includes at least one of: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time.
Optionally, the time interval range of the historical operating information is a preset time length traced from the current time forward.
Optionally, the historical operation information includes the response time length to the data request; the obtaining of the historical operation information of each object storage gateway in the gateway list includes: respectively sending a first data request to each object storage gateway in the gateway list; respectively receiving first data responses of the object storage gateways to the first data requests; and determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
In a second aspect, an embodiment of the present invention further provides a data interaction apparatus for model training, including: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a gateway list corresponding to a target server, the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways; the scoring unit is used for scoring each object storage gateway in the gateway list according to a preset rule; and the request unit is used for responding to a data request of the model training task and requesting data to the target server through the object storage gateway with the highest score in the gateway list.
Optionally, the obtaining unit is specifically configured to pull a gateway list to each object storage gateway corresponding to the target server through a cache manager of a computing node deployed with the model training task.
Optionally, the scoring unit includes: the acquisition module is used for acquiring historical operation information of each object storage gateway in the gateway list; and the scoring module is used for scoring each object storage gateway in the gateway list according to the historical operation information.
Optionally, the historical operating information includes at least one of: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time.
Optionally, the time interval range of the historical operating information is a preset time length traced from the current time forward.
Optionally, the historical operation information includes the response time length to the data request; the acquisition module includes: a sending submodule, configured to send a first data request to each object storage gateway in the gateway list; the receiving submodule is used for respectively receiving first data responses of the object storage gateways to the first data requests; and the determining submodule is used for determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
In a third aspect, an embodiment of the present invention further provides an electronic device, including: the device comprises a shell, a processor, a memory, a circuit board and a power circuit, wherein the circuit board is arranged in a space enclosed by the shell, and the processor and the memory are arranged on the circuit board; a power supply circuit for supplying power to each circuit or device of the electronic apparatus; the memory is used for storing executable program codes; the processor executes a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute any data interaction method for model training provided by the embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing one or more programs, which are executable by one or more processors to implement any of the data interaction methods for model training provided by the embodiments of the present invention.
The data interaction method, the data interaction device, the electronic equipment and the storage medium for model training provided by the embodiments of the present invention can acquire a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, the gateway list includes at least two object storage gateways, then scores each object storage gateway in the gateway list according to a preset rule, and requests the data from the target server through an object storage gateway with the highest score in the gateway list in response to a data request of the model training task. Therefore, the most suitable object storage gateway can be selected from the gateway list according to the preset rule to perform data interaction with the object storage server, and therefore the data interaction efficiency between the model training server and the object storage server is effectively improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a data interaction application scenario for model training in an embodiment of the present invention;
FIG. 2 is a flowchart of a data interaction method for model training according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating a setting position of a load balancing algorithm in the data interaction method for model training according to an embodiment of the present invention;
FIG. 4 is a detailed flowchart of a data interaction method for model training according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a data interaction device for model training according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In machine learning, on the one hand, a computer with powerful computing power is required for model training, and on the other hand, sufficient data samples are also required for computer learning. These two tasks may be performed by different servers, respectively. For example, model training may be performed by a model training server, while training data may be provided by an object storage server. The model training server may interact with the object storage server through the object storage gateway to obtain data from the object storage server and perform model training using the data. Meanwhile, the trained model can be output to an object storage server through an object storage gateway for storage.
However, there may be multiple object storage gateways in the object storage server, and how to select the gateway most suitable for the current data interaction is related to the data interaction efficiency between the model training server and the object storage server. To this end, embodiments of the present invention provide a data interaction method for model training.
Technical ideas, embodiments and advantageous technical effects of the embodiments of the present invention will be described in detail below with reference to specific examples in order to enable those skilled in the art to better understand the technical ideas, embodiments and advantageous technical effects of the examples.
Fig. 1 is a schematic application scenario diagram of a data interaction method for model training according to an embodiment of the present invention, and fig. 2 is a flowchart of the data interaction method for model training. With reference to fig. 1 and fig. 2, a data interaction method for model training provided by an embodiment of the present invention may include:
s11, acquiring a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways;
the target server may refer to a server in which data required for the model training task is stored, may refer to a certain server, and may also refer to a service cluster including a plurality of service nodes. In the embodiment of the invention, the target server is based on the object storage architecture and is an object storage server. The model training server may access the object store server through the object store gateway. Each target server may be served by a plurality of object storage gateways. The model training server may obtain a list of these object storage gateways to learn which object storage gateways are available for use.
S12, scoring each object storage gateway in the gateway list according to a preset rule;
after the gateway list is obtained, in order to find the gateway most suitable for the current use, in this step, the gateway score may be stored for each object in the gateway list according to a preset rule. The preset rules may include any scoring rules that can reflect the performance and/or working state of each object storage gateway.
And S13, responding to the data request of the model training task, and requesting data from the target server through the object storage gateway with the highest score in the gateway list.
The running of the model training task requires a code and a data set, and corresponding data can be read from the object memory. The model obtained by model training can also be stored in the object storage server. Therefore, the data request of the model training task may refer to the model training task reading data from the object storage server, or may refer to the model training task writing data to the object storage server.
Based on the score obtained in step S12, when the model training task issues a data request, the data request may be forwarded to the object storage gateway with the highest score in this step, so that the data request is responded through the object storage gateway with the highest score.
The data interaction method for model training provided by the embodiment of the invention can acquire a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, the gateway list comprises at least two object storage gateways, then each object storage gateway in the gateway list is scored according to a preset rule, and in response to a data request of the model training task, the data is requested from the target server through the object storage gateway with the highest score in the gateway list. Therefore, the most suitable object storage gateway can be selected from the gateway list according to the preset rule to perform data interaction with the object storage server, and therefore the data interaction efficiency between the model training server and the object storage server is effectively improved.
In addition, each model training task can flexibly select the most suitable object storage gateway, so that load balance of all object storage gateways can be realized, the problem of unreasonable resource utilization common in NAS and cluster file systems can be avoided on one hand, and reasonable nodes can be automatically selected for data reading on the other hand, and the performance maximization of the system is ensured.
Specifically, the step S11 of acquiring the gateway list corresponding to the target server may specifically include: and pulling a gateway list from each object storage gateway corresponding to the target server through a cache manager of the computing node with the model training task. The computing node may be provided with a cache manager, and the cache manager requests data from the target server. Before requesting data from the target server, the cache manager may first pull a gateway list from each object storage gateway corresponding to the target server. Optionally, the time for pulling the gateway list may be pulling when the computing node is started, or may be pulling every preset time after the computing node is started, for example, pulling every 1 hour.
Specifically, in an embodiment of the present invention, the scoring, according to a preset rule, each object storage gateway in the gateway list in step S12 may include:
acquiring historical operation information of each object storage gateway in the gateway list;
and scoring each object storage gateway in the gateway list according to the historical operation information.
That is, the score of the object storage gateway is based on historical data of the gateway when the gateway actually operates, so that the performance and/or the state of the gateway can be truly reflected. The historical operating information may include a variety of different information types, and different information types may correspond to different scoring rules.
Optionally, in an embodiment of the present invention, the historical operation information may include one or more of the following: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time. The longer the response time length of the data request is, the lower the corresponding score is, and the shorter the response time length is, the higher the corresponding score is. The longer the stable operation time of the gateway is, the higher the corresponding score is, and the shorter the stable operation time is, the lower the corresponding score is. The higher the failure frequency of the gateway, the lower the corresponding score, and the lower the failure frequency, the higher the corresponding score. If the data set B is included in the past data request of the gateway a, the score of the gateway a may be increased according to a preset proportion or a preset amplitude when the data set B is requested again next time.
In order to comprehensively reflect the influence of various historical operating information on the scoring of the gateway, in one embodiment of the invention, a functional relationship between the scoring and various historical operating information can be constructed, so that the scoring of the gateway can be quickly determined according to the functional relationship. The higher the score, the more efficient the data interaction through the gateway.
Since the historical operation information represents the operation information of each gateway in a period of time in the past, the selection of the start-stop node in the period of time can directly determine the specific historical operation information content and can also influence the scoring of the gateways. In order that the historical operating information can properly describe and evaluate whether each gateway is suitable for the current data request, in one embodiment of the invention, the time period of the historical operating information ranges from the current time to the previous time by a preset time length, such as one week, 72 hours, 24 hours, 1 hour and the like.
In many types of historical operation information, the response time of the gateway to the data request is important information, and the response speed of the gateway to the data can be directly reflected. In order to obtain the response time length of the gateway to the data request, in an embodiment of the present invention, the step S11 of obtaining the historical operating information of each object storage gateway in the gateway list specifically includes:
respectively sending a first data request to each object storage gateway in the gateway list;
respectively receiving first data responses of the object storage gateways to the first data requests;
and determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
For example, in an embodiment of the present invention, if three object storage gateways RGW1, RGW2, and RGW3 are included in the gateway list, the first data requests may be sent to the three object storage gateways, and the content of the three first data requests may be the same or different. After receiving the first data request, the three object storage gateways can search data corresponding to the request in the object storage server, and then feed back the searched data to the model training task so as to respond to the first data request. Assume that the time difference between the data reception time and the request transmission time corresponding to RGW1 is 0.3 seconds, the time difference corresponding to RGW2 is 0.5 seconds, and the time difference corresponding to RGW3 is 0.1 seconds. If only the response time length of the data request is taken as the consideration for selecting the object storage gateway, the RGW3 with the shortest response time length can be selected for data interaction. After the model training tasks selecting the RGW3 for data interaction are gradually increased, the response time of the RGW3 is gradually increased, so that the subsequent model training tasks do not select the RGW3 for data interaction, and the load balance of the object storage gateway is realized.
In this embodiment, the response time of the object storage gateway to a single data request is calculated, and further, an average value of the response time of the object storage gateway to a plurality of data requests in a period of time may be calculated, and a corresponding object storage gateway is selected according to the average value.
Through the use of historical operation information, each model training task can select the most suitable object storage gateway, and therefore load balancing of the object storage gateways is achieved. As shown in fig. 3, in one embodiment of the invention, such a load balancing algorithm may be integrated as a plug-in or development kit (SDK) into a cache manager of a file system (e.g., a file system in user space file system) or a cache manager of an object store (e.g., S3), such that both the file system and the object store may fulfill data requests based on the load balancing algorithm.
The following describes in detail the data interaction method for model training provided by the embodiment of the present invention with specific embodiments.
As shown in fig. 4, a data interaction method for model training provided by an embodiment of the present invention may include:
s201, acquiring a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways;
s202, respectively sending a first data request to each object storage gateway in the gateway list;
s203, respectively receiving first data responses of the object storage gateways to the first data requests;
s204, determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request;
s205, obtaining historical operation information of each object storage gateway in the gateway list, wherein the historical operation information comprises the response time of each object storage gateway to a data request;
s206, scoring each object storage gateway in the gateway list according to the historical operation information;
s207, sending a data request by the model training task to request data from a target server;
and S208, responding to a data request of the model training task, and requesting data from the target server through the object storage gateway with the highest score in the gateway list.
Correspondingly, the embodiment of the invention also provides a data interaction device for model training, which can effectively improve the data interaction efficiency of the model training task.
As shown in fig. 5, a data interaction apparatus for model training provided by an embodiment of the present invention may include:
an obtaining unit 31, configured to obtain a gateway list corresponding to a target server, where the target server is an object storage server, the target server stores data required by a model training task, and the gateway list includes at least two object storage gateways;
a scoring unit 32, configured to score each object storage gateway in the gateway list according to a preset rule;
a requesting unit 33, configured to request, in response to a data request of the model training task, data from the target server through an object storage gateway with a highest score in the gateway list.
The data interaction device for model training provided by the embodiment of the invention can acquire a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, the gateway list comprises at least two object storage gateways, then each object storage gateway in the gateway list is scored according to a preset rule, and in response to a data request of the model training task, the data is requested from the target server through the object storage gateway with the highest score in the gateway list. Therefore, the most suitable object storage gateway can be selected from the gateway list according to the preset rule to perform data interaction with the object storage server, and therefore the data interaction efficiency between the model training server and the object storage server is effectively improved.
In addition, each model training task can flexibly select the most suitable object storage gateway, so that load balance of all object storage gateways can be realized. On one hand, the common problem of unreasonable resource utilization in the NAS and the cluster file system can be avoided, on the other hand, reasonable nodes can be automatically selected for data reading, and the performance maximization of the system is guaranteed.
Optionally, the obtaining unit 31 may be specifically configured to pull a gateway list to each object storage gateway corresponding to the target server through a cache manager of a computing node deployed with the model training task.
Optionally, the scoring unit 32 may include:
the acquisition module is used for acquiring historical operation information of each object storage gateway in the gateway list;
and the scoring module is used for scoring each object storage gateway in the gateway list according to the historical operation information.
Optionally, the historical operating information includes at least one of: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time.
Optionally, the time interval range of the historical operating information is a preset time length traced from the current time forward.
Optionally, the historical operation information includes the response time length to the data request;
the acquisition module includes:
a sending submodule, configured to send a first data request to each object storage gateway in the gateway list;
the receiving submodule is used for respectively receiving first data responses of the object storage gateways to the first data requests;
and the determining submodule is used for determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
Correspondingly, the embodiment of the invention also provides the electronic equipment, which can effectively improve the data interaction efficiency of model training.
As shown in fig. 6, an electronic device provided in an embodiment of the present invention may include: the device comprises a shell 51, a processor 52, a memory 53, a circuit board 54 and a power circuit 55, wherein the circuit board 54 is arranged inside a space enclosed by the shell 51, and the processor 52 and the memory 53 are arranged on the circuit board 54; a power supply circuit 55 for supplying power to each circuit or device of the electronic apparatus; the memory 53 is used to store executable program code; the processor 52 executes a program corresponding to the executable program code by reading the executable program code stored in the memory 53, so as to execute the data interaction method for model training provided in any of the foregoing embodiments.
For specific execution processes of the above steps by the processor 52 and further steps executed by the processor 52 by running the executable program code, reference may be made to the description of the foregoing embodiments, and details are not described herein again.
The above electronic devices exist in a variety of forms, including but not limited to:
(1) a mobile communication device: such devices are characterized by mobile communications capabilities and are primarily targeted at providing voice, data communications. Such terminals include: smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) Ultra mobile personal computer device: the equipment belongs to the category of personal computers, has calculation and processing functions and generally has the characteristic of mobile internet access. Such terminals include: PDA, MID, and UMPC devices, etc., such as ipads.
(3) A portable entertainment device: such devices can display and play multimedia content. This type of device comprises: audio, video players (e.g., ipods), handheld game consoles, electronic books, and smart toys and portable car navigation devices.
(4) A server: the device for providing the computing service comprises a processor, a hard disk, a memory, a system bus and the like, and the server is similar to a general computer architecture, but has higher requirements on processing capacity, stability, reliability, safety, expandability, manageability and the like because of the need of providing high-reliability service.
(5) And other electronic equipment with data interaction function.
Accordingly, an embodiment of the present invention further provides a computer-readable storage medium, where one or more programs are stored, and the one or more programs can be executed by one or more processors to implement any one of the data interaction methods for model training provided in the foregoing embodiments, so that corresponding technical effects can also be achieved, which have been described in detail above and are not described herein again.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "comprising", without further limitation, means that the element so defined is not excluded from the group consisting of additional identical elements in the process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments.
In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
For convenience of description, the above devices are described separately in terms of functional division into various units/modules. Of course, the functionality of the units/modules may be implemented in one or more software and/or hardware implementations of the invention.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (14)
1. A data interaction method for model training, comprising:
acquiring a gateway list corresponding to a target server, wherein the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways;
scoring each object storage gateway in the gateway list according to a preset rule;
and responding to a data request of the model training task, and requesting data from the target server through an object storage gateway with the highest score in the gateway list.
2. The method of claim 1, wherein the obtaining the gateway list corresponding to the target server comprises:
and pulling a gateway list from each object storage gateway corresponding to the target server through a cache manager of the computing node with the model training task.
3. The method of claim 1, wherein the scoring each object storage gateway in the gateway list according to a preset rule comprises:
acquiring historical operation information of each object storage gateway in the gateway list;
and scoring each object storage gateway in the gateway list according to the historical operation information.
4. The method of claim 3, wherein the historical operational information comprises at least one of: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time.
5. The method of claim 3, wherein the period of the historical operating information ranges from a current time back to a preset time.
6. The method of claim 3, wherein the historical operational information includes the duration of the response to the data request;
the obtaining of the historical operation information of each object storage gateway in the gateway list includes:
respectively sending a first data request to each object storage gateway in the gateway list;
respectively receiving first data responses of the object storage gateways to the first data requests;
and determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
7. A data interaction apparatus for model training, comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a gateway list corresponding to a target server, the target server is an object storage server, the target server stores data required by a model training task, and the gateway list comprises at least two object storage gateways;
the scoring unit is used for scoring each object storage gateway in the gateway list according to a preset rule;
and the request unit is used for responding to a data request of the model training task and requesting data to the target server through the object storage gateway with the highest score in the gateway list.
8. The apparatus according to claim 7, wherein the obtaining unit is specifically configured to pull a gateway list to each object storage gateway corresponding to the target server through a cache manager of a compute node deployed with the model training task.
9. The apparatus according to claim 7, wherein the scoring unit comprises:
the acquisition module is used for acquiring historical operation information of each object storage gateway in the gateway list;
and the scoring module is used for scoring each object storage gateway in the gateway list according to the historical operation information.
10. The apparatus of claim 9, wherein the historical operational information comprises at least one of: the response time to the data request, the stable operation time of the gateway, the failure frequency of the gateway and the data set range related to the data request of the previous time.
11. The apparatus of claim 9, wherein the period of the historical operating information ranges from a current time to a preset time.
12. The apparatus of claim 9, wherein the historical operating information comprises the duration of the response to the data request;
the acquisition module includes:
a sending submodule, configured to send a first data request to each object storage gateway in the gateway list;
the receiving submodule is used for respectively receiving first data responses of the object storage gateways to the first data requests;
and the determining submodule is used for determining the response time of each object storage gateway to the data request according to the time difference between the receiving time of each group of first data response and the corresponding sending time of the first data request.
13. An electronic device, characterized in that the electronic device comprises: the device comprises a shell, a processor, a memory, a circuit board and a power circuit, wherein the circuit board is arranged in a space enclosed by the shell, and the processor and the memory are arranged on the circuit board; a power supply circuit for supplying power to each circuit or device of the electronic apparatus; the memory is used for storing executable program codes; the processor executes a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the data interaction method for model training of any one of the preceding claims 1 to 6.
14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs which are executable by one or more processors to implement the data interaction method for model training of any one of the preceding claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011595881.8A CN112733892B (en) | 2020-12-28 | 2020-12-28 | Data interaction method and device for model training |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011595881.8A CN112733892B (en) | 2020-12-28 | 2020-12-28 | Data interaction method and device for model training |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112733892A true CN112733892A (en) | 2021-04-30 |
| CN112733892B CN112733892B (en) | 2024-06-21 |
Family
ID=75610190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011595881.8A Active CN112733892B (en) | 2020-12-28 | 2020-12-28 | Data interaction method and device for model training |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112733892B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107992888A (en) * | 2017-11-29 | 2018-05-04 | 深圳市智物联网络有限公司 | The recognition methods of operation of industrial installation and server |
| US20180198841A1 (en) * | 2007-06-12 | 2018-07-12 | Icontrol Networks, Inc. | Generating risk profile using data of home monitoring and security system |
| CN108521656A (en) * | 2018-03-23 | 2018-09-11 | 深圳市信锐网科技术有限公司 | A kind of Lora communication means and system |
| CN109814868A (en) * | 2018-12-13 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Network transmission analogy method, device, computer equipment and storage medium |
| CN109816509A (en) * | 2018-12-14 | 2019-05-28 | 平安科技（深圳）有限公司 | Generation method, terminal device and the medium of scorecard model |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| CN110990138A (en) * | 2019-12-04 | 2020-04-10 | 北京三快在线科技有限公司 | Resource scheduling method, device, server and storage medium |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN111258958A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
| CN111258959A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
| CN111506367A (en) * | 2020-04-20 | 2020-08-07 | 东云睿连(武汉)计算技术有限公司 | Multi-cluster artificial intelligence online service method and system |
| CN111797294A (en) * | 2020-07-09 | 2020-10-20 | 上海商汤智能科技有限公司 | Visualization method and related equipment |
| CN112085208A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Method and device for model training by using cloud |
| CN112087487A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Model training task scheduling method and device, electronic equipment and storage medium |
| CN112084017A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Memory management method and device, electronic equipment and storage medium |
-
2020
- 2020-12-28 CN CN202011595881.8A patent/CN112733892B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180198841A1 (en) * | 2007-06-12 | 2018-07-12 | Icontrol Networks, Inc. | Generating risk profile using data of home monitoring and security system |
| CN107992888A (en) * | 2017-11-29 | 2018-05-04 | 深圳市智物联网络有限公司 | The recognition methods of operation of industrial installation and server |
| CN108521656A (en) * | 2018-03-23 | 2018-09-11 | 深圳市信锐网科技术有限公司 | A kind of Lora communication means and system |
| CN109814868A (en) * | 2018-12-13 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Network transmission analogy method, device, computer equipment and storage medium |
| CN109816509A (en) * | 2018-12-14 | 2019-05-28 | 平安科技（深圳）有限公司 | Generation method, terminal device and the medium of scorecard model |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| CN110990138A (en) * | 2019-12-04 | 2020-04-10 | 北京三快在线科技有限公司 | Resource scheduling method, device, server and storage medium |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN111258958A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
| CN111258959A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
| CN111506367A (en) * | 2020-04-20 | 2020-08-07 | 东云睿连(武汉)计算技术有限公司 | Multi-cluster artificial intelligence online service method and system |
| CN111797294A (en) * | 2020-07-09 | 2020-10-20 | 上海商汤智能科技有限公司 | Visualization method and related equipment |
| CN112085208A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Method and device for model training by using cloud |
| CN112087487A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Model training task scheduling method and device, electronic equipment and storage medium |
| CN112084017A (en) * | 2020-07-30 | 2020-12-15 | 北京聚云科技有限公司 | Memory management method and device, electronic equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| SOTTILARE, R等: "Component interaction within the Generalized Intelligent Framework for Tutoring (GIFT) as a model for adaptive instructional system standards", 《THE ADAPTIVE INSTRUCTIONAL SYSTEM (AIS) STANDARDS WORKSHOP OF THE 14TH INTERNATIONAL CONFERENCE OF THE INTELLIGENT TUTORING SYSTEMS (ITS) CONFERENCE》, 30 June 2018 (2018-06-30), pages 55 - 62 * |
| 韩雪颖: "建筑中庭室内物理环境智能控制系统研究", 《中国优秀硕士学位论文全文数据库 工程科技II辑》, no. 1, 15 January 2020 (2020-01-15), pages 038 - 1722 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112733892B (en) | 2024-06-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112087487B (en) | Scheduling method and device of model training task, electronic equipment and storage medium | |
| CN106250182B (en) | File processing method and device and electronic equipment | |
| CN105487760A (en) | Method and device for displaying message and electronic equipment | |
| CN107391108B (en) | Notification bar information correction method and device and electronic equipment | |
| CN104765793A (en) | Software recommending method and server | |
| WO2014146441A1 (en) | Method, server and system for processing task data | |
| CN111258958A (en) | Data acquisition method, data providing method and device | |
| CN111258965A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112131457A (en) | Information recommendation method, device and system and storage medium | |
| CN105991583A (en) | Game application recommendation method, application server, terminal and system | |
| CN110652728A (en) | Game resource management method and device, electronic equipment and storage medium | |
| CN112084017A (en) | Memory management method and device, electronic equipment and storage medium | |
| CN112085208B (en) | Method and device for training model by cloud | |
| CN113965402A (en) | Configuration method and device of firewall security policy and electronic equipment | |
| CN105320777A (en) | Application program recommendation method and device | |
| CN112749127A (en) | Data providing method and system for model training | |
| CN111258959A (en) | Data acquisition method, data providing method and device | |
| CN112733892A (en) | Data interaction method and device for model training | |
| CN105677926A (en) | Local search result display method and device and electronic equipment | |
| US9280384B2 (en) | Method, server and system for processing task data | |
| CN110719367A (en) | Cloud mobile phone friend recommendation method, device, equipment and storage medium | |
| CN115454670A (en) | Computer process information enumeration method and device and electronic equipment | |
| CN109840182A (en) | A kind of resource monitoring method, device and electronic equipment | |
| CN113987335A (en) | Data collection method and device, electronic equipment and storage medium | |
| CN113609349A (en) | Rental vehicle inventory detection method, apparatus and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |