CN110245003A - A kind of machine learning uniprocessor algorithm arranging system and method - Google Patents
A kind of machine learning uniprocessor algorithm arranging system and method Download PDFInfo
- Publication number
- CN110245003A CN110245003A CN201910493696.9A CN201910493696A CN110245003A CN 110245003 A CN110245003 A CN 110245003A CN 201910493696 A CN201910493696 A CN 201910493696A CN 110245003 A CN110245003 A CN 110245003A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- machine learning
- application container
- container
- uniprocessor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Abstract
The present invention provides a kind of machine learning uniprocessor algorithm arranging system and method.The system features are to include: machine learning algorithm Depending module, are used for predefined machine learning algorithm container mirror image, and store the machine learning algorithm container mirror image of creation；Application container platform, for creating application container based on the machine learning algorithm container mirror image；Customized business algoritic module, for business algorithmic code to be stored in code storage or is stored in distributed file system with static file；Algorithm environment frame, for being embedded in the application container, method and interface that specially a kind of programming language Python is realized.It realizes the grade creation of machine learning algorithm environment second, guarantee that algorithm is finished, supports the effects of more algorithm task loads.
Description
Technical field
The present invention relates to computer technology, in particular to a kind of machine learning uniprocessor algorithm arranging system, method and algorithm
Operation method.
Background technique
Docker is the Application Container technology based on LXC, as general container scheme, has and controls in host
The resource allocation effect of container application, including control core cpu quantity, memory, network and disk volume size.One complete
Docker be generally made of following components: dockerClient client, Docker Daemon finger daemon,
Docker Image mirror image, DockerContainer container.Docker is using C/S framework Docker daemon as server-side
Receive the request from client, and handles these requests (creation, operation, distribution container).Client and server-side can both transport
Row on one machine, can also be communicated by socket RESTful API.Docker daemon is generally in place
Master host running background waits the message to be received from client.Docker client then provides for user a series of executable
Order, user are realized with these orders with Docker daemon interaction.
Hadoop YARN is the scheduling system of distributed type assemblies resource management and distribution, supports resource queue's isolation and meter
The ability of operator node grouping management can support the resource bid for managing a variety of distributed algorithms and frame.Hadoop YARN is
One universal resource management system can provide unified resource management and scheduling for upper layer application, it is introduced as cluster in benefit
Big advantages are brought with rate, resource unified management and data sharing etc..
Jekins is a kind of code automatic deployment frame, integrated applied to software continuous, the volume in automated software exploitation
It translates, distribute, disposing, test job process.It can be installed by yum, or downloading war packet and pass through the quick reality such as docker container
Existing installation and deployment, can facilitate web interface configuration management.
Kubernates is for container automatic deployment, extension and the open source system for managing containerization application program.
Container does not have the ability of data and state that migration application generates, it is suitble to the application statelessly relied on, still
For machine learning uniprocessor algorithm calculating for, need save and shift data output, container do not have data environment migration and
The ability of data resource management；
And Hadoop YARN can only manage developing according to YARN resource interface for task, and cannot be compatible with and largely deposit
Uniprocessor algorithm resource management；
In addition, Jekins is only used as compiling automatically, distributes, disposes, the target of test code is realized, and cannot complete
The management that algorithm application is inputted, exported；
In addition, conventional individual running environment is unsatisfactory for algorithm stability requirement, then arithmetic result is lost for running environment collapse,
And traditional application deployment way is to install application by plug-in unit or script.Disadvantage of this is that the operation of application, match
Set, manage, all life cycles will bind with current operation system, do so the upgrading update/rollback etc. for being unfavorable for application
Operation, naturally it is also possible to certain functions are realized by way of creating virtual machine, but virtual machine is very heavy, being unfavorable for can
Transplantability.
Machine learning uniprocessor algorithm layout operation method, often the NameSpace in operating system layer of multi-tenant is isolated
System-level isolation after environment, user isolation including operating system layer and operating system virtualization, but such operation side
Method:
1. the running environment fast construction of multi-tenant multitask (uniprocessor algorithm calculating) cannot be supported, resource overhead is big；
2. environment parameter flexible configuration is not supported, it is instant to take, it recycles in time, may be programmed managerial difference
Summary of the invention
To solve the above-mentioned problems, an aspect of of the present present invention provides a kind of machine learning uniprocessor algorithm arranging system,
It is characterized in that, comprising: machine learning algorithm Depending module is used for predefined machine learning algorithm container mirror image, and stores creation
The machine learning algorithm container mirror image；Application container platform, for being created based on the machine learning algorithm container mirror image
Application container；Customized business algoritic module, for business algorithmic code to be stored in code storage or is deposited with static file
It is stored in distributed file system, the business algorithmic code completes outputting and inputting for algorithm in the application container；Algorithm
Environment framework, for being embedded in the application container, method and interface that specially a kind of programming language Python is realized.
Further, machine learning uniprocessor algorithm arranging system as the aforementioned, the algorithm environment frame include: algorithm fortune
Row master control, for the algorithm task resource description integration into the application container will to be distributed；
The application container platform includes Container Management master control, is held for perceiving and disposing the application according to external request
Device；
The application container includes operation agency, for being responsible for and algorithm operation master control communication, Processing Algorithm environment
It initializes and passes algorithm state and result description back to algorithm operation master control in algorithm end of run；And algorithm mould
Block, for receiving the operation parameter that distributes of agency and running feedback.
Further, machine learning uniprocessor algorithm arranging system as the aforementioned, the algorithm operation master control description integration
Algorithm task resource includes algorithm parameter, algorithm dependence, algorithm resource.
Further, machine learning uniprocessor algorithm arranging system as the aforementioned, the machine learning algorithm Depending module is also
Include: external software packet, is stored in the base management system of corresponding software class or distributed text is purely stored in document form
Part system, for supporting the algorithm environment frame to complete machine learning algorithm.
Further, machine learning uniprocessor algorithm arranging system as the aforementioned, the algorithm environment frame further include: log
And monitoring unit, the program language for collecting and monitoring output journal information access the content that object is sent；And storage is single
Member, for the program language access object of access algorithm parameter and the distributed file system of access analysis data content.
Another aspect provides it is a kind of it is preceding it is any as described in machine learning uniprocessor algorithm method of combination, it is special
Sign is, comprising: mirror image foundation step is based on Container Management tool, and the environment of machine learning algorithm is supported in pre-preparation and creation
Mirror image；Business algorithmic code applying step, the definition business algorithmic code module pre-preparation algorithm service application code are used
In outputting and inputting for completion algorithm；Requirement command step, according to external algorithm assignment instructions demand, Xiang Suoshu algorithm operation master
Parameter collection, arrangement before controlling submission task；And application container foundation step, the application container platform are based on matched institute
Mirror image creation application container is stated, realizes the programmable management of algorithm operation resource environment.
Further, further include Resource Calculation and environmental test step before the application container foundation step, be used for
Conformity calculation is described to the required resource of application container creation and the environment needed to application container creation is examined
It looks into.
Further, the Resource Calculation and environmental test step further include that algorithm parameter is passed to step, by related algorithm
Parameter is incoming to be used with being supplied to specific algorithm for operation agency；Algorithm relies on library and lists step, and machine learning is calculated
The external program and code that method operation and calculating rely on list environment installation kit and script for preparing may rely on；Algorithm money
Step is applied in source, and application meets the algorithm resource needed for needing for realizing algorithm operation.
Further, the application container foundation step includes: that mirror image pulls step, according to algorithm requirements, judges to match
Mirror image and pull；Application container deploying step, from the interim running environment of mirror image dynamic creation pulled, by application container
Management tool selects host, and specifies the parameter of this operation algorithm；Algorithm environment initialization step, the application container
The algorithm environment frame is initialized after host starting；And algorithm operating procedure, the algorithm environment
After frame completes initialization, machine learning algorithm is adjusted, the output that algorithm generates is passed back by operation agency；Step is terminated,
Algorithm end of run in the application container, the application container exit, and resource is withdrawn, and notify resource and environmental test strategy,
Resource is conceded for new calculating.
It further, further include Resource Calculation and environmental test judgment step before the application container foundation step, institute
It states in Resource Calculation and environmental test step, if resource description conformity calculation and the environmental test needed to application container creation
Normally, then step is pulled into the mirror image, if abnormal, enters the termination step.
Further, the algorithm environment initialization step include: the algorithm environment frame acquisition algorithm parameter and under
It is downloaded in the application container local；The algorithm environment frame downloads to the external software packet in the application container i.e.
When dispose；Algorithm environment frame machine learning algorithm itself program downloads in the application container.
Further, it needs to be pre-configured with algorithm in the mirror image foundation step to rely on library and be embedded in the machine learning
Environment framework.
Further, the file access in the algorithm environment frame is used in the business algorithmic code applying step
Object, output journal object get parms information object to complete outputting and inputting for algorithm.
Further, in the business algorithmic code applying step, the business algorithm application code, which needs to upload, to be protected
It deposits to the distributed file system, and obtains unique access download path.
Further, the requirement command step further include: path obtaining step is walked in business algorithmic code application
In the algorithm saved in rapid, the selected algorithm for needing to run obtains unique access download path；Resource parameters list step
Suddenly, acquisition algorithm running environment requirement, lists algorithm resource parameters；ID obtaining step, after obtaining the resource parameters, storage
Into the storage unit, the corresponding unique ID of resource parameters of this storage is obtained.
Further, this method further include: container is restarted automatically step, and the application container is non-in the host environment
The Container Management tool is restarted automatically the application container when normal interruption.
Further, machine learning uniprocessor algorithm method of combination as the aforementioned further includes that the algorithm operating procedure includes:
Operation agency distributes step, and the operation agency distributes the algorithm parameter to the algoritic module；Algoritic module communication step
Suddenly, the algoritic module is acted on behalf of with the operation after the completion of operation and carries out communications feedback；State outcome passes step, algorithm fortune back
At the end of row, algorithm state and result description are passed back.
Reach realization the present invention provides a kind of machine learning uniprocessor algorithm arranging system, method and supports uniprocessor algorithm
The programmable management of computing resource environment, the purpose for supporting data isolation, running environment isolation, while the present invention realizes 1. machines
The device learning algorithm environment second grade creation, support programming interface set environment parameter, including dependent software package version, CPU, memory,
Training data is specified；2. guaranteeing that algorithm has executed because external system reason arbitrarily terminates and restarts when supporting algorithm environment operation
Finish；3. cooperating existing container cluster administrative skill, the present invention can achieve resilient expansion computing resource, support more algorithm tasks
The management and running ability of load.
Detailed description of the invention
Fig. 1 is the overall framework schematic diagram of machine learning uniprocessor algorithm arranging system of the invention.
Fig. 2 is the schematic diagram that application container process is created in machine learning uniprocessor algorithm method of combination of the invention.
Fig. 3 is the schematic diagram of algorithm operational process in machine learning uniprocessor algorithm method of combination of the invention.
Specific embodiment
Exemplary embodiments of the present invention are illustrated below in conjunction with attached drawing, it should be understood that provide these embodiment party
Formula is used for the purpose of making those skilled in the art can better understand that realizing the present invention in turn, and not limit in any way
The scope of the present invention.
[machine learning uniprocessor algorithm arranging system]
Fig. 1 is the overall framework schematic diagram of machine learning uniprocessor algorithm arranging system of the invention, and Fig. 2 is machine of the invention
Device learns the schematic diagram of uniprocessor algorithm arranging system operation creation application container process, and Fig. 3 is machine learning single machine of the invention
The schematic diagram of the algorithm operation method of algorithm layout application container.Fig. 1, Fig. 2 and Fig. 3 embody machine learning of the invention jointly
The structure composition of uniprocessor algorithm arranging system.
As shown in Figure 1, machine learning uniprocessor algorithm arranging system includes machine learning algorithm Depending module 1, application container
Platform 2, customized business algoritic module 3 and algorithm environment frame 4, wherein machine learning algorithm Depending module 1 is for predefining
Machine learning algorithm container mirror image interacts, the machine created in present system with the mirror image warehouse system of Container Management class tool
Device study mirror image is stored in wherein, and machine learning algorithm Depending module 1 includes but is not limited to SK-learn mould in the present invention
The external softwares packets such as block, WEKA module, scipy module, numpy module, be stored in the base management system of corresponding software class or
Distributed file system is purely stored in document form, for supporting the algorithm environment frame to complete machine learning algorithm,
It can certainly use other that other modules or class or data set etc. of predefined machine learning algorithm container mirror image may be implemented.
Wherein, mirror image is a kind of cured container environment configurations, and be static derived from the corresponding technology entities of Image in Docker
, multiple mirror image trustships have unique id to identify in the mirror image warehouse of concentration.
Application container platform 2 is used to create application container based on the machine learning algorithm container mirror image, and application container is flat
It can include but is not limited to control interface module, scheduler module, engine module, network module, monitoring module etc. in platform 2, when
It so also may include other such as control modules.Wherein, application container is real derived from the corresponding technology of Container in Docker
Body, it is an interim dynamic running environment being created that from mirror image, occupies and provides the computing resource of the host of operation.
Customized business algoritic module 3 is for being stored in code storage for business algorithmic code or being stored with static file
In distributed file system, business algorithmic code completes outputting and inputting for algorithm in application container.
Algorithm environment frame 4 is for being embedded in application container, method and connect that specially a kind of programming language Python is realized
Mouthful, the program language of access object, offer proxy database connection in the program language including providing file type data is visited
It asks object, the program language access object of output journal information is provided, the program language access object of acquisition algorithm parameter is provided；
Algorithm environment frame 4 further includes collection and monitoring unit, collects and monitor algorithm operating status in application container for providing
Service, the specially one log monitoring service based on message-oriented middleware monitor the log information program language access pair
As the content of transmission；Algorithm environment frame 4 further includes storage unit, for providing the service for saving Machine Learning Parameter, specifically
For a parameter management service based on distributed key assignments storage engines, the ginseng saved and inquiry machine learning algorithm is run is supported
It counts, in the trustship algorithm environment frame, acquisition algorithm parameter program language is accessed in the data in this service of object accesses
Hold.
Here, although to include machine learning algorithm Depending module 1, application container platform 2, customized business algoritic module
3 and the equal hardware configurations of algorithm environment frame 4 be presented the overall architecture of machine learning uniprocessor algorithm arranging system of the present invention, but this
The machine learning uniprocessor algorithm arranging system of invention is to realize that those skilled in the art answer by the following each step that will be described
When the realization for understanding each step is not limited by hardware configuration, therefore machine learning uniprocessor algorithm arranging system of the invention is not
It is confined to the hardware configuration illustrated in application documents, as long as the structure for covering the function that the present invention can be realized belongs to this
Invent range claimed.
[machine learning uniprocessor algorithm arranging system running environment]
As shown in Fig. 2, the algorithm operation master control 21 in application container needs to describe during following creation application containers
Integration algorithm task resource algorithm parameter 211, algorithm rely on 212, algorithm resource 213.Wherein algorithm parameter 211 is joined including data
Several and systemic parameter two parts, data parameters are supplied to specific algorithm as agency using file object and use, systemic parameter
It is then saved in the service of the machine learning algorithm parameter of storage unit offer.
It is machine learning algorithm operation and the external program and code for calculating dependence, including machine word that algorithm, which relies on 212,
Say running environment, third party's program library etc., preparing when pre-preparation and creation support the mirror image of the environment of machine learning algorithm can
The environment installation kit and script that can be relied on.
Algorithm resource 213 indicates the calculating core amounts of algorithm operation needs, amount of memory, and computing resource is that have total amount limit
System, resource needed for needing when multitask running to distribute according to need and check whether satisfaction.
The algorithm running environment that 3 entities such as 212, algorithm resource 213 form is relied on by algorithm above parameter 211, algorithm,
Each machine learning algorithm issues operation request, and the programmable management of specific implementation algorithm resource environment is that an algorithm is executable
Strategy:
When existing algorithms library and computing resource meet applied condition, then operation is distributed by Container Management interface and calculated
Otherwise method task refuses simultaneously termination process；
Computing resource total amount is fixed, but available quantity is relative dynamic, therefore the strategy has needed recycling in time
The resource of exiting for task is completed, task releases computing resource when exiting automatically, needs the state of timely updating and is new
Task makes a decision preparation.
Here, although to include that algorithm parameter 211, algorithm rely on the 212, parameter resources such as algorithm resource 213 this are presented
Invention machine learning uniprocessor algorithm arranging system running environment, but the operation of machine learning uniprocessor algorithm arranging system of the invention
It is to be realized by the following each step that will be described, it should be understood by those skilled in the art that the realization of each step is not provided by parameter
The limitation in source, therefore the operation of machine learning uniprocessor algorithm arranging system of the invention is not limited to the ginseng illustrated in application documents
Number resource, as long as the parameter resource for covering the function that the present invention can be realized belongs to present invention model claimed
It encloses.
[machine learning uniprocessor algorithm method of combination]
It include holding in container platform as shown in figure 3, including that algorithm runs master control 21 in algorithm environment frame when algorithm is run
Device manages master control 22, and application container includes: operation agency 23 and algoritic module 24.
Algorithm operation master control 21 is used to distribute the algorithm task resource description integration into the application container, all machines
Device learning algorithm is registered in advance to be belonged in the range of present invention needs realization in algorithm master control, and algorithm is stored in known to master control
Position, can be the either central code library in path of distributed storage file system.
Container Management master control 22 is all to dispose machine learning for perceiving and according to external request application deployment container
The host of algorithm is perceived by Container Management master control and is responsible for according to external request deployment container, its realization can rely on container
Manage class tool.
Operation agency 23 is for being responsible for algorithm operation master control communication, Processing Algorithm context initialization and in algorithm
Algorithm state and result description are passed back to the algorithm operation master control when end of run, are responsible for and algorithm operation master control communication.Fortune
The relationship of row agency 23 and application container: it is the program executed after application container creates prior to algorithm；Operation agency
23 functions are responsible for Processing Algorithm context initialization, including from algorithm master control acquisition algorithm itself program, distribute in container, obtain
It takes parameter and executes state to algorithm master control feedback algorithm.
Algoritic module 24, for receiving the operation parameter that distributes of agency and running feedback.
Here, although to include algorithm operation master control 21, Container Management master control 22, operation agency 23 and algoritic module 24 etc.
Module runs module, but machine learning single machine of the invention machine learning uniprocessor algorithm arranging system algorithm of the invention is presented
The operation of algorithm arranging system algorithm is realized by the following each step that will be described, it should be understood by those skilled in the art that each step
Rapid realization is not limited by module, therefore the operation of the algorithm of machine learning uniprocessor algorithm arranging system of the invention not office
It is limited to the module illustrated in application documents, as long as the module for covering the function that the present invention can be realized belongs to institute of the present invention
Claimed range.
[machine learning uniprocessor algorithm method of combination]
Illustrate the operation method of machine learning uniprocessor algorithm arranging system of the invention below, comprising:
Mirror image foundation step S1, is based on Container Management tool, and the environment of machine learning algorithm is supported in pre-preparation and creation
Mirror image, wherein containing trustship algorithm environment frame 4, the external software packet that machine learning algorithm relies on.
Here, the machine learning algorithm Depending module 1 in mirror image foundation step S1 corresponding diagram 1 is used for predefined engineering
Container mirror image is practised, is interacted with the mirror image warehouse system of Container Management class tool, the machine learning mirrored storage of creation is in wherein；
In addition there are also the external software packet that machine learning relies on, it is stored in the base management system of corresponding software class, naturally it is also possible to purely
With the dependence of document form, it is stored in distributed file system.
Business algorithmic code applying step S2, customized 3 pre-preparation algorithm service application code of business algorithmic code module,
Wherein algorithm application code needs to realize based on trustship algorithm environment frame 4, uses the file access object in frame, output day
Will object, get parms information object, completes outputting and inputting for algorithm, handles data parameters, using file object as operation
Agency is supplied to specific algorithm use；Wherein, the application code of business algorithmic code applying step S2 needs to upload to save and extremely divide
Cloth file system, and obtain a unique access download path；Here, business algorithmic code applying step S2 corresponds to Fig. 1
In customized business algorithmic code module 3, business algorithmic code is stored in code storage or is stored in point with static file
Cloth file system.
Path obtaining step S3, it is selected to need to run in the algorithm saved in business algorithmic code applying step S2
Algorithm, obtain unique access download path.
Resource parameters list step S4, and algorithm resource parameters are listed in the requirement of acquisition algorithm running environment.
Data Identification step S5 is analyzed, analysis data used in business algorithm upload to storage unit, obtain access path；
Here, path obtaining step S3, resource parameters list step S4, ID obtaining step S5 and are referred to as requirement command step
Suddenly, for the parameter collection, arrangement according to external algorithm assignment instructions demand, before running 21 submission task of master control to algorithm.
Application container foundation step S6, application container platform 2 are based on matched mirror image and create application container, realize algorithm fortune
The programmable management of row resource environment is based on container CLI command interface, based on the matched mirror image creation of path obtaining step S3
Application container, when creating container can the obtained algorithm path afferent pathway obtaining step S3, ID obtaining step S5 obtain it is unique
The relevant start-up parameter of ID and CPU, memory, to realize the programmable management of algorithm operation resource environment.
Here, the detailed process of application container foundation step S6 will be explained later.
Container is restarted automatically step S7, is based on Container Management master control 22, and algorithm application container is improper in host environment
When interruption, Container Management tool is restarted automatically application container, reaches machine learning algorithm application environment stability height, does not lose knot
The effect of fruit.
Machine learning uniprocessor algorithm arranging system operation method according to the present invention, especially application container foundation step
S6, realizes the invention effect of machine learning algorithm environment second grade creation, and supports programming interface set environment parameter, including
Dependent software package version, CPU, memory, training data are specified etc..
[creation application container process]
Fig. 2 is the schematic diagram of machine learning uniprocessor algorithm arranging system operation creation application container process of the invention.Under
Face describes creation application container process of the invention according to fig. 2.Include:
Algorithm parameter is passed to step S611, related algorithm parameter 211 is passed to be used to run agency 23 and be supplied to specifically
Algorithm uses；
Algorithm relies on library and lists step S612, machine learning algorithm is run and calculated the external program relied on and code column
Out for preparing the environment installation kit and script that may rely on；
Algorithm resource bid step S613, application meet the algorithm resource 213 needed for needing for realizing algorithm operation.
Resource Calculation and environmental test step S61, it is whole for the required resource that application container creates to be described
Total environment calculated and needed to application container creation checks.
Resource Calculation and environmental test judgment step S614, to the result of the S61 in Resource Calculation and environmental test step
Judged, if resource description conformity calculation and to application container creation need environmental test it is normal, enter mirror image draw
Step S62 is taken, if abnormal, enter and terminates step S66.
Mirror image pulls step S62, and according to above description, mirror image is a kind of static resource, during algorithm relies on, such as language ring
Border can save in mirror image in advance, therefore, when creating container, first according to algorithm requirements, judge that matched mirror image goes to create
Container；The present invention claims all mirror images needs to be pre-configured with, and in addition to algorithm relies on library, to be also embedded in machine learning initialization context
Frame.
Application container deploying step S63, container are to occupy host from the interim running environment of the mirror image dynamic creation pulled
The computing resource of machine, for corresponding to Container Management, generally there are multiple hosts for operation, based on connecing for device management tool
Mouth (as described in Fig. 1 Docker container platform module), container only provides when disposing calculates core, memory requirements and specified mirror
The host of picture, specific deployment container is selected by Container Management tool；Deployment container also needs to specify this operation algorithm herein
Parameter；
Algorithm environment initialization step S64, container, because having been inserted into initialization context frame, obtain after host starting
After algorithm parameter, the file object that data parameters transfer to environment framework to act on behalf of downloads to local in container；Machine learning algorithm
The dependence of static file packet is also downloaded in container by environment framework to be disposed immediately；Machine learning algorithm itself program is by environment framework
It downloads in container.
Algorithm operating procedure S65 has adjusted machine after algorithm environment initialization step S64 completes environment framework initialization
Learning algorithm, the output that algorithm generates are passed back by environment framework agency.
Here, the detailed step of the operation of algorithm involved in algorithm operating procedure S65 is discussed below.
Step S66, algorithm end of run in container are terminated, container exits, and resource is withdrawn, and notifies resource and environmental test plan
Slightly, resource is conceded for new calculating；
[algorithm operational process]
Fig. 3 is the schematic diagram of machine learning uniprocessor algorithm method of combination of the invention.It is described below according to Fig. 3 of the invention
Algorithm operation method.Include:
Resource consolidation step S650 (not shown), algorithm run master control 21 and describe to integrate by the algorithm task resource distributed,
Here, algorithm task resource includes algorithm parameter 211, algorithm dependence 212, algorithm resource 213.
Application container starting step S651, algorithm run master control 21 to 22 transmission algorithm resource parameters of Container Management master control.
Application container deploying step S652, to host deployment container after 22 receiving algorithm resource parameters of Container Management master control
The application container is set to run.
Operation agency distributes step S653, and operation agency 23 distributes algorithm parameter into algoritic module 24.
Algoritic module communication steps S654, algoritic module 24 carry out communications feedback with operation agency 23 after the completion of operation.
Here, operation agency distributes step S653 and algoritic module communication steps S654 and is referred to as application container operation step
Suddenly, after container operation, algorithm operation agency 23 runs 21 acquisition algorithm parameter 211 of master control to algorithm, in container at the beginning of algorithm
Beginningization file object, log object and parameter object；File, log, parameter object are all the specific generations that algorithm is obtained and exported
Reason, algorithm obtain operating parameter by operation agency 23.
State outcome passes step S655 back, when algorithm end of run, runs master control 21 to algorithm by operation agency 23 and passes
Return algorithm state and result description.
Algorithm operation method according to the present invention, the Container Management master control based on dependence give compatible container management master
Control deployment, the cooperative approach for running agency, algorithm master control for expanding container resource, being capable of elastic telescopic host based on container master control
Machine quantity, the characteristic for being restarted automatically the improper container exited realize when supporting algorithm environment operation because of external system reason
It arbitrarily terminates and restarts, guarantee the invention effect that algorithm is finished；And it may be implemented to cooperate existing container cluster management
Technology can achieve the invention effect of resilient expansion computing resource, the management and running ability for supporting more algorithm task loads.
Claims (19)
1. a kind of machine learning uniprocessor algorithm arranging system characterized by comprising
Machine learning algorithm Depending module is used for predefined machine learning algorithm container mirror image, and stores the machine of creation
Learning algorithm container mirror image；
Application container platform, for creating application container based on the machine learning algorithm container mirror image；And
Customized business algoritic module, for business algorithmic code to be stored in code storage or is stored in point with static file
Cloth file system, the business algorithmic code complete outputting and inputting for algorithm in the application container；
Algorithm environment frame, for being embedded in the application container, control business algorithmic code executes in the application container.
2. machine learning uniprocessor algorithm arranging system according to claim 1, which is characterized in that the algorithm environment frame
It include: algorithm operation master control, for the algorithm task resource description integration into the application container will to be distributed.
3. machine learning uniprocessor algorithm arranging system according to claim 1, which is characterized in that the application container platform
It include: Container Management master control, for perceiving and disposing the application container according to external request.
4. machine learning uniprocessor algorithm arranging system according to claim 3, which is characterized in that the application container packet
It includes:
Operation agency is used to be responsible for algorithm operation master control communication, Processing Algorithm context initialization and in algorithm operation
At the end of to the algorithm operation master control pass back algorithm state and result description；
Algoritic module, for receiving the operation parameter that distributes of agency and running feedback.
5. machine learning uniprocessor algorithm arranging system according to claim 2, which is characterized in that the algorithm runs master control
The algorithm task resource of description integration includes algorithm parameter, algorithm dependence, algorithm resource.
6. -5 machine learning uniprocessor algorithm arranging system described in any one according to claim 1, which is characterized in that the machine
Device learning algorithm Depending module further include:
External software packet is stored in the base management system of corresponding software class or is purely stored in distributed document with document form
System, for supporting the algorithm environment frame to complete machine learning algorithm.
7. -5 machine learning uniprocessor algorithm arranging system described in any one according to claim 1, which is characterized in that the calculation
Method environment framework further include:
Log and monitoring unit, the program language for collecting and monitoring output journal information access the content that object is sent；With
And
Storage unit, for the program language access object of access algorithm parameter and the distributed document of access analysis data content
System.
8. a kind of machine learning uniprocessor algorithm method of combination characterized by comprising
Mirror image foundation step, predefined machine learning algorithm container mirror image, and store the machine learning algorithm container of creation
Mirror image；
Business algorithmic code applying step, predefined algorithm service application code；
Requirement command step, algorithm environment frame are parameter collection before submission task, whole according to external algorithm assignment instructions demand
Manage situation；
Application container foundation step creates application container based on the matched mirror image.
9. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that created in the application container
Building step further includes before Resource Calculation and environmental test step, comprising: the algorithm environment frame creates the application container
Conformity calculation is described in the required resource built and the environment needed to application container creation checks.
10. machine learning uniprocessor algorithm method of combination according to claim 9, which is characterized in that the Resource Calculation and
Environmental test step further include:
Algorithm parameter is passed to step, and related algorithm parameter is passed to；
Algorithm relies on library and lists step, runs and calculates the external program of dependence for machine learning algorithm and code is listed and is used for standard
The standby environment installation kit and script that may rely on；
Algorithm resource bid step, application meet the algorithm resource needed for needing for realizing algorithm operation.
11. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that the application container wound
Building step includes:
Mirror image pulls step, according to algorithm requirements, judges matched mirror image and pulls；
Application container deploying step, from the interim running environment of mirror image dynamic creation pulled, by application container management tool
Host is selected, and specifies the parameter of this operation algorithm；
Algorithm environment initialization step, the application container carry out just the algorithm environment frame after host starting
Beginningization；And
Algorithm operating procedure after the algorithm environment frame completes initialization, runs machine learning algorithm, what output algorithm generated
Data；
Step, algorithm end of run in the application container are terminated, the application container exits.
12. machine learning uniprocessor algorithm method of combination according to claim 11, which is characterized in that the mirror image pulls step
Before rapid further include: Resource Calculation and environmental test judgment step, in the Resource Calculation and environmental test judgment step, if money
The Source Description conformity calculation and environmental test needed to application container creation is normal, then pull step into the mirror image, if
It is abnormal, then enter the termination step.
13. machine learning uniprocessor algorithm method of combination according to claim 11, which is characterized in that at the beginning of the algorithm environment
Beginningization step includes:
The algorithm environment frame acquisition algorithm parameter simultaneously downloads to local in the application container；
The external software packet is downloaded in the application container and is disposed immediately by the algorithm environment frame；
The algorithm environment frame downloads to machine learning algorithm itself program in the application container.
14. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that created in the mirror image
It needs to be pre-configured with algorithm in step to rely on library and be embedded in the algorithm environment frame.
15. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that in the business algorithm
In code applying step, using in the algorithm environment frame file access object, output journal object, get parms information
Object completes outputting and inputting for algorithm.
16. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that in the business algorithm
In code applying step, the business algorithm application code, which needs to upload, to be saved to distributed file system, and obtains unique visit
Ask download path.
17. machine learning uniprocessor algorithm method of combination according to claim 8, which is characterized in that the requirement command step
Suddenly further include:
Path obtaining step, in the algorithm saved in the business algorithmic code applying step, the selected calculation for needing to run
Method obtains unique access download path；
Resource parameters list step, and algorithm resource parameters are listed in the requirement of acquisition algorithm running environment；
Data Identification step is analyzed, analysis data used in business algorithm upload to storage unit, obtain access path.
18. any machine learning uniprocessor algorithm method of combination of 1-17 according to claim 1, which is characterized in that further include:
Container is restarted automatically step, application container Container Management tool in the improper interruption of the host environment
It is restarted automatically the application container.
19. a kind of any machine learning uniprocessor algorithm method of combination of 1-17 according to claim 1, which is characterized in that institute
Stating algorithm operating procedure includes:
Operation agency distributes step, and the operation agency of the application container distributes algorithm parameter to the algorithm of the application container
Module；
Algoritic module communication steps, the algoritic module of the application container after the completion of operation with the operation generation of the application container
Reason carries out communications feedback.
State outcome passes step back, when algorithm end of run, passes algorithm state and result description back.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910493696.9A CN110245003A (en) | 2019-06-06 | 2019-06-06 | A kind of machine learning uniprocessor algorithm arranging system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910493696.9A CN110245003A (en) | 2019-06-06 | 2019-06-06 | A kind of machine learning uniprocessor algorithm arranging system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110245003A true CN110245003A (en) | 2019-09-17 |
Family
ID=67886388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910493696.9A Pending CN110245003A (en) | 2019-06-06 | 2019-06-06 | A kind of machine learning uniprocessor algorithm arranging system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245003A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780987A (en) * | 2019-10-30 | 2020-02-11 | 上海交通大学 | Deep learning classroom analysis system and method based on container technology |
| CN111199430A (en) * | 2020-01-08 | 2020-05-26 | 中信银行股份有限公司 | Service solution generation method and device and electronic equipment |
| CN111290778A (en) * | 2020-02-06 | 2020-06-16 | 网易（杭州）网络有限公司 | AI model packaging method, platform and electronic equipment |
| CN111506402A (en) * | 2020-03-31 | 2020-08-07 | 上海氪信信息技术有限公司 | Computer task scheduling method, device, equipment and medium for machine learning modeling |
| CN111552524A (en) * | 2020-05-06 | 2020-08-18 | Oppo（重庆）智能科技有限公司 | Plug-in loading method and device and computer readable storage medium |
| CN112311605A (en) * | 2020-11-06 | 2021-02-02 | 北京格灵深瞳信息技术有限公司 | Cloud platform and method for providing machine learning service |
| CN112817581A (en) * | 2021-02-20 | 2021-05-18 | 中国电子科技集团公司第二十八研究所 | Lightweight intelligent service construction and operation support method |
| CN114064213A (en) * | 2021-11-16 | 2022-02-18 | 四川启睿克科技有限公司 | Kubernets container environment-based rapid arranging service method and system |
| CN114064155A (en) * | 2022-01-17 | 2022-02-18 | 深圳市华付信息技术有限公司 | Container-based algorithm calling method, device, equipment and storage medium |
| CN114115857A (en) * | 2021-10-29 | 2022-03-01 | 北京邮电大学 | Method and system for constructing automatic production line of machine learning model |
| CN114638373A (en) * | 2020-12-15 | 2022-06-17 | Aptiv技术有限公司 | Managing machine learning environment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030217195A1 (en) * | 2002-05-16 | 2003-11-20 | Sun Microsystems, Inc. | Method and apparatus for managing host-based data services using CIM providers |
| CN106657248A (en) * | 2016-11-01 | 2017-05-10 | 山东大学 | Docker container based network load balancing system and establishment method and operating method thereof |
| CN107450961A (en) * | 2017-09-22 | 2017-12-08 | 济南浚达信息技术有限公司 | A kind of distributed deep learning system and its building method, method of work based on Docker containers |
| CN107480027A (en) * | 2017-07-07 | 2017-12-15 | 上海诺悦智能科技有限公司 | A kind of distributed deep learning operational system |
| CN107733977A (en) * | 2017-08-31 | 2018-02-23 | 北京百度网讯科技有限公司 | A kind of cluster management method and device based on Docker |
| CN108958892A (en) * | 2018-08-14 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method and apparatus creating the container for deep learning operation |
| CN109146084A (en) * | 2018-09-06 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of method and device of the machine learning based on cloud computing |
-
2019
- 2019-06-06 CN CN201910493696.9A patent/CN110245003A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030217195A1 (en) * | 2002-05-16 | 2003-11-20 | Sun Microsystems, Inc. | Method and apparatus for managing host-based data services using CIM providers |
| CN106657248A (en) * | 2016-11-01 | 2017-05-10 | 山东大学 | Docker container based network load balancing system and establishment method and operating method thereof |
| CN107480027A (en) * | 2017-07-07 | 2017-12-15 | 上海诺悦智能科技有限公司 | A kind of distributed deep learning operational system |
| CN107733977A (en) * | 2017-08-31 | 2018-02-23 | 北京百度网讯科技有限公司 | A kind of cluster management method and device based on Docker |
| CN107450961A (en) * | 2017-09-22 | 2017-12-08 | 济南浚达信息技术有限公司 | A kind of distributed deep learning system and its building method, method of work based on Docker containers |
| CN108958892A (en) * | 2018-08-14 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method and apparatus creating the container for deep learning operation |
| CN109146084A (en) * | 2018-09-06 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of method and device of the machine learning based on cloud computing |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780987A (en) * | 2019-10-30 | 2020-02-11 | 上海交通大学 | Deep learning classroom analysis system and method based on container technology |
| CN111199430A (en) * | 2020-01-08 | 2020-05-26 | 中信银行股份有限公司 | Service solution generation method and device and electronic equipment |
| CN111290778A (en) * | 2020-02-06 | 2020-06-16 | 网易（杭州）网络有限公司 | AI model packaging method, platform and electronic equipment |
| CN111290778B (en) * | 2020-02-06 | 2024-03-01 | 网易（杭州）网络有限公司 | AI model packaging method, platform and electronic equipment |
| CN111506402B (en) * | 2020-03-31 | 2023-06-27 | 上海氪信信息技术有限公司 | Computer task scheduling method, device, equipment and medium for machine learning modeling |
| CN111506402A (en) * | 2020-03-31 | 2020-08-07 | 上海氪信信息技术有限公司 | Computer task scheduling method, device, equipment and medium for machine learning modeling |
| CN111552524B (en) * | 2020-05-06 | 2023-10-13 | Oppo（重庆）智能科技有限公司 | Plug-in loading method and device and computer readable storage medium |
| CN111552524A (en) * | 2020-05-06 | 2020-08-18 | Oppo（重庆）智能科技有限公司 | Plug-in loading method and device and computer readable storage medium |
| CN112311605A (en) * | 2020-11-06 | 2021-02-02 | 北京格灵深瞳信息技术有限公司 | Cloud platform and method for providing machine learning service |
| CN112311605B (en) * | 2020-11-06 | 2023-12-22 | 北京格灵深瞳信息技术股份有限公司 | Cloud platform and method for providing machine learning service |
| CN114638373A (en) * | 2020-12-15 | 2022-06-17 | Aptiv技术有限公司 | Managing machine learning environment |
| CN112817581A (en) * | 2021-02-20 | 2021-05-18 | 中国电子科技集团公司第二十八研究所 | Lightweight intelligent service construction and operation support method |
| CN114115857A (en) * | 2021-10-29 | 2022-03-01 | 北京邮电大学 | Method and system for constructing automatic production line of machine learning model |
| CN114115857B (en) * | 2021-10-29 | 2024-04-05 | 北京邮电大学 | Machine learning model automatic production line construction method and system |
| CN114064213A (en) * | 2021-11-16 | 2022-02-18 | 四川启睿克科技有限公司 | Kubernets container environment-based rapid arranging service method and system |
| CN114064155A (en) * | 2022-01-17 | 2022-02-18 | 深圳市华付信息技术有限公司 | Container-based algorithm calling method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245003A (en) | A kind of machine learning uniprocessor algorithm arranging system and method | |
| Carzaniga et al. | Designing distributed applications with mobile code paradigms | |
| CN105979009B (en) | A kind of increase load automatic balancing method for cloud application container | |
| CN108139935B (en) | The extension of the resource constraint of service definition container | |
| US20180004503A1 (en) | Automated upgradesystem for a service-based distributed computer system | |
| CN110658794B (en) | Manufacturing execution system | |
| CN109582441A (en) | For providing system, the method and apparatus of container service | |
| CN106155759B (en) | Virtual machine batch creation method and device | |
| CN109302483A (en) | A kind of management method and system of application program | |
| CN112585919A (en) | Method for managing application configuration state by using cloud-based application management technology | |
| US10970649B2 (en) | Automated reinforcement-learning-based application manager that uses local agents | |
| CN103414761A (en) | Mobile terminal cloud resource scheduling method based on Hadoop framework | |
| CN109194538A (en) | Test method, device, server and storage medium based on distributed coordination | |
| CN104813614A (en) | Asynchronous Framework For Management Of IAAS | |
| CN111324571A (en) | Container cluster management method, device and system | |
| WO2022037612A1 (en) | Method for providing application construction service, and application construction platform, application deployment method and system | |
| CN111984269A (en) | Method for providing application construction service and application construction platform | |
| CN111984270A (en) | Application deployment method and system | |
| CN109117259A (en) | Method for scheduling task, platform, device and computer readable storage medium | |
| CN113031993A (en) | Application upgrading method and device based on cluster container | |
| CN113672352A (en) | Method and device for deploying federated learning task based on container | |
| CN116010027A (en) | Method for managing task processing cluster, method for executing task and container cluster | |
| CN113742033A (en) | Kubernetes cluster federal system and implementation method thereof | |
| Guo et al. | Mobile agent‐based service migration in mobile edge computing | |
| JP2024501005A (en) | Management method and device for container clusters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190917 |
|
| RJ01 | Rejection of invention patent application after publication |