CN109522089A - Based on the distributed view of virtualized environment as recognition methods - Google Patents
Based on the distributed view of virtualized environment as recognition methods Download PDFInfo
- Publication number
- CN109522089A CN109522089A CN201811302975.4A CN201811302975A CN109522089A CN 109522089 A CN109522089 A CN 109522089A CN 201811302975 A CN201811302975 A CN 201811302975A CN 109522089 A CN109522089 A CN 109522089A
- Authority
- CN
- China
- Prior art keywords
- function
- recognition methods
- virtualized environment
- data
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of based on the distributed view of virtualized environment as recognition methods the following steps are included: S1: pretreatment visible image data；S2: identification target view carries out image classification and target detection as data；S3: multiserver node, the deployment of more identification containers and GPU resource distribution management are carried out by internal web；S4: production mirror image corresponding with image classification function, target detection function classification, and the load of the resource allocation of custom-built system, mirror image, container instance and container internal services self-starting function.The present invention passes through the integration to multiple deep learning frames, reach the quick instantiation in a platform environment to multiple deep learning frame model abilities, and by the GPU resource of platform mechanism virtual management allotment multiserver node, functional module each in project independent mirror image container of opposing is instantiated by virtualization scheme, and then reaches and quickly carries out project deployment and the foundation of clustering functionality on single machine or multimachine.
Description
Technical field
The present invention relates to technical field of virtualization, know more particularly to a kind of distributed view picture based on virtualized environment
Other method.
Background technique
Universal with mobile Internet and 4G network, the media information that people touch daily is more and more numerous and more jumbled, by
The even many people of mobile Internet can become the producer of media information.Therefore, the security control of new media content is become
At a urgent task, internet visible image data very large for scale, manual examination and verification are not obviously able to satisfy
Demand.With the fast development of GPU operation and deep neural network for image recognition technology, pass through machine learning training
The scheme for carrying out visible image identification shows more and more important practical value in very more fields.
The deep learning frame of current several relatively mainstreams, such as ensorflow, mxnet, caffe, each frame is to ring
Border dependence, network parameter debugging, model training, model capability instantiation have oneself process and mutually it is incompatible.
When not virtualized, will pre-process if necessary, the frames such as caffe, mxnet, tensorflow
Model instantiated on the same server, that must be complete by the condition depended of these frames on this server
Portion successively installs, and needs to solve complex environment and relies on conflict, if the environment of current server is moved to other one
Platform server, that just must all re-start condition depended installation, this can be one under large-scale cluster server conditions
Part very hard work；
During projects, in order to multiple deep learning frames compatible in a set of platform, model is quickly carried out
The instantiation of trained and model capability becomes very stubborn problem, for GPU resource virtualization also without more mature scheme, because
This is badly in need of a unified platform specification to solve the environmental warfare of more frames, and model training, model capability instantiation is allowed to become
It is simpler, and each model of manage and dispatch uses the distribution of GPU resource on multiple server nodes.
Summary of the invention
It is an object of the invention to overcome the deficiencies of the prior art and provide a kind of distributed views based on virtualized environment
As recognition methods is reached in a platform environment by the integration to multiple deep learning frames to multiple deep learning frames
The quick instantiation of frame model capability, while virtual management is carried out by GPU resource of the platform mechanism to multiserver node
Functional module each in project independent mirror image container of opposing is instantiated by virtualization scheme, and then reached by allotment
Project deployment and the foundation of clustering functionality are quickly carried out on single machine or multimachine.
The purpose of the present invention is achieved through the following technical solutions: the distributed view picture based on virtualized environment is known
Other method, comprising the following steps:
S1: visible image data are pre-processed, and obtain target view as data；
S2: identification target view carries out image classification and target detection, carries out frame docking, mould with unified process as data
The instantiation of type ability, process security operations；
S3: multiserver node, the deployment of more identification containers and GPU resource distribution management are carried out by internal web；
S4: production mirror image corresponding with image classification function, target detection function classification, and pass through engineering script custom-built system
Resource allocation function, mirror image load function, container instance function and container internal services self-starting function.
The step 1 includes following sub-step:
S101: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S102: basic data stream is provided for service management framework basic document by PRE_BASE and is supported, and realizes and feeds back with MT
Communication；
S103: on the basis of the PRE_BASE, formulate and realize that picture pre-processes, at video preprocessor by PreInstance
The primitive rule of reason.
The step 1 further includes the transcoding work of visible image data, using GPU hardware accelerated mode by visible image data
Switch to the visible image data of object format.
PreInstance frame is libkt_if.so in the step S103, and the libkt_if.so is according between the time
Every and key frame carry out pumping frame, and frame data are saved in local disk or memcache server.
The step S2 includes following sub-step:
S201: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S202: providing basic data stream by RP_BASE for service management framework basic document and support, and realizes that feedback is logical with MT
Letter；
S203: on the basis of the RP_BASE, by DarknetInstance, CaffeInstace,
Caffe2Instance carries out corresponding network load, picture parsing, picture recognition process.
The beneficial effects of the present invention are:
1) it is integrated by the deep learning frame to current main-stream, reaches the rapid deployment of a variety of frame model recognition capabilities
And instantiation, virtualization and manage and dispatch are carried out based on GPU resource of the customized development of platform to GPU multimachine assembly, together
When be based on virtualization scheme, can will pretreatment, model recognition capability be packaged into independent container, can be in single machine or multimachine feelings
It is rapidly performed by deployment under condition, realizes the purpose of multiple deep learning frame image recognition integration and distributed arithmetic, effectively
The overall calculation time has been saved, computational efficiency is substantially increased.
2) multiserver node, the deployment of more identification containers and GPU resource distribution management are carried out by internal web, borrowed
Virtual management allotment is carried out by GPU resource of the platform mechanism to multiserver node, accomplishes the cooperative scheduling of multimachine GPU resource
Management.
3) by the utilization to virtualization technology, related service and condition depended can be all isolated, for more
Kind deep learning frame (it is more complicated relying on environment, dispose and its be easy to cause environmental warfare) can be integrated quickly, reach
It, can to the flexible deployment on single machine or multimachine and application quickly instantiation, while by mature distributed type assemblies Managed Solution
To accomplish large-scale mirror image container application layout management to multi node server.
Detailed description of the invention
Fig. 1 is server cluster distribution schematic diagram；
Fig. 2 is the schematic diagram for pre-processing visible image data procedures；
Fig. 3 is the schematic diagram for identifying visible image data procedures；
Fig. 4 is the flow diagram of virtualization scheme.
Specific embodiment
Below in conjunction with embodiment, technical solution of the present invention is clearly and completely described, it is clear that described
Embodiment is only a part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, this field
Technical staff's every other embodiment obtained under the premise of not making the creative labor belongs to what the present invention protected
Range.
Refering to fig. 1-4, the present invention provides a kind of technical solution: the distributed view picture identification side based on virtualized environment
Method, comprising the following steps:
S1: visible image data are pre-processed, and obtain target view as data；
As shown in Fig. 2, the step 1 includes following sub-step:
S101: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S102: basic data stream is provided for service management framework basic document by PRE_BASE and is supported, and realizes and feeds back with MT
Communication；
S103: on the basis of the PRE_BASE, formulate and realize that picture pre-processes, at video preprocessor by PreInstance
The primitive rule of reason, i.e. PreInstance are the specific services that the associated frame members based on PRE_BASE parent are realized.
PreInstance frame is libkt_if.so in the step S103, and the libkt_if.so is according between the time
Every and key frame carry out pumping frame, and frame data are saved in local disk or memcache server, libkt_if.so is
Base library is pre-processed, python converting interface is its python interposer.
The step 1 further includes the transcoding work of visible image data, using GPU(graphics processor) hardware-accelerated mode
The visible image data that visible image data are switched to object format substitute software algorithm using hardware to make full use of hardware to consolidate
Some rapid charaters improve computational efficiency, further include other bottom libraries further, the bottom library be Cuda.so,
opencv.so、Ffmpeg.so。
S2: identification target view carries out image classification and target detection, carries out frame pair with unified process as data
It connects, model capability instantiation, process security operations, and then has unified platform specification and carry out mainstream deep learning frame
Frame docking and model capability instantiation, unified outbound data input and output, unified process safety protecting mechanism, unified money
Source control and allotment mechanism are reached in a platform environment by the integration to multiple deep learning frames to multiple depth
The quick instantiation of learning framework model capability；
As shown in figure 3, the step S2 includes following sub-step:
S201: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S202: providing basic data stream by RP_BASE for service management framework basic document and support, and realizes that feedback is logical with MT
Letter；
S203: on the basis of the RP_BASE, by DarknetInstance, CaffeInstace,
Caffe2Instance carries out corresponding network load, picture parsing, picture recognition process.
Cfg. Ini, MT, RP_BASE, DarknetInstance, CaffeInstace, Caffe2Instance group
At basic platform services external member.
Further, deep learning frame foundation library includes Darknet.s, Caffe.so, Caffe2.so, and python turns
Connecing layer is its python converting interface, and GPU and other bottom libraries include Cuda.so, opencv.so etc..
As shown in Figure 1 and Figure 4, multiserver node, the deployment of more identification containers and GPU S3: are carried out by internal web
Business datum to be treated is stored in inside the file of server, then carries out bulk management by resource allocation management, by
Platform mechanism carries out virtual management allotment to the GPU resource of multiserver node, accomplishes the cooperative scheduling pipe of multimachine GPU resource
Reason；
S4: production mirror image corresponding with function classification, and the resource allocation function by being engineered script custom-built system, mirror image add
Carry function, container instance function and container internal services self-starting function.
By the virtualization scheme, functional module each in project can be opposed to independent mirror image container to carry out example
Change, and then reach and quickly carry out project deployment and the foundation of clustering functionality on single machine or multimachine, by each deep learning frame,
The modules such as pretreatment all stand alone as unique mirror image, and quick container instance then can be carried out on multiple server nodes
Change, the simpler safety that deployment management becomes.
In addition, related service and condition depended can be all isolated by the utilization to virtualization technology, for
A variety of deep learning frames (it is more complicated relying on environment, dispose and its be easy to cause environmental warfare) can quickly be integrated,
Reach on single machine or multimachine flexible deployment and application quickly instantiation, while by mature distributed type assemblies Managed Solution,
It can accomplish large-scale mirror image container application layout management to multi node server.
The present invention is integrated by the deep learning frame to current main-stream, reaches a variety of frame model recognition capabilities
Rapid deployment and instantiation are virtualized and are managed based on GPU resource of the customized development of platform to GPU multimachine assembly
Scheduling, while it being based on virtualization scheme, pretreatment, model recognition capability can be packaged into independent container, it can be in single machine
Or deployment is rapidly performed by the case of multimachine, realize the mesh of multiple deep learning frame image recognition integration and distributed arithmetic
, it is effectively saved the overall calculation time, substantially increases computational efficiency.
The above is only a preferred embodiment of the present invention, it should be understood that the present invention is not limited to described herein
Form should not be regarded as an exclusion of other examples, and can be used for other combinations, modifications, and environments, and can be at this
In the text contemplated scope, modifications can be made through the above teachings or related fields of technology or knowledge.And those skilled in the art institute into
Capable modifications and changes do not depart from the spirit and scope of the present invention, then all should be in the protection scope of appended claims of the present invention
It is interior.
Claims (5)
1. based on the distributed view of virtualized environment as recognition methods, it is characterised in that: the following steps are included:
S1: visible image data are pre-processed, and obtain target view as data；
S2: identification target view carries out image classification and target detection, carries out frame docking, mould with unified process as data
The instantiation of type ability, process security operations；
S3: multiserver node, the deployment of more identification containers and GPU resource distribution management are carried out by internal web；
S4: production mirror image corresponding with image classification function, target detection function classification, and pass through engineering script custom-built system
Resource allocation function, mirror image load function, container instance function and container internal services self-starting function.
2. it is according to claim 1 based on the distributed view of virtualized environment as recognition methods, it is characterised in that: it is described
Step 1 include following sub-step:
S101: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S102: basic data stream is provided for service management framework basic document by PRE_BASE and is supported, and realizes and feeds back with MT
Communication；
S103: on the basis of the PRE_BASE, formulate and realize that picture pre-processes, at video preprocessor by PreInstance
The primitive rule of reason.
3. it is according to claim 1 or 2 based on the distributed view of virtualized environment as recognition methods, it is characterised in that:
The step 1 further includes the transcoding work of visible image data, and visible image data are switched to target using GPU hardware accelerated mode
The visible image data of format.
4. it is according to claim 2 based on the distributed view of virtualized environment as recognition methods, it is characterised in that: it is described
PreInstance frame is libkt_if.so in step S103, the libkt_if.so according to time interval and key frame into
Row takes out frame, and frame data are saved in local disk or memcache server.
5. it is according to claim 1 based on the distributed view of virtualized environment as recognition methods, it is characterised in that: it is described
Step S2 includes following sub-step:
S201: the configuration file when platform server node is read from Cfg. ini by MT, then according to the need of configuration file
It asks and successively drives related service；
S202: providing basic data stream by RP_BASE for service management framework basic document and support, and realizes that feedback is logical with MT
Letter；
S203: on the basis of the RP_BASE, by DarknetInstance, CaffeInstace,
Caffe2Instance carries out corresponding network load, picture parsing, picture recognition process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811302975.4A CN109522089A (en) | 2018-11-02 | 2018-11-02 | Based on the distributed view of virtualized environment as recognition methods |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811302975.4A CN109522089A (en) | 2018-11-02 | 2018-11-02 | Based on the distributed view of virtualized environment as recognition methods |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109522089A true CN109522089A (en) | 2019-03-26 |
Family
ID=65776024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811302975.4A Pending CN109522089A (en) | 2018-11-02 | 2018-11-02 | Based on the distributed view of virtualized environment as recognition methods |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109522089A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780987A (en) * | 2019-10-30 | 2020-02-11 | 上海交通大学 | Deep learning classroom analysis system and method based on container technology |
| CN113656151A (en) * | 2021-08-20 | 2021-11-16 | 上海熠知电子科技有限公司 | Deep learning computing power virtualization method |
| CN113656150A (en) * | 2021-08-20 | 2021-11-16 | 上海熠知电子科技有限公司 | Deep learning computing power virtualization system |
| WO2022062304A1 (en) * | 2020-09-25 | 2022-03-31 | 亮风台（上海）信息科技有限公司 | Method and device for deploying image recognition service on container cloud |
| CN115359299A (en) * | 2022-08-25 | 2022-11-18 | 上海人工智能创新中心 | Image target detection method, device and equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107450961A (en) * | 2017-09-22 | 2017-12-08 | 济南浚达信息技术有限公司 | A kind of distributed deep learning system and its building method, method of work based on Docker containers |
| CN108055158A (en) * | 2017-12-19 | 2018-05-18 | 深圳供电局有限公司 | Power grid image recognition system and method |
| US20180197297A1 (en) * | 2016-09-13 | 2018-07-12 | Intelligent Fusion Technology, Inc | System and method for detecting and tracking multiple moving targets based on wide-area motion imagery |
-
2018
- 2018-11-02 CN CN201811302975.4A patent/CN109522089A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180197297A1 (en) * | 2016-09-13 | 2018-07-12 | Intelligent Fusion Technology, Inc | System and method for detecting and tracking multiple moving targets based on wide-area motion imagery |
| CN107450961A (en) * | 2017-09-22 | 2017-12-08 | 济南浚达信息技术有限公司 | A kind of distributed deep learning system and its building method, method of work based on Docker containers |
| CN108055158A (en) * | 2017-12-19 | 2018-05-18 | 深圳供电局有限公司 | Power grid image recognition system and method |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110780987A (en) * | 2019-10-30 | 2020-02-11 | 上海交通大学 | Deep learning classroom analysis system and method based on container technology |
| WO2022062304A1 (en) * | 2020-09-25 | 2022-03-31 | 亮风台（上海）信息科技有限公司 | Method and device for deploying image recognition service on container cloud |
| CN113656151A (en) * | 2021-08-20 | 2021-11-16 | 上海熠知电子科技有限公司 | Deep learning computing power virtualization method |
| CN113656150A (en) * | 2021-08-20 | 2021-11-16 | 上海熠知电子科技有限公司 | Deep learning computing power virtualization system |
| CN115359299A (en) * | 2022-08-25 | 2022-11-18 | 上海人工智能创新中心 | Image target detection method, device and equipment |
| CN115359299B (en) * | 2022-08-25 | 2024-06-11 | 上海人工智能创新中心 | Image target detection method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109522089A (en) | Based on the distributed view of virtualized environment as recognition methods | |
| US11093216B2 (en) | Automatic discovery of microservices from monolithic applications | |
| CN113115241B (en) | Industrial Internet system based on industrial brain | |
| Martín et al. | Kafka-ML: Connecting the data stream with ML/AI frameworks | |
| CN106250782B (en) | A kind of data permission control method and device based on SQL statement parsing | |
| CN107924393A (en) | Distributed server system for language understanding | |
| CN100547545C (en) | The method and system that is used for the application fractionation of network edge calculating | |
| CN106657314A (en) | Cross-data center data synchronization system and method | |
| CN105893045A (en) | Service assembly loading method and device and service assembly running system | |
| CN107179903B (en) | System and method for developing mobile-based applications using model-driven development | |
| CN110737653A (en) | micro-service-based enterprise data processing system and method | |
| Cruz et al. | A DDS‐based middleware for quality‐of‐service and high‐performance networked robotics | |
| CN110362363A (en) | A method of model realization controls terminal applies when based on operation | |
| CN112955865A (en) | Static coordination of application view hierarchies | |
| Yu et al. | Modeling and analysis of mobile cloud computing based on bigraph theory | |
| CN114596046A (en) | Integrated platform based on unified digital model of business center station and data center station | |
| CN108345536A (en) | A kind of dispositions method, device and the equipment of continuous integrating environment | |
| Han et al. | ECCVideo: A scalable edge cloud collaborative video analysis system | |
| CN114115821A (en) | Application development method and platform, application deployment method and node, system and equipment | |
| CN109857374A (en) | The development approach and device of mobile application | |
| CN116578423B (en) | Task processing method, automatic question answering method and image generation method | |
| CN107368300B (en) | MapReduce-based data summarization system and method | |
| CN102999329A (en) | Active directory interface developing method based on extensive markup language mapping configuration | |
| US20120137044A1 (en) | Method and apparatus for providing persistent computations | |
| CN114780211B (en) | Method for managing a secure container and system based on a secure container |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220519 Address after: 518000 22nd floor, building C, Shenzhen International Innovation Center (Futian science and Technology Plaza), No. 1006, Shennan Avenue, Xintian community, Huafu street, Futian District, Shenzhen, Guangdong Province Applicant after: Shenzhen wanglian Anrui Network Technology Co.,Ltd. Address before: Floor 4-8, unit 5, building 1, 333 Yunhua Road, high tech Zone, Chengdu, Sichuan 610041 Applicant before: CHENGDU 30KAITIAN COMMUNICATION INDUSTRY Co.,Ltd. |
|
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190326 |