CN115661723B - Multi-scene monitoring method based on Haesi SD3403 - Google Patents
Multi-scene monitoring method based on Haesi SD3403 Download PDFInfo
- Publication number
- CN115661723B CN115661723B CN202211586312.6A CN202211586312A CN115661723B CN 115661723 B CN115661723 B CN 115661723B CN 202211586312 A CN202211586312 A CN 202211586312A CN 115661723 B CN115661723 B CN 115661723B
- Authority
- CN
- China
- Prior art keywords
- model
- file
- platform
- training
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a multi-scene monitoring method based on Haisi SD 3403. A method for generating binary files needed by a platform includes using yolov3 to train targets to obtain prototxt files and cafemeodel files on the basis of a caffe frame at a server side, compiling configuration codes according to formats of objects to be predicted and preprocessing operation during training to enable the configuration codes to be converted into an om model supported by an SD3403 platform, enabling the om model to be deployed to the platform at the moment and carrying out reasoning to convert the om model under different scenes into the binary files, carrying out parameter setting on the om model by using a Qt model conversion tool during conversion to generate the binary files needed by the platform, and transmitting all the binary files into the SD3403 platform.
Description
Technical Field
The invention relates to the technical field of graphical communication, in particular to a multi-scene monitoring method based on Haisi SD 3403.
Background
The Haisi SD3403 platform is a professional SoC chip developed aiming at high definition/ultra-high definition (1080 p/4M/5M/4K) IPC product application, integrates an ARM A55 four-core processor and a neural network reasoning engine, supports various intelligent algorithm applications, a neural network acceleration engine, a calculation power of 4Tops INT8, and supports a complete API and a tool chain. By means of strong calculation power and rich interfaces, multiple detection algorithm models can be migrated on the platform, and meanwhile, the model compression tool under the platform is used to realize quantification of the models, so that the model reasoning speed is increased, and algorithm optimization is realized.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a Haisi SD3403 platform and a mask framework, a yolov3 model obtained by training on a server is converted, so that a model type supported by the SD3403 is obtained, in order to improve the calculation speed, the platform can quantize the converted model, namely, the weight and the data of the model are subjected to low-bit processing, so that the finally generated network model is lighter, and the aims of saving the storage space of the network model, reducing the transmission delay, improving the calculation efficiency and improving the performance are fulfilled. In order to improve the diversity of monitoring scenes, the video streams in different scenes can be analyzed by utilizing multiple channels of the platform.
By deploying the detection model on the platform, the application scene of visual detection can be expanded, the application volume is reduced, and the flexibility is improved.
(II) technical scheme
In order to achieve the purpose, the invention provides the following technical scheme: a multisite monitoring method based on Haisi SD3403 is based on a Haisi SD3403 platform and a caffe frame, before model migration, a model and a weight file of the cafe frame need to be trained at a server side firstly, the model on the server is converted through a model conversion tool of the Haisi platform according to a format of an object to be tested and a preprocessing mode during training, if the reasoning speed of the model on the platform is to be accelerated, the converted model needs to be compressed by using a model compression tool, aiming at different scenes, a plurality of models can be converted and uploaded to the Haisi SD3403 platform, so as to facilitate management and generate different models, a model packaging tool is made according to a Qt for an interface reserved by a program, so that model parameters (such as model size, model description and the like) are conveniently set, and a model meeting requirements is generated.
Preferably, when a single model is used for reasoning, the server side is based on a coffee framework, yolov3 is used for training a target to obtain a prototxt file and a coffee model file, and a configuration code is compiled according to the format (yuv 420SP or RGB, etc.) of an object to be predicted and preprocessing operation during training; if the reasoning speed needs to be accelerated, the prototxt file and the coffee file need to be quantized, the quantized files are converted into an om model by using the same method, and then the om model is deployed to a platform, so that the accelerated reasoning can be carried out.
Preferably, the single model inference includes the steps of:
s1, configuring various parameters during training and testing, including training strategies, learning rate change rates and model storage frequency parameters;
s2, splicing training models under a non-Caffe framework and generating a parameter data file in a Prototxt text format under a Caffe model;
s3, describing a network structure file of a test network layer, converting the file into an om file, and storing a data file of a compressed prototxt training model;
s4, describing a network structure file of a ca ffeemodel training network layer, converting the ca ffeemodel file into an om file, and storing a data file of a compressed ca ffeemodel training model;
and S5, after the converted om model file is compressed, the om model file is deployed on the SD3403 platform.
Preferably, the om model conversion is to simplify the derived model diagram by using an onnx tool, set environment variables of the onnx, modify a slice operator of the model by using an accessory script, configure a Haesi development environment, configure the environment variables, convert a network model of an open-source framework and a defined single operator description file by using an atc into an offline model supported by the Haesi, and realize optimization of operator scheduling and data rearrangement of weights in the model conversion process.
Preferably, the prototxt file is designed with an object to be optimized, a training network for learning and a testing network for evaluation, parameters are updated by optimizing the iteration of forward and backward algorithms, the testing network is evaluated periodically, the states of a model and a slover are displayed in the optimizing process, the forward algorithm is called to calculate a final output value in each iteration process, the backward algorithm is called to calculate the gradient of each layer, the gradient is used for parameter learning according to a selected slover method, and the learning rate and the corresponding state of each iteration are recorded and stored.
Preferably, the caffiedel file stores the structure of the model, the weight parameters and the offset information obtained by training, and the parameters and the information of the structure of the entire training network, and the caffiedel file is a group of files used for classification in the model testing stage.
Preferably, when a plurality of the models are used for reasoning, the om models in different scenes need to be converted into binary files, when the conversion is performed, a Qt model conversion tool is used for parameter setting (model size, model description, operation platform and the like) on the om models, the binary files required by the platform can be conveniently generated, finally, all the binary files are transmitted to the SD3403 platform, and the purpose of multi-scene monitoring is achieved by setting channels corresponding to the models in the configuration files.
Preferably, the reasoning for a plurality of said models comprises the steps of:
s1, converting models of different scenes into binary files;
s2, setting parameters of the model by using a Qt model conversion tool;
and S3, transmitting the binary file into a Haisi SD3403 platform.
Preferably, the binary file conversion gives a function of file class attribute through a Qt meta-object system, judges whether a child node exists according to an incoming file node, the child node is a composite attribute, the child node is a simple attribute, the simple attribute is not, extracts the content of the child node, assigns the content to an object instance, continues to process the next child node, dynamically generates a corresponding object instance according to the attribute and the name of the child node, assembles the content of the generated object instance, judges the name of the attribute of the current own point to be recorded in which type of information, gives a parent instance object according to the attribute method and the value of the object instance, converts the file content in a recursive mode, traverses all attributes of the class, and forms the binary file.
Preferably, the model parameters are set by setting environment variables on the Qt platform, loading a control for displaying the model on the Qt interface, modifying the size of the data of the model by an inputtable auxiliary information input box control, receiving the data by using image data transmitted by a class built in the Qt, displaying and storing the data, and converting the image into an RGB image format during the display process.
Preferably, the method comprises the steps of programming by multiple channels, wherein one channel is responsible for acquiring a video signal and displaying an acquired image in real time, the processing channel is responsible for transmitting the acquired image to a platform by a network, format conversion and display of the image are realized by Qt programming in the platform, video data is shared in the acquisition channel and the transmission channel, a circular queue is constructed and used for storing the data shared by the acquisition channel and the transmission channel, reading and writing operations of the queue are mutually exclusive by using a mutual exclusion lock, synchronization of the two channels is realized by using two semaphores, a buffer block size is generated according to a video format, a video buffer pool attribute is set, a video buffer pool is initialized, a control parameter is configured, a binding relationship between an input channel and an input device is set according to input video mode information, the input channel is prompted to start operation, the channel attributes are set, including image size, maximum bit rate and frame rate, an encoding channel is generated according to the channel number and the channel attributes, pictures and videos are started to be received, denoising, deinterlacing and scaling are performed on the received video data, the received video data are processed, the received videos are bound by calling control, deployment and monitoring of the deployment and monitoring channels are provided in the process of the monitoring quality, monitoring scenes can be respectively, and monitoring scenes can be stopped, and monitoring scenes can be respectively transmitted to a user, and the monitoring scene is completed according to the monitoring scene is completed.
(III) advantageous effects
The invention provides a multi-scene monitoring method based on Haisi SD3403, which has the following beneficial effects:
(1) The Haesi SD3403 platform is small in size, strong in calculation capacity, convenient to deploy and space-saving.
(2) The custom degree of the caffe framework is higher, operators can be added in a custom mode, and the network is more flexible.
(3) Different models are deployed in different channels, so that the diversity of monitoring scenes can be improved.
(4) Different models can be conveniently generated using the model encapsulation tool made with Qt.
Drawings
Fig. 1 is a single model deployment flow chart of a multi-scene monitoring method based on haisi SD 3403.
Fig. 2 is a flowchart of a multi-model deployment of a multi-scene detection method based on haisi SD 3403.
Detailed Description
All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Examples
In order to achieve the purpose of the invention, the invention provides a technical scheme that: a multi-scene monitoring method based on Haisi SD 3403.
Further, when a single model is used for reasoning, the server side is based on a noise frame, yolov3 is used for training a target, a prototxt file and a noise file are obtained, and a configuration code is compiled according to the format (yuv 420SP or RGB, etc.) of an object to be predicted and preprocessing operation during training; if the reasoning speed needs to be increased, the prototxt file and the ca ffemodel file need to be quantized, the quantized files are converted into an om model by the same method, and then the om model is deployed to the platform, so that the accelerated reasoning can be carried out.
Further, the single model inference comprises the steps of:
s1, configuring various parameters during training and testing, including training strategies, learning rate change rates and model storage frequency parameters;
s2, splicing training models under a non-Caffe framework and generating a parameter data file in a Prototxt text format under a Caffe model;
s3, describing a network structure file of a test network layer, converting the file into an om file, and storing a data file of a compressed prototxt training model;
s4, describing a network structure file of a ca ffeemodel training network layer, converting the ca ffeemodel file into an om file, and storing a data file of a compressed ca ffeemodel training model;
and S5, after the converted om model file is compressed, deploying the om model file on a platform of the SD 3403.
Further, the om model conversion is to simplify the derived model diagram by using an onnx tool, set environment variables of the onnx, modify a slice operator of the model by using an accessory script, configure a Haesi development environment, configure the environment variables, convert a network model of an open-source framework and a defined single operator description file by using an atc into an offline model supported by the Haesi, and can realize optimization of operator scheduling and data rearrangement of weight values in the model conversion process.
Furthermore, an object to be optimized, a training network for learning and a testing network for evaluation are designed in the prototxt file, parameters are updated through iteration of forward and backward algorithms, the testing network is evaluated regularly, states of a model and a slover are displayed in the optimization process, in each iteration process, a forward algorithm is called to calculate a final output value, a backward algorithm is called to calculate a gradient of each layer, according to a selected slover method, parameter learning is performed by using the gradient, and the learning rate and the corresponding state of each iteration are recorded and stored.
Further, the ca ffemodel file stores the structure of the model, the weight parameters and the offset information obtained by training, and the parameters and the information of the structure of the whole training network, and is a group of files used for classification in the model testing stage.
Furthermore, when a plurality of models are used for reasoning, the om model under different scenes needs to be converted into a binary file, when the conversion is performed, a Qt model conversion tool is used for parameter setting (model size, model description, operation platform and the like) on the om model, the binary file required by the platform can be conveniently generated, finally, all the binary files are transmitted to the SD3403 platform, and the purpose of multi-scene monitoring is achieved by setting channels corresponding to the models in the configuration file.
Further, the reasoning of a plurality of said models comprises the steps of:
s1, converting models of different scenes into binary files;
s2, setting parameters of the model by using a Qt model conversion tool;
s3, transmitting the binary file into a Haisi SD3403 platform;
further, the binary file conversion is characterized in that a file class attribute function is given through a Qt meta-object system, whether a child node exists or not is judged according to an incoming file node, the child node is a composite attribute, the child node is a simple attribute, the simple attribute is not, the content of the child node is extracted and assigned to an object instance, the next child node is processed continuously, a corresponding object instance is dynamically generated according to the attribute and the name of the child node, the generated object instance is subjected to content assembly, the attribute name of the current own point is judged to be recorded in which type of information, a parent instance object is given according to the attribute method and the value of the object instance, the file content is converted in a recursive mode, and all attributes of classes are formed into a binary file.
Furthermore, the model parameters are set by setting environment variables on the Qt platform, loading a control for displaying the model on the Qt interface, modifying the size of the data of the model by an inputtable auxiliary information input box control, receiving the data by utilizing image data transmitted by a class built in the Qt, displaying and storing the data, and converting the image into an RGB image format in the display process.
Further, the method includes multi-channel programming, wherein one channel is responsible for acquiring video signals and displaying the acquired images in real time, a processing channel is responsible for transmitting the acquired images to a platform through a network, qt programming is utilized in the platform to achieve format conversion and display of the images, video data shared in the acquisition channel and the transmission channel is constructed to form a circular queue, the circular queue is used for storing the data shared by the acquisition channel and the transmission channel, a mutual exclusion lock is utilized to perform mutual exclusion of reading and writing operations of the queue, synchronization of the two channels is achieved through two semaphores, a buffer block size is generated according to the format of the video, video buffer pool attributes are set, a video buffer pool is initialized, control parameters are configured, binding relation between an input channel and input equipment is set according to input video mode information, the input channel is prompted to operate, the channel attributes are set, coding channels are generated according to the size, the maximum bit rate and the frame rate according to the channel number and the channel attributes, pictures and the videos are started to be received, denoising, deinterlacing and scaling are performed on the received video data, deployment and feedback are provided in the process of deployment and monitoring of service quality, monitoring scenes can be respectively transmitted to a user, monitoring scene is completed, and monitoring scenes can be monitored, and monitoring scenes can be respectively transmitted to a user, and monitoring scenes can be monitored, and the user can be monitored.
In summary, the multi-scene monitoring method based on Haisi SD3403 can conveniently generate different models through the model encapsulation tool made by Qt.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. A multi-scene monitoring method based on Haisi SD3403 is characterized in that: when a single model is inferred, the server side is based on a coffee framework, yolov3 is used for training a target to obtain a prototxt file and a coffee file, and configuration codes are compiled according to the format of an object to be predicted and preprocessing operation during training to convert the configuration codes into an om model supported by an SD3403 platform; if the reasoning speed needs to be accelerated, the prototxt file and the coffee file need to be quantized, the quantized files are converted into an om model by using the same method, and then the om model is deployed to a platform, so that the accelerated reasoning can be carried out.
2. The method of claim 1, wherein: the single model inference includes the steps of:
s1, configuring various parameters during training and testing, including training strategies, learning rate change rates and model storage frequency parameters;
s2, splicing training models under a non-Caffe framework and generating a parameter data file in a Prototxt text format under a Caffe model;
s3, the prototxt is a network structure file for describing various parameters during testing network training, the file is converted into an om file, and a data file of a compressed prototxt training model is stored;
s4, storing relevant parameters and specific information of model weight by the ca ffeemodel, converting the ca ffeemodel file into an om file, and storing a data file of a compressed ca ffeemodel training model;
and S5, after the converted om model file is compressed, deploying the om model file on a platform of the SD 3403.
3. The method of claim 1, wherein: the om model conversion is characterized in that an onnx tool is used for simplifying the derived model diagram, environment variables of the onnx are set, a slice operator of the model is modified by an accessory script, a Haesi development environment is configured, the environment variables are configured, an atc is used for converting the network model of the open-source framework and the defined single operator description file into an offline model supported by the Haesi, and optimization of operator scheduling and data rearrangement of weight values can be realized in the model conversion process.
4. The method of claim 1, wherein: the prototxt file is used for designing an object needing to be optimized, a training network used for learning and a testing network used for evaluation, optimizing is carried out through iteration of forward and backward algorithms to update parameters, the testing network is evaluated regularly, states of a model and a slope are displayed in the optimizing process, in each iteration process, the forward algorithm is called to calculate a final output value, the backward algorithm is called to calculate gradient of each layer, parameter learning is carried out through the gradient according to a selected slope method, and the learning rate and the corresponding state of each iteration are recorded and stored.
5. The method of claim 1, wherein: the caffieldoel file stores the structure of the model, the weight parameters and the offset information obtained by training, and the parameters and the information of the structure of the whole training network, and is a group of files used for classification in the model testing stage.
6. The method of claim 1, wherein: when a plurality of models are reasoned, the om models in different scenes need to be converted into binary files, when the conversion is carried out, parameter setting is carried out on the om models by using a Qt model conversion tool, the binary files required by the platform can be conveniently generated, finally, all the binary files are transmitted into the SD3403 platform, and the purpose of multi-scene monitoring is achieved by setting channels corresponding to the models in the configuration files.
7. The method of claim 6, wherein: the reasoning of a plurality of said models comprises the steps of:
s1, converting models of different scenes into binary files;
s2, setting parameters of the model by using a Qt model conversion tool;
and S3, transmitting the binary file into a Haisi SD3403 platform.
8. The method of claim 6, wherein: the binary file conversion is characterized in that a file class attribute function is given through a Qt meta-object system, whether a child node exists or not is judged according to an incoming file node, the child node is a composite attribute, the child node is a simple attribute, the simple attribute is not, the content of the child node is extracted and assigned to an object instance, the next child node is processed continuously, a corresponding object instance is dynamically generated according to the attribute and the name of the child node, the generated object instance is subjected to content assembly, the attribute name of the current point is judged to be recorded in the current type of information, a parent class instance object is given according to the attribute method and the value of the object instance, the file content is converted in a recursive mode, and all attributes of classes are traversed to form a binary file.
9. The method of claim 6, wherein: the model parameters are set by setting environment variables on a Qt platform, loading a control for displaying the model on a Qt interface, modifying the size of the data of the model through an auxiliary information input box control, receiving the data by utilizing class-transmitted image data built in the Qt, displaying and storing the data, and converting the image into an RGB image format in the display process.
10. The method of claim 6, wherein: the method comprises the steps of multi-channel programming, wherein one channel is responsible for acquiring a video signal and displaying an obtained image in real time, a processing channel is responsible for transmitting the acquired image to a platform through a network, qt programming is utilized in the platform to realize format conversion and display of the image, video data is shared in the acquisition channel and the transmission channel, a circular queue is constructed to store data shared by the acquisition channel and the transmission channel, a lock is utilized to perform exclusive read-write operation of the queue, two semaphores are utilized to realize synchronization of the two channels, a buffer block size is generated according to the format of the video, video buffer pool attributes are set, a video buffer pool is initialized, control parameters are configured, a binding relationship between an input channel and input equipment is set according to input video mode information, the input channel is prompted to start to operate, the channel attributes including the image size, the maximum code rate and the channel generated according to the channel number and the channel attributes, the video and the video are started to be received, denoising, deinterlacing and zooming are performed on the received video data, sharpening processing is performed, binding is called through control, monitoring and binding is provided with service quality monitoring and feedback in the transmission process, monitoring channels, and monitoring scenes are respectively transmitted to the different scenes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211586312.6A CN115661723B (en) | 2022-12-11 | 2022-12-11 | Multi-scene monitoring method based on Haesi SD3403 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211586312.6A CN115661723B (en) | 2022-12-11 | 2022-12-11 | Multi-scene monitoring method based on Haesi SD3403 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115661723A CN115661723A (en) | 2023-01-31 |
| CN115661723B true CN115661723B (en) | 2023-03-28 |
Family
ID=85019205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211586312.6A Active CN115661723B (en) | 2022-12-11 | 2022-12-11 | Multi-scene monitoring method based on Haesi SD3403 |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115661723B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10726634B2 (en) * | 2018-05-04 | 2020-07-28 | Microsoft Technology Licensing, Llc | Generating and providing platform agnostic scene files in an intermediate format |
| CN111860483B (en) * | 2020-07-21 | 2022-08-23 | 中国科学院光电技术研究所 | Target detection method based on Haisi platform |
| CN212953668U (en) * | 2020-08-12 | 2021-04-13 | 成都睿瞳科技有限责任公司 | Intelligent device for detecting scene personnel of elevator car |
| CN114386578A (en) * | 2022-01-12 | 2022-04-22 | 西安石油大学 | Convolution neural network method implemented on Haisi non-NPU hardware |
-
2022
- 2022-12-11 CN CN202211586312.6A patent/CN115661723B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115661723A (en) | 2023-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109891897B (en) | Method for analyzing media content | |
| CN111242071B (en) | Attention remote sensing image target detection method based on anchor frame | |
| CN112565777B (en) | Deep learning model-based video data transmission method, system, medium and device | |
| KR20230131878A (en) | Apparatus and method for point cloud processing | |
| CN108334895A (en) | Sorting technique, device, storage medium and the electronic device of target data | |
| CN114691765A (en) | Data processing method and device in artificial intelligence system | |
| CN115328319B (en) | Intelligent control method and device based on light-weight gesture recognition | |
| CN113723279A (en) | Multi-target tracking acceleration method based on time-space optimization in edge computing environment | |
| KR102342526B1 (en) | Method and Apparatus for Video Colorization | |
| CN115131849A (en) | Image generation method and related device | |
| CN115661723B (en) | Multi-scene monitoring method based on Haesi SD3403 | |
| CN113010469B (en) | Image feature extraction method, device and computer readable storage medium | |
| CN117354467A (en) | Intelligent optimized transmission system for image data | |
| CN115063673B (en) | Model compression method, image processing method and device and cloud equipment | |
| EP4224860A1 (en) | Processing a time-varying signal using an artificial neural network for latency compensation | |
| CN110398957A (en) | Automatic Pilot behavior prediction method, apparatus, computer equipment and storage medium | |
| CN114723398A (en) | Stage creative arrangement method, stage creative arrangement device and electronic equipment | |
| EP4164221A1 (en) | Processing image data | |
| CN115511802A (en) | Marine target detection system based on Springboot frame | |
| CN115270404A (en) | Design system and method for space-based information system efficiency simulation architecture | |
| CN114610482A (en) | Model edge end deployment method based on multi-task fusion | |
| US20030026338A1 (en) | Automated mask selection in object-based video encoding | |
| CN112102830B (en) | Coarse granularity instruction identification method and device | |
| CN112486677B (en) | Data graph transmission method and device | |
| CN117115589A (en) | Cloud edge end collaborative machine vision algorithm based self-training method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |