CN113158968A - Embedded object cognitive system based on image processing - Google Patents
Embedded object cognitive system based on image processing Download PDFInfo
- Publication number
- CN113158968A CN113158968A CN202110505690.6A CN202110505690A CN113158968A CN 113158968 A CN113158968 A CN 113158968A CN 202110505690 A CN202110505690 A CN 202110505690A CN 113158968 A CN113158968 A CN 113158968A
- Authority
- CN
- China
- Prior art keywords
- embedded
- image
- terminal
- module
- image processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001149 cognitive effect Effects 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 title claims abstract description 37
- 238000012549 training Methods 0.000 claims abstract description 65
- 238000000034 method Methods 0.000 claims description 43
- 238000005096 rolling process Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 230000004927 fusion Effects 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 230000019771 cognition Effects 0.000 claims description 4
- 239000000463 material Substances 0.000 claims description 4
- 230000037213 diet Effects 0.000 claims description 3
- 235000005911 diet Nutrition 0.000 claims description 3
- 238000004148 unit process Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 abstract description 5
- 239000010410 layer Substances 0.000 description 17
- 230000005540 biological transmission Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000012935 Averaging Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000002902 bimodal effect Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003930 cognitive ability Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides an embedded object cognitive system based on image processing. In this embodiment, the embedded object recognition system based on image processing includes an image acquisition module, a model training module, a model terminal deployment module, and a terminal inference module. The system can acquire images in real time and obtain a recognition result after reasoning through a lightweight convolutional neural network model in a model training module; the system can greatly reduce the requirement on hardware resources while ensuring the accuracy and reasoning speed of object identification, and realize identification and classification of different types of objects.
Description
Technical Field
The invention relates to the technical field of embedded artificial intelligence, in particular to an embedded object cognitive system based on image processing.
Background
Embedded Artificial Intelligence (EAI) is a product of deeply fusing Embedded computer technology, Artificial Intelligence technology and actual requirements in each application scene. Besides the technical advantages of artificial intelligence, the embedded artificial intelligence has the characteristics of excellent real-time performance, applicability, robustness and stability of the embedded technology.
The traditional embedded intelligent software and hardware platform takes a cloud server as a center, original data collected by a terminal are transmitted to the cloud, data storage and analysis are completed at the cloud, the embedded terminal only realizes data collection and performs corresponding operation on an output result, and one-time data circulation is completed. The intelligent embedded software and hardware platform with cloud computing as a core has the problems of high cost, poor real-time performance, data privacy and the like, and can not meet most of the requirements of practical application. With the development of the technology, the processing capacity of the terminal is more and more powerful, and emerging technologies such as edge computing and fog computing are proposed, which aim at overcoming the defects of an embedded intelligent software and hardware platform of a cloud core. However, the method only cuts the propagation process of the network model into a terminal part and a cloud part, the embedded terminal does not have complete cognitive ability, and the complete inference process still needs the calculation of the cloud part.
Therefore, in view of the above technical problems, it is necessary to provide an embedded object recognition system based on image processing. The system collects images in real time, and obtains a recognition result after reasoning through a lightweight convolutional neural network model in a model training module; the system can greatly reduce the requirement on hardware resources while ensuring the accuracy and reasoning speed of object identification, and realize identification and classification of different types of objects.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide an embedded object recognition system based on image processing, where the system acquires images in real time and obtains a recognition result after reasoning through a lightweight convolutional neural network model in a model training module; the system can greatly reduce the requirement on hardware resources while ensuring the accuracy and reasoning speed of object identification, and realize identification and classification of different types of objects.
In order to achieve the above purpose, the technical solutions provided by the embodiments of the present invention are as follows: an image processing-based embedded object recognition system comprises: the image acquisition module is used for acquiring the image characteristics of the training object; the model training module is connected with the image acquisition module, takes the image characteristics obtained by the image acquisition module as training materials, and adopts a preset algorithm to generate a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework; the model terminal deployment module is used for deploying the cognitive model parameter component obtained by the model training module on the embedded terminal; and the terminal reasoning module is used for carrying out target object cognition according to the image of the target object acquired by the image acquisition module and by adopting the cognitive model parameter component provided by the model terminal deployment module.
As a further improvement of the invention, the model training module comprises two modes: a PC model training mode and an embedded terminal real-time reasoning training mode.
As a further improvement of the invention, the PC model training mode comprises the following steps: the embedded terminal acquires image characteristics and transmits the image characteristics to the PC terminal; the PC terminal establishes a data set according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in a model training module; and deploying the cognitive model parameter component to an embedded terminal in a burning mode.
As a further improvement of the invention, the embedded terminal real-time reasoning training mode comprises the following steps: the embedded terminal acquires image characteristics; the terminal reasoning module carries out image processing according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in the model training module; and storing the cognitive model parameter component to an embedded terminal.
As a further improvement of the invention, the hardware configuration in the model terminal deployment module is different according to different parameter properties in the embedded object cognitive system.
As a further improvement of the invention, the constant parameters are stored in the FLASH memory, and the variable parameters are stored in the RAM memory.
As a further improvement of the present invention, the constant parameters include filter diet, BIAS parameters, and propagation structure function; the variable parameters include image characteristics, input variables, and output variables.
As a further improvement of the invention, the embedded object cognitive system replaces the reading and writing of the dynamic array in the RAM with the erasing and reading and writing of the continuous address space of the preset FLASH designated sector.
As a further improvement of the invention, the parameter format used in the preset algorithm in the model training module is in a multi-dimensional array form in C language.
As a further improvement of the invention, the image acquisition module adopts an optimized camera driving algorithm to drive the camera so as to acquire the image characteristics of the training object.
As a further improvement of the present invention, the image acquisition module includes an image processing unit, and the image processing unit processes the original image of the training object obtained by the image acquisition module by using a threshold filtering method to obtain the image features of the training object.
As a further improvement of the invention, the model training module adopts a fusion rolling convolution algorithm to generate a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework.
The invention has the following advantages:
the embodiment of the invention aims to provide an embedded object cognitive system based on image processing, which acquires images in real time and obtains a recognition result after reasoning through a lightweight convolutional neural network model in a model training module. Furthermore, a model training module in the system adopts a fusion rolling convolution algorithm, so that the requirement of the embedded system on the image area size space is effectively optimized. Furthermore, the system drives the camera by adopting an optimized camera driving algorithm in the image acquisition process, so that the image reading and displaying speed is effectively improved. The system can greatly reduce the requirement on hardware resources while ensuring the accuracy and reasoning speed of object identification, and realize identification and classification of different types of objects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic block diagram of an embedded object recognition system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data flow model of the embedded object recognition system of the embodiment shown in FIG. 1;
FIG. 3(a) is a schematic flow chart of the PC model training mode in the embodiment shown in FIG. 1;
FIG. 3(b) is a schematic flow chart of a real-time inference training mode of the embedded terminal in the embodiment shown in FIG. 1;
FIG. 4 is a diagram of a hardware configuration distribution framework in a model terminal deployment module in the embodiment shown in FIG. 1;
FIG. 5 is a flowchart of optimizing a camera drive algorithm in an embodiment of the present invention;
fig. 6(a), 6(b), and 6(c) are schematic diagrams of the fusion rolling convolution algorithm in the embodiment of the present invention.
Description of reference numerals:
100. embedded object recognition system 10, image acquisition module 20 and model training module
30. Model terminal deployment module 40 and terminal reasoning module
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, a schematic block diagram of an embedded object recognition system based on image processing according to an embodiment of the present invention is provided. In this embodiment, the embedded object recognition system 100 based on image processing includes an image acquisition module 10, a model training module 20, a model terminal deployment module 30, and a terminal inference module 40. The image acquisition module 10 is used for acquiring image features of a training object. The model training module 20 is connected to the image acquisition module 10, and generates a cognitive model parameter component that can be directly compiled and used in an embedded engineering framework by using the image features acquired by the image acquisition module 10 as training materials and using a preset algorithm. The model terminal deployment module 30 is configured to deploy the cognitive model parameter component obtained by the model training module 20 to the embedded terminal. The terminal reasoning module 40 performs target object cognition by using the cognitive model parameter component provided by the model terminal deployment module 30 according to the image of the target object acquired by the image acquisition module 10.
With continued reference to fig. 2, the image acquisition module 10 first acquires feature data of a corresponding object as a material for training the cognitive model; after a sufficient amount of image data is collected, the model training module 20 trains a model, and finally a cognitive model parameter component which can be directly compiled and used under a general embedded engineering framework is generated through a relevant algorithm; the model terminal deployment module 30 deploys the cognitive model parameter components obtained by the model training module 20 on the embedded terminal, that is, after recompilation and burning, a new cognitive model is deployed on the terminal, and at this time, the terminal can perform cognition on the target object through the terminal reasoning module 40 to obtain a cognitive result.
In the overall application system, the model training module 20 can be divided into two modes according to the function executed by the system and the data direction transmission: a PC model training mode and an embedded terminal real-time reasoning training mode.
The flow of the PC model training mode is shown in fig. 3 (a). The PC model training mode comprises the following steps: the embedded terminal acquires image characteristics and transmits the image characteristics to the PC terminal, namely the image acquisition module 10 acquires image characteristic data of a corresponding object and transmits the image characteristics to the PC terminal; the PC terminal establishes a data set according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in a model training module; and deploying the cognitive model parameter component to an embedded terminal in a burning mode.
The flow of the embedded terminal real-time inference training mode is shown in fig. 3 (b). The embedded terminal real-time reasoning training mode comprises the following steps: the embedded terminal acquires image characteristics, namely the image acquisition module 10 acquires image characteristic data of a corresponding object; the terminal reasoning module carries out image processing according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in the model training module; and storing the cognitive model parameter component to an embedded terminal.
The physical resource memory of the MCU in the embedded system is divided into volatile memory and non-volatile memory. A Random Access Memory (RAM) and a Flash Memory (Flash EEPROM Memory) are representative devices of the RAM and the Flash Memory, respectively. The RAM is used for providing temporary data such as local variables and the like required by the running process of the processor, and the FLASH stores read-only data for the running of the program and the program. The size of the RAM space is typically much smaller than that of FLASH, and for the STM32L431RC chip, the size of RAM is 64KB, while the size of FLASH is 256 KB.
The size of the RAM of the main control chip determines the performance of the system for processing data, in the inference process of the network model, temporary data generated in the inference process of each layer of sub-network does not influence follow-up, and layer-by-layer propagation of the network model is realized only through the output characteristic matrix between each layer of the feedforward neural network, so that the total data quantity of all parameters used by a single-layer network is not larger than the space size of the RAM of the main control chip. The resource consumption of the network model in the embedded terminal is also divided into the deployed parameter model and the extra data consumption generated by the propagation of the runtime model. In the forward propagation process of the network model from input to output, the total data resources involved are five data categories, namely, input images, network model parameters, temporary space generated in the operation process, input and output in the transmission process of each layer and output of the final model.
In this embodiment, the image processing-based embedded object recognition system 100 designs a reasonable model resource configuration architecture according to the spatial resource characteristics of the embedded terminal chip. The hardware configuration in the model terminal deployment module 30 differs according to the difference of the parameter properties in the embedded object recognition system. Specifically, the constant parameters are stored in the FLASH memory, and the variable parameters are stored in the RAM memory. Model parameters with different characteristics are adapted to different physical storage resources in the embedded terminal, so that a resource allocation method for terminal operation is rationalized, and unnecessary resource consumption is reduced.
As shown in fig. 4, in this embodiment, the constant parameters include filter diet, BIAS parameters, and propagation structure function; the variable parameters include image characteristics, input variables, and output variables. For network model parameters which are not changed in the propagation process, such as convolution kernel parameters, bias and the like, because the network model parameters usually occupy the maximum storage space and cannot be updated and iterated in the network model reasoning, the system is stored in FLASH in a constant array form, and the FLASH has larger resource space compared with RAM and is more suitable for storing such data. As the structure of the input and output in the transmission process of each layer of network is fixed, the input and output in the embedded terminal exist in a multidimensional array with fixed size, and only the value of each array member changes in the transmission process, the system is stored in the RAM in a dynamic array mode, so that the operation speed of the system is facilitated. Since the input image and the output array are processed in common with other software layer components, the system is stored in RAM in a global array.
In embedded systems, the size of the RAM is relatively small, but the RAM is responsible for the main data computation. Meanwhile, in the process of network model reasoning, the resources occupied by each layer are different, and the difference of the space resources occupied by different network structures of each layer is larger. Taking VGG16[54] network as an example, the network parameters and inputs/outputs used in the first layer convolutional layer occupy nearly 100KB of space, while the final fully-connected neural network layer only occupies less than 1KB of computational space. How to cut the network layer occupying the largest resources and reduce the space occupation of the network layer by other methods is one of the problems which are urgently needed to be solved at present.
Preferably, the embedded object recognition system 100 replaces the reading and writing of the dynamic array in the RAM with the erasing and reading and writing of the continuous address space of the preset FLASH designated sector. Therefore, the consumption of the RAM resource by the embedded object cognitive system 100 is reduced, the applicability of the main control chip to the size of the network model is improved, and the inference precision of the model is improved by replacing small time loss. The specific algorithm for erasure replacement is shown in table 1:
due to the difference of compiling frameworks, a general low-resource embedded terminal cannot directly support neural network algorithm libraries such as Keras and the like. In this embodiment, the embedded object recognition system 100 first reads parameters of an H5 format file generated by training a network model fitting platform, and then generates a C language version component of the model parameters that can be directly compiled in an embedded engineering framework according to a designed algorithm.
Through data analysis of the H5 file, the data format in the H5 file is found to be a tree structure and is divided into two types of data, namely weight data and bias data. The expression form of the specific data of the convolution kernel element a in the h row and w column of the nth dimension of the l-layer network is shown as formula 1:
a ═ layerl _ n _ h _ w (formula 1)
The data of the kth bias term b of the l-th network is expressed in a form shown in formula 2:
b ═ layerl _ bias _ k (formula 2)
Therefore, the position of the data can be directly positioned, and a theoretical basis is provided for the design of the preset algorithm in the embodiment of the invention. The embedded object cognitive system 100 designs a model inference parameter format conversion algorithm to convert parameters used by model inference into a multi-dimensional array form in the C language in a storage form, and converts the algorithm model into an embedded engineering component. Namely, the parameter format used in the preset algorithm in the model training module 20 is in the form of a multidimensional array in C language.
The model parameters are converted into a multi-dimensional array form in C language, then the commonalities in the embedded engineering are extracted, the attached header file components and the elements such as variable declarations needed at the beginning of the file are added, the elements are converted into a general embedded engineering component form, the embedded engineering component form can be directly deployed at an embedded terminal, and model parameter data support is provided for terminal reasoning.
Compared with the traditional environment sensor acquisition process, the image data acquisition process is more complex and the data amount processed in a complicated way is more huge. Aiming at the problems, the embedded object recognition system 100 designs an image acquisition acceleration algorithm and a feature extraction algorithm, and ensures that a terminal can quickly acquire high-quality image data. In a preferred embodiment, the image acquisition module 10 adopts an optimized camera driving algorithm to drive the camera to acquire the image characteristics of the training object.
Taking the image format QVGA as an example, the size of the image transmitted to the cache chip by the camera is 80 × 60 pixels, and 4800 times of pixel point information data needs to be read, so the judgment method for the effectiveness of each pixel and the speed increase brought by the amplification of the communication flow between the camera and the cache chip are considerable. As shown in fig. 5, a flowchart of optimizing a camera driving algorithm in an embodiment of the present invention. The embedded object recognition system 100 acquires that one pixel needs to output 2 times of clock signals, namely 4 times of GPIO high/low level signals, and reads a complete QVGA image and needs to output 19200 times of clock signals. The system then first performs an accelerated optimization operation for this operation of outputting the clock signal.
The traditional embedded GPIO operation usually calls a packaged GPIO interface function, and the interface function usually places parameter checking and judging operation in the function in order to ensure the robustness of a program. For example, as the HAL bottom package library of french semiconductor corporation, a pull-up operation of a GPIO requires a series of operations such as port correctness verification, GPIO register verification, etc. The operations avoid irregular parameter input, and ensure the reusability and the robustness of the function. However, a single GPIO operation for image acquisition is not appropriate and also increases the burden on system operation. The embedded object recognition system 100 sets the input/output register corresponding to the communication GPIO port of the camera module, so that the transmission and judgment of parameters are avoided, and the efficiency and speed of image acquisition are improved.
The image of the pixel size acquired by the camera unit in the image acquisition module 10 is cropped and compressed. Therefore, the occupied space of the image data can be reduced under the condition of ensuring that the image characteristic data is not lost as much as possible. Different from the traditional image processing algorithm for processing the two-dimensional array, the image data acquisition module transmits pixel by pixel during transmission, so that the embedded terminal performs related operations on the one-dimensional image array in the process of acquiring images. Assume that the image input dimension of the employed image processing algorithm is H × H pixel size, and the algorithm compresses QVGA size image data into H × H format size. In this embodiment, the compression algorithm firstly cuts the collected image to 60 × 60, and then determines whether the ordinal number n of the input pixel is the target pixel: if yes, storing the position Ax, y in the corresponding target two-dimensional array, otherwise, discarding. The specific compression algorithm is shown in table 2:
further, the embodiment of the invention also performs LCD display acceleration. The process of displaying single pixel point data on the LCD is also a process of communicating the chip and the LCD through a Serial Peripheral Interface (SPI). The process of point-by-point display is changed into the process of firstly setting an LCD display area and then calling the SPI to directly send the pixel data to the LCD, so that the process of setting coordinates for sending every time is omitted. In order to utilize MCU resource to the maximum extent, LCD display adopts the method of point-by-point display, namely directly displaying on LCD after receiving and finishing a pixel point data, and displaying a complete image on LCD by only occupying and reusing the resource of a single pixel point. Meanwhile, the conventional LCD display pixel function is to position and display each displayed pixel, that is, to determine the relative position of the display on the LCD and then display a corresponding pixel. But this is inefficient for the case of image display where the designated area is repeated multiple times. Each pixel point displayed by the image has a certain position continuous relation with the pixel points displayed before and after, and the complete image can be displayed without positioning each pixel point.
Preferably, the image acquisition module 10 further comprises an image processing unit (not shown in the figures). The image processing unit processes the original image of the training object obtained by the image acquisition module by adopting a threshold filtering method to obtain the image characteristics of the training object, thereby effectively filtering the image background and keeping the image characteristics of the target object as much as possible.
In this embodiment, the threshold filtering method specifically includes an edge averaging method, a bimodal averaging method, and a bimodal valley method. The edge averaging method carries out averaging operation on all edge pixel points, and the obtained average value is used as a threshold value to carry out filtering operation on the image. The algorithm of the bimodal mean value method is based on the idea of iterative updating. The algorithm firstly processes the occurrence frequency of each gray value in an input image into a histogram form, and then performs double-peak judgment on the histogram, namely whether two local maximum values appear or not, if so, the average of the two local maximum values is taken as a filtering threshold, otherwise, each data point is smoothed with the span of N, meanwhile, the smoothing frequency N is given, and if the upper limit N is exceeded, the image cannot be filtered. After obtaining two gray values with the maximum frequency, the double-peak valley bottom method does not take the two middle gray values, but takes the lowest valley between the two peaks, that is, the gray value with the lowest occurrence frequency between the two gray values, as a threshold value to filter the image.
The conventional network model usually processes the whole image data by first acquiring the image data and then inputting the image data into the network, which is reasonable under the condition of high resources. In the case of low resources, if the input image is large, such as the 224 × 224 pixel size image commonly used by network models, a single image occupies 49KB of space. In this reduced system space, this also affects the number of model parameters, ultimately reducing the recognition accuracy of the system. How to minimize the space occupied by the input image is the key to improve the space utilization efficiency of the system. To address this problem, the model training module 20 in the embedded object recognition system 100 uses a dynamic rolling convolution algorithm to generate a cognitive model parameter component that can be directly compiled and used in an embedded engineering framework. Aiming at the controllability and the resolvability of the embedded terminal in the image acquisition process, the system fuses the image acquisition process with the first layer of convolution layer of the network, and a dynamic rolling convolution algorithm is designed.
The dynamic rolling convolution algorithm divides the fusion rolling convolution algorithm into the following steps according to the resolvable and controllable properties of the image acquisition process of the embedded terminal camera:
(1) and acquiring pixel point data of k + H lines in front of the image, and storing the pixel point data in the corresponding two-dimensional array G [ H +1] [ S ]. k is epsilon [1, T-H-1 ]. The value of S is the single line size of the acquired data, and the image data is temporarily stopped from being received through the corresponding control interface after the acquisition is finished.
(2) Convolution operation is carried out on the convolution kernel A [ H ] [ H ] and the line from the kth line to the (k + H-1) th line of the G, the obtained feature layer array is stored in the kth line of the feature layer array according to the sequence, and the traditional method is consistent as shown in fig. 6(a) and fig. 6 (b).
(3) In order to save memory space, only one additional variable is defined for exchange, elements in adjacent k-th and k + 1-th rows are exchanged sequentially according to a sequence until all elements are exchanged, and an original k-th row data specific exchange method is discarded, which is adopted in the present document and is shown in fig. 6 (c).
(4) And opening the read enable of the cache chip, continuing to read the image data of the k + H +1 th line, and storing the image data in the k + H th line of the G.
(5) And (4) increasing the k value by 1, and repeating the steps (1), (2), (3) and (4) in sequence until the convolution is completed to obtain a complete output array.
Regarding the aspect of space resource consumption, the storage space of the input image required by the fusion convolution algorithm of the embodiment of the present invention is sx (H +1), while the space occupied by the conventional convolution method is sxt.
The specific code process of the fusion rolling convolution algorithm of the embodiment of the present invention is shown in table 3:
the embedded object cognitive system based on image processing provided by the embodiment of the invention can acquire images in real time and obtain a recognition result after reasoning through a lightweight convolutional neural network model in the model training module. Furthermore, a model training module in the system adopts a fusion rolling convolution algorithm, so that the requirement of the embedded system on the image area size space is effectively optimized. Furthermore, the system drives the camera by adopting an optimized camera driving algorithm in the image acquisition process, so that the image reading and displaying speed is effectively improved. The system can greatly reduce the requirement on hardware resources while ensuring the accuracy and reasoning speed of object identification, and realize identification and classification of different types of objects.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (10)
1. An embedded object recognition system based on image processing, comprising:
the image acquisition module is used for acquiring the image characteristics of the training object; the image acquisition module adopts an optimized camera driving algorithm to drive a camera so as to acquire the image characteristics of a training object;
the model training module is connected with the image acquisition module, takes the image characteristics obtained by the image acquisition module as training materials, and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework by adopting a fusion rolling convolution algorithm;
the model terminal deployment module is used for deploying the cognitive model parameter component obtained by the model training module on the embedded terminal;
and the terminal reasoning module is used for carrying out target object cognition according to the image of the target object acquired by the image acquisition module and by adopting the cognitive model parameter component provided by the model terminal deployment module.
2. The image processing-based embedded object recognition system of claim 1, wherein the model training module comprises two modes: a PC model training mode and an embedded terminal real-time reasoning training mode.
3. The embedded object recognition system based on image processing as claimed in claim 2, wherein the PC model training mode comprises the steps of:
the embedded terminal acquires image characteristics and transmits the image characteristics to the PC terminal;
the PC terminal establishes a data set according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in a model training module;
and deploying the cognitive model parameter component to an embedded terminal in a burning mode.
4. The image processing-based embedded object recognition system according to claim 2, wherein the embedded terminal real-time inference training mode comprises the steps of:
the embedded terminal acquires image characteristics;
the terminal reasoning module carries out image processing according to the image characteristics and generates a cognitive model parameter component which can be directly compiled and used under an embedded engineering framework according to a preset algorithm in the model training module;
and storing the cognitive model parameter component to an embedded terminal.
5. The image processing-based embedded object recognition system of claim 1, wherein the hardware configuration in the model terminal deployment module is different according to different parameter properties in the embedded object recognition system.
6. The embedded object recognition system based on image processing as claimed in claim 5, wherein the constant parameters are stored in a FLASH memory, and the variable parameters are stored in a RAM memory.
7. The embedded object recognition system based on image processing as claimed in claim 6, wherein the constant parameters include filter diet, BIAS BIAS parameters, propagation structure function; the variable parameters include image characteristics, input variables, and output variables.
8. The embedded object recognition system based on image processing as claimed in claim 6, wherein the embedded object recognition system replaces the reading and writing of the dynamic array in the RAM with the erasing and reading and writing of the continuous address space of the preset FLASH designated sector.
9. The image processing-based embedded object recognition system of claim 1, wherein the parameter format used in the preset algorithm in the model training module is a multidimensional array in C language.
10. The embedded object recognition system based on image processing as claimed in claim 1, wherein the image capturing module includes an image processing unit, and the image processing unit processes the original image of the training object obtained by the image capturing module by using a threshold filtering method to obtain the image features of the training object.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110505690.6A CN113158968A (en) | 2021-05-10 | 2021-05-10 | Embedded object cognitive system based on image processing |
| PCT/CN2021/122781 WO2022237061A1 (en) | 2021-05-10 | 2021-10-09 | Embedded object cognitive system based on image processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110505690.6A CN113158968A (en) | 2021-05-10 | 2021-05-10 | Embedded object cognitive system based on image processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113158968A true CN113158968A (en) | 2021-07-23 |
Family
ID=76874165
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110505690.6A Pending CN113158968A (en) | 2021-05-10 | 2021-05-10 | Embedded object cognitive system based on image processing |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113158968A (en) |
| WO (1) | WO2022237061A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022237061A1 (en) * | 2021-05-10 | 2022-11-17 | 苏州大学 | Embedded object cognitive system based on image processing |
| CN115526217A (en) * | 2022-11-28 | 2022-12-27 | 陕西公众电气股份有限公司 | Partial discharge mode identification method and system based on embedded platform |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103595456A (en) * | 2013-10-16 | 2014-02-19 | 南京邮电大学 | Method for achieving multimedia sensor network data transmission system |
| CN109685017A (en) * | 2018-12-26 | 2019-04-26 | 中山大学 | A kind of ultrahigh speed real-time target detection system and detection method based on light weight neural network |
| US20190188562A1 (en) * | 2017-12-15 | 2019-06-20 | International Business Machines Corporation | Deep Neural Network Hardening Framework |
| CN110349146A (en) * | 2019-07-11 | 2019-10-18 | 中原工学院 | The building method of fabric defect identifying system based on lightweight convolutional neural networks |
| WO2020164270A1 (en) * | 2019-02-15 | 2020-08-20 | 平安科技（深圳）有限公司 | Deep-learning-based pedestrian detection method, system and apparatus, and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108399362B (en) * | 2018-01-24 | 2022-01-07 | 中山大学 | Rapid pedestrian detection method and device |
| CN113158968A (en) * | 2021-05-10 | 2021-07-23 | 苏州大学 | Embedded object cognitive system based on image processing |
| CN113112431B (en) * | 2021-05-10 | 2023-08-15 | 苏州大学 | Image processing method in embedded system |
-
2021
- 2021-05-10 CN CN202110505690.6A patent/CN113158968A/en active Pending
- 2021-10-09 WO PCT/CN2021/122781 patent/WO2022237061A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103595456A (en) * | 2013-10-16 | 2014-02-19 | 南京邮电大学 | Method for achieving multimedia sensor network data transmission system |
| US20190188562A1 (en) * | 2017-12-15 | 2019-06-20 | International Business Machines Corporation | Deep Neural Network Hardening Framework |
| CN109685017A (en) * | 2018-12-26 | 2019-04-26 | 中山大学 | A kind of ultrahigh speed real-time target detection system and detection method based on light weight neural network |
| WO2020164270A1 (en) * | 2019-02-15 | 2020-08-20 | 平安科技（深圳）有限公司 | Deep-learning-based pedestrian detection method, system and apparatus, and storage medium |
| CN110349146A (en) * | 2019-07-11 | 2019-10-18 | 中原工学院 | The building method of fabric defect identifying system based on lightweight convolutional neural networks |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022237061A1 (en) * | 2021-05-10 | 2022-11-17 | 苏州大学 | Embedded object cognitive system based on image processing |
| CN115526217A (en) * | 2022-11-28 | 2022-12-27 | 陕西公众电气股份有限公司 | Partial discharge mode identification method and system based on embedded platform |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022237061A1 (en) | 2022-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109815868B (en) | Image target detection method and device and storage medium | |
| TWI746865B (en) | Deep vision processor | |
| CN113469073B (en) | SAR image ship detection method and system based on lightweight deep learning | |
| CN106778928B (en) | Image processing method and device | |
| CN111797983A (en) | Neural network construction method and device | |
| CN113158968A (en) | Embedded object cognitive system based on image processing | |
| CN112163601B (en) | Image classification method, system, computer device and storage medium | |
| CN112529165B (en) | Deep neural network pruning method, device, terminal and storage medium | |
| CN110807362A (en) | Image detection method and device and computer readable storage medium | |
| CN110222718A (en) | The method and device of image procossing | |
| CN112132145A (en) | Image classification method and system based on model extended convolutional neural network | |
| CN109982088B (en) | Image processing method and device | |
| CN111767947A (en) | Target detection model, application method and related device | |
| CN113780357A (en) | Maize leaf disease and pest moving end identification method based on transfer learning and MobileNet | |
| CN111833360A (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN113177956B (en) | Semantic segmentation method for unmanned aerial vehicle remote sensing image | |
| CN114067225A (en) | Unmanned aerial vehicle small target detection method and system and storable medium | |
| EP4024343A1 (en) | Viewpoint image processing method and related device | |
| CN113112431A (en) | Image processing method in embedded system | |
| CN114170113B (en) | Three-dimensional modeling method and system for unmanned aerial vehicle aerial surveying and mapping | |
| CN115577747A (en) | High-parallelism heterogeneous convolutional neural network accelerator and acceleration method | |
| CN114219757B (en) | Intelligent damage assessment method for vehicle based on improved Mask R-CNN | |
| CN102902972B (en) | Human behavior characteristic extraction method and system and abnormal behavior detection method and system | |
| CN113963178A (en) | Method, device, equipment and medium for detecting infrared dim and small target under ground-air background | |
| WO2023028866A1 (en) | Image processing method and apparatus, and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210723 |