CN110503600A - Feature point detecting method, device, electronic equipment and readable storage medium storing program for executing - Google Patents
Feature point detecting method, device, electronic equipment and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN110503600A CN110503600A CN201910788794.5A CN201910788794A CN110503600A CN 110503600 A CN110503600 A CN 110503600A CN 201910788794 A CN201910788794 A CN 201910788794A CN 110503600 A CN110503600 A CN 110503600A
- Authority
- CN
- China
- Prior art keywords
- feature point
- image
- feature
- model
- hidden layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000000605 extraction Methods 0.000 claims abstract description 72
- 238000012545 processing Methods 0.000 claims abstract description 54
- 238000001514 detection method Methods 0.000 claims abstract description 45
- 238000013138 pruning Methods 0.000 claims abstract description 12
- 230000008569 process Effects 0.000 claims abstract description 8
- 238000013139 quantization Methods 0.000 claims description 42
- 210000002569 neuron Anatomy 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 8
- 238000007667 floating Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000011002 quantification Methods 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 description 6
- 210000000697 sensory organ Anatomy 0.000 description 5
- 230000035945 sensitivity Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the present application provides a kind of feature point detecting method, device, electronic equipment and readable storage medium storing program for executing, is related to computer vision field.This method obtains multiple first connection weights that the feature point extraction model pre-established includes first；Then, it is based on the feature point extraction model, obtains the corresponding multiple characteristics of image to be processed of image to be detected, wherein the feature point extraction model is the model for first passing through model beta pruning processing in advance；Then, based on the characteristic point quantitative model pre-established, quantification treatment is carried out to each characteristics of image to be processed, obtains multiple to convolved image feature, quantification treatment is carried out to each first connection weight, obtains multiple second connection weights；Finally, obtaining target feature point to each described to convolved image feature and each second connection weight carries out process of convolution.In this way, carrying out feature extraction based on light-weighted feature point extraction model, while guaranteeing detection accuracy detection rates are got a promotion.
Description
Technical Field
The present application relates to the field of computer vision, and in particular, to a feature point detection method, apparatus, electronic device, and readable storage medium.
Background
Face recognition is a biometric technology for identity recognition based on facial feature information of a person. A series of related technologies, also commonly referred to as face recognition or face recognition, collect an image or video stream containing a human face with a camera or a video camera, automatically detect and track the human face in the image, and then perform face recognition on the detected human face.
A complete face recognition process comprises five main components, namely face detection, face feature point positioning, face alignment, face feature extraction and corresponding recognition, verification and classification algorithms. The positioning of the face feature points is to accurately position the positions of five sense organs of a face, even facial contours, provides a large amount of accurate geometric information for the whole automatic identification program, and is a data basis of the whole program.
In the video portrait beautification, five sense organs of the face and each area of the face in each frame of image are distributed differently, and different beautification operations are required to be carried out on different areas in the beautification process, so that the detection and the positioning of the characteristic points of the face of the five sense organs of the face are a very critical step. In addition, the video has extremely high real-time requirements on image processing, and in a mobile terminal, the requirements on the storage space occupied by the algorithm model and the algorithm power consumption are also strict. How to improve the accuracy and the calculation rate of the algorithm model deployed at the mobile terminal is the key point of the current research.
Disclosure of Invention
In view of the foregoing, the present application provides a feature point detection method, apparatus, electronic device, and readable storage medium.
The embodiment of the application can be realized as follows:
in a first aspect, an embodiment of the present application provides a feature point detection method, where the method includes:
obtaining a plurality of first connection weights contained in a pre-established feature point extraction model, wherein the feature point extraction model is a model subjected to model pruning in advance;
acquiring a plurality of to-be-processed image features corresponding to the to-be-detected image based on the feature point extraction model;
based on a pre-established feature point quantization model, performing quantization processing on each image feature to be processed to obtain a plurality of image features to be convolved, and performing quantization processing on each first connection weight to obtain a plurality of second connection weights;
and performing convolution processing on the image features to be convolved and the second connection weights to obtain target feature points.
Further, the step of performing convolution processing on each image feature to be convolved and each second connection weight to obtain a target feature point includes:
for each hidden layer of the feature point extraction model, taking a target feature point obtained by the last hidden layer of the hidden layer as an image feature to be convolved input into the hidden layer, and performing convolution processing according to the image feature to be convolved input into the hidden layer, the image feature to be convolved output by the hidden layer and a second connection weight corresponding to the hidden layer by using the following formula to obtain the target feature point corresponding to the hidden layer:
q3＝Z3+M*(q1-Z1)(q2-Z2)
wherein M is a constant, q1For inputting the image features to be convolved of the hidden layer, q2For the second connection weight, q3Is a target feature point, Z1For inputting a first parameter, Z, corresponding to a feature of the image to be convolved of the hidden layer2A second parameter, Z, corresponding to said second connection weight3And outputting a third parameter corresponding to the image feature to be convolved for the hidden layer.
Further, the feature point extraction model is obtained by training through the following steps:
acquiring a pre-stored sample data set;
training the constructed convolutional neural network by using the sample data set to obtain an initial characteristic point extraction model, wherein the convolutional neural network comprises a plurality of hidden layers, and each hidden layer comprises a plurality of neurons;
and updating the first connection weight of the neurons between the hidden layers based on a preset loss function to obtain the feature point extraction model.
Further, the step of updating the first connection weights of the neurons between the hidden layers based on a preset loss function to obtain the feature point extraction model includes:
calculating gradients of the characteristic diagrams output by the hidden layers based on a preset loss function, obtaining a minimum gradient in the multiple gradients, and obtaining a target neuron corresponding to the minimum gradient;
deleting the target neuron in the initial characteristic point extraction model and the connection weight corresponding to the target neuron;
and continuing training the initial characteristic point extraction model after deletion processing until the total time length of the obtained initial characteristic point extraction model from the input of the sample image to the output of the corresponding image characteristic is less than or equal to the preset time length, and taking the initial characteristic point extraction model as the characteristic point extraction model.
Further, the sample data set comprises a plurality of sample images, and the feature point quantization model is obtained by:
inputting each sample image into the feature point extraction model to obtain a feature map corresponding to the sample image output by each hidden layer contained in the feature point extraction model;
for each hidden layer, obtaining the maximum value and the minimum value of the characteristic element values of a plurality of characteristic graphs;
respectively carrying out moving average processing on the maximum values and the minimum values to obtain maximum pixel values and minimum pixel values after the moving average processing; and constructing a characteristic point quantization model according to the maximum pixel value and the minimum pixel value.
Further, the step of performing moving average processing on the feature map output by each hidden layer to obtain a feature map after moving average processing includes:
for each feature map output by the hidden layer, calculating the moving average value of the feature map at the current moment according to the moving average value of the feature map at the previous moment and the real value of the current moment by the following formula:
Vt＝C*Vt-1+(1-C)*Ht
wherein, VtIs the moving average of the maximum or minimum at time t, C is a constant, Vt-1Is a moving average of the maximum or minimum values at time t-1, HtThe true value of the maximum value or the minimum value at the moment t;
and constructing and obtaining a characteristic diagram after the moving average processing according to the moving average value obtained at each moment.
Further, the feature point quantization model is:
r＝S*(q-Z)
wherein,b is the maximum pixel value, a is the minimum pixel value, r is floating point data before quantization, q is fixed point data after quantization,int () is rounded.
In a second aspect, an embodiment of the present application provides a feature point detection apparatus, including:
the system comprises an acquisition module, a first connection weight calculation module and a second connection weight calculation module, wherein the acquisition module is used for acquiring a plurality of first connection weights contained in a pre-established feature point extraction model, and the feature point extraction model is a model subjected to model pruning in advance;
the characteristic point extraction model is used for acquiring a plurality of to-be-processed image characteristics corresponding to the to-be-detected image;
the characteristic point quantization model is used for performing quantization processing on the image characteristics to be processed to obtain a plurality of image characteristics to be convolved, and performing quantization processing on the first connection weights to obtain a plurality of second connection weights;
and the convolution module is used for performing convolution processing on the image features to be convolved and the second connection weights to obtain target feature points.
In a third aspect, an embodiment of the present application provides an electronic device, where the device includes a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processor and the memory communicate with each other through the bus, and the processor executes the machine-readable instructions to perform the steps of the feature point detection method according to any one of the foregoing embodiments.
In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored, and the computer program, when executed, implements the feature point detection method described in any one of the foregoing embodiments.
The embodiment of the application provides a feature point detection method, a feature point detection device, electronic equipment and a readable storage medium. Therefore, the feature extraction is carried out based on the light-weight feature point extraction model, and the speed of detecting the image feature points is improved while the detection precision is ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure;
fig. 2 is a flowchart of a feature point detection method according to an embodiment of the present application;
fig. 3 is a flowchart illustrating a sub-step of a feature point detection method according to an embodiment of the present application;
fig. 4 is a second flowchart illustrating sub-steps of a feature point detection method according to an embodiment of the present application;
fig. 5 is a third flow chart of sub-steps in a feature point detection method according to an embodiment of the present application;
fig. 6 is a functional block diagram of a feature point detection apparatus according to an embodiment of the present application.
Icon: 100-an electronic device; 110-a memory; 120-a processor; 130-feature point detection means; 131-an acquisition module; 132-feature point extraction model; 133-feature point quantification model; 134-convolution module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.
The inventor researches and discovers that in the video portrait beautification, five sense organs of the face and each area of the face in each frame of image are distributed differently, and different beautification operations are required to be carried out on different areas in the beautification process, so that the detection and the positioning of the facial feature points of the person on the five sense organs of the face are a very critical step. In addition, the video has extremely high real-time requirements on image processing, and in a mobile terminal, the requirements on the storage space occupied by the algorithm model and the algorithm power consumption are also strict.
In the prior art, the face feature point detection method based on the deep learning method of the convolutional neural network has become the mainstream in the industry, and for the real-time application of the mobile terminal, how to reduce the memory and the calculation amount required by the calculation of a large number of parameters related to the convolutional neural network and how to improve the detection speed is very important.
In the existing face feature point detection algorithm, in order to improve the detection speed, a small-sized face feature point detection model with fewer layers and fewer parameter channels is usually designed, and redundant parameters are eliminated by combining a pruning technology, so that the effect of reducing the calculated amount is achieved. However, when the cropping rate is lower than a certain threshold, the accuracy of the face feature point detection model is seriously affected. This becomes an obstacle that restricts the model from further reduction, and severely restricts the deployment of the human face feature point detection model at the mobile terminal.
Based on the above findings, the present application provides a feature point detection method, apparatus, electronic device, and readable storage medium, where the method performs model pruning (model pruning, i.e., deleting the weights corresponding to unimportant features and neurons and only retaining useful weights and neurons) on a trained convolutional neural network, so as to reduce the size of the model on the basis of ensuring the precision of the convolutional neural network model, and at the same time, performs quantization processing on a feature map obtained from the pruned model, and changes floating-point data into fixed-point data, so as to further reduce the amount of computation and accelerate the operation rate. The above method is explained in detail below.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an electronic device 100 according to an embodiment of the present disclosure. The device may include a processor 120, a memory 110, a feature point detection apparatus 130, and a bus, where the memory 110 stores machine-readable instructions executable by the processor 120, and when the electronic device 100 runs, the processor 120 and the memory 110 communicate with each other through the bus, and the processor 120 executes the machine-readable instructions and performs the steps of the feature point detection method.
The memory 110, the processor 120, and other components are electrically connected to each other directly or indirectly to enable signal transmission or interaction.
For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The feature point detecting device 130 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware (firmware). The processor 120 is configured to execute an executable module stored in the memory 110, such as a software functional module or a computer program included in the feature point detection apparatus 130.
The Memory 110 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
The processor 120 may be an integrated circuit chip having signal processing capabilities. The processor 120 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and so on.
But may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In the embodiment of the present application, the memory 110 is used for storing a program, and the processor 120 is used for executing the program after receiving the execution instruction. The method defined by the process disclosed in any of the embodiments of the present application can be applied to the processor 120, or implemented by the processor 120.
In the embodiment of the present application, the electronic device 100 may be, but is not limited to, a smart phone, a Personal Computer (PC), a tablet computer, or other devices having a processing function.
It will be appreciated that the configuration shown in figure 1 is merely illustrative. Electronic device 100 may also have more or fewer components than shown in FIG. 1, or a different configuration than shown in FIG. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.
Referring to fig. 2, fig. 2 is a flowchart of a feature point detection method according to an embodiment of the present disclosure. The specific flow shown in fig. 2 is described in detail below.
And S1, obtaining a plurality of first connection weights contained in a pre-established feature point extraction model, wherein the feature point extraction model is a model subjected to model pruning in advance.
And S2, acquiring a plurality of to-be-processed image features corresponding to the to-be-detected image based on the feature point extraction model.
As an embodiment, referring to fig. 3, the pre-established feature point extraction model is obtained by training through the following steps:
and S21, acquiring a pre-stored sample data set.
Taking the feature point of the extracted face as an example, the sample data set includes a plurality of face sample images, and each face sample image is marked with face feature point marking information.
And S22, training the constructed convolutional neural network by using the sample data set to obtain an initial feature point extraction model, wherein the convolutional neural network comprises a plurality of hidden layers, and each hidden layer comprises a plurality of neurons.
The embodiment of the application also comprises an input layer and an output layer. The input layer, the hidden layer and the output layer are connected in sequence through the neurons.
The process and principle of training the convolutional neural network by using the face sample images and the face feature point labeling information in the sample data set can refer to the prior art, and are not described herein again.
And S23, updating the first connection weight of the neurons between the hidden layers based on a preset loss function, and obtaining the feature point extraction model.
Alternatively, referring to fig. 4, updating the first connection weights of the neurons between the hidden layers to obtain the feature point extraction model may be implemented through steps S231 to S233.
S231, calculating gradients of the characteristic diagrams output by the hidden layers based on a preset loss function, obtaining a minimum gradient in the gradients, and obtaining a target neuron corresponding to the minimum gradient.
The predetermined loss function may be a square loss function (square loss function). L (y, f (x)) ((y-f (x)))2Where y is the predicted value (the feature of the output image to be processed), and f (x) is the actual value (the input sample data set).
Calculating the sensitivity of the output layer from the loss function L Wherein,is the net input to the output layer,output characteristics of the last layer of the hidden layer, f' () is a derivation operation, ai nRepresenting the activation value of the ith neuron of the output layer.
Taking the fully connected layer as an example, the sensitivity of each hidden layer is calculated from the output layer to the previous time
Wherein,for net input of the l-th hidden layer, WlIs the gradient of the l-th layer,sensitivity of the l +1 layer.
Calculating the corresponding gradient Δ W from the sensitivitiesl：
Wherein, ai lRepresents the activation value of the ith neuron of the l-th hidden layer.
S232, deleting the target neuron in the initial characteristic point extraction model and the connection weight corresponding to the target neuron.
And S233, continuing training the initial feature point extraction model after the deletion processing until the total time length of the obtained initial feature point extraction model from the input of the sample image to the output of the corresponding image feature is less than or equal to the preset time length, and taking the initial feature point extraction model as the feature point extraction model.
It should be noted that when there is an edge in the image, there must be a large gradient value, and conversely, when there is a relatively smooth portion in the image, the gray value change is small, and the corresponding gradient is also small. Therefore, the gradient of each feature map is obtained, the minimum gradient value is found, and the corresponding weight and the neuron are deleted, so that redundant parameters can be deleted, the accuracy is guaranteed, meanwhile, a feature point extraction model is simplified, and the operation rate is accelerated.
And S3, based on the pre-established feature point quantization model, performing quantization processing on the features of the images to be processed to obtain a plurality of features of the images to be convolved, and performing quantization processing on the first connection weights to obtain a plurality of second connection weights.
Alternatively, referring to fig. 5, the feature point quantization model is obtained by training through the following steps:
and S31, inputting each sample image into the feature point extraction model, and obtaining the feature map corresponding to the sample image output by each hidden layer included in the feature point extraction model.
S32, for each hidden layer, obtaining a maximum value and a minimum value of feature element values of a plurality of feature maps;
s33, respectively carrying out moving average processing on the maximum values and the minimum values to obtain maximum pixel values and minimum pixel values after the moving average processing;
as an embodiment, performing the moving average processing on each of the plurality of maximum values and the plurality of minimum values may be implemented by:
for each hidden layer, calculating the moving average value of the maximum value at the current moment according to the following formula according to the moving average value of the maximum value at the previous moment and the actual value of the maximum value at the current moment, and calculating the moving average value of the minimum value at the current moment according to the following formula according to the moving average value of the minimum value at the previous moment and the actual value of the minimum value at the current moment for each hidden layer:
Vt＝C*Vt-1+(1-C)*Ht
wherein, VtIs the moving average of the maximum or minimum at time t, C is a constant, Vt-1Is a moving average of the maximum or minimum values at time t-1, HtThe actual value of the maximum value or the minimum value at the time t, that is, the maximum value or the minimum value corresponding to the time t.
And finally, obtaining the maximum pixel value and the minimum pixel value after the moving average processing according to the moving average value obtained at each moment. In the embodiment of the present application, C may be 0.999.
For example, when the sample image is 1000, 1000 feature maps may be obtained for each hidden layer, and obtaining the maximum value and the minimum value in each feature map may obtain 1000 maximum values max (i) ([ 1, 1000], and 1000 minimum values min (j) ([ 1, 1000 ]).
And performing moving average processing on the 1000 maximum values and the minimum values according to the formula to obtain maximum pixel values and minimum pixel values after the moving average processing.
It can be understood that performing the moving average processing on the feature map is equivalent to adding a shadow to each pixel value in the feature map, and the pixel value changes and the shadow follows the pixel value. Calculating the average value of each pixel value over a period of time increases the generalization of the model.
And S34, constructing a characteristic point quantization model according to the maximum pixel value and the minimum pixel value.
The characteristic point quantization model is as follows:
r＝S*(q-Z)
wherein,b is the maximum pixel value, a is the minimum pixel value, r is floating point data before quantization, q is fixed point data after quantization,int () is rounded.
It should be noted that, in the feature point quantization model in the embodiment of the present application, the essence is the mapping relationship from floating point type data to fixed point type data. By the characteristic point quantization model, the data type of the parameter needing quantization can be converted from a floating point type to a corresponding fixed point type.
As an implementation manner, in the embodiment of the present application, floating point type data is converted into fixed point type data of 8 bits. In practical applications, the converted length may be selected according to actual requirements, such as 16-bit fixed point data or 32-bit fixed point data.
It can be understood that, while one floating-point type data is 4 bytes and one 8-bit fixed-point type data only occupies one byte, in the embodiment of the present application, the floating-point type data is converted into the 8-bit fixed-point type data through the characteristic point quantization model, which not only compresses the storage space required by the model to 1/4, but also reduces the memory space occupied by the output characteristic diagram of the whole model to 1/4 in theory.
The memory space is reduced, the time for loading data from the memory by the register in the mobile terminal is greatly shortened, and meanwhile, a single instruction in a 128-bit register can run 4 pieces of 32-bit single-precision floating-point data and 16 pieces of 8-bit fixed-point data. Obviously, the operation rate of the 8-bit fixed-point type data is faster than that of the floating-point type data.
Meanwhile, in the embodiment of the present application, a plurality of to-be-processed image features and first connection weights are quantized in the manner in S3, floating point data are converted into fixed point data, and the corresponding convolution operation manner should be changed accordingly, so that in the embodiment of the present application, method steps for performing convolution processing on the to-be-convolved image features and second connection weights after being converted into fixed point data to obtain target feature points are further provided, and the method steps are described in detail below.
And S4, performing convolution processing on the image features to be convolved and the second connection weights to obtain target feature points.
Optionally, for each hidden layer of the feature point extraction model, taking a target feature point obtained by a previous hidden layer of the hidden layer as an image feature to be convolved input into the hidden layer, and performing convolution processing according to the image feature to be convolved input into the hidden layer, the image feature to be convolved output by the hidden layer, and a second connection weight corresponding to the hidden layer by using the following formula to obtain a target feature point corresponding to the hidden layer:
q3＝Z3+M*(q1-Z1)(q2-Z2)
wherein M is a constant, q1For inputting the image features to be convolved of the hidden layer, q2For the second connection weight, q3Is a target feature point, Z1For inputting a first parameter, Z, corresponding to a feature of the image to be convolved of the hidden layer2A second parameter, Z, corresponding to said second connection weight3And outputting a third parameter corresponding to the image feature to be convolved for the hidden layer.
It will be appreciated that the above formula is for a multiplication calculation of a single feature element by a single weight. And during calculation, calculating according to the single characteristic elements and the single weights respectively included in the image features to be convolved input into the hidden layer, the image features to be convolved output by the hidden layer and the second connection weights corresponding to the hidden layer.
Wherein,in the embodiment of the present application, M is floating point data, and M can also be written as M0*2-n，M0Is an integer of 2-nCan be calculated by means of shifting, and n is an integer.
And for each hidden layer, calculating the characteristics of the image to be convolved input into the hidden layer, the second connection weight and the characteristics of the image to be convolved output by the hidden layer.
The method specifically comprises the following steps:
wherein, a1For the minimum pixel value corresponding to the image feature to be convolved which is input into the hidden layer, b1And inputting the maximum pixel value corresponding to the image feature to be convolved of the hidden layer.
Wherein, a2For the smallest second connection in the second connection weightReceiving a weight, b2Is the largest of the second connection weights.
a3Minimum pixel value corresponding to the image feature to be convolved which is output by the hidden layer and is subjected to the moving average processing, b3And outputting the minimum pixel value corresponding to the image feature to be convolved after the moving average processing for the hidden layer.
Z1The minimum pixel value and the maximum pixel value (obtained by the calculation of the method steps corresponding to the above-mentioned S32-S33) corresponding to the image features to be convolved which are input into the hidden layer are calculated. The method specifically comprises the following steps:
int () is rounded, a1For the minimum pixel value corresponding to the image feature to be convolved which is input into the hidden layer, b1And inputting the maximum pixel value corresponding to the image feature to be convolved of the hidden layer.
Z2And calculating the minimum second connection weight and the maximum second connection weight in the second connection weights.
The method specifically comprises the following steps:
int () is rounded, a2As the smallest of the second connection weights, b2Is the largest of the second connection weights.
Z3The minimum pixel value and the maximum pixel value (obtained by the calculation of the method steps corresponding to the above-mentioned S32-S33) corresponding to the image feature to be convolved output by the hidden layer are obtained by calculation.
int () is rounded, a3Minimum pixel value corresponding to the feature of the image to be convolved, b, output for the hidden layer3And outputting the minimum pixel value corresponding to the image feature to be convolved for the hidden layer.
According to the feature point detection method provided by the embodiment of the application, a plurality of to-be-processed image features corresponding to an image to be detected are obtained through a feature point extraction model which is established in advance after model pruning processing, quantization processing is carried out on feature data of the to-be-processed image and first connection weights of a feature point extraction model through a feature point quantization model which is established in advance, and then target feature points of the image to be detected are calculated through a convolution mode. Therefore, the feature extraction is carried out based on the light-weight feature point extraction model, and the speed of detecting the image feature points is improved while the detection precision is ensured.
Referring to fig. 6, an embodiment of the present application also provides a feature point detection apparatus 130, including:
the obtaining module 131 is configured to obtain a plurality of first connection weights included in a pre-established feature point extraction model.
And the feature point extraction model 132 is used for acquiring a plurality of to-be-processed image features corresponding to the to-be-detected image, wherein the feature point extraction model is a model which is subjected to model pruning in advance.
The feature point quantization model 133 is configured to perform quantization processing on each to-be-processed image feature to obtain a plurality of to-be-convolved image features, and perform quantization processing on each first connection weight to obtain a plurality of second connection weights.
And the convolution module 134 is configured to perform convolution processing on each to-be-convolved image feature and each second connection weight to obtain a target feature point.
It can be understood that, for the specific operation method of each functional module in the embodiment of the present application, reference may be made to the detailed description of the corresponding step in the foregoing method embodiment, and repeated descriptions are not repeated here.
An embodiment of the present application also provides a readable storage medium, in which a computer program is stored, and when the computer program is executed, the above feature point detection method is implemented.
In summary, the embodiment of the present application provides a feature point detection method, an apparatus, an electronic device 100, and a readable storage medium, in which a feature point extraction model established in advance after model pruning is used to obtain a plurality of to-be-processed image features corresponding to an image to be detected, a feature point quantization model established in advance is used to perform quantization processing on the to-be-processed image features and a first connection weight of the feature point extraction model, and a target feature point of the image to be detected is calculated in a convolution manner. Therefore, the feature extraction is carried out based on the light-weight feature point extraction model, and the speed of detecting the image feature points is improved while the detection precision is ensured.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of feature point detection, the method comprising:
obtaining a plurality of first connection weights contained in a pre-established feature point extraction model, wherein the feature point extraction model is a model subjected to model pruning in advance;
acquiring a plurality of to-be-processed image features corresponding to the to-be-detected image based on the feature point extraction model;
based on a pre-established feature point quantization model, performing quantization processing on each image feature to be processed to obtain a plurality of image features to be convolved, and performing quantization processing on each first connection weight to obtain a plurality of second connection weights;
and performing convolution processing on the image features to be convolved and the second connection weights to obtain target feature points.
2. The method according to claim 1, wherein the step of performing convolution processing on each of the image features to be convolved and each of the second connection weights to obtain the target feature point comprises:
for each hidden layer of the feature point extraction model, taking a target feature point obtained by the last hidden layer of the hidden layer as an image feature to be convolved input into the hidden layer, and performing convolution processing according to the image feature to be convolved input into the hidden layer, the image feature to be convolved output by the hidden layer and a second connection weight corresponding to the hidden layer by using the following formula to obtain the target feature point corresponding to the hidden layer:
q3＝Z3+M*(q1-Z1)(q2-Z2)
wherein M is a constant, q1For inputting the image features to be convolved of the hidden layer, q2For the second connection weight, q3Is a target feature point, Z1For inputting a first parameter, Z, corresponding to a feature of the image to be convolved of the hidden layer2A second parameter, Z, corresponding to said second connection weight3And outputting a third parameter corresponding to the image feature to be convolved for the hidden layer.
3. The feature point detection method according to claim 1, wherein the feature point extraction model is trained by:
acquiring a pre-stored sample data set;
training the constructed convolutional neural network by using the sample data set to obtain an initial characteristic point extraction model, wherein the convolutional neural network comprises a plurality of hidden layers, and each hidden layer comprises a plurality of neurons;
and updating the first connection weight of the neurons between the hidden layers based on a preset loss function to obtain the feature point extraction model.
4. The method according to claim 3, wherein the step of updating the first connection weights of the neurons between the hidden layers based on a preset loss function to obtain the feature point extraction model comprises:
calculating gradients of the characteristic diagrams output by the hidden layers based on a preset loss function, obtaining a minimum gradient in the multiple gradients, and obtaining a target neuron corresponding to the minimum gradient;
deleting the target neuron in the initial characteristic point extraction model and the connection weight corresponding to the target neuron;
and continuing training the initial characteristic point extraction model after deletion processing until the total time length of the obtained initial characteristic point extraction model from the input of the sample image to the output of the corresponding image characteristic is less than or equal to the preset time length, and taking the initial characteristic point extraction model as the characteristic point extraction model.
5. The method according to claim 3, wherein the sample data set includes a plurality of sample images, and the feature point quantization model is obtained by:
inputting each sample image into the feature point extraction model to obtain a feature map corresponding to the sample image output by each hidden layer contained in the feature point extraction model;
for each hidden layer, obtaining the maximum value and the minimum value in the characteristic element values of a plurality of characteristic graphs in the hidden layer;
respectively carrying out moving average processing on the maximum values and the minimum values to obtain maximum pixel values and minimum pixel values after the moving average processing;
and constructing a characteristic point quantization model according to the maximum pixel value and the minimum pixel value.
6. The method according to claim 5, wherein the step of performing a moving average process on each of the plurality of maximum values and the plurality of minimum values to obtain maximum pixel values and minimum pixel values after the moving average process comprises:
for each hidden layer, calculating the moving average value of the maximum value at the current moment according to the following formula according to the moving average value of the maximum value at the previous moment and the actual value of the maximum value at the current moment, and calculating the moving average value of the minimum value at the current moment according to the following formula according to the moving average value of the minimum value at the previous moment and the actual value of the minimum value at the current moment for each hidden layer:
Vt＝C*Vt-1+(1-C)*Ht
wherein, VtIs the moving average of the maximum or minimum at time t, C is a constant, Vt-1The moving average value of the maximum value or the minimum value at the moment t-1 is obtained, and Ht is the true value of the maximum value or the minimum value at the moment t;
and obtaining the maximum pixel value and the minimum pixel value after the moving average processing according to the moving average value obtained at each moment.
7. The feature point detection method according to claim 5, wherein the feature point quantization model is:
r＝S*(q-Z)
wherein,b is the maximum pixel value, a is the minimum pixel value, r is floating point data before quantization, q is fixed point data after quantization,int () is rounded.
8. A feature point detection apparatus, characterized in that the apparatus comprises:
the system comprises an acquisition module, a first connection weight calculation module and a second connection weight calculation module, wherein the acquisition module is used for acquiring a plurality of first connection weights contained in a pre-established feature point extraction model, and the feature point extraction model is a model subjected to model pruning in advance;
the characteristic point extraction model is used for acquiring a plurality of to-be-processed image characteristics corresponding to the to-be-detected image;
the characteristic point quantization model is used for performing quantization processing on the image characteristics to be processed to obtain a plurality of image characteristics to be convolved, and performing quantization processing on the first connection weights to obtain a plurality of second connection weights;
and the convolution module is used for performing convolution processing on the image features to be convolved and the second connection weights to obtain target feature points.
9. An electronic device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the feature point detection method according to any one of claims 1-7.
10. A readable storage medium, characterized in that a computer program is stored therein, which when executed implements the feature point detection method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910788794.5A CN110503600B (en) | 2019-08-26 | 2019-08-26 | Feature point detection method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910788794.5A CN110503600B (en) | 2019-08-26 | 2019-08-26 | Feature point detection method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110503600A true CN110503600A (en) | 2019-11-26 |
| CN110503600B CN110503600B (en) | 2022-10-28 |
Family
ID=68589415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910788794.5A Active CN110503600B (en) | 2019-08-26 | 2019-08-26 | Feature point detection method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110503600B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111882040A (en) * | 2020-07-30 | 2020-11-03 | 中原工学院 | Convolutional neural network compression method based on channel number search |
| CN113762495A (en) * | 2020-06-04 | 2021-12-07 | 合肥君正科技有限公司 | Method for improving precision of low bit quantization model of convolutional neural network model |
| CN115272706A (en) * | 2022-07-28 | 2022-11-01 | 腾讯科技（深圳）有限公司 | Image processing method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104023216A (en) * | 2014-05-28 | 2014-09-03 | 清华大学 | Face video compression method |
| KR101903127B1 (en) * | 2017-05-12 | 2018-10-01 | 이래에이엠에스 주식회사 | Gaze estimation method and apparatus |
| CN109472307A (en) * | 2018-11-07 | 2019-03-15 | 郑州云海信息技术有限公司 | A kind of method and apparatus of training image disaggregated model |
| CN109934226A (en) * | 2019-03-13 | 2019-06-25 | 厦门美图之家科技有限公司 | Key area determines method, apparatus and computer readable storage medium |
-
2019
- 2019-08-26 CN CN201910788794.5A patent/CN110503600B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104023216A (en) * | 2014-05-28 | 2014-09-03 | 清华大学 | Face video compression method |
| KR101903127B1 (en) * | 2017-05-12 | 2018-10-01 | 이래에이엠에스 주식회사 | Gaze estimation method and apparatus |
| CN109472307A (en) * | 2018-11-07 | 2019-03-15 | 郑州云海信息技术有限公司 | A kind of method and apparatus of training image disaggregated model |
| CN109934226A (en) * | 2019-03-13 | 2019-06-25 | 厦门美图之家科技有限公司 | Key area determines method, apparatus and computer readable storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113762495A (en) * | 2020-06-04 | 2021-12-07 | 合肥君正科技有限公司 | Method for improving precision of low bit quantization model of convolutional neural network model |
| CN111882040A (en) * | 2020-07-30 | 2020-11-03 | 中原工学院 | Convolutional neural network compression method based on channel number search |
| CN111882040B (en) * | 2020-07-30 | 2023-08-11 | 中原工学院 | Convolutional neural network compression method based on channel number search |
| CN115272706A (en) * | 2022-07-28 | 2022-11-01 | 腾讯科技（深圳）有限公司 | Image processing method and device, computer equipment and storage medium |
| CN115272706B (en) * | 2022-07-28 | 2024-08-09 | 腾讯科技（深圳）有限公司 | Image processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110503600B (en) | 2022-10-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111401516B (en) | Searching method for neural network channel parameters and related equipment | |
| US10726244B2 (en) | Method and apparatus detecting a target | |
| CN111950638B (en) | Image classification method and device based on model distillation and electronic equipment | |
| CN109671020B (en) | Image processing method, device, electronic equipment and computer storage medium | |
| CN110503600B (en) | Feature point detection method and device, electronic equipment and readable storage medium | |
| CN111401339B (en) | Method and device for identifying age of person in face image and electronic equipment | |
| CN111914908B (en) | Image recognition model training method, image recognition method and related equipment | |
| CN113869282B (en) | Face recognition method, hyper-resolution model training method and related equipment | |
| CN107730514A (en) | Scene cut network training method, device, computing device and storage medium | |
| CN111104830A (en) | Deep learning model for image recognition, training device and method of deep learning model | |
| CN113011532B (en) | Classification model training method, device, computing equipment and storage medium | |
| CN111144285A (en) | Fat and thin degree identification method, device, equipment and medium | |
| CN112818821A (en) | Human face acquisition source detection method and device based on visible light and infrared light | |
| CN111126049B (en) | Object relation prediction method, device, terminal equipment and readable storage medium | |
| CN110210279B (en) | Target detection method, device and computer readable storage medium | |
| CN110659641B (en) | Text recognition method and device and electronic equipment | |
| CN114120454A (en) | Training method and device of living body detection model, electronic equipment and storage medium | |
| CN110210314B (en) | Face detection method, device, computer equipment and storage medium | |
| CN111414823A (en) | Human body feature point detection method and device, electronic equipment and storage medium | |
| CN113706580B (en) | Target tracking method, system, equipment and medium based on relevant filtering tracker | |
| CN115619729A (en) | Face image quality evaluation method and device and electronic equipment | |
| CN112084874B (en) | Object detection method and device and terminal equipment | |
| CN112699809B (en) | Vaccinia category identification method, device, computer equipment and storage medium | |
| CN114373071A (en) | Target detection method and device and electronic equipment | |
| CN112489687A (en) | Speech emotion recognition method and device based on sequence convolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |