CN111539470A - Image processing method, device, computer equipment and storage medium - Google Patents

Abstract

This solution relates to an image processing method. The method includes: acquiring an image to be processed; the image to be processed contains a food area; searching for an initial target detection algorithm, and adjusting parameters in the initial target detection algorithm to obtain a target detection algorithm; and extracting the target detection algorithm according to the target detection algorithm. Food area, and extract the visual features in the food area through the deep neural network algorithm; classify the images to be processed according to the visual features. By adjusting the parameters in the initial target detection algorithm, the accuracy of extracting the food area in the image to be processed can be improved, while the deep neural network algorithm has good visual feature recognition ability and has a better effect in food image processing. The accuracy of image processing can be improved.

Description

Image processing method, device, computer equipment and storage medium

technical field

The present invention relates to the field of computer technology, and in particular, to an image processing method, an apparatus, a computer device and a storage medium.

Background technique

The development of computer technology has brought a lot of communication and application of image information, and image retrieval and classification technology has also been widely used. Image retrieval or classification is often carried out through text, and image processing is carried out based on text recognition, that is, text information such as title, time, environment, etc. is used to describe the image. During image processing, text matching is used for retrieval or classification. Since image retrieval and classification through text can no longer meet people's daily needs, image processing methods based on image content have been studied. Food image processing is widely used in many fields such as food and health. Food images can further help people evaluate the calories of food, analyze people's eating habits, and provide personalized services.

Because the food pictures captured not only contain the visual information of the food itself, but also contain various background information, the traditional image processing methods often have the problem of low accuracy in image classification or retrieval.

SUMMARY OF THE INVENTION

Based on this, in order to solve the above technical problems, an image processing method, device, computer equipment and storage medium are provided, which can improve the accuracy of image processing.

An image processing method, the method comprising:

acquiring an image to be processed; the image to be processed includes a food area;

Find an initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain a target detection algorithm;

Extract the food area in the to-be-processed image according to the target detection algorithm, and extract the visual features in the food area through a deep neural network algorithm;

The images to be processed are classified according to the visual features.

In one embodiment, the adjustment of parameters in the initial target detection algorithm to obtain a target detection algorithm includes:

obtaining a database of images, and looking up food images from the database of images;

generating a food image dataset from the food images;

The target detection algorithm is obtained by adjusting the parameters in the initial target detection algorithm using the food image data set.

In one embodiment, the extracting the food area in the to-be-processed image according to the target detection algorithm includes:

Obtain candidate frames of the to-be-processed image respectively according to the target detection algorithm;

Calculate the candidate frame proportion of each of the candidate frames;

Extract the food area in the to-be-processed image according to the candidate frame weight.

In one of the embodiments, the extraction of visual features in the food area through a deep neural network algorithm includes:

Obtain a specific gravity threshold, and use the food area corresponding to the candidate frame specific gravity higher than the specific gravity threshold as a candidate area;

According to the candidate frame weight of the candidate area, obtain the candidate frame coordinates of the candidate area;

Extract the sub-features of the candidate region according to the candidate frame coordinates;

According to the sub-features, the visual features in the food region are obtained.

In one embodiment, the classifying the to-be-processed image according to the visual feature includes:

Extracting the color feature in the visual feature, and extracting the shape feature in the visual feature;

According to the color feature and the shape feature, the image to be processed is classified to obtain the category of the image to be processed.

In one embodiment, the classification of the image to be processed according to the color feature and the shape feature to obtain the category of the image to be processed includes:

According to the color feature and the shape feature, obtain the edge feature map of the to-be-processed image;

Calculate the number of edge points according to the edge feature map, and obtain an image histogram according to the number of edge points;

The image histogram is input into an image classifier to obtain the category of the image to be processed.

In one embodiment, the method further includes:

acquiring an image to be retrieved, and extracting image features of the image to be retrieved;

Perform image retrieval in the image library according to the image features to obtain an initial retrieval image;

calculating the similarity between the initial retrieval image and the to-be-retrieved image;

The target retrieval image is obtained according to the similarity.

An image processing device, the device comprising:

an image acquisition module for acquiring an image to be processed; the image to be processed includes a food area;

a parameter adjustment module, used to find an initial target detection algorithm, and adjust parameters in the initial target detection algorithm to obtain a target detection algorithm;

a feature extraction module, used for extracting the food area in the to-be-processed image according to the target detection algorithm, and extracting visual features in the food area through a deep neural network algorithm;

An image classification module, configured to classify the to-be-processed image according to the visual feature.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

acquiring an image to be processed; the image to be processed includes a food area;

Find an initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain a target detection algorithm;

Extract the food area in the to-be-processed image according to the target detection algorithm, and extract the visual features in the food area through a deep neural network algorithm;

The images to be processed are classified according to the visual features.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring an image to be processed; the image to be processed includes a food area;

Find an initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain a target detection algorithm;

Extract the food area in the to-be-processed image according to the target detection algorithm, and extract the visual features in the food area through a deep neural network algorithm;

The images to be processed are classified according to the visual features.

The above-mentioned image processing method, device, computer equipment and storage medium obtain a target detection algorithm by acquiring an image to be processed; the to-be-processed image includes a food area; searching for an initial target detection algorithm, and adjusting parameters in the initial target detection algorithm; The target detection algorithm extracts the food area in the image to be processed, and extracts the visual features in the food area through the deep neural network algorithm; the image to be processed is classified according to the visual features. By adjusting the parameters in the initial target detection algorithm, the accuracy of extracting the food area in the image to be processed can be improved, while the deep neural network algorithm has good visual feature recognition ability and has a better effect in food image processing. The accuracy of image processing can be improved.

Description of drawings

Fig. 1 is the application environment diagram of the image processing method in one embodiment;

2 is a schematic flowchart of an image processing method in one embodiment;

3 is a schematic flowchart of extracting visual features in a food area in one embodiment;

Figure 4 is a schematic diagram of the analysis of the accuracy comparison results of different methods in the experiment;

Figure 5 is a schematic diagram of the classification results of randomly selected dishes using 15 different methods in the experiment;

Figure 6 is a schematic diagram of using Precision@K to compare the retrieval performance of different methods in the experiment;

Figure 7 is a schematic diagram of comparing the retrieval performance of different methods using MPA@K in the experiment;

8 is a structural block diagram of an image processing apparatus in one embodiment;

Figure 9 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The image processing method provided by the embodiment of the present application can be applied to the application environment shown in FIG. 1 . As shown in FIG. 1 , the application environment includes a computer device 110 . The computer device 110 can acquire the image to be processed; the image to be processed contains the food area; the computer device 110 can search for the initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm; the computer device 110 can detect the target according to the The algorithm extracts the food area in the image to be processed, and extracts the visual features in the food area through a deep neural network algorithm; the computer device 110 can classify the image to be processed according to the visual features. The computer device 110 may be, but is not limited to, various personal computers, notebook computers, smart phones, robots, unmanned aerial vehicles, tablet computers, portable wearable devices, and the like.

In one embodiment, as shown in Figure 2, an image processing method is provided, comprising the following steps:

Step 202, acquiring an image to be processed; the image to be processed includes a food area.

The image to be processed may be an image that needs to be classified and retrieved. The computer device can display a display interface, and the user can input the image to be processed through the display interface. The image to be processed may contain food regions.

Step 204 , searching for an initial target detection algorithm, and adjusting parameters in the initial target detection algorithm to obtain a target detection algorithm.

The initial target detection algorithm may be used for target detection. Specifically, the initial target detection algorithm may be a convolutional neural network algorithm (CNN, Convolutional Neural Networks).

A neural network consists of a set of neural nodes that are divided into many layers, each connected by a full connection, that is, each node is associated with all nodes in the previous layer. The input of each node is the output of all nodes in the previous layer, and the current node uses the sum of its weight coefficient and the input data as the output of the node. The weight parameter is obtained through training and represents the important proportion of the input node corresponding to the previous layer in the task. If the weight value is larger, the output value of the corresponding node will be increased, affecting the deeper network.

Convolutional neural networks have three core ideas, namely local receptive fields, weight sharing, and pooling. Among them, the local receiving field, that is, each neural node is only related to the local image area, and has nothing to do with the global image. Neural nodes can learn some basic features of local receiver neighborhoods, such as edges and corners. At the same time, since each node only needs to be associated with a small local area in the image, the weight parameters that need to be trained can be reduced. Weight sharing can be used to indicate that in a convolutional neural network, the weights of the same convolution kernel in different local regions of the input image are the same. That is, if the primary feature detector is able to detect a feature in one area of the image, then the detector is also effective elsewhere in the image, thereby being able to overcome variations such as image warping or offset. Pooling refers to reducing the dimensionality of the image to reduce the size of the image. The exact position of the learned image features is not important, only the relative position needs to be recorded. In addition, since different images may have problems such as offset and deformation, the precise location of features will have a certain impact on the next step of learning.

As a kind of pre-feedback neural network, convolutional neural network mainly includes convolutional layer, pooling layer, fully connected layer and loss function layer. Convolutional layers perform convolution operations on the input data, and each convolutional layer contains multiple convolution kernels. A convolution kernel is essentially a set of filter matrices with fixed weights. The convolution operation is a convolution kernel that performs an inner product operation on the input data through a sliding window. The convolution layer is the embodiment of the idea of local receptive field and weight sharing in the network. Each convolution kernel can also be called a "filter", which is used to learn a class of features of the input image. Different convolution kernels can obtain different features. , such as color depth, outline, etc. Generally speaking, shallower convolutional layers can only extract some low-level visual features, such as shape, edge, texture, etc., while deeper networks can extract some high-level visual-semantic features for target tasks through previous low-level features.

The pooling layer downsamples the input data. There are many forms of nonlinear pooling functions, such as max pooling MaxPooling, average pooling MeanPooling, Gaussian pooling GaussPooling, etc. Among them, max pooling is the most common, and max pooling divides the input image into several rectangular areas, and is Each subregion outputs the maximum value. The pooling layer will continue to reduce the size of the data space, so the number of parameters and the amount of computation will also be reduced, which also inhibits overfitting to a certain extent.

The fully connected layer and the loss function layer correspond to the classical neural network. The fully connected layer can be regarded as the hidden layer of the neural network, and the loss function layer is the training target of the whole network.

The parameter training of the convolutional neural network is the same as that of the neural network, and the back-propagation algorithm is still used. The algorithm first sends the training data to the network to obtain the network output, then calculates the error by calculating the network output and the target output through the loss function, and inversely calculates the residuals of each layer of the network according to the obtained error to update the parameters.

The computer equipment can adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm. Among them, the target detection algorithm can be an R-CNN network, and the R-CNN network can realize the target detection technology based on algorithms such as convolutional neural network (CNN), linear regression and support vector machine (SVM).

Step 206 , extract the food area in the image to be processed according to the target detection algorithm, and extract the visual features in the food area through the deep neural network algorithm.

Specifically, the computer device can detect the food area in the image to be processed by using the target detection algorithm, so as to extract the food area in the image to be processed. The deep neural network algorithm can be a CNN deep neural network, and the computer equipment can extract visual features in the food area through the CNN deep neural network.

Step 208 , classify the images to be processed according to the visual features.

In this embodiment, the computer device obtains the image to be processed; the image to be processed contains food areas; searches for the initial target detection algorithm and adjusts the parameters in the initial target detection algorithm to obtain the target detection algorithm; extracts the target detection algorithm according to the target detection algorithm The food area in the image is processed, and the visual features in the food area are extracted by the deep neural network algorithm; the image to be processed is classified according to the visual feature. By adjusting the parameters in the initial target detection algorithm, the accuracy of extracting the food area in the image to be processed can be improved, while the deep neural network algorithm has good visual feature recognition ability and has a better effect in food image processing. The accuracy of image processing can be improved.

In one embodiment, an image processing method provided may further include a process of obtaining a target detection algorithm, and the specific process includes: acquiring an image database, and searching for food images from the image database; generating a food image data set according to the food images; using The food image dataset adjusts the parameters in the initial target detection algorithm to obtain the target detection algorithm.

A plurality of images of different types can be stored in the image database, and the computer equipment can identify the images in the image database and mark the images containing food. After the computer device acquires the image database, the marked image, that is, the food image, can be found from the image database. The computer device can obtain a set of food images according to the food images, thereby generating a food image data set, and then use the food image data set to adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm.

其中，i是小的anchor指数，p_i表示anchor i的预测概率，如果anchor为正，则p^*i为1，否则p^*i为0。t_i表示预测外围的四个参数坐标，t^*i为正anchor对应地面truthbox的坐标矢量，分类损失Lcls为两类(目标和非目标)的对数损失。Since most food images contain irrelevant background information, only extracting the features of the food area will reduce the influence of the background information of the food image. Therefore, it is necessary to detect the food area in the image. R-CNN can detect the image in the image. food area. Specifically, the computer equipment can first select images of food-related categories from the visual gene bank, and then fine-tune the R-CNN using the selected food image dataset. The formula of the target detection algorithm is:

Among them, i is the small anchor index, pi represents the predicted probability of anchor _i , if the anchor is positive, then p ^* i is 1, otherwise p ^* i is 0. t _i represents the four parameter coordinates of the prediction periphery, t ^* i is the coordinate vector of the positive anchor corresponding to the ground truthbox, and the classification loss Lcls is the logarithmic loss of the two categories (target and non-target).

In one embodiment, an image processing method provided may further include a process of extracting a food area, and the specific process includes: separately acquiring candidate frames of the image to be processed according to a target detection algorithm; calculating the candidate frame proportions of each candidate frame; Candidate frame weights extract food regions in the image to be processed.

After fine-tuning the initial detection algorithm, the computer device can use the fine-tuned target detection algorithm to calculate candidate frames of the image to be processed and the proportion of each candidate frame, so as to extract the food region in the image to be processed.

As shown in FIG. 3, in one embodiment, an image processing method provided may further include a process of obtaining visual features, and the specific steps include:

In step 302, a specific gravity threshold is obtained, and a food region corresponding to a candidate frame with a specific gravity higher than the specific gravity threshold is used as a candidate region.

The weight threshold may be a preset value for judging the weight of the candidate frame. The computer device may obtain the weight threshold, and compare the weight of the candidate frame with the weight threshold to obtain a comparison result. When the comparison result obtained by the computer device is that the proportion of the candidate frame is higher than the proportion threshold, the computer device may take the food area corresponding to the proportion of the candidate frame as the candidate area.

Step 304: Obtain the candidate frame coordinates of the candidate region according to the candidate frame weight of the candidate region.

Step 306 , extract the sub-features of the candidate region according to the coordinates of the candidate frame.

Since the proportion of candidate frames in the candidate region is higher than the proportion threshold, the computer equipment can extract the features of the FC7 layer by using the AlexNet network based on the coordinates of the candidate frames, and mark the extracted features of the FC7 layer as sub-features.

Step 308: Obtain visual features in the food area according to the sub-features.

The computer equipment can concatenate the sub-features to obtain visual features in the food area.

In one embodiment, the provided image processing method may further include a process of classifying the images to be processed, and the specific process includes: extracting color features in visual features, and extracting shape features in visual features; feature, classify the image to be processed, and obtain the category of the image to be processed.

Color features are one of the most widely used features in the fields of image classification and image retrieval because they are relatively intuitive. The RGB color space adopts the principle of mixing three colors of R, G, and B through the color display of the image. According to the principle of the three primary colors, the original monochrome can be added and mixed: F=r[R]+g[G]+b [B]. The RGB model can be represented by a three-dimensional Cartesian coordinate system. The three coordinate axes R, G, and B represent the three primary colors of red, blue, and green, respectively. The origin of the coordinate axis represents black. Any color can be represented by a linear combination of the three primary colors. . Although the RGB color space can represent any color, in the Cartesian coordinate system, the RGB coordinate values of two similar colors may be very different, so when the color feature similarity matching is performed on the image in the color space, The RGB color space will be converted to other color spaces. Because the general representation of the image is RGB, the RGB color space needs to be converted to other color spaces during the conversion process. For example, to convert the RGB space to the HSV space, the conversion formula is:

其中，r、g和b是RGB笛卡尔空间中的坐标值，max和min是三者的最大值和最小值，H、S和V表示HSV空间模型中的色调、饱和度和亮度。

Among them, r, g, and b are the coordinate values in the RGB Cartesian space, max and min are the maximum and minimum values of the three, and H, S, and V represent the hue, saturation, and brightness in the HSV space model.

Shape features can be used to represent the contours of objects in an image. The computer equipment can extract the shape features in the visual features, so as to classify the images to be processed according to the color features and the shape features, and obtain the categories of the images to be processed.

In one embodiment, the provided image processing method may further include a process of obtaining the category of the image to be processed, and the specific process includes: obtaining an edge feature map of the image to be processed according to color features and shape features; calculating according to the edge feature map The number of edge points is obtained, and the image histogram is obtained according to the number of edge points; the image histogram is input into the image classifier to obtain the category of the image to be processed.

When performing image classification, it is necessary to increase the similarity matching of shape features. Shape features are mainly reflected in the contour features and edge features of the image. At present, there are many algorithms for image edge detection, such as Sobel algorithm, Canny algorithm, Laplacian algorithm, etc., all of which are used to extract edge features of images.

Edge detection is the first step in extracting image shape features. Computer equipment can extract image shape features based on edge feature maps. Commonly used methods for extracting shape features are: using the variable of angle to calculate the number of edge points to draw a histogram, and calculate the edge detection image; use the area, eccentricity, shape parameters, and shape feature histogram as the description of the image, and input the histogram into the image. In the trained classifier, the category information of the image is obtained. Among them, the computer equipment can extract visual features from all the food images in the training set and the test set, and then train the classifier through the training set to obtain the classification result based on the classification model after training.

In one embodiment, the provided image processing method may further include a process of retrieving images, and the specific process includes: acquiring an image to be retrieved, and extracting image features of the image to be retrieved; performing image retrieval in an image library according to the image features, Obtain the initial retrieval image; calculate the similarity between the initial retrieval image and the image to be retrieved; obtain the target retrieval image according to the similarity.

The computer device may acquire an image to be retrieved, wherein the image to be retrieved may be an image to be searched entered by a user. The computer equipment can use the CNN algorithm to extract the image features of the image to be retrieved, convert the image features into a 1×4096 vector, and perform image retrieval in the image database according to the image features to obtain the initial retrieval image. Next, the computer device may calculate the similarity between the initial retrieval image and the to-be-retrieved image according to the extracted image features. In this embodiment, the Euclidean distance formula or the determined angle can be used to characterize the similarity between images, because the eigenvector matrix is unified, the greater the product between the two pairs, the more similar the two images are. . The calculation formula is: a·b=|a|×|b|×cosθ. Among them, a and b represent two eigenvectors, |a| and |b| can represent the modulus value of the two eigenvectors, and cosθ is the angle between the two eigenvectors. The larger the dot product, the more the two eigenvectors are. resemblance. The computer equipment can sort the images according to the size of the dot product, and select the first N image objects from the image database to retrieve the images.

Next, the computer device can classify the above N images according to the BOW model. Since the accuracy rate of the BOW classification model is above 50%, and only one feature of the image is used, that is, the SIFT feature, the classification accuracy rate can be greatly improved by the method of multi-feature and multi-core. Since the accuracy of CNN is at least above 70%, most of the images belong to the same category, and only a few images belong to different categories. Different categories are eliminated, and only images in large categories are retained. The computer device can feed back the final search result obtained after the preliminary search result is processed by the BOW model to the user to obtain the final search result.

其中，i是样本的序列号，j是样本向量的维数，

是第一个样本的第一维特征值。In one embodiment, the essence of image retrieval is to find one or more points in the database that are closest to the index term, and these points may be referred to as neighbors. Taking the image retrieval algorithm based on spectral hashing as an example, given m samples of n variables, an m×n matrix is constructed, and the value of n is generally large, so there is usually redundant data, which is difficult for humans to identify. For a vector matrix composed of the features of each sample, some elements in the vector matrix are indistinguishable. For example, if one element in all samples is 0, or very close to 0, then this element of the feature is indistinguishable. distinguishable. This element can be ignored when distinguishing. It is necessary to find the element with large variation between samples. The variation of the element can be characterized by the size of the variance, and only the elements with large variance are retained, thereby reducing data redundancy and effectively simplifying the operation. Assuming a total of m samples, each of size n-dimensional, calculate the mean of each feature:

where i is the serial number of the sample, j is the dimension of the sample vector,

is the first dimension eigenvalue of the first sample.

θ(x,y)＝αtan2(L(x,y+1)-L(x,y-1)/L(x+1,y)-L(x-11,y))，由于SIFT特征点的数量庞大，直接将BOW模型作为码本，数据量大，计算复杂度高。因此，需要聚类算法k-means或支持向量机对SIFT特征进行聚类。In one embodiment, the BOW model is a commonly used model in text classification. It uses a feature vector composed of words to represent documents, while ignoring the grammar, grammar and word order of documents. Due to the uniqueness of words in documents, documents' Features can be represented by it. The BOW model classifies images based on SIFT features. SIFT features not only have the invariance of displacement, scale, and deformation, but also the invariance of illumination, that is, they still have a good detection effect on similar images with large differences in brightness, which can make up for CNN. Insufficient lighting characteristics. The steps of extracting the SIFT features of the image may include: establishing a fixed scale space, the main purpose is to describe the scale invariance of the image, the definition of the image scale space is: L(x,y,σ)=G(x,y,σ)× (x, y), where x, y are the spatial coordinates of the image pixels, and the size of σ determines the smoothness of the image; then, detect extreme points in the DOG scale space; followed by feature point selection, by fitting three-dimensional The quadratic function determines the position and scale of key points, and removes pixels with asymmetric local curvature, thereby improving the stability of matching and anti-noise ability; finally, a 128-dimensional feature vector is assigned to each feature point. The formulas for calculating the gradient modulus and direction of each point are:

θ(x,y)=αtan2(L(x,y+1)-L(x,y-1)/L(x+1,y)-L(x-11,y)), due to SIFT feature points There is a huge amount of data, and the BOW model is directly used as a codebook, which has a large amount of data and high computational complexity. Therefore, clustering algorithm k-means or support vector machine is needed to cluster SIFT features.

其中，i是码本的序列号，x_i是对应于第一码本的SIFT特征的统计，而Yi是正规化值。After clustering the SIFT features into k classes, put the centroid of each class into a word bag, the codebook size of the word bag model is k, construct an empty array of size k, and calculate each image in the test data set The feature points of , that is, each feature in the distance codebook. The obtained word frequency histogram can be normalized according to the formula, the formula is:

where i is the sequence number of the codebook, _xi is the statistic corresponding to the SIFT feature of the first codebook, and Yi is the normalized value.

In one embodiment, an experiment was carried out on this scheme. The experimental process and results are as follows: the original dish data set contains 117,504 images and 11,611 dish categories, and the dish category with the number of images greater than or equal to 15 is selected, and finally 233 dishes and 49168 images, the dataset is called dish-233.

In order to utilize Fast R-CNN to detect food image regions, it is necessary to perform fast fine-tuning of R-CNN. For this, a dataset of food images with region boundary labels is required. The vision database contains 108,077 calibrated images, including a large number of food images. So using the category names of the Dish-233 dataset to translate it into English as query words, a list of query dictionaries is constructed, and the food images are picked from the visual dataset. In order to obtain more food pictures, further select the category information of other food databases, select food pictures, and then perform further manual filtering to remove non-food pictures, and finally get 10641 food pictures and corresponding calibration areas, called VisGenome- 11K.

For the parameter settings of the model, in the Fast R-CNN training process, the minimum (mini-batch) batch is to extract 256 anchors from the image, and the number of iterations is 80,000. In the first 60000 iterations, the learning rate is set to 0.001, and after 20000 iterations, the learning rate is set to 0.0001, the momentum parameter is set to 0.9, and the weight decay parameter is set to 0.0005. When fine-tuning the Alex-Net model, the initial learning rate was set to 0.001, the learning rate was adjusted to 0.1 before every 20 epochs, and the maximum number of iterations was set to 60 epochs. The VisGenome-11K image set is divided into two parts, 80% for the training set and 20% for the validation set. Use VisGenome-11K to fine-tune the faster R-CNN, then use the fine-tuned faster R-CNN model to perform region detection on the food images of the Dish dataset, get the food regions in the image, and then apply the pre-trained Alex- Net model extracts visual features of food regions from images, and after region detection and visual feature extraction are performed on all images, it is used for retrieval and classification of food images.

Accuracy and MAP are commonly used evaluation metrics in information retrieval. In order to verify the effectiveness of the method in this paper, CNN-G, CNN-G-F and Fast R-CNN-G are compared: CNN-G mainly uses 7-layer Alex network to directly extract visual features of global images; compared with CNN-G , CNN-G-F first uses the training set to fine-tune the Alex network, and then uses the fine-tuning network to extract the visual features of the entire image; Fast R-CNN-G directly uses the Fast R-CNN network to detect candidate food regions in the image, and then uses the fine-tuning network to detect candidate food regions in the image. The AlexNet network extracts visual features for the highest weighted proposals. Among them, methods CNN-G and CNN-G-F do not use Fast R-CNN for region detection, and Fast R-CNN-G method further illustrates the impact of fine-tuning Fast RCNN by VisGenome-11K.

For the image classification task, 75% of the dataset for each class is used as the training set and 25% of the dataset is used as the test set. Since the classification task is single-label, the accuracy rate is used as the evaluation index. The accuracy comparison results of different methods are shown in Table 1, and the analysis of the accuracy comparison results of different methods is shown in Figure 4.

Table 1:

Method Accursdy rate CNN-G 0.356 CNN-G-F 0.704 Faster-R-CNN-G 0.748 Artucle method 0.754

As shown in Fig. 4, Fast R-CNN-G improves the performance by 5% compared to CNN-G-F.

In one embodiment, the classification results of dishes of 15 different methods are randomly selected for analysis, and the results are shown in Table 2 and FIG. 5 .

Table 2:

Combining Table 2 and Figure 5, the classification results of 15 randomly selected dishes under different methods can be obtained. The best classification result accuracy of CNN-G method is 0.41, the best classification result accuracy of CNN-G-F method is 0.73, and the best classification result accuracy of Fast R-CNN-G method is 0.78. It can be seen that the Fast R-CNN-G method has the best classification performance in most experiments.

Retrieval experiments are carried out on the Dish-233 dataset, using Precision@K and MAP@K (K represents the number of candidate images returned during the retrieval process), K={1, 20, 40, 60, 80100}, to conduct experiments, The retrieval results of four of the two indexes are shown in Figures 6 and 7.

According to Figure 6 and Figure 7, it can be concluded that CNN-G-F has better retrieval performance than CNN-G, indicating that fine-tuning the Alex-Net network can obtain visual features more suitable for the Dish dataset; Fast R-CNN-G is faster Fast, better than CNN-G-F, indicating that using fast R-CNN to detect food image regions can effectively reduce the interference of background information of food images, thereby improving retrieval performance; indicating that fine-tuning with VisGenome11K can improve the accuracy of food image detection.

It should be understood that although the steps in the above-mentioned flowcharts are displayed in sequence according to the arrows, these steps are not necessarily executed in the sequence indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in each of the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. These sub-steps or stages The order of execution of the steps is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 8, an image processing apparatus is provided, including: an image acquisition module 810, a parameter adjustment module 820, a feature extraction module 830, and an image classification module 840, wherein:

The image acquisition module 810 is used for acquiring an image to be processed; the image to be processed includes a food area.

The parameter adjustment module 820 is used to find the initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm.

The feature extraction module 830 is configured to extract the food region in the image to be processed according to the target detection algorithm, and extract the visual features in the food region through the deep neural network algorithm.

The image classification module 840 is used for classifying the images to be processed according to the visual features.

In one embodiment, the parameter adjustment module 820 is further configured to acquire an image database, and search for food images from the image database; generate a food image data set according to the food images; use the food image data set to adjust the parameters in the initial target detection algorithm to obtain Object detection algorithm.

In one embodiment, the feature extraction module 830 is further configured to obtain candidate frames of the image to be processed according to the target detection algorithm; calculate the candidate frame proportions of each candidate frame; and extract food regions in the to-be-processed image according to the candidate frame proportions.

In one embodiment, the feature extraction module 830 is further configured to obtain a weight threshold, and use a food area corresponding to a candidate frame weight higher than the weight threshold as a candidate area; according to the candidate frame weight of the candidate area, obtain a candidate frame of the candidate area Coordinates; extract the sub-features of the candidate area according to the candidate frame coordinates; obtain the visual features in the food area according to the sub-features.

In one embodiment, the image classification module 840 is further configured to extract the color feature in the visual feature, and extract the shape feature in the visual feature; classify the image to be processed according to the color feature and the shape feature to obtain the category of the image to be processed.

In one embodiment, the image classification module 840 is further configured to obtain the edge feature map of the image to be processed according to the color feature and the shape feature; calculate the number of edge points according to the edge feature map, and obtain an image histogram according to the number of edge points; Input to the image classifier to get the category of the image to be processed.

In one embodiment, an image processing apparatus provided may further include an image retrieval module for acquiring an image to be retrieved and extracting image features of the image to be retrieved; performing image retrieval in an image library according to the image features to obtain an initial retrieval image; calculate the similarity between the initial retrieval image and the image to be retrieved; obtain the target retrieval image according to the similarity.

In one embodiment, a computer device is provided, and the computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 9 . The computer equipment includes a processor, memory, a network interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program implements an image processing method when executed by a processor. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 9 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:

Obtain the image to be processed; the image to be processed contains food areas;

Find the initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm;

Extract the food area in the image to be processed according to the target detection algorithm, and extract the visual features in the food area through the deep neural network algorithm;

Classify images to be processed based on visual features.

In one embodiment, the processor further implements the following steps when executing the computer program: acquiring an image database, and searching for food images from the image database; generating a food image data set according to the food images; adjusting the initial target detection algorithm using the food image data set parameters to obtain the target detection algorithm.

In one embodiment, the processor further implements the following steps when executing the computer program: respectively acquiring candidate frames of the image to be processed according to the target detection algorithm; calculating the candidate frame proportions of each candidate frame; food area.

In one embodiment, the processor further implements the following steps when executing the computer program: obtaining a specific gravity threshold, and using a food area corresponding to a candidate frame proportion higher than the specific gravity threshold as a candidate area; The candidate frame coordinates of the region; the sub-features of the candidate region are extracted according to the candidate frame coordinates; the visual features in the food region are obtained according to the sub-features.

In one embodiment, the processor further implements the following steps when executing the computer program: extracting color features in the visual features, and extracting shape features in the visual features; classifying the images to be processed according to the color features and the shape features, and obtaining the to-be-processed images The category of the image.

In one embodiment, the processor also implements the following steps when executing the computer program: obtaining the edge feature map of the image to be processed according to the color feature and the shape feature; calculating the number of edge points according to the edge feature map, and obtaining the image histogram according to the number of edge points; Input the image histogram into the image classifier to obtain the category of the image to be processed.

In one embodiment, the processor also implements the following steps when executing the computer program: acquiring an image to be retrieved, and extracting image features of the image to be retrieved; performing image retrieval in an image library according to the image features to obtain an initial retrieval image; calculating an initial retrieval image The similarity between the image and the image to be retrieved; the target retrieval image is obtained according to the similarity.

In one embodiment, a computer-readable storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Obtain the image to be processed; the image to be processed contains food areas;

Find the initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain the target detection algorithm;

Extract the food area in the image to be processed according to the target detection algorithm, and extract the visual features in the food area through the deep neural network algorithm;

Classify images to be processed based on visual features.

In one embodiment, the computer program, when executed by the processor, further implements the following steps: acquiring an image database, and looking up food images from the image database; generating a food image dataset based on the food images; adjusting an initial object detection algorithm using the food image dataset The parameters in , get the target detection algorithm.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: respectively acquiring candidate frames of the image to be processed according to the target detection algorithm; calculating the candidate frame proportions of each candidate frame; food area.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining a specific gravity threshold, and using a food area corresponding to a candidate frame proportion higher than the specific gravity threshold as a candidate area; according to the candidate frame proportion of the candidate area, obtaining The candidate frame coordinates of the candidate region; the sub-features of the candidate region are extracted according to the candidate frame coordinates; the visual features in the food region are obtained according to the sub-features.

In one embodiment, the computer program further implements the following steps when executed by the processor: extracting color features in the visual features, and extracting shape features in the visual features; classifying the images to be processed according to the color features and the shape features, and obtaining the to-be-processed images The class to handle the image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining an edge feature map of the image to be processed according to the color feature and the shape feature; calculating the number of edge points according to the edge feature map, and obtaining an image histogram according to the number of edge points ; Input the image histogram into the image classifier to obtain the category of the image to be processed.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: acquiring an image to be retrieved, and extracting image features of the image to be retrieved; performing image retrieval in an image library according to the image features to obtain an initial retrieved image; calculating an initial retrieval image The similarity between the retrieved image and the image to be retrieved; the target retrieval image is obtained according to the similarity.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. an image processing method, is characterized in that, described method comprises: acquiring an image to be processed; the image to be processed includes a food area; Find an initial target detection algorithm, and adjust the parameters in the initial target detection algorithm to obtain a target detection algorithm; Extract the food area in the to-be-processed image according to the target detection algorithm, and extract the visual features in the food area through a deep neural network algorithm; The images to be processed are classified according to the visual features. 2. The method according to claim 1, wherein the adjusting parameters in the initial target detection algorithm to obtain a target detection algorithm, comprising: obtaining a database of images, and looking up food images from the database of images; generating a food image dataset from the food images; The target detection algorithm is obtained by adjusting the parameters in the initial target detection algorithm using the food image data set. 3. The method according to claim 1, wherein the extracting the food region in the to-be-processed image according to the target detection algorithm comprises: Obtain candidate frames of the to-be-processed image respectively according to the target detection algorithm; Calculate the candidate frame proportion of each of the candidate frames; Extract the food area in the to-be-processed image according to the candidate frame weight. 4. The method according to claim 3, wherein the extraction of visual features in the food area by a deep neural network algorithm comprises: Obtain a specific gravity threshold, and use the food area corresponding to the candidate frame specific gravity higher than the specific gravity threshold as a candidate area; According to the candidate frame weight of the candidate area, obtain the candidate frame coordinates of the candidate area; Extract the sub-features of the candidate region according to the candidate frame coordinates; According to the sub-features, the visual features in the food region are obtained. 5. The method according to claim 1, wherein the classifying the to-be-processed image according to the visual feature comprises: Extracting the color feature in the visual feature, and extracting the shape feature in the visual feature; According to the color feature and the shape feature, the image to be processed is classified to obtain the category of the image to be processed. 6. The method according to claim 5, wherein, according to the color feature and the shape feature, classifying the to-be-processed image to obtain the class of the to-be-processed image, comprising: According to the color feature and the shape feature, obtain the edge feature map of the to-be-processed image; Calculate the number of edge points according to the edge feature map, and obtain an image histogram according to the number of edge points; The image histogram is input into an image classifier to obtain the category of the image to be processed. 7. The method of claim 1, wherein the method further comprises: acquiring an image to be retrieved, and extracting image features of the image to be retrieved; Perform image retrieval in the image library according to the image features to obtain an initial retrieval image; calculating the similarity between the initial retrieval image and the to-be-retrieved image; The target retrieval image is obtained according to the similarity. 8. An image processing device, wherein the device comprises: an image acquisition module for acquiring an image to be processed; the image to be processed includes a food area; a parameter adjustment module, used to find an initial target detection algorithm, and adjust parameters in the initial target detection algorithm to obtain a target detection algorithm; a feature extraction module, used for extracting the food area in the to-be-processed image according to the target detection algorithm, and extracting visual features in the food area through a deep neural network algorithm; An image classification module, configured to classify the to-be-processed image according to the visual feature. 9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when the processor executes the computer program . 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images