WO2019233271A1

WO2019233271A1 - Image processing method, computer readable storage medium and electronic device

Info

Publication number: WO2019233271A1
Application number: PCT/CN2019/087678
Authority: WO
Inventors: 陈岩
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-06-08
Filing date: 2019-05-21
Publication date: 2019-12-12
Also published as: CN108805198A; CN108805198B

Abstract

An image processing method, comprising: acquiring an image to be processed; identifying the image to be processed to obtain an image category label for the image to be processed and a corresponding category confidence level, the category confidence level being used to express the credibility which is identified for the image category label; acquiring weather data and a capture time from when the image to be processed was captured; and according to the weather data and capture time, adjust the category confidence level to obtain a target category confidence level corresponding to the image category label.

Description

Image processing method, computer-readable storage medium, and electronic device

Cross-reference to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on June 08, 2018, with application number 201810588360.6, and the invention name is "image processing method, device, computer-readable storage medium, and electronic device". Incorporated by reference in this application.

Technical field

The present application relates to the field of computer technology, and in particular, to an image processing method, a computer-readable storage medium, and an electronic device.

Background technique

Smart devices can capture images through the camera, or they can acquire images through transmission with other smart devices. There can be many scenes for image shooting, such as beach, snow, night, etc. There may also be many target objects in the captured image, such as cars, people, animals, and so on. Generally, images taken in different scenes have different color characteristics, and different target objects have different performance characteristics.

Summary of the Invention

According to various embodiments of the present application, an image processing method, a computer-readable storage medium, and an electronic device are provided.

An image processing method, the method includes:

Obtaining images to be processed;

Identify the image to be processed, and obtain an image classification label of the image to be processed and a corresponding classification confidence level, where the classification confidence level is used to indicate the credibility of the image classification label;

Obtaining weather data and shooting time when shooting the image to be processed;

The classification confidence level is adjusted according to the weather data and shooting time to obtain a target classification confidence level corresponding to the image classification label.

An image processing device includes:

An image acquisition module, configured to acquire an image to be processed;

An image recognition module, configured to identify the image to be processed, to obtain an image classification label of the image to be processed, and a corresponding classification confidence level, where the classification confidence level is used to indicate the credibility of the image classification label. degree;

A data acquisition module, configured to acquire weather data and a shooting time when shooting the image to be processed;

The confidence adjustment module is configured to adjust the classification confidence according to the weather data and shooting time to obtain a target classification confidence corresponding to the image classification label.

A computer-readable storage medium on which a computer program is stored is characterized in that, when the computer program is executed by a processor, the following operations are performed:

Obtaining images to be processed;

An electronic device includes a memory and a processor. The memory stores computer-readable instructions. When the instructions are executed by the processor, the processor causes the processor to perform the following operations:

Obtaining images to be processed;

The above image processing method, computer-readable storage medium, and electronic device can identify the image to be processed to obtain the image classification label level classification confidence, and then obtain the weather data and shooting time when the image to be processed is captured, and adjust according to the weather data and shooting time Classification confidence. In this way, when the image is recognized, the recognition result of the image can be adjusted according to the weather data and shooting time when the image was actually taken, so that the recognition result is more in line with the characteristics of the current environment, and the obtained recognition result is more accurate, which improves the image. Processing accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application or the prior art more clearly, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without paying creative work.

FIG. 1 is an application environment diagram of an image processing method in an embodiment.

FIG. 2 is a flowchart of an image processing method according to an embodiment.

FIG. 3 is a flowchart of an image processing method in another embodiment.

FIG. 4 is a schematic diagram of training a neural network model in one embodiment.

FIG. 5 is a flowchart of an image processing method according to another embodiment.

FIG. 6 is a flowchart of an image processing method according to another embodiment.

FIG. 7 is a flowchart of an image processing method according to another embodiment.

FIG. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment.

FIG. 9 is a schematic diagram of an image processing circuit in one embodiment.

Detailed ways

In order to make the purpose, technical solution, and advantages of the present application clearer, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

It can be understood that the terms “first”, “second”, and the like used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first client may be referred to as the second client, and similarly, the second client may be referred to as the first client. Both the first client and the second client are clients, but they are not the same client.

FIG. 1 is an application environment diagram of an image processing method in an embodiment. As shown in FIG. 1, the application environment includes a terminal 102 and a server 104. The image to be processed may be transmitted between the terminal 102 and the server 104, and the image to be processed may be classified and processed. In one embodiment, the terminal 102 can capture several images to be processed, obtain weather data and shooting time when shooting the images to be processed, and then send the images to be processed, weather data, and shooting time to the server 104. A classification algorithm for classifying images is stored in the server 104, and the received images to be processed can be identified to obtain image classification labels and corresponding classification confidences of the images to be processed. The server 104 can adjust the classification confidence level according to the weather data and the shooting time to obtain the target classification confidence level corresponding to the image classification label. Finally, the recognition result of the image to be processed is adjusted according to the target classification confidence, and the adjusted recognition result is sent to the terminal 102. The terminal 102 may process the image to be processed according to the obtained recognition result. The terminal 102 is an electronic device located at the outermost periphery of a computer network and is mainly used for inputting user information and outputting processing results. For example, the terminal 102 may be a personal computer, a mobile terminal, a personal digital assistant, or a wearable electronic device. The server 104 is a device for responding to a service request while providing a computing service, and may be, for example, one or more computers. In other embodiments provided in this application, the foregoing application environment may further include only the terminal 102 or the server 104, which is not limited herein.

FIG. 2 is a flowchart of an image processing method according to an embodiment. As shown in FIG. 2, the image processing method includes operations 202 to 208. among them:

In operation 202, an image to be processed is obtained.

In one embodiment, cameras may be installed on the electronic device internally or externally, and the positions and number of the cameras are not limited. For example, you can install one camera on the front of the phone and two cameras on the back of the phone. The process of capturing an image is generally divided into two phases: a preview phase and a shooting phase. During the preview phase, the camera will capture images at a certain interval. The captured images will not be stored, but will be displayed for users to view. The user adjusts the shooting angle, light and other parameters according to the displayed image. When a shooting instruction input by the user is detected, the shooting phase is entered, and the electronic device stores the next frame image after receiving the shooting instruction as the final captured image.

The electronic device can store the captured image and send it to the server or other electronic devices. It can be understood that the image to be processed obtained in this embodiment is not limited to being taken by the electronic device itself, but may also be sent by other electronic devices or downloaded through the network. After obtaining the images, the electronic device can process the images immediately or store the images in a folder in a unified manner. After the images stored in the folder reach a certain number, the stored images are processed in a unified manner. The electronic device may store the acquired images in an album, and when the number of images stored in the album is greater than a certain number, processing of the images in the album is triggered.

In operation 204, the image to be processed is identified, and an image classification label and a corresponding classification confidence level of the image to be processed are obtained. The classification confidence level is used to indicate the credibility of the image classification label.

The image classification label can be used to indicate the specific classification of the shooting scene of the image to be processed. For example, the shooting scene of an image can be divided into one of landscape, beach, blue sky, green grass, snow, backlight, sunrise / sunset, fireworks, spotlight, and so on. The electronic device can obtain an image classification label by identifying the shooting scene of the image to be processed.

Specifically, the electronic device can recognize the image to be processed by using a classification algorithm model, and obtain an image classification label of the image to be processed. Before the classification algorithm model is used for recognition, it is usually necessary to train the classification algorithm model through a training image set to obtain an algorithm model that can accurately identify it. In general, the more training images contained in the training image set, the more accurate the classification algorithm model finally obtained. After the classification algorithm model is trained, the image to be processed is input into the trained classification algorithm model, and the classification result is output through the classification algorithm model.

In one embodiment, the classification algorithm model can identify the image classification label corresponding to the image to be processed and the corresponding classification confidence level. The classification confidence level is used to indicate the credibility of the image classification label, and the greater the classification confidence level is. , The more accurate the recognition result is. Generally, multiple application scenarios can be defined for the image in advance, and multiple classification labels can be defined according to the multiple application scenarios. The classification algorithm model can calculate the probability that the image corresponds to each classification label. The greater the probability of a classification label, the greater the likelihood that an image is an application scenario corresponding to the classification label. The classification label with the highest probability can be used as the image classification label of the image, and the probability corresponding to the image classification label is the classification confidence. The above classification algorithm model may be, but is not limited to, a K nearest neighbor classification model, a nearest neighbor classification model, a naive Bayes classification model, a neural network model, and the like.

Operation 206: Obtain weather data and a shooting time when shooting the image to be processed.

When an electronic device captures an image, it can obtain the weather data and the shooting time when the image is captured, and store the weather data and the shooting time at the same time. When the electronic device transmits the image to be processed, the weather data and the shooting time can be transmitted simultaneously with the image to be processed.

In one embodiment, when the electronic device detects a shooting instruction, it will control the camera to capture an image. After capturing the image, it calls a dedicated weather interface to obtain weather data, and at the same time obtains the current time as the shooting time when shooting the image. For example, the acquired weather data can be divided into sunny, cloudy, cloudy, and rainy days. The obtained shooting time can be "11:53 on May 26, 2018".

Operation 208: Adjust the classification confidence level according to the weather data and the shooting time to obtain the target classification confidence level corresponding to the image classification label.

It is understandable that when images are taken in different weather and time, the captured images have different characteristics, and the recognition results of the electronic devices also have different credibility. For example, the image collected on a sunny day is relatively light, the captured image can well restore the original characteristics of the object, and the recognition result of the image by the electronic device is more accurate. The images collected during rainy weather are relatively dark, and the captured images deviate from the original characteristics of the object. Similarly, images taken at noon more accurately reflect the original characteristics of objects than images taken in the evening.

In the embodiment provided by this application, if the classification confidence corresponding to the image classification label obtained by image recognition is too low, such a recognition result is generally considered to be unreliable, and the obtained recognition result will be directly discarded. Because the recognition result is affected by the weather and shooting time, in order to prevent the recognition result from being mishandled, the recognition result can be adjusted by weather data and shooting time. The adjusted recognition result can more accurately reflect the real shooting of the image Scenes. For example, if the image is taken on a sunny day, the classification confidence obtained through recognition can be appropriately increased, and if the image is shot in a rainy weather, the classification confidence obtained through recognition can be appropriately reduced.

The image processing method provided in the foregoing embodiment may recognize an image to be processed to obtain an image classification tag level classification confidence, and then obtain weather data and a shooting time when the image to be processed is captured, and adjust the classification confidence according to the weather data and the shooting time. In this way, when the image is recognized, the recognition result of the image can be adjusted according to the weather data and shooting time when the image was actually taken, so that the recognition result is more in line with the characteristics of the current environment, and the obtained recognition result is more accurate, improving the image Processing accuracy.

FIG. 3 is a flowchart of an image processing method in another embodiment. As shown in FIG. 3, the image processing method includes operations 302 to 312. among them:

In operation 302, an image to be processed is obtained.

When an electronic device processes an image, it can be triggered automatically or manually by a user. An automatic triggering condition may be preset, and when the automatic triggering condition is satisfied, operation 302 is performed. For example, when the number of updated images in the electronic device reaches a preset number, it starts to acquire the stored images and starts processing the images. Or each time when a specified time is reached, the image to be processed is started to be acquired, and the image to be processed is processed.

Generally, the larger the number of images to be processed, the larger the memory consumed during image processing and the longer the time consumed. The processing capacity of the terminal is generally relatively limited, so when the terminal needs to process a large number of images in batches, it can send the images and corresponding weather data to the server together with the shooting time. The server recognizes the image and returns the recognition result to the terminal. The terminal processes the image according to the received recognition result.

In operation 304, the image to be processed is taken as an input of a neural network model, and a reference confidence level corresponding to at least one preset classification label is calculated through the neural network model.

In one embodiment, the image to be processed may be identified by a neural network model. Before recognition, the neural network model needs to be trained. The electronic device can obtain a training image set used to train a neural network model, and the training images in the training image set all have labels for labeling scene categories. During the training process, the training image in the training image set is used as the input of the neural network model, and the recognition result of the training image is obtained. The recognition result is compared with the pre-labeled label, and the parameters of the neural network model are adjusted according to the comparison result. To make the recognition result of the neural network model more accurate.

Before training the model, the shooting scenes of the image are predefined, and each shooting scene corresponds to a preset classification label. According to the neural network model, any one of the predefined shooting scenes can be identified. After the neural network model is trained, the image to be processed that needs to be identified is used as the input of the neural network model. The neural network model is used to calculate the reference confidence level corresponding to each of the predefined classification labels, and the final classification of the image is determined according to the reference confidence level. result.

FIG. 4 is a schematic diagram of training a neural network model in one embodiment. As shown in FIG. 4, when training a neural network model, a training image with a category label is used as an input to the neural network model, thereby completing the training of the neural network model. After the neural network model is trained, a loss function can be obtained. In the recognition process, the image can be recognized by the trained neural network model, and the confidence of each category can be calculated by the loss function, and the final classification result is determined according to the obtained confidence.

Specifically, the neural network model may be stored in an electronic device in advance. When an image to be processed is obtained, the neural network model is used to perform recognition processing on the image to be processed. It can be understood that the neural network model generally occupies the storage space of the electronic device, and when processing a large number of images, the storage capacity requirements of the electronic device are also relatively high. When processing the to-be-processed image on the terminal, it can be processed through the neural network model stored locally on the terminal, or the to-be-processed image can be sent to the server for processing through the neural network model stored on the server.

Because the processing capability of the terminal is generally limited, the server can send the trained neural network model to the terminal after the neural network model is trained, and the terminal does not need to train the neural network model. At the same time, the neural network model stored in the terminal can be a compressed model, so that the compressed model will occupy less resources, but the corresponding recognition accuracy will be lower. The terminal can decide whether to perform the recognition processing locally on the terminal or the recognition processing on the server according to the number of images to be processed. After the terminal obtains the image to be processed, it counts the number of images of the image to be processed. If the number of images exceeds the preset upload number, the terminal uploads the image to be processed to the server, and performs recognition processing on the server. After processing by the server, the recognition result is sent to the terminal.

Operation 306: Determine the image classification label corresponding to the image to be processed from the preset classification labels according to the reference confidence level, and use the reference confidence level corresponding to the image classification label as the classification confidence level.

The preset classification labels are predefined, and the reference confidence corresponding to each preset classification label is calculated by a neural network model. The larger the reference confidence level of the preset classification label, the greater the possibility that the image should correspond to the preset classification label. According to the reference confidence level, the final recognized image classification label can be determined.

Specifically, the preset classification label with the highest reference confidence can be used as the image classification label corresponding to the image to be processed, and the maximum reference confidence can be used as the classification confidence corresponding to the image classification label. For example, if the predefined preset classification labels include landscape, night, and snow, and the image is used as the input of the neural network model, the reference confidences corresponding to the three preset classification labels are 0.85, 0.1, and 0.05, respectively. The preset classification label “Landscape” with the highest confidence is used as the image recognition result. That is, the image classification label of the image is "landscape", and the corresponding classification confidence is 0.85.

Operation 308: Determine whether the classification confidence level exceeds a preset confidence level range.

In operation 310, if yes, the weather data and the shooting time when the image to be processed is captured are acquired.

In the recognition result, if the classification confidence of the finally obtained image classification label exceeds a certain range, the classification result is considered to be unreliable, and the electronic device may directly discard the unreliable classification result. However, in order to prevent the recognition result from being discarded because the obtained classification confidence is inaccurate, the classification confidence can be adjusted according to weather data and shooting time to make the obtained classification confidence more accurate.

Specifically, the classification confidence can be divided into a trusted value range and an untrusted value range. When the classification confidence is within the trusted value range, the classification result is considered reliable; when the classification confidence is within the untrusted value range The results are considered unreliable. When the classification confidence is within the preset confidence range, it is considered that the classification confidence adjusted according to the added value of the confidence is the same as the above-mentioned value range of the classification confidence before the adjustment, so that adjusting the classification confidence does not affect the reliability Judging the result, it is not necessary to adjust the classification confidence.

For example, when the classification confidence is [0,0.4], the classification result is considered to be unreliable; when the classification confidence is (0.4,1], the classification result is considered to be reliable. The minimum value of the confidence increase is 0.4, and the maximum value is 1.6. It can be understood that when the value of the classification confidence is [0,0.25], it is immediately multiplied by the maximum value of the confidence increase of 1.6, and the value of the adjusted classification confidence is still within the range of unreliable values. Without affecting the judgment of the reliability of the classification result. Therefore, when the classification confidence is within the preset confidence range [0, 0.25], no adjustment is needed.

Operation 312: Determine the confidence value increase according to the weather data and the shooting time, and adjust the classification confidence value according to the confidence value increase to obtain the target classification confidence value corresponding to the image classification label.

You can determine the confidence increase of the classification confidence based on the weather data and shooting time, and then adjust the classification confidence based on the obtained confidence increase. Specifically, the operation of adjusting the confidence level of the classification may include:

Operation 502: Determine a first confidence value increase according to weather data, and determine a second confidence value increase according to a shooting time.

After obtaining the weather data and the shooting time, the increments for adjusting the classification confidence can be obtained according to the weather data and the shooting time, respectively. Specifically, the first confidence value increase may be determined according to the weather data, and the second confidence value increase may be determined according to the shooting time. The first confidence value increase and the second confidence value increase may be related to each other or independent of each other, which is not limited herein. Confidence increment refers to the increment that adjusts the confidence of the classification. It can be negative or positive.

The correspondence relationship between the weather data and the first confidence value increase may be defined in advance, and the first confidence value increase may be obtained according to the weather data. The correspondence between the shooting time and the added value of the second confidence is defined in advance, and the second added value of the confidence can be obtained according to the shooting time. For example, the definition of weather data includes "00", "01", "10", and "11", which respectively represent the weather "sunny", "cloudy", "cloudy", and "rainy", and the corresponding first confidence value increases respectively. It is 1.2, 1.1, 0.9, 0.8.

Operation 504: Multiply the result of multiplying the first confidence value by the second confidence value by the classification confidence to obtain the target classification confidence corresponding to the image classification label.

The classification confidence is adjusted according to the obtained first confidence value increase and the second confidence value increase, or a total confidence value increase may be generated according to the first confidence value increase and the second confidence value increase, and then adjusted according to the total confidence value increase Classification confidence. When adjusting the classification confidence, it can be adjusted by means of superposition or by means of product, which is not limited here.

In this embodiment, the first confidence value increase and the second confidence value increase may be multiplied, and then the result of multiplying the first confidence value increase with the second confidence value increase and the classification confidence may be multiplied to obtain image classification. The target classification confidence corresponding to the label. For example, if the first confidence increase is 1.2, the second confidence increase is 0.9, and the classification confidence is 0.8, the target classification confidence is 1.2 * 0.9 * 0.8 = 0.864 by adjusting the product method.

In one embodiment, when different image classification labels are identified, the algorithms for adjusting the classification confidence may be different, that is, the manners of obtaining the added value of the confidence may be different. Then specific:

Operation 602: Obtain a first correspondence relationship according to the image classification label, and determine a first confidence increase value corresponding to the weather data according to the first correspondence relationship.

When the recognition result obtained by the electronic device is different, the algorithm for adjusting the confidence level of the classification may also be different. The correspondence between the weather data and the first confidence value increase under different recognition results can be defined in advance, and the first correspondence relationship can be obtained according to the identified image classification labels. The first correspondence relationship is the correspondence between the weather data and the first confidence value increase. Therefore, the first confidence value increase corresponding to the weather data can be obtained according to the first correspondence relationship.

Operation 604: Obtain a second correspondence relationship according to the image classification label, and determine a second confidence increase value corresponding to the shooting time according to the second correspondence relationship.

The correspondence between the shooting time and the added value of the second confidence value under different recognition results is predefined, and the second correspondence can be obtained according to the identified image classification labels. The second correspondence is the correspondence between the shooting time and the value of the second confidence value. A first confidence value increase corresponding to the shooting time can be obtained according to the second correspondence relationship.

In the embodiment provided by the present application, the classification confidence degree may indicate the credibility of the image recognition result. Generally, the obtained classification confidence has a certain value range, and the adjusted classification confidence cannot exceed the value range. specific:

Operation 702: Calculate a reference classification confidence level according to the confidence value increment and the classification confidence level.

Operation 704: if the confidence value is negative, determine whether the reference classification confidence is less than a preset lower confidence value; if so, use the lower confidence value as the target classification confidence corresponding to the image classification label; if not, refer to The classification confidence is used as the target classification confidence corresponding to the image classification label.

Define the value range of the classification confidence, the maximum cannot exceed the upper confidence limit, and the minimum cannot exceed the lower confidence limit. If the obtained confidence increase is negative, the classification confidence adjusted according to the confidence increase will decrease, that is, the adjusted classification confidence cannot be less than the lower confidence limit. Specifically, first, the classification confidence is adjusted according to the added value of the confidence, and a reference classification confidence is calculated. If the confidence increase is negative, the obtained reference classification confidence is compared with the lower confidence limit. If the obtained reference classification confidence is less than the lower confidence limit, the lower confidence limit is used as the target classification confidence; if the obtained reference classification confidence is greater than the lower confidence limit, the reference classification confidence is used as the target classification Confidence.

In operation 706, if the confidence increase value is a positive number, determine whether the reference classification confidence is greater than a preset upper confidence limit; if so, use the upper confidence limit as the target classification confidence corresponding to the image classification label. The reference classification confidence is used as the target classification confidence corresponding to the image classification label.

If the obtained confidence increase is a positive number, the classification confidence adjusted according to the confidence increase will become larger, that is, the adjusted classification confidence cannot be greater than the upper confidence limit. Specifically, first, the classification confidence is adjusted according to the added value of the confidence, and a reference classification confidence is calculated. If the confidence increase is a positive number, the obtained reference classification confidence is compared with the upper confidence limit. If the obtained reference classification confidence is greater than the upper confidence limit, the upper confidence limit is used as the target classification confidence; if the obtained reference classification confidence is less than the upper confidence limit, the reference classification confidence is used as the target classification Confidence.

For example, the image classification label obtained by identifying the image may be one of landscape, beach, blue sky, green grass, snow, backlight, sunrise / sunset, fireworks, and spotlight. If the identified image is one of the above image classification labels, the confidence value obtained based on the weather data is: if the weather is sunny, the confidence value is increased to 1.2; if the weather is cloudy, the confidence value is increased to 1.1; if the weather is cloudy It is cloudy and the confidence increase is 0.9; if the weather is rainy, the confidence increase is 0.8.

If the image classification label of the identified image is one of landscape, beach, blue sky, green grass, snow, and backlight, the confidence value obtained based on the shooting time is: if the shooting time is from 07:00 to 10:00, the confidence is If the shooting time is from 10:00 to 14:00, the confidence value will increase to 1.2; if the shooting time is from 14:00 to 17:00, the confidence value will increase to 1.1; if the shooting time is 19:00 ～ 21: 00, the confidence increase value is 0.9; if the shooting time is from 21:00 to 02:00, the confidence increase value is 0.8; if the shooting time is from 20:00 to 05:00, the confidence increase value is 0.9; The built-in reliability increase value is 1 for other time periods.

If the image classification label of the identified image is one of night scene, fireworks, and spotlight, the confidence value obtained based on the shooting time is: if the shooting time is from 07:00 to 10:00, the confidence value is increased to 0.9; if shooting If the time is from 10:00 to 14:00, the confidence value will increase to 0.8; if the shooting time is from 14:00 to 17:00, the confidence value will increase to 0.9; if the shooting time is from 19:00 to 21:00, the confidence will be increased. If the shooting time is from 21:00 to 23:00, the confidence value will increase to 1.2; If the shooting time is from 23:00 to 05:00, the confidence value will increase to 1.1; the built-in reliability value will be added to other time periods Is 1.

If the image classification label of the identified image is sunrise / sunset, the confidence value obtained according to the shooting time is: if the shooting time is from 05:00 to 07:00, the confidence value is increased to 1.2; if the shooting time is at 17:00 ～ 20: 00, the confidence increase value is 1.2; the built-in reliability increase value is 0.8 in other time periods. Obtain the confidence value increase according to the corresponding relationship, and multiply the confidence value increase by the classification confidence value to obtain the target classification confidence value. The range of the obtained target classification confidence value is [0,1].

After the generated image classification label, the image to be processed can be labeled according to the image classification label, so that the user can search the image according to the generated classification label. For example, the images to be processed may be classified and displayed, which is convenient for users to view the images to be processed. The search box can also be displayed on the display interface. The user can enter a search keyword through the search box, and the electronic device can search for a pending image containing the search keyword in the classification label for display.

The electronic device may also classify the image to be processed according to the image classification label, and perform the classification processing on the image to be processed. An image processing algorithm is acquired according to the image classification label, and the image to be processed is processed according to the acquired image processing algorithm. For example, when it is recognized as a landscape, the image saturation may be recognized, and when it is recognized as a night landscape, the brightness of the image may be appropriately increased.

The image processing method provided in the foregoing embodiment may recognize an image to be processed to obtain an image classification tag level classification confidence, and then obtain weather data and a shooting time when the image to be processed is captured, and adjust the classification confidence according to the weather data and the shooting time. In this way, when the image is recognized, the recognition result of the image can be adjusted according to the weather data and shooting time when the image was actually taken, so that the recognition result is more in line with the characteristics of the current environment, and the obtained recognition result is more accurate, which improves the image. Processing accuracy.

It should be understood that although the operations in the flowcharts of FIG. 2, FIG. 3, FIG. 5, FIG. 6, and FIG. 7 are sequentially displayed as indicated by the arrows, these operations are not necessarily performed sequentially in the order indicated by the arrows. Unless explicitly stated in this article, there is no strict order in which these operations can be performed, and these operations can be performed in other orders. Moreover, at least a part of the operations in FIG. 2, FIG. 3, FIG. 5, FIG. 6, and FIG. 7 may include multiple sub-operations or multiple stages. These sub-operations or stages are not necessarily performed at the same time, but may be performed at the same time. At different times, the execution order of these sub-operations or phases is not necessarily sequential, but can be performed in turn or alternately with at least a part of the sub-operations or phases of other operations or other operations.

FIG. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment. As shown in FIG. 8, the image processing apparatus 800 includes an image acquisition module 802, an image recognition module 804, a data acquisition module 806, and a confidence adjustment module 808. among them:

The image acquisition module 802 is configured to acquire an image to be processed.

An image recognition module 804 is configured to identify the image to be processed, and obtain an image classification label of the image to be processed and a corresponding classification confidence level, where the classification confidence level is used to indicate that the image classification label can be identified as an image classification label. Trust degree.

A data acquisition module 806 is configured to acquire weather data and a shooting time when the image to be processed is captured.

The confidence adjustment module 808 is configured to adjust the classification confidence according to the weather data and the shooting time to obtain a target classification confidence corresponding to the image classification label.

The image processing apparatus provided in the foregoing embodiment may recognize an image to be processed to obtain an image classification tag level classification confidence, and then obtain weather data and a shooting time when the image to be processed is captured, and adjust the classification confidence according to the weather data and the shooting time. In this way, when the image is recognized, the recognition result of the image can be adjusted according to the weather data and shooting time when the image was actually taken, so that the recognition result is more in line with the characteristics of the current environment, and the obtained recognition result is more accurate, improving the image Processing accuracy.

In one embodiment, the image recognition module 804 is further configured to use the to-be-processed image as an input of a neural network model, and calculate a reference confidence level corresponding to at least one preset classification label through the neural network model; according to the reference confidence level, The image classification label corresponding to the image to be processed is determined from the preset classification labels, and the reference confidence level corresponding to the image classification label is used as the classification confidence level.

In one embodiment, the data acquisition module 806 is further configured to determine whether the classification confidence exceeds a preset confidence range; if so, obtain weather data and a shooting time when the image to be processed is captured.

In one embodiment, the confidence adjustment module 808 is further configured to determine a confidence value increase based on weather data and shooting time, and adjust the classification confidence value according to the confidence value increase to obtain a target classification confidence corresponding to the image classification label. degree.

In one embodiment, the confidence adjustment module 808 is further configured to determine a first confidence value increase according to the weather data, and determine a second confidence value increase according to the shooting time; and to combine the first confidence value increase with the The result of the multiplication of the second confidence value is multiplied with the classification confidence value to obtain the target classification confidence value corresponding to the image classification label.

In one embodiment, the confidence adjustment module 808 is further configured to obtain a first correspondence relationship according to the image classification tag, and determine a first confidence increase value corresponding to the weather data according to the first correspondence relationship; according to the image The classification label acquires a second correspondence relationship, and determines a second confidence increase value corresponding to the shooting time according to the second correspondence relationship.

In one embodiment, the confidence adjustment module 808 is further configured to calculate a reference classification confidence based on the confidence increase and classification confidence; if the confidence increase is negative, determine whether the reference classification confidence is less than A preset confidence level lower limit; if it is, the confidence level lower limit is used as the target classification confidence level corresponding to the image classification label; otherwise, the reference classification confidence level is used as the target corresponding to the image classification label level Classification confidence; if the added confidence value is a positive number, determine whether the reference classification confidence is greater than a preset upper confidence limit; if so, use the upper confidence limit as the image classification label corresponding The target classification confidence of, if otherwise, the reference classification confidence is used as the target classification confidence corresponding to the image classification label.

The division of each module in the above image processing apparatus is for illustration only. In other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the above image processing apparatus.

An embodiment of the present application further provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, causing the processors to perform the image processing provided by the foregoing embodiments method.

A computer program product containing instructions, which when run on a computer, causes the computer to execute the image processing method provided by the above embodiments.

An embodiment of the present application further provides an electronic device. The above electronic device includes an image processing circuit. The image processing circuit may be implemented by hardware and / or software components, and may include various processing units that define an ISP (Image Signal Processing) pipeline. FIG. 9 is a schematic diagram of an image processing circuit in one embodiment. As shown in FIG. 9, for ease of description, only aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 9, the image processing circuit includes an ISP processor 940 and a control logic 950. The image data captured by the imaging device 910 is first processed by the ISP processor 940, which analyzes the image data to capture image statistical information that can be used to determine and / or one or more control parameters of the imaging device 910. The imaging device 910 may include a camera having one or more lenses 912 and an image sensor 914. The image sensor 914 may include a color filter array (such as a Bayer filter). The image sensor 914 may obtain the light intensity and wavelength information captured by each imaging pixel of the image sensor 914, and provide a set of Image data. The sensor 920 (such as a gyroscope) may provide parameters (such as image stabilization parameters) of the acquired image processing to the ISP processor 940 based on the interface type of the sensor 920. The sensor 920 interface may use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the foregoing interfaces.

In addition, the image sensor 914 may also send the original image data to the sensor 920, and the sensor 920 may provide the original image data to the ISP processor 940 based on the interface type of the sensor 920, or the sensor 920 stores the original image data in the image memory 930.

The ISP processor 940 processes the original image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 940 may perform one or more image processing operations on the original image data and collect statistical information about the image data. The image processing operations may be performed with the same or different bit depth accuracy.

The ISP processor 940 may also receive image data from the image memory 930. For example, the sensor 920 interface sends the original image data to the image memory 930, and the original image data in the image memory 930 is then provided to the ISP processor 940 for processing. The image memory 930 may be a part of a memory device, a storage device, or a separate dedicated memory in an electronic device, and may include a DMA (Direct Memory Access) feature.

When receiving raw image data from the image sensor 914 interface or from the sensor 920 interface or from the image memory 930, the ISP processor 940 may perform one or more image processing operations, such as time-domain filtering. The processed image data may be sent to the image memory 930 for further processing before being displayed. The ISP processor 940 receives processing data from the image memory 930 and performs image data processing on the processing data in the original domain and in the RGB and YCbCr color spaces. The image data processed by the ISP processor 940 may be output to the display 970 for viewing by the user and / or further processed by a graphics engine or a GPU (Graphics Processing Unit). In addition, the output of the ISP processor 940 can also be sent to the image memory 930, and the display 970 can read image data from the image memory 930. In one embodiment, the image memory 930 may be configured to implement one or more frame buffers. In addition, the output of the ISP processor 940 may be sent to an encoder / decoder 960 to encode / decode image data. The encoded image data can be saved and decompressed before being displayed on the display 970 device. The encoder / decoder 960 may be implemented by a CPU or a GPU or a coprocessor.

The statistical data determined by the ISP processor 940 may be sent to the control logic 950 unit. For example, the statistical data may include image information of the image sensor 914 such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, and lens 912 shading correction. The control logic 950 may include a processor and / or a microcontroller that executes one or more routines (such as firmware). The one or more routines may determine the control parameters of the imaging device 910 and the ISP processing according to the received statistical data. Parameters of the controller 940. For example, the control parameters of the imaging device 910 may include sensor 920 control parameters (such as gain, integration time for exposure control, image stabilization parameters, etc.), camera flash control parameters, lens 912 control parameters (such as focus distance for focusing or zooming), or these A combination of parameters. ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), and lens 912 shading correction parameters.

The following is an image processing method provided by the above embodiment using the image processing technology in FIG. 9.

Any reference to memory, storage, database, or other media used in this application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR, SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation manners of the present application, and their descriptions are more specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of this application patent shall be subject to the appended claims.

Claims

An image processing method, the method includes:

Obtaining images to be processed;

Identify the image to be processed, and obtain an image classification label of the image to be processed and a corresponding classification confidence level, where the classification confidence level is used to indicate the credibility of the image classification label;

Obtaining weather data and shooting time when shooting the image to be processed;

The classification confidence level is adjusted according to the weather data and shooting time to obtain a target classification confidence level corresponding to the image classification label.
The method according to claim 1, wherein the step of identifying the image to be processed to obtain an image classification label and a corresponding classification confidence of the image to be processed comprises:

Using the to-be-processed image as an input to a neural network model, and calculating a reference confidence level corresponding to at least one preset classification label through the neural network model;

An image classification label corresponding to the image to be processed is determined from the preset classification labels according to the reference confidence level, and a reference confidence level corresponding to the image classification label is used as a classification confidence level.
The method according to claim 1, wherein the acquiring the weather data and the shooting time when the image to be processed is captured comprises:

Judging whether the classification confidence degree exceeds a preset confidence range;

If yes, weather data and shooting time when the image to be processed is captured are obtained.
The method according to any one of claims 1 to 3, wherein the adjusting the classification confidence level according to the weather data and shooting time to obtain a target classification confidence level corresponding to the image classification label includes: :

The confidence value increase is determined according to the weather data and the shooting time, and the classification confidence value is adjusted according to the confidence value increase to obtain the target classification confidence value corresponding to the image classification label.
The method according to claim 4, characterized in that said determining the confidence value increase according to the weather data and the shooting time, and adjusting the classification confidence value according to the confidence value increase to obtain the target classification corresponding to the image classification label Confidence, including:

Determining a first confidence value increase according to the weather data, and determining a second confidence value increase according to the shooting time;

Multiplying the result of multiplying the first confidence value increase with the second confidence value increase with the classification confidence to obtain the target classification confidence corresponding to the image classification label.
The method according to claim 5, wherein the determining a first confidence value increase based on the weather data comprises:

Obtaining a first correspondence relationship according to the image classification label, and determining a first confidence increase value corresponding to the weather data according to the first correspondence relationship;

The determining a second confidence value increase according to the shooting time includes:

A second correspondence relationship is obtained according to the image classification label, and a second confidence increase value corresponding to the shooting time is determined according to the second correspondence relationship.
The method according to claim 4, wherein the adjusting the classification confidence according to the added value of the confidence to obtain the target classification confidence corresponding to the image classification label comprises:

Calculating the reference classification confidence level according to the added confidence level and the classification confidence level;

If the added confidence value is negative, determine whether the reference classification confidence is less than a preset lower confidence value, and if so, use the lower confidence value as the target classification confidence corresponding to the image classification label. If otherwise, the reference classification confidence is used as the target classification confidence corresponding to the image classification label;

If the added confidence value is a positive number, determine whether the reference classification confidence is greater than a preset upper confidence limit value; if so, use the upper confidence limit value as a target classification confidence corresponding to the image classification label Degree, if otherwise, the reference classification confidence level is used as the target classification confidence level corresponding to the image classification label.
A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, the following operations are implemented:

Obtaining images to be processed;

Identify the image to be processed, and obtain an image classification label of the image to be processed and a corresponding classification confidence level, where the classification confidence level is used to indicate the credibility of the image classification label;

Obtaining weather data and shooting time when shooting the image to be processed;

The classification confidence level is adjusted according to the weather data and shooting time to obtain a target classification confidence level corresponding to the image classification label.
The computer-readable storage medium according to claim 8, wherein the computer program is executed by a processor to identify the image to be processed to obtain an image classification label and a corresponding classification of the image to be processed. During confidence, the following operations are also performed:

Using the to-be-processed image as an input to a neural network model, and calculating a reference confidence level corresponding to at least one preset classification label through the neural network model;

An image classification label corresponding to the image to be processed is determined from the preset classification labels according to the reference confidence level, and a reference confidence level corresponding to the image classification label is used as a classification confidence level.
The computer-readable storage medium according to claim 8, wherein when the computer program is executed by the processor to obtain weather data and shooting time when the image to be processed is captured, the following operations are further performed:

Judging whether the classification confidence degree exceeds a preset confidence range;

If yes, weather data and shooting time when the image to be processed is captured are obtained.
The computer-readable storage medium according to any one of claims 8 to 10, wherein the computer program is executed by a processor and the classification confidence is adjusted according to the weather data and shooting time to obtain the When describing the target classification confidence corresponding to the image classification label, the following operations are also performed:

The confidence value increase is determined according to the weather data and the shooting time, and the classification confidence value is adjusted according to the confidence value increase to obtain the target classification confidence value corresponding to the image classification label.
The computer-readable storage medium of claim 11, wherein the computer program is executed by a processor to determine a confidence value increase based on weather data and shooting time, and adjust the classification confidence value according to the confidence value increase. When obtaining the target classification confidence corresponding to the image classification label, the following operations are also performed:

Determining a first confidence value increase according to the weather data, and determining a second confidence value increase according to the shooting time;

Multiplying the result of multiplying the first confidence value increase with the second confidence value increase with the classification confidence to obtain the target classification confidence corresponding to the image classification label.
The computer-readable storage medium according to claim 12, wherein, when the computer program is executed by the processor and the first confidence value increase is determined according to the weather data, the following operations are further performed:

Obtaining a first correspondence relationship according to the image classification label, and determining a first confidence increase value corresponding to the weather data according to the first correspondence relationship;

The determining a second confidence value increase according to the shooting time includes:

A second correspondence relationship is obtained according to the image classification label, and a second confidence increase value corresponding to the shooting time is determined according to the second correspondence relationship.
The computer-readable storage medium according to claim 11, wherein the computer program is executed by a processor to adjust the classification confidence according to the added value of the confidence to obtain a target classification corresponding to the image classification label. During confidence, the following operations are also performed:

Calculating the reference classification confidence level according to the added confidence level and the classification confidence level;

If the added confidence value is negative, determine whether the reference classification confidence is less than a preset lower confidence value, and if so, use the lower confidence value as the target classification confidence corresponding to the image classification label. If otherwise, the reference classification confidence is used as the target classification confidence corresponding to the image classification label;

If the added confidence value is a positive number, determine whether the reference classification confidence is greater than a preset upper confidence limit value; if so, use the upper confidence limit value as a target classification confidence corresponding to the image classification label Degree, if otherwise, the reference classification confidence level is used as the target classification confidence level corresponding to the image classification label.
An electronic device includes a memory and a processor. The memory stores computer-readable instructions. When the instructions are executed by the processor, the processor causes the processor to perform the following operations:

Obtaining images to be processed;

Identify the image to be processed, and obtain an image classification label of the image to be processed and a corresponding classification confidence level, where the classification confidence level is used to indicate the credibility of the image classification label;

Obtaining weather data and shooting time when shooting the image to be processed;

The classification confidence level is adjusted according to the weather data and shooting time to obtain a target classification confidence level corresponding to the image classification label.
The electronic device according to claim 15, wherein when the processor executes the recognition of the image to be processed to obtain an image classification label and a corresponding classification confidence of the image to be processed, the processor further executes As follows:

Using the to-be-processed image as an input to a neural network model, and calculating a reference confidence level corresponding to at least one preset classification label through the neural network model;

An image classification label corresponding to the image to be processed is determined from the preset classification labels according to the reference confidence level, and a reference confidence level corresponding to the image classification label is used as a classification confidence level.
The electronic device according to claim 15, wherein when the processor executes the acquiring the weather data and the shooting time when shooting the image to be processed, the processor further performs the following operations:

Judging whether the classification confidence degree exceeds a preset confidence range;

If yes, weather data and shooting time when the image to be processed is captured are obtained.
The electronic device according to any one of claims 15 to 17, wherein the processor executes the adjustment of the classification confidence according to the weather data and a shooting time to obtain a corresponding value of the image classification label. When the target classification is confidence, the following operations are also performed:

The confidence value increase is determined according to the weather data and the shooting time, and the classification confidence value is adjusted according to the confidence value increase to obtain the target classification confidence value corresponding to the image classification label.
The electronic device according to claim 18, wherein the processor executes the determination of a confidence increase value according to weather data and shooting time, and adjusts the classification confidence value according to the confidence value increase to obtain the image When the target classification confidence corresponding to the classification label, the following operations are also performed:

Determining a first confidence value increase according to the weather data, and determining a second confidence value increase according to the shooting time;

Multiplying the result of multiplying the first confidence value increase with the second confidence value increase with the classification confidence to obtain the target classification confidence corresponding to the image classification label.
The electronic device according to claim 18, wherein when the processor executes the adjustment of the classification confidence according to the added value of the confidence to obtain the target classification confidence corresponding to the image classification label, the processor further executes As follows:

Calculating the reference classification confidence level according to the added confidence level and the classification confidence level;

If the added confidence value is negative, determine whether the reference classification confidence is less than a preset lower confidence value, and if so, use the lower confidence value as the target classification confidence corresponding to the image classification label. If otherwise, the reference classification confidence is used as the target classification confidence corresponding to the image classification label;

If the added confidence value is a positive number, determine whether the reference classification confidence is greater than a preset upper confidence limit value; if so, use the upper confidence limit value as a target classification confidence corresponding to the image classification label Degree, if otherwise, the reference classification confidence level is used as the target classification confidence level corresponding to the image classification label.