CN116888621A - Image detection method and device and electronic equipment - Google Patents

Image detection method and device and electronic equipment Download PDF

Info

Publication number
CN116888621A
CN116888621A CN202180093086.5A CN202180093086A CN116888621A CN 116888621 A CN116888621 A CN 116888621A CN 202180093086 A CN202180093086 A CN 202180093086A CN 116888621 A CN116888621 A CN 116888621A
Authority
CN
China
Prior art keywords
image
algorithm
image processing
image data
parameters
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202180093086.5A
Other languages
Chinese (zh)
Inventor
邱珏沁
柳海波
闻明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN116888621A publication Critical patent/CN116888621A/en
Pending legal-status Critical Current

Links

Classifications

    • G06T5/60—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Abstract

The embodiment of the application provides an image detection method, an image detection device and electronic equipment, wherein the image detection method comprises the following steps: acquiring image data to be detected by a first camera device; processing the image data to be detected by using an image processing algorithm to generate a processed image; inputting the processed image into an image detection model to obtain a detection result; the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result. According to the image detection method provided by the application, when the new shooting device is combined with the trained image detection model, the accuracy of image detection model reasoning can be improved.

Description

Image detection method and device and electronic equipment Technical Field
The embodiment of the application relates to the technical field of artificial intelligence, in particular to an image detection method, an image detection device and electronic equipment.
Background
With the development of scientific technology, artificial intelligence (AI, artificial Intelligence) technology has been improved. In some artificial intelligence techniques, machine learning methods are generally used to construct initial models of various structures, such as neural network models, support vector machine models, decision tree models, and the like. Then, training the initial model through the training sample to realize functions such as image detection, voice recognition and the like. Currently, in computer vision technology based on image detection, a neural network is generally trained to obtain a perception model, so as to realize image detection tasks such as scene recognition, object detection or image segmentation.
In related computer vision technology, sample image data for training a neural network and image data to be detected are generally collected by different camera modules. Because of the significant differences in the manufacturing process, photoelectric response function, noise level and the like of different camera modules, the sensing model has larger deviation between the detection result and the real result in the image detection process. Therefore, when a new camera module is combined with an already trained perception model, how to efficiently improve the accuracy of the detection result of the perception model is a problem to be solved.
Disclosure of Invention
According to the image detection method, the image detection device and the electronic equipment, when the new shooting device is combined with the trained image detection model, the accuracy of image detection model reasoning can be improved.
In order to achieve the above purpose, the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides an image detection method, including: acquiring image data to be detected by a first camera device; processing the image data to be detected by using an image processing algorithm to generate a processed image; inputting the processed image into an image detection model to obtain a detection result; the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result.
According to the embodiment of the application, the parameters of the image processing algorithm for executing a plurality of image processing processes are adjusted, and the parameters are adjusted based on the image detection model, so that the style of the image obtained after the image processing of the image acquired by the first image pickup device is consistent with the style of the sample image data for training the image detection model, and therefore, the difference between the image data acquired by the image pickup device and the characteristic distribution of the sample image data for training the image detection model in a high-dimensional space is reduced, and the accuracy of reasoning of the image detection model is improved.
Based on the first aspect, in a possible implementation manner, the image detection model is obtained by training a neural network on second sample image data acquired by the second camera device.
Based on the first aspect, in a possible implementation manner, the parameters of the image processing algorithm are determined by: comparing the detection result with the labeling information of the first sample image data to obtain a comparison result; iteratively adjusting parameters of the image processing algorithm based on errors between the detection result and labeling information of the sample image data; and when the preset condition is met, saving parameters of the image processing algorithm.
The preset conditions herein may include, but are not limited to: the error is less than or equal to a preset threshold, or the number of iterations is greater than or equal to a preset threshold.
Based on the first aspect, in a possible implementation manner, the comparison result is an error, and iteratively adjusting parameters of the image processing algorithm based on the comparison result includes: constructing a target loss function based on the error between the comparison result and the labeling information of the first sample image data, wherein the target loss function comprises parameters to be adjusted in the image processing algorithm; and iteratively adjusting parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm based on the target loss function.
Based on the first aspect, in a possible implementation manner, the processing of the image data and the first sample image data includes at least one of the following: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, or image noise reduction.
Based on the first aspect, in a possible implementation manner, the image processing algorithm is executed by an image signal processor; and the parameters of the image processing algorithm include at least one of: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device; boundary coordinates of a neutral color region in the white balance correction algorithm in the image; target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm; a contrast threshold in a contrast enhancement algorithm; edge enhancement factors in the image edge enhancement algorithm; and spatial domain gaussian parameters and pixel value domain gaussian parameters in an image noise reduction algorithm.
Based on the first aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm further include: and generating a weight coefficient of the neural network of the image processing model.
Based on the first aspect, in a possible implementation manner, the labeling information of the first sample image data is manually labeled; the method further comprises: the first sample image data is converted into a color image suitable for manual labeling.
Based on the first aspect, in a possible implementation manner, the image detection model is used for performing at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
In a second aspect, an embodiment of the present application provides a parameter adjustment method for image processing, including: performing image processing on first sample image data by using an image processing algorithm to generate first image data, wherein the first sample image data is acquired by a first camera device; inputting the first image data into a pre-trained image detection model to obtain a detection result; comparing the error between the detection result and the labeling information of the first sample image data to obtain a comparison result; and adjusting parameters of the image processing algorithm based on the comparison result.
Based on the second aspect, in a possible implementation manner, the image detection model is obtained by training a neural network on second sample image data acquired by the second camera device.
Based on the second aspect, in a possible implementation manner, the comparison result is an error, and iteratively adjusting parameters of the image processing algorithm based on the comparison result includes: constructing a target loss function based on the error between the detection result and the labeling information of the first sample image data, wherein the target loss function comprises parameters to be adjusted in the image processing algorithm; and iteratively adjusting parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm based on the target loss function.
Based on the second aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, or image noise reduction.
Based on the second aspect, in a possible implementation manner, the parameters of the image processing algorithm include at least one of the following: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device; boundary coordinates of a neutral color region in the white balance correction algorithm in the image; target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm; a contrast threshold in a contrast enhancement algorithm; edge enhancement factors in the image edge enhancement algorithm; and spatial domain gaussian parameters and pixel value domain gaussian parameters in an image noise reduction algorithm.
Based on the second aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include: and generating a weight coefficient of the neural network of the image processing model.
Based on the second aspect, in a possible implementation manner, the labeling information of the first sample image data is manually labeled; the method further comprises: the first sample image data is converted into a color image suitable for manual labeling.
Based on the second aspect, in a possible implementation manner, the image detection model is used for performing at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
In a third aspect, an embodiment of the present application provides an image detection apparatus including: the acquisition module is configured to acquire image data to be detected through the first camera device; the processing module is configured to process the image data to be detected by utilizing an image processing algorithm to generate a processed image; the detection module is configured to input the processed image into an image detection model to obtain a detection result; the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result.
Based on the third aspect, in a possible implementation manner, the image detection model is obtained by training a neural network on second sample image data acquired by the second image capturing device.
Based on the third aspect, in a possible implementation manner, the parameters of the image processing algorithm are determined by a parameter adjustment module, and the parameter adjustment module includes: a comparing sub-module configured to compare the detection result of the first sample image data with the labeling information of the first sample image data to obtain the comparison result; an adjustment sub-module configured to iteratively adjust parameters of the image processing algorithm based on the comparison result; and the storage submodule is configured to store parameters of the image processing algorithm when preset conditions are met.
Based on the third aspect, in a possible implementation manner, the comparison result is an error, and the adjustment submodule is further configured to: constructing a target loss function based on the detection result of the first sample image data and the error between the labeling information of the first sample image data, wherein the target loss function comprises parameters to be updated in the image processing algorithm; and based on the target loss function, iteratively updating parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm.
Based on the third aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following image processing flows: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, and image noise reduction.
Based on the third aspect, in a possible implementation manner, the parameters of the image processing algorithm include at least one of the following: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device; boundary coordinates of a neutral color region in the white balance correction algorithm in the image; target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm; a contrast threshold in a contrast enhancement algorithm; edge enhancement factors in the image edge enhancement algorithm; and spatial domain gaussian parameters and pixel value domain gaussian parameters in an image noise reduction algorithm.
Based on the third aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include: and generating a weight coefficient of the neural network of the image processing model.
Based on the third aspect, in a possible implementation manner, the labeling information of the first sample image data is manually labeled; the method further comprises: the first sample image data is converted into a color image suitable for manual labeling.
Based on the third aspect, in a possible implementation manner, the image detection model is used for performing at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: the first camera device is used for collecting image data to be detected; the image signal processor is used for processing the image data to be detected by utilizing an image processing algorithm to generate a processed image; the artificial intelligent processor is used for inputting the processed image into an image detection model to obtain a detection result; the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result.
In a fifth aspect, embodiments of the present application provide an image detection apparatus comprising one or more processors and memory; the memory is coupled to the processor, the memory is used for storing one or more programs; the one or more processors are configured to run the one or more programs to implement the method as described in the first aspect.
In a sixth aspect, an embodiment of the present application provides a parameter adjustment device for image processing, the parameter adjustment device for image processing including one or more processors and a memory; the memory is coupled to the processor, the memory is used for storing one or more programs; the one or more processors are configured to run the one or more programs to implement the method as described in the second aspect.
In a seventh aspect, embodiments of the present application provide a computer readable storage medium having stored therein a computer program for implementing the method according to the first or second aspect when executed by at least one processor.
In an eighth aspect, embodiments of the present application provide a computer program product for implementing a method according to the first or second aspect when executed by at least one processor.
It should be understood that the second to eighth aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding possible embodiments are similar, and are not repeated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.
Fig. 1 is a schematic structural diagram of a terminal according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an ISP and AI processor combined to execute an image processing flow according to an embodiment of the present application;
FIG. 3 is a schematic view of a vehicle according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a system architecture including an electronic device for parameter debugging of an image processing algorithm, provided by an embodiment of the present application;
FIG. 5 is a flowchart of a method for debugging parameters of an image processing algorithm according to an embodiment of the present application;
FIG. 6 is a flowchart of a specific application of a method for debugging parameters of an image processing algorithm according to an embodiment of the present application;
FIG. 7 is a flowchart of an image detection method according to an embodiment of the present application;
fig. 8 is a schematic diagram of an image processing apparatus according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a parameter tuning device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the following description, reference is made to the accompanying drawings which form a part hereof and which show by way of illustration specific aspects in which embodiments of the application may be practiced. It is to be understood that embodiments of the application may be used in other aspects and may include structural or logical changes not depicted in the drawings. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present application is defined by the appended claims. For example, it should be understood that the disclosure in connection with the described methods may be equally applicable to a corresponding apparatus or system for performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding apparatus may comprise one or more units, such as functional units, to perform the one or more described method steps (e.g., one unit performing one or more steps, or multiple units each performing one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, if a specific apparatus is described based on one or more units such as a functional unit, for example, the corresponding method may include one step to perform the functionality of the one or more units (e.g., one step to perform the functionality of the one or more units, or multiple steps each to perform the functionality of one or more units, even if such one or more steps are not explicitly described or illustrated in the figures). Further, it is to be understood that features of the various exemplary embodiments and/or aspects described herein may be combined with each other, unless explicitly stated otherwise.
The image detection method can be applied to the field of computer vision, and a scene in which an image detection model obtained by training sample images acquired by other shooting equipment is combined with new shooting equipment.
Referring to fig. 1, fig. 1 shows a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 1, the electronic device 100 may be a User Equipment (UE), such as a mobile phone, a tablet computer, a smart screen, or various types of devices such as an image capturing device. Furthermore, the electronic device 100 may also be a vehicle. An imaging device 101 may be provided in the electronic apparatus 100 for capturing image data. The electronic device 100 may further comprise or be integrated in a module, a chip, a chipset, a circuit board or a component of the electronic device, which chip or chipset or the circuit board on which the chip or chipset is mounted may operate under the necessary software driving. The electronic device 100 includes one or more processors, such as an image signal processor (ISP, image Signal Processor) 102 and an AI processor 103. Alternatively, the one or more processors may be integrated within one or more chips, which may be considered a chipset, also referred to as a System on a Chip (SOC) when the one or more processors are integrated within the same Chip. In addition to the one or more processors, the electronic device 100 also includes one or more other necessary components, such as memory and the like.
The imaging device 101 shown in fig. 1 may be a monocular camera. Alternatively, the image pickup apparatus 101 may include a plurality of cameras, and the cameras may be physically integrated in one image pickup apparatus or may be physically separated from the plurality of image pickup apparatuses. And shooting a plurality of images at the same time through a plurality of cameras, and processing according to the images to obtain an image to be detected. Of course, the imaging device 101 may be other, and the embodiment of the present application is not limited specifically. In particular implementations, the camera 101 may acquire image data in real time or periodically. Such as 3s, 5s, 10s, etc. The image capturing apparatus 101 may also capture image data in other ways, and embodiments of the present application are not particularly limited. After the image data is acquired by the image pickup device 101, the image data may be transferred to the ISP102.
As with the ISP102 shown in fig. 1, a plurality of hardware modules or running necessary software programs may be provided to process image data and communicate with the AI processor 103. ISP102 may be provided as a component alone or integrated into other digital logic devices including, but not limited to: CPU (central processing unit ), GPU (graphics processor, graphics Processing Unit) or DSP (digital signal processor, digital Signal Processing). Illustratively, the CPU, GPU, and DSP are all processors within a system-on-chip. ISP102 may perform a number of image processing procedures that may include, but are not limited to: dark current correction, response nonlinearity correction, shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, edge enhancement, noise reduction, color correction, and the like.
It should be noted that ISP102 performs the above-described plurality of image processing procedures by executing an image processing algorithm. Each of the plurality of image processing procedures described above may be regarded as an independent image processing procedure, and thus, the image processing algorithm for executing each image processing procedure may be regarded as independent. Based on this, ISP102 may include a plurality of logic modules. Examples include, but are not limited to: dark current correction module, response non-linearity correction module, shading correction module, demosaicing module, etc. Each logic module is configured to perform an image detection process. Each logic module may use a specific hardware structure, and a plurality of logic modules may share a set of hardware structures, which is not limited in the embodiment of the present application. Further, the one or more image processing processes are typically performed sequentially. For example, after the image data acquired by the image pickup apparatus 101 is supplied to the ISP, processing procedures such as dark current correction, response nonlinearity correction, shading correction, demosaicing, white balance correction …, and the like may be sequentially performed. It should be noted that, the sequence of the image processing process executed by the ISP is not limited in the embodiment of the present application, for example, white balance correction may be executed first, and then demosaicing may be executed.
The AI processor 103 as shown in FIG. 1 may include a dedicated Neural processor such as a Neural Network Processor (NPU) including, but not limited to, a convolutional Neural network processor, a tensor processor, or a Neural processing engine. The AI processor may be provided as a component alone or integrated into other digital logic devices including, but not limited to: CPU, GPU or DSP. The AI processor 103 may run an image detection model that is trained on the deep neural network based on the sample image dataset S1. The image detection model may perform specific detection tasks. This particular detection task may include, but is not limited to: labeling of a detection frame, identification of a target object, prediction of confidence, prediction of a target object motion track, image segmentation and the like. It should be noted that the image detection model is deployed in the AI processor 103 shown in fig. 1 after the offline end training is completed. The off-line end here may be regarded as a server device or a device for model training.
In one possible implementation, in addition to the AI processor 103 running an image detection model to perform image detection operations, the AI processor 103 may also perform one or more image processing operations, which may include, but are not limited to: demosaicing, white balance correction, tone mapping, contrast enhancement, edge enhancement, noise reduction, color correction. At this time, the AI processor 103 may also have one or more image processing models running therein, each of which is used to perform a certain specific image processing procedure. In this possible implementation, the image data acquired from the image capturing apparatus 101 may undergo a plurality of image processing procedures to generate a final image processing result, and the AI processor 103 may execute one or more of the image processing procedures, that is, the ISP102 may execute one or more of the image processing procedures corresponding to the one or more image processing operations. Here, the AI processor 103 may perform different image processing procedures from the ISP102, and in addition, the AI processor 103 and the ISP102 may perform the same image processing procedures, for example, perform further enhancement processing, which is not limited in this embodiment. In this possible implementation, each image processing model may be obtained by training the neural network using a machine learning method based on the sample image dataset S3. The sample image dataset S3 includes: a plurality of sample image data H, reference image data I corresponding to each of the plurality of sample image data H. The reference image data I is used for comparing the image data, and the image data compared with the reference image data I is: and the neural network to be trained carries out image processing on the sample image data H to obtain image data. It should be noted that the reference image data I and the corresponding sample image data H are presented with the same scene. The presentation of the same scenario described herein can be understood as: the reference image data I and the corresponding sample image data H represent the same target object and the position of the target object in the image. Furthermore, the image processing procedure performed by the image processing model is different, and the sample image data set S3 for training the image processing model is also different. For example, when the Image processing model is used to perform an Image processing procedure for demosaicing, each sample Image data H included in the sample Image data set S3 is Image data of a single channel RAW Image Format (RAW) of a size of a×b×1, each reference Image data I corresponding to each sample Image data is RGB Image data of a size of a×b×3, where a is a longitudinal pixel value of an Image, B is a transverse pixel value of an Image, 1 is a single channel (e.g., R channel, G channel, or B channel), and 3 is a three channel (RGB channel). For another example, when the image processing model is used for performing the image processing flow of white balance correction, each sample image data H included in the sample image data set S3 is single-channel RAW image data of a size of a×b×1, each reference image data I corresponding to each sample image data is single-channel image data of a size of a×b×1, and the sample image data set S3. The reference image data I and its corresponding sample image data H have the same scene but different white balance values. The training process of the image processing model will be described below taking the image processing process of performing demosaicing by the image processing model as an example. Each sample image data H included in the sample image data set S3 is input to the neural network to be trained, respectively, to obtain processed image data. A loss function is constructed based on the error between the processed image data and the reference image data I. The loss function may include, for example, but is not limited to: mean Absolute Error (MAE) loss function or Mean Square Error (MSE) loss function. The loss function comprises weight coefficients of each layer of network in the neural network to be trained. And iteratively adjusting the weight coefficient value of each layer of network in the neural network to be trained by using a back propagation algorithm and a gradient descent algorithm based on the constructed loss function until the error between the processed image data output by the image processing model and the reference image data I is smaller than or equal to a preset threshold value or the iteration times are smaller than the preset threshold value, and storing the weight coefficient value of each layer of network in the neural network to be trained. At this time, the neural network to be trained is an image processing model.
When the ISP102 and the AI processor 103 are combined to perform the image processing, the ISP102 may be provided with a plurality of ports, and the ISP102 may also be provided with a plurality of ports, and the ISP102 may supply the processed image data a to the AI processor 103 through one of the plurality of ports, and the AI processor 103 generates the image data B after processing the image data a and supplies the image data B to the ISP102 through one of the plurality of ports. Taking the example of the AI processor 103 executing a demosaicing, a manner in which the ISP102 and the AI processor 103 are combined will be described with reference to fig. 2. In fig. 2, ISP102 acquires image data from image pickup device 101, performs three image processing flows of dark current correction, response non-linear correction, and shading correction on the acquired image data, and generates image data a, which is supplied to input port Vai of AI processor 103 through port Vio. The image processing model executed by the AI processor 103 performs demosaicing processing on the image data a to generate image data B, and supplies the image data B to the input port Vii of the ISP102 through the output port Vao. The ISP102 performs the subsequent image processing procedures such as white balance correction and color correction on the image data B input from the input port Vii, and generates image data C to be input to the AI processor 103. The image detection model executed by the AI processor 103 may perform image detection processing on the image data C.
Based on the image processing manner in which the ISP102 and the AI processor 103 are combined, the AI processor 103 may include one or more, and when the AI processor 103 includes one, an image processing model for performing image processing and an image detection model for performing image detection may be provided in the same AI processor 103, and when the AI processor 103 includes a plurality of, an image processing model for performing image processing and an image detection model for performing image detection may be provided in different AI processors 103.
The parameters in the image processing algorithm executed by the ISP102 and the parameters in the image processing model executed by the AI processor 103 in the embodiment of the present application are obtained by debugging based on the sample image dataset S2 acquired by the image capturing device 101 and the image detection result of the image detection model. The method for debugging each parameter in the image processing algorithm and the image processing model executed by the ISP102 is specifically referred to as an embodiment shown in fig. 5 hereinafter.
In general, the sample image data set S1 for training the image detection model run in the AI processor 103 is collected by large data, and an image pickup apparatus for collecting the sample image data set S1 is a different image pickup apparatus from the image pickup apparatus 101 shown in fig. 1. Because the characteristics of the manufacturing process, the photoelectric response function, the noise level and the like of different image capturing devices have significant differences, the style of the sample image data D in the sample image dataset S1 is different from the style of the image obtained by performing image processing on the image data collected by the image capturing device 101, so that the feature distribution of the image data collected by the image capturing device 101 and the feature distribution of the sample image data D in the sample image dataset S1 in a high-dimensional space have significant differences, and when the image detection model is deployed in the AI processor 103, the deviation between the detection result and the real result is large in the detection process of the image data collected by the image capturing device 101, and the detection accuracy of the image detection model deployed in the AI processor 103 is reduced.
Based on this, the embodiment of the present application adjusts the parameters of the image processing algorithm for executing the plurality of image processing procedures (or adjusts the parameters of the image processing algorithm for executing the plurality of image processing procedures and the parameters of the image processing model) by keeping the parameters of the image detection model unchanged, so that the style of the image obtained after the image processing of the image acquired by the image capturing device 101 is consistent with the style of the sample image data D in the sample image dataset S1, thereby reducing the difference between the image data acquired by the image capturing device 101 and the feature distribution of the sample image data D in the sample image dataset S1 in the high-dimensional space, and being beneficial to improving the accuracy of the reasoning of the image detection model.
In the conventional computer vision technology, in order to adapt an already trained image detection model to a new image capturing device so as to improve the accuracy of the detection result of the image detection model, a new data set is generally required to be used for retraining or fine-tuning the already trained image detection model. However, the process of collecting and labeling new data sets consumes a lot of manpower and material resources, and the process of retraining the image detection model necessarily requires that the image detection model "forgets" some learned knowledge, so that the historical data cannot be utilized maximally. According to the embodiment of the application, the parameters of the trained image detection model are kept unchanged, and the difference between the characteristic distribution of the image data acquired by the image pickup device 101 and the characteristic distribution of the sample image data D in a high-dimensional space is reduced by adjusting the parameters of the image processing algorithm for executing a plurality of image processing processes (or adjusting the parameters of the image processing algorithm for executing a plurality of image processing processes and the parameters of the image processing model), so that compared with the mode of retraining the image detection model in the prior art, the embodiment of the application can save time and calculation force cost required for retraining and fine-tuning the image detection model without any modification of the trained image detection model; in addition, because the embodiment of the application adjusts the parameters of the image processing algorithm for executing a plurality of image processing processes, the image processing algorithm does not execute the image detection process, so that the training can be completed by adopting fewer training samples, the number of manual labeling on the training samples can be reduced, and the debugging period when the image detection model is combined with a new camera device is shortened.
The following describes an application scenario in the embodiment of the present application in more detail with reference to a schematic structural diagram of the electronic device 100 shown in fig. 1 by taking an autopilot scenario as an example. Referring to fig. 3, fig. 3 is a schematic diagram illustrating a vehicle 300 according to an embodiment of the application.
Components coupled to the vehicle 300 or included in the vehicle 300 may include a control system 10, a propulsion system 20, and a sensor system 30. It should be appreciated that the vehicle 300 may also include additional systems, which are not described in detail herein. The control system 10 may be configured to control the operation of the vehicle 300 and its components. The ISP102 and the AI processor 103 shown in fig. 1 may be provided in the control system 10, and in addition, the control system 10 may further include a central processing unit, a memory, etc., for storing instructions and data required for the respective processors to operate. Propulsion system 20 may be used to provide powered movement of vehicle 300, which may include, but is not limited to: an engine/generator, an energy source, a transmission, and wheels. The sensor system 104 may include, but is not limited to: a global positioning system, an inertial measurement unit, a lidar sensor, or a millimeter wave radar sensor, such as the camera 101 shown in fig. 1, may be provided to the sensor system 30. The components and systems of vehicle 300 may be coupled together by a system bus, network, and/or other connection mechanism to operate in an interconnected manner with other components internal and/or external to their respective systems. In particular operation, the components of the vehicle 300 cooperate with one another to implement a variety of autopilot functions. The autopilot functionality may include, but is not limited to: blind spot detection, parking assistance or lane change assistance, etc.
In the process of realizing the above-described autopilot function, the image pickup device 101 may periodically collect image data and supply the collected image data to the ISP102. The ISP102 (or the image processing models in the ISP102 and the AI processor 103) processes the image data by performing a plurality of image processing procedures, converts the processed image data into image data that can be recognized or calculated by the image detection model running in the AI processor 103, and supplies the image data to the AI processor 103, thereby causing the AI processor 103 to perform reasoning or detection of a specific task, and generates a detection result. Other components (e.g., a CPU that performs decisions) in the control system 10 control other devices or components to perform the autopilot function based on the detection results of the AI processor 103.
In general, some parts may not be produced by a manufacturer of a production vehicle, and a trained image detection model is ordered by a manufacturer a, and an image capturing device is ordered by a manufacturer B, so that in order to enable the image detection model to more accurately detect image data acquired by the image capturing device, the training method described in the embodiment of the present application may be used to debug an image processing algorithm or parameters of the image processing model for executing an image processing procedure. For another example, manufacturers are upgrading certain models of vehicles and are required to replace imaging devices of different models from those of the previous configuration. At this time, in order to match the replaced image capturing device with the image detection model, so that the detection result of the image detection model for detecting the image data acquired by the new image capturing device is more accurate, the training method according to the embodiment of the present application may also be used to debug the parameters of the image processing algorithm or the image processing model for executing the image processing flow.
It should be noted that, the parameter tuning of the image processing algorithm or the image processing model may be performed on the offline side (or may be performed by training in a server or a device for performing model training). After the parameter debugging of the image processing algorithm is finished, the image processing algorithm can be deployed in the ISP of the terminal. When some image processing procedures are performed by the image processing model, the image processing model may be deployed in the AI processor 103 after the parameter debugging in the image processing model is completed. Referring to fig. 4, a system architecture diagram 400 including an electronic device for parameter debugging of an image processing algorithm according to an embodiment of the present application is shown.
In fig. 4, a system architecture 400 includes the image pickup apparatus 101, a parameter debugging device 401, a storage device 402, and a display device 403. The image pickup apparatus 101 is configured to acquire a plurality of sample image data E, and store the acquired sample image data E to the storage device 402. The imaging device 101 is the same (or the same) imaging device as the imaging device 101 shown in fig. 1. The storage device 402 may include, but is not limited to: read-only memory or random access memory, etc. Which is used to store sample image data E. Further, the storage device 402 may also store therein executable programs and data of an image processing algorithm for performing an image processing process, and executable programs and data of an image detection model for performing image detection. The parameter debugging device 401 may run the image processing algorithm and the image detection model, and the parameter debugging device 401 may also call the sample image data E, the executable program and data for executing the image processing algorithm like the processing procedure, and the executable program and data for executing the image detection model of the image detection from the storage device 101 to debug the parameters of the image processing algorithm. In addition, the parameter debugging device 401 may also store the data generated by the operation and the debugging result after each parameter debugging of the image processing algorithm to the storage device 402. In addition, the parameter debugging device 401 and the storage device 402 may also be provided with I/O ports for data interaction with the display device 403. The display device 403 may include a display means such as a screen to annotate the sample image data E. Specifically, the parameter tuning device 401 may acquire the sample image data E from the storage device 402, perform image processing on the sample image data E, and provide the sample image data E to the display device 403 for presentation in the display device 403. The user annotates the sample image data E via the display device 403 and stores the annotation information of the sample image data E to the storage device 402.
In the embodiment of the present application, in order to facilitate the processing of the sample image data E by the image processing algorithm and the detection of the image by the image detection model, the sample image data E output by the image capturing device 101 is high-bit-depth (for example, 16bit, 20bit or 24 bit) single-channel linear RAW image data, the dynamic range of which is far greater than the dynamic range that can be displayed by the display, and in addition, the sample image data E is a color filter array (CFA, color filter array) image, which does not have color information, so that it is difficult for the labeling personnel to identify each target object from the sample image data E output by the image capturing device 101. Based on this, in the embodiment of the present application, the parameter tuning device 401 further runs an image processing algorithm T, which is used to process the sample image data E to generate a color image, such as an RGB image, that can be presented in the display and has suitable brightness and color, so as to facilitate the labeling of the target object presented in the sample image data E by the labeling personnel. The image processing procedure performed by the image processing algorithm T may include, but is not limited to: systematic error correction, global tone mapping, demosaicing, or white balance correction. The parameters in the image processing algorithm T do not need to be adjusted, and the image processing algorithm T can be realized by adopting a traditional image processing algorithm. It should be noted that, in the embodiment of the present application, the image processing algorithm T is configured to process the sample image data E to generate image data that can be displayed in a display for labeling by a labeling person; the image processing algorithm in the embodiment of the present application is used for processing the sample image data E to generate image data for image detection by the image detection model, and parameters thereof need to be adjusted.
Based on the schematic structural diagram of the electronic device 100 shown in fig. 1, the application scenario of the vehicle described in fig. 3, and the system architecture 400 shown in fig. 4, a parameter debugging method for image processing will be described in detail below with reference to fig. 5 and 6.
Referring to fig. 5, a flow 500 of a parameter debugging method of an image processing algorithm according to an embodiment of the application is shown. It should be noted that, the execution subject of the parameter debugging method for image processing described in the embodiment of the present application may be the parameter debugging device 401 shown in fig. 4. As shown in fig. 5, the parameter debugging method for image processing includes the steps of:
step 501, based on the sample image data set S2, each sample image data E in the sample image data set S2 is processed by an image processing algorithm to generate a plurality of image data F.
The sample image data set S2 includes a plurality of sample image data E and labeling information of each sample image data E. Wherein each sample image data E in the sample image dataset S2 is acquired by the image capturing device 101 as shown in fig. 1. The labeling information of the sample image data E is labeled based on the detection contents performed by the image detection model. For example, when the image detection model is used to perform target detection, the annotation information of the sample image data E may include the target object and the position of the target object in the second sample image; when the image detection model is used to perform pedestrian intent detection, the annotation information of the sample image data E may include the target object and motion information of the target object.
The image processing algorithms herein are used to perform one or more image processing procedures. The one or more image processing procedures include, but are not limited to: dark current correction, response nonlinearity correction, lens shading correction, demosaicing, white balance correction, tone mapping, noise reduction, contrast enhancement or edge enhancement, and the like. It should be noted that the one or more image processing procedures are typically performed sequentially. The execution order of the image processing process in the embodiment of the present application is not particularly limited.
Step 502, detecting the image data F by using the image detection model, and generating a detection result.
Here, the image detection model may perform at least one of the following detection: target detection, lane line detection, pedestrian intention detection, or the like. The image detection model is trained on the deep neural network based on the image dataset S1. The image data D in the image data set S1 is acquired by an image pickup device other than the image pickup device 101. In addition, the training method of the image detection model is a conventional technology, and is not described herein.
In step 503, parameters of the image processing algorithm are adjusted based on the detection result and the labeling information of the sample image data E.
In one possible implementation, the parameters of the image processing algorithm may be adjusted using a machine learning approach. A second possible implementation is described in detail below.
A loss function is constructed based on an error between the detection result of each sample image data E in the sample image data set S2 and the labeling information of the sample image data E. The loss function may include, but is not limited to: cross entropy functions, etc. The back propagation algorithm and the gradient descent algorithm are then used to adjust parameters of an image processing module in the image processing algorithm that is used to perform one or more image processing flows. The gradient descent algorithm may specifically include, but is not limited to: SGD, adam, etc. When back propagation is performed based on the preset loss function, a gradient of the preset loss function with respect to each parameter in the image processing algorithm can be calculated by using a chain rule.
Specifically, the image processing algorithms for executing the respective image processing flows are independent of each other. When the image processing flow executed by the image processing algorithm is the flow of dark current correction, response nonlinear correction, lens shading correction, demosaicing, white balance correction, noise reduction, contrast enhancement or edge enhancement in sequence, and the like, the parameter of the image processing algorithm for each flow is adjusted by adopting the back propagation algorithm, the parameter which is propagated and adjusted first is the parameter of the image processing algorithm for executing the edge enhancement, and then the parameter of the image processing algorithm which is adjusted in sequence is the parameter of the image processing algorithm for executing the contrast enhancement, the parameter of the image processing algorithm for executing the noise reduction, the parameter of the image processing algorithm for executing the white balance correction, and the like. It will be appreciated that embodiments of the present application may include more or fewer image processing flows, and accordingly may include more or fewer parameters to be adjusted. In addition, since the order of the image processing flow executed by the image processing algorithm is not limited in the embodiment of the present application, when the back propagation algorithm is used to adjust each parameter of the image processing algorithm, the first-adjusted parameter and the last-adjusted parameter are not particularly limited in the embodiment of the present application.
It should be noted that, when the parameters of the image processing algorithm are adjusted by adopting the back propagation algorithm based on the preset loss function, each parameter in the image detection model is kept unchanged.
Step 504, determining whether the loss value of the preset loss function is less than or equal to a preset threshold. If the loss value of the preset loss function is smaller than or equal to a preset threshold value, saving parameters of an image processing algorithm; if the loss value of the predetermined loss function is greater than the predetermined threshold, step 505 is performed.
Step 505, determining whether the number of iterative adjustment of parameters of the image processing algorithm is greater than or equal to a preset threshold. If the number of times of iteratively adjusting the parameters of the image processing algorithm is greater than or equal to the preset threshold, the parameters of the image processing algorithm are saved, and if the number of times of iteratively adjusting the parameters of the image processing algorithm is less than the preset threshold, steps 501-505 are continued.
In one possible implementation of the present application, one or more of the image processing flows described above may all be implemented by conventional image processing algorithms. When the one or more image processing flows described above are implemented by a conventional image processing algorithm, it is assumed that the one or more image processing flows include dark current correction, response nonlinearity correction, lens shading correction, demosaicing, white balance correction, tone mapping, noise reduction, contrast enhancement, and edge enhancement. At this time, the parameters of the image processing algorithm to be adjusted may include, but are not limited to: parameters of an image processing algorithm for performing lens shading correction, parameters of an image processing algorithm for performing white balance correction, parameters of an image processing algorithm for performing tone mapping, parameters of an image processing algorithm for performing contrast enhancement, parameters of an image processing algorithm for performing edge enhancement, and parameters of an image processing algorithm for performing noise reduction. It should be noted that, when the parameters of the image processing algorithm are adjusted by using the machine learning method, the image detection algorithm in the embodiment of the application has differentiability so as to be guided based on the chain rule back propagation. The parameters of each image processing algorithm to be adjusted are described in detail below.
The lens shading correction is used for correcting illumination attenuation of the image edge area due to the increase of the incidence angle of the main light rays, and a polynomial is used for fitting an illumination attenuation curved surface, wherein the independent variable of the polynomial is the distance between each pixel of the image and the optical center of the image pickup device. Therefore, in the image processing algorithm for performing lens shading correction, the parameter to be adjusted is the distance between each pixel of the image and the optical center of the image pickup device, that is, the value of the argument in the polynomial.
The execution flow of the white balance correction is as follows: the neutral color pixel searching algorithm is used for screening neutral color areas in the image, and boundary coordinates of the neutral color areas in the image are determined based on screening results. The pixel values in the screened neutral areas are then weighted using the luminance channels of the image, generating a binarized neutral color pixel mask (neutral pixel mask). Next, each (near) neutral color pixel is weighted averaged using the neutral color pixel mask to obtain an estimate of the light source color in the image. And finally, calculating the ratio between RGB channels of the light source color to obtain a white balance correction coefficient corresponding to the image, and applying the white balance correction coefficient to the original image to obtain the image after white balance correction. Therefore, in the image processing algorithm for performing white balance correction, the parameter to be adjusted is the boundary coordinates of the neutral color region in the image.
The tone mapping is used for receiving a linear image with high bit depth, converting the linear image into a nonlinear image, simultaneously completing the compression of the image bit depth and outputting an 8-bit image. When the global gamma function is used as the tone mapping function, the trainable parameter is a gamma parameter; when a logarithmic transformation algorithm is used for carrying out dynamic range compression on the linear image, the trainable parameter is the base number of logarithmic transformation; when using a more complex tone mapping model, such as the retinex model based on the human eye dynamic range response, the trainable parameters are the target luminance parameter (key), the target saturation parameter (saturation), and the filter kernel parameters used to generate the low pass filtered image.
Contrast enhancement is used to enhance the contrast of an image. Specifically, the contrast of the image may be locally adjusted using a CLAHE (contrast limited adaptive histogram equalization ) algorithm. The CLAHE algorithm contains two adjustable parameters: contrast threshold parameters and sub-image block sizes for histogram statistics. In one possible implementation manner of the embodiment of the present application, the size of the sub-image block may be fixed, and only the contrast threshold parameter may be adjusted. Still further, the size of the sub-image block may be fixed to the size of the input image.
In the edge enhancement of the image, firstly, gaussian filtering is carried out on a Y-channel image in the received image to obtain a low-pass Y-channel image Y L The method comprises the steps of carrying out a first treatment on the surface of the The original Y-channel image and the low-pass Y-channel image are combined L The difference image between them being the high-frequency signal in the image, i.e. Y HF ＝Y-Y L The high frequency signal generally corresponds to an edge region in the image; by amplifying the intensity of the high frequency signal and superimposing it on the low-pass Y-channel image Y L In (3), an edge-enhanced image Y can be obtained E I.e. Y E ＝Y L +α·(Y-Y L ) Where α is an edge enhancement factor, the degree of image edge enhancement increases as α increases. Among the parameters for performing the image processing algorithm for edge enhancement, the parameter to be adjusted is the edge enhancement factor α.
In image denoising, a bilateral filter (noise reduction algorithm) is generally adopted. In the bilateral filtering noise reduction algorithm, the trainable parameters may include: a spatial domain gaussian kernel parameter σs for controlling the relationship between the noise reduction intensity and the spatial distance, and a pixel value domain gaussian kernel parameter σr for controlling the relationship between the noise reduction intensity and the response value difference.
In another possible implementation manner, part of the one or more image processing flows may be implemented by the image processing model. For example, the above-described image processing procedures of demosaicing, white balance correction, tone mapping, noise reduction, contrast enhancement, and edge enhancement are implemented by an image processing model. The image processing model may include a plurality of types, each of which is used to perform a particular image processing operation. For example, the noise-canceled image processing model is used to perform the noise-canceled image processing operation, and the demosaiced image processing model is used to perform the demosaiced image processing operation. Each image processing model can be obtained by training a multi-layer neural network by using a training sample by adopting a traditional neural network training method, and the training method is not repeated in the embodiment of the application. The operation of each layer in each image processing model can be described by the mathematical expression y=a (W x ten b). Where W is the weight, x is the input vector (i.e., input neuron), b is the bias data, y is the output vector (i.e., output neuron), and a is a constant. The work of each layer in the physical layer deep neural network can be understood as completing the transformation of the input space into the output space (i.e., the row space to the column space of the matrix) by five operations on the input space (set of input vectors), including 1, dimension up/down; 2. zoom in/out; 3. rotating; 4. translating; 5. "bending". Wherein operations 1, 2, 3 are completed by W x, operation 4 is completed by +b, and operation 5 is completed by a (). The term "space" is used herein to describe two words because the image being processed is not a single thing, but rather a class of things, space referring to the collection of all individuals of such things. Where W is a weight vector, each value in the vector representing a weight value of a neuron in the layer neural network. The vector W determines the spatial transformation of the input space into the output space described above, i.e. the weights W of each layer control how the space is transformed. When image data is processed using one or more image processing models, the image processing algorithm is adjusted to be a weight matrix of all layers forming the image processing model (a weight matrix formed by vectors W of many layers). It should be noted that the one or more image processing models are pre-trained at the offline side. When the parameter debugging method is adopted to adjust the parameters of the image processing algorithm, the parameters of the neural network used for forming the image processing model are only required to be finely adjusted, so that the style characteristics of the image obtained after the image processing model processes the image are similar to the style characteristics of sample image data used for training the image detection model. Thereby improving the detection accuracy of the image detection model.
Based on the parameter debugging method of the image processing algorithm shown in fig. 5, the parameter debugging method of the image processing algorithm according to the embodiment of the present application will be described in more detail below by taking the detection function executed by the image detection model as the target object identification as an example. Referring to fig. 6, fig. 6 is a schematic diagram showing a specific application of the parameter debugging method for image processing according to the embodiment of the application. The parameter debugging method for image processing described in fig. 6 may be the parameter debugging device 401 shown in fig. 4.
In step 601, a plurality of sample image data E are acquired by the image pickup device 101. The plurality of sample image data E is high bit depth (e.g., 16bit, 20bit, or 24 bit) single channel linear RAW image data.
In step 602, the sample image data E is processed using an image processing algorithm T, and processed image data F is generated for presentation in a display screen. The image data F is a color image such as an RGB image.
And 603, manually labeling the image data F to obtain labeling information of each sample of image data E. Since the detection performed by the image detection model is target detection, the annotation information of the sample image data includes the category of the target object presented by the sample image data and the position in the sample image data.
In step 604, the sample image data is processed using an image processing algorithm to generate image data D.
Step 605, inputting the image data D into the image detection model to obtain an image detection result, where the image detection result includes a location area of the preset target object in the sample image data and a probability value of the preset target object.
Step 606, constructing a loss function based on the image detection result and the labeling information of the sample image data E.
The image detection model is assumed to be represented by N (), and thereforeWherein the method comprises the steps ofRepresenting the loss function of the image detection model. Y is Y out Representing image data, i.e. images, input by an image detection modelThe processing algorithm processes the image data that is ultimately output by executing a plurality of image processing procedures.
In step 607, it is determined whether the loss value of the loss function reaches a preset threshold. If the preset threshold is not reached, step 508 is performed, and if the preset threshold is reached, the parameters of the image processing algorithm are saved.
In step 608, the parameters of the image detection algorithm are adjusted using a back propagation algorithm and a gradient descent algorithm.
In the following, more detailed description will be given by taking as an example the adjustment of parameters of an image processing algorithm for executing an edge enhancement procedure. The edge enhancement of the image is to amplify and superimpose the high-frequency signal in the image into the low-pass image to realize the edge sharpening of the image, namely: ye=yl+α· (Y-YL). The meaning of each parameter in the formula is detailed in the related description in the edge enhancement of the image, and is not repeated here. Among the parameters of the image processing algorithm for performing edge enhancement, the parameter to be adjusted is an edge enhancement factor α. Let P (·) denote all image operations after image edge enhancement, i.e., Y out ＝P(Y E ). The image obtained after all the image operations before edge enhancement is Y, and under the condition that all parameters in P (-) are kept unchanged, the gradient of an objective function L relative to an edge enhancement factor alpha can be obtained according to a chain rule:
according to SGD algorithm, after one iteration, alpha is updated to a value alpha (t+1) The method comprises the following steps:
wherein alpha is (t+1) At the current value alpha of alpha (t) Values after stepping a distance in the opposite direction of its gradient, delta represents the learning rate for controlStep sizes of each iteration are made. Substituting the formula (2) into the formula (1) to obtain the updated value of the edge enhancement factor alpha:
assuming that the current value of the edge enhancement factor alpha is alpha (0) The value of the edge enhancement factor α after one iteration is:
similarly, the adjustment process of the parameters of the image detection algorithm for performing other image processing flows may be similar to the adjustment process of the parameters of the image detection algorithm for performing edge enhancement, and will not be described here again.
As shown in fig. 6, the parameter debugging method is repeated from step 604 to step 608, that is, the parameter in the image processing algorithm is iteratively adjusted multiple times, so as to obtain the optimal parameter value for enabling the loss function L to reach the minimum value in the image processing algorithm.
Based on the electronic device 100 shown in fig. 1 and the parameter debugging methods for image processing shown in fig. 5 and 6, the embodiment of the application further provides an image detection method. Referring to fig. 7, a flow 700 of an image detection method according to an embodiment of the application is shown. The image detection method shown in fig. 7 may be executed by the ISP processor and the AI processor shown in fig. 1. As shown in fig. 7, the image detection method includes the steps of:
in step 701, image data to be detected is acquired by the image pickup device 101.
Step 702, processing image data to be detected by using an image processing algorithm, and generating a processed image.
In step 703, the processed image is input to an image detection model, and a detection result is obtained.
The parameters of the image processing algorithm in step 702 may be obtained after being debugged by using the parameter debugging method for image processing as described in fig. 5.
In one possible implementation, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
It will be appreciated that the electronic device, in order to achieve the above-described functions, comprises corresponding hardware and/or software modules for performing the respective functions. The present application can be implemented in hardware or a combination of hardware and computer software, in conjunction with the example algorithm steps described in connection with the embodiments disclosed herein. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The present embodiment may divide the above one or more processors into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules described above may be implemented in hardware. It should be noted that, in this embodiment, the division of the modules is schematic, only one logic function is divided, and another division manner may be implemented in actual implementation.
In the case of dividing the respective functional modules with the respective functions, fig. 8 shows a schematic diagram of one possible composition of the image detection apparatus 800 involved in the above-described embodiment, as shown in fig. 8, the image detection apparatus 800 may include: acquisition module 801, processing module 802 and detection module 803. An acquisition module 801 configured to acquire image data to be detected by a first image pickup device; a processing module 802 configured to process the image data to be detected using an image processing algorithm, and generate a processed image; a detection module 803 configured to input the processed image to an image detection model to obtain a detection result; the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result.
In one possible implementation, the image detection model is obtained by training a neural network on second sample image data acquired by the second image capturing device.
In a possible implementation, the parameters of the image processing algorithm are determined by a parameter adjustment module (not shown in the figure) comprising: a comparing sub-module (not shown in the figure) configured to compare the detection result of the first sample image data with the labeling information of the first sample image data to obtain the comparison result; an adjustment sub-module (not shown in the figure) configured to iteratively adjust parameters of the image processing algorithm based on the comparison result; a saving sub-module (not shown in the figure) configured to save the parameters of the image processing algorithm when a preset condition is satisfied.
In one possible implementation, the comparison result is an error, and the adjustment sub-module (not shown in the figure) is further configured to: constructing a target loss function based on the detection result of the first sample image data and the error between the labeling information of the first sample image data, wherein the target loss function comprises parameters to be updated in the image processing algorithm; and based on the target loss function, iteratively updating parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm.
In one possible implementation, the image processing algorithm includes at least one of: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, and image noise reduction.
In one possible implementation, the parameters of the image processing algorithm include at least one of: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device; boundary coordinates of a neutral color region in the white balance correction algorithm in the image; target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm; a contrast threshold in a contrast enhancement algorithm; edge enhancement factors in the image edge enhancement algorithm; and spatial domain gaussian parameters and pixel value domain gaussian parameters in an image noise reduction algorithm.
In one possible implementation, the image processing algorithm is performed by a trained image processing model; the parameters of the image processing algorithm include: and generating a weight coefficient of the neural network of the image processing model.
In one possible implementation, the labeling information of the first sample image data is manually labeled; the apparatus further comprises: a conversion module (not shown in the figure) is configured to convert the first sample image data into a color image suitable for manual labeling.
In one possible implementation, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
The image detection apparatus 800 provided in this embodiment is used to perform the image detection method performed by the electronic device 100, and can achieve the same effects as the implementation method described above. The respective modules corresponding to fig. 8 above may be implemented in software, hardware, or a combination of both, and for example, each module may be implemented in software to drive the ISP102 and the AI processor 103 in the electronic device 100 shown in fig. 1. Alternatively, each module may comprise two parts, a corresponding processor and corresponding driver software.
In the case of dividing the respective functional modules with the respective functions, fig. 9 shows one possible composition diagram of the parameter debugging apparatus 900 for image processing involved in the above-described embodiment, as shown in fig. 9, the parameter debugging apparatus 900 for image processing may include: a processing module 901, a detection module 902, a comparison module 903 and an adjustment module 904. A processing module 901 configured to perform image processing on first sample image data by using an image processing algorithm, to generate first image data, where the first sample image data is acquired by a first image capturing device; a detection module 902 configured to input the first image data into a pre-trained image detection model to obtain a detection result; a comparing module 903 configured to compare an error between the detection result and the labeling information of the first sample image data, to obtain a comparison result; an adjustment module 904 configured to adjust parameters of the image processing algorithm based on the comparison result.
In one possible implementation, the image detection model is obtained by training a neural network on second sample image data acquired by the second image capturing device.
In one possible implementation, the comparison result is an error, and the adjustment module is configured to: constructing a target loss function based on the error between the detection result and the labeling information of the first sample image data, wherein the target loss function comprises parameters to be adjusted in the image processing algorithm; and iteratively adjusting parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm based on the target loss function.
In one possible implementation, the image processing algorithm includes at least one of: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, or image noise reduction.
In one possible implementation, the parameters of the image processing algorithm include at least one of: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device; boundary coordinates of a neutral color region in the white balance correction algorithm in the image; target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm; a contrast threshold in a contrast enhancement algorithm; edge enhancement factors in the image edge enhancement algorithm; and spatial domain gaussian parameters and pixel value domain gaussian parameters in an image noise reduction algorithm.
In one possible implementation, the image processing algorithm is performed by a trained image processing model; the parameters of the image processing algorithm include: and generating a weight coefficient of the neural network of the image processing model.
In one possible implementation, the labeling information of the first sample image data is manually labeled; the apparatus further comprises: a conversion module (not shown in the figure) is configured to convert the first sample image data into a color image suitable for manual labeling.
In one possible implementation, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
In the case of an integrated unit, the image detection device 800 may include at least one processor and a memory. The at least one processor may invoke all or a portion of the computer program stored in the memory to control and manage the actions of the electronic device 100, for example, to support the electronic device 100 to execute the steps executed by the respective modules. The memory may be used to support the electronic device 100 for storing program codes and data, etc. The processor may implement or execute the various exemplary logic modules described in connection with the present disclosure, which may be one or more microprocessor combinations that implement computing functions, including, for example, but not limited to, the image signal processor 101 and the AI processor 103 shown in fig. 1. In addition, the microprocessor combination may also include a central processing unit, a controller, and the like. In addition, the processor may include other programmable logic devices, transistor logic devices, discrete hardware components, or the like in addition to the processors shown in fig. 1. The memory may include Random Access Memory (RAM), read Only Memory (ROM), and the like. The random access memory may include volatile memory (e.g., SRAM, DRAM, DDR (double data rate SDRAM, double Data Rate SDRAM) or SDRAM, etc.) and nonvolatile memory. The RAM may store therein data (such as an image processing algorithm or the like) and parameters necessary for the operation of the ISP102 and the AI processor 103, intermediate data generated by the operation of the ISP102 and the AI processor 103, image data processed by the ISP102, an output result after the operation of the AI processor 103, and the like. The ROM may store executable programs of the ISP102 and the AI processor 103. The above-described components may perform their respective tasks by loading an executable program. The memory-stored executable program may perform the image detection method as described in fig. 7.
In the case of an integrated unit, the image detection apparatus 900 may include at least one processor and a storage device. Wherein at least one processor may invoke all or part of the computer program stored in the memory to control and manage the actions of the parameter debugging device 401 as shown in fig. 4, for example, may be used to support the parameter debugging device 401 to perform the steps performed by the above-mentioned respective modules. The memory may be used to support the execution of stored program code and data by parameter debugging device 401, and the like. The processor may implement or execute the various exemplary logic modules described in connection with the present disclosure, which may be a combination of one or more microprocessors that perform the computing functions, including, for example, but not limited to, central processing units and controllers, and the like. In addition, the processor may also include other programmable logic devices, transistor logic devices, discrete hardware components, or the like. The memory may include Random Access Memory (RAM), read only memory ROM, and the like. The random access memory may include volatile memory (e.g., SRAM, DRAM, DDR (double data rate SDRAM, double Data Rate SDRAM) or SDRAM, etc.) and nonvolatile memory. The RAM may store therein data (such as an image processing algorithm or the like) and parameters required for the operation of the parameter debugging device 401, intermediate data generated by the operation of the parameter debugging device 401, an output result after the operation of the parameter debugging device 401, and the like. The read only memory ROM may have stored therein an executable program of the parameter debugging device 401. The above-described components may perform their respective tasks by loading an executable program. The memory-stored executable program may perform the parameter adjustment method as described in fig. 5 or fig. 6.
The present embodiment also provides a computer-readable storage medium having stored therein computer instructions that, when executed on a computer, cause the computer to perform the above-described related method steps to implement the image detection method of the image detection apparatus 800 in the above-described embodiment, or implement the parameter adjustment method of the parameter adjustment apparatus 900 in the above-described embodiment.
The present embodiment also provides a computer program product, which when run on a computer, causes the computer to perform the above-described related steps to implement the image detection method of the image detection apparatus 800 in the above-described embodiment or implement the parameter adjustment method of the parameter adjustment apparatus 900 in the above-described embodiment.
The computer readable storage medium or the computer program product provided in this embodiment is used to execute the corresponding method provided above, and therefore, the advantages achieved by the method can refer to the advantages in the corresponding method provided above, and will not be described herein.
It will be appreciated by those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned readable storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims (22)

  1. An image detection method, comprising:
    acquiring image data to be detected by a first camera device;
    processing the image data to be detected by using an image processing algorithm to generate a processed image, wherein parameters of the image processing algorithm are obtained by comparing labeling information of first sample image data acquired by the first camera device with a detection result of the first sample image data by using an image detection model and adjusting the detection result based on the comparison result;
    And inputting the processed image into the image detection model to obtain a detection result.
  2. The image detection method according to claim 1, wherein the image detection model is obtained by performing neural network training on second sample image data acquired by the second image pickup device.
  3. The image detection method according to claim 1 or 2, characterized in that the parameters of the image processing algorithm are determined by:
    comparing the detection result of the first sample image data with the labeling information of the first sample image data to obtain a comparison result;
    iteratively adjusting parameters of the image processing algorithm based on the comparison result;
    and when the preset condition is met, saving parameters of the image processing algorithm.
  4. The image detection method according to claim 3, wherein the comparison result is an error, and the iteratively adjusting parameters of the image processing algorithm based on the comparison result includes:
    constructing a target loss function based on the detection result of the first sample image data and the error between the labeling information of the first sample image data, wherein the target loss function comprises parameters to be adjusted in the image processing algorithm;
    And iteratively adjusting parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm based on the target loss function.
  5. The image detection method according to any one of claims 1-4, wherein the image processing algorithm comprises at least one of: dark current correction algorithm, lens shading correction algorithm, demosaicing algorithm, white balance correction algorithm, tone mapping algorithm, contrast enhancement algorithm, image edge enhancement algorithm, and image noise reduction algorithm.
  6. The image detection method according to claim 5, wherein the parameters of the image processing algorithm include at least one of:
    the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device;
    boundary coordinates of a neutral color region in the white balance correction algorithm in the image;
    target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm;
    a contrast threshold in a contrast enhancement algorithm;
    edge enhancement factors in the image edge enhancement algorithm; and
    spatial domain gaussian parameters and pixel value domain gaussian parameters in the image noise reduction algorithm.
  7. The image detection method according to claim 5, wherein the image processing algorithm is performed by a trained image processing model; the parameters of the image processing algorithm include:
    And generating a weight coefficient of the neural network of the image processing model.
  8. The image detection method according to any one of claims 1 to 7, wherein the labeling information of the first sample image data is manually labeled; and
    the method further comprises the steps of:
    the first sample image data is converted into a color image suitable for manual labeling.
  9. The image detection method according to any one of claims 1-8, wherein the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
  10. An image detection apparatus, comprising:
    the acquisition module is configured to acquire image data to be detected through the first camera device;
    the processing module is configured to process the image data to be detected by utilizing an image processing algorithm to generate a processed image, wherein parameters of the image processing algorithm are obtained by comparing labeling information of first sample image data acquired by the first camera device with a detection result of an image detection model on the first sample image data and adjusting the detection result based on the comparison result;
    And the detection module is configured to input the processed image into the image detection model to obtain a detection result.
  11. The image detection apparatus according to claim 10, wherein the image detection model is obtained by neural network training of second sample image data acquired by the second image pickup means.
  12. The image detection apparatus according to claim 10 or 11, wherein the parameters of the image processing algorithm are determined by a parameter adjustment module comprising:
    a comparing sub-module configured to compare the detection result of the first sample image data with the labeling information of the first sample image data to obtain the comparison result;
    an adjustment sub-module configured to iteratively adjust parameters of the image processing algorithm based on the comparison result;
    and the storage submodule is configured to store parameters of the image processing algorithm when preset conditions are met.
  13. The image detection apparatus according to claim 12, wherein the comparison result is an error, the adjustment submodule being further configured to:
    constructing a target loss function based on the detection result of the first sample image data and the error between the labeling information of the first sample image data, wherein the target loss function comprises parameters to be updated in the image processing algorithm;
    And based on the target loss function, iteratively updating parameters of the image processing algorithm by using a back propagation algorithm and a gradient descent algorithm.
  14. The image detection apparatus according to any one of claims 10 to 13, wherein the image processing algorithm comprises at least one of the following image processing flows: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, and image noise reduction.
  15. The image detection apparatus according to claim 14, wherein the parameters of the image processing algorithm include at least one of:
    the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the image pickup device;
    boundary coordinates of a neutral color region in the white balance correction algorithm in the image;
    target brightness, target saturation and filter kernel parameters for generating a low-pass filtered image in a tone mapping algorithm;
    a contrast threshold in a contrast enhancement algorithm;
    edge enhancement factors in the image edge enhancement algorithm; and
    spatial domain gaussian parameters and pixel value domain gaussian parameters in the image noise reduction algorithm.
  16. The image detection apparatus according to claim 14, wherein the image processing algorithm is performed by a trained image processing model; the parameters of the image processing algorithm include:
    And generating a weight coefficient of the neural network of the image processing model.
  17. The image detection apparatus according to any one of claims 10 to 16, wherein the labeling information of the first sample image data is manually labeled; and
    the apparatus further comprises:
    a conversion module configured to convert the first sample image data into a color image suitable for manual annotation.
  18. The image detection apparatus according to any one of claims 10-17, wherein the image detection model is configured to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting confidence and predicting a target object motion trail.
  19. An electronic device, comprising:
    the first camera device is used for collecting image data to be detected;
    the image signal processor is used for processing the image data to be detected by utilizing an image processing algorithm to generate a processed image;
    the artificial intelligent processor is used for inputting the processed image into an image detection model to obtain a detection result;
    the parameters of the image processing algorithm are obtained by comparing the labeling information of the first sample image data acquired by the first image pickup device with the detection result of the image detection model on the first sample image data and adjusting the parameters based on the comparison result.
  20. An image detection apparatus, comprising:
    one or more processors and memory;
    the memory is coupled to the processor, the memory is used for storing one or more programs;
    the one or more processors are configured to run the one or more programs to implement the method of any of claims 1-9.
  21. A computer-readable storage medium, in which a computer program is stored which, when being executed by at least one processor, is adapted to carry out the method according to any one of claims 1-9.
  22. A computer program product for implementing the method according to any of claims 1-9 when the computer program product is executed by at least one processor.
CN202180093086.5A 2021-03-01 2021-03-01 Image detection method and device and electronic equipment Pending CN116888621A (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/078478 WO2022183321A1 (en) 2021-03-01 2021-03-01 Image detection method, apparatus, and electronic device

Publications (1)

Publication Number Publication Date
CN116888621A true CN116888621A (en) 2023-10-13

Family

ID=83153711

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202180093086.5A Pending CN116888621A (en) 2021-03-01 2021-03-01 Image detection method and device and electronic equipment

Country Status (2)

Country Link
CN (1) CN116888621A (en)
WO (1) WO2022183321A1 (en)

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9378406B2 (en) * 2012-06-15 2016-06-28 Seref Sagirouglu System for estimating gender from fingerprints
CN108171748B (en) * 2018-01-23 2021-12-07 哈工大机器人（合肥）国际创新研究院 Visual identification and positioning method for intelligent robot grabbing application
CN109961443A (en) * 2019-03-25 2019-07-02 北京理工大学 Liver neoplasm dividing method and device based on the guidance of more phase CT images
CN110458095B (en) * 2019-08-09 2022-11-18 厦门瑞为信息技术有限公司 Effective gesture recognition method, control method and device and electronic equipment
CN112101328A (en) * 2020-11-19 2020-12-18 四川新网银行股份有限公司 Method for identifying and processing label noise in deep learning

Also Published As

Publication number Publication date
WO2022183321A1 (en) 2022-09-09

Similar Documents

Publication Publication Date Title
CN111741211B (en) Image display method and apparatus
CN108780508B (en) System and method for normalizing images
CN106845487B (en) End-to-end license plate identification method
US20220188999A1 (en) Image enhancement method and apparatus
WO2021164234A1 (en) Image processing method and image processing device
KR102459221B1 (en) Electronic apparatus, method for processing image thereof and computer-readable recording medium
CN110276767A (en) Image processing method and device, electronic equipment, computer readable storage medium
WO2020152521A1 (en) Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures
WO2020001196A1 (en) Image processing method, electronic device, and computer readable storage medium
CN112581379A (en) Image enhancement method and device
CN110807384A (en) Small target detection method and system under low visibility
CN111209858A (en) Real-time license plate detection method based on deep convolutional neural network
CN113129236A (en) Single low-light image enhancement method and system based on Retinex and convolutional neural network
CN115239581A (en) Image processing method and related device
CN116757986A (en) Infrared and visible light image fusion method and device
Saleem et al. A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset
CN116917954A (en) Image detection method and device and electronic equipment
WO2023029559A1 (en) Data processing method and apparatus
CN116129150A (en) Unmanned aerial vehicle target tracking method integrating global tracker and local tracker
CN116888621A (en) Image detection method and device and electronic equipment
Wischow et al. Monitoring and Adapting the Physical State of a Camera for Autonomous Vehicles
CN113781375A (en) Vehicle-mounted vision enhancement method based on multi-exposure fusion
CN115170456A (en) Detection method and related equipment
CN114372944B (en) Multi-mode and multi-scale fused candidate region generation method and related device
US11941871B2 (en) Control method of image signal processor and control device for performing the same

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination