WO2022183321A1

WO2022183321A1 - Image detection method, apparatus, and electronic device

Info

Publication number: WO2022183321A1
Application number: PCT/CN2021/078478
Authority: WO
Inventors: 邱珏沁; 柳海波; 闻明
Original assignee: 华为技术有限公司
Priority date: 2021-03-01
Filing date: 2021-03-01
Publication date: 2022-09-09
Also published as: CN116888621A

Abstract

The embodiments of the present application provide an image detection method, an apparatus and an electronic device. The image detection method comprises: acquiring, by means of a first camera apparatus, image data to be detected; using an image processing algorithm to process said image data, so as to generate a processed image; and inputting the processed image into an image detection model to obtain a detection result, the parameters of the image processing algorithm being obtained by comparing annotation information of first sample image data acquired by the first camera apparatus with the detection result of the image detection model on the first sample image data and performing adjustment on the basis of the comparison result. The image detection method provided in the present application improves the accuracy of inference of an image detection model when a new photographing apparatus is combined with a trained image detection model.

Description

Image detection method, device and electronic device

technical field

The embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to an image detection method, apparatus, and electronic device.

Background technique

With the development of science and technology, artificial intelligence (AI, Artificial Intelligence) technology has been improved by leaps and bounds. In some artificial intelligence technologies, machine learning methods are usually used to construct initial models of various structures, such as neural network models, support vector machine models, and decision tree models. Then, the initial model is trained by training samples to realize functions such as image detection, speech recognition, etc. At present, in the computer vision technology based on image detection, the neural network is usually trained to obtain a perception model to realize image detection tasks such as scene recognition, object detection or image segmentation.

In related computer vision technologies, the sample image data used for training the neural network and the image data to be detected are usually collected by different camera modules. Due to the significant differences in the manufacturing process, photoelectric response function and noise level of different camera modules, there is a large deviation between the detection results of the perception model and the real results in the image detection process. Therefore, when a new camera module is combined with an already trained perception model, how to efficiently improve the accuracy of the detection result of the perception model is a problem that needs to be solved.

SUMMARY OF THE INVENTION

The image detection method, device and electronic device provided by the present application can improve the accuracy of image detection model inference when a new photographing device is combined with an already trained image detection model.

To achieve the above object, the application adopts the following technical solutions:

In a first aspect, an embodiment of the present application provides an image detection method, the image detection method includes: collecting image data to be detected by a first camera device; processing the image data to be detected by using an image processing algorithm to generate a processing The processed image is input into the image detection model to obtain the detection result; wherein, the parameters of the image processing algorithm are the annotation information of the first sample image data collected by the first camera device by comparing It is obtained by adjusting the detection result of the first sample image data with the image detection model and based on the comparison result.

In the embodiment of the present application, the parameters of the image processing algorithms used to execute multiple image processing processes are adjusted based on the image detection model, so that the image obtained after image processing is performed on the image collected by the first camera device is The style is consistent with the style of the sample image data for training the image detection model, thereby reducing the difference between the image data collected by the camera device and the feature distribution in the high-dimensional space of the sample image data for training the image detection model. There are It is beneficial to improve the accuracy of image detection model inference.

Based on the first aspect, in a possible implementation manner, the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.

Based on the first aspect, in a possible implementation manner, the parameters of the image processing algorithm are determined by the following steps: comparing the detection result with the annotation information of the first sample image data to obtain the comparison Result: Based on the error between the detection result and the annotation information of the sample image data, the parameters of the image processing algorithm are iteratively adjusted; when the preset conditions are satisfied, the parameters of the image processing algorithm are saved.

The preset conditions here may include but are not limited to: the error is less than or equal to a preset threshold, or the number of iterations is greater than or equal to a preset threshold.

Based on the first aspect, in a possible implementation manner, the comparison result is an error, and the iteratively adjusting the parameters of the image processing algorithm based on the comparison result includes: based on the comparison result and the first The error between the annotation information of the sample image data, construct a target loss function, wherein the target loss function includes the parameters to be adjusted in the image processing algorithm; based on the target loss function, using the back propagation algorithm and Gradient descent algorithm, iteratively adjust the parameters of the image processing algorithm.

Based on the first aspect, in a possible implementation manner, the processing of the image data and the first sample image data includes at least one of the following: dark current correction, lens shading correction, demosaicing, and white balance correction , tone mapping, contrast enhancement, image edge enhancement, or image noise reduction.

Based on the first aspect, in a possible implementation manner, the image processing algorithm is executed by an image signal processor; and the parameters of the image processing algorithm include at least one of the following: each pixel of the image in the lens shading correction algorithm The distance from the optical center of the camera; the boundary coordinates of the neutral color region in the image in the white balance correction algorithm; the target brightness and target saturation in the tone mapping algorithm, and the filter kernel parameters used to generate the low-pass filtered image; contrast ratio The contrast threshold in the enhancement algorithm; the edge enhancement factor in the image edge enhancement algorithm; and the spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the image noise reduction algorithm.

Based on the first aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm further include: a neural network for generating the image processing model weight factor.

Based on the first aspect, in a possible implementation manner, the annotation information of the first sample image data is manually annotated; and the method further includes: converting the first sample image data into suitable Human-annotated color images.

Based on the first aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence levels, and prediction of motion trajectories of target objects .

In a second aspect, an embodiment of the present application provides a parameter adjustment method for image processing. The parameter adjustment method for image processing includes: using an image processing algorithm to perform image processing on first sample image data to generate a first image data, wherein the first sample image data is collected by a first camera; input the first image data into a pre-trained image detection model to obtain a detection result; compare the detection result with the first The error between the annotation information of a sample image data is obtained, and a comparison result is obtained; based on the comparison result, the parameters of the image processing algorithm are adjusted.

Based on the second aspect, in a possible implementation manner, the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.

Based on the second aspect, in a possible implementation manner, the comparison result is an error, and the iteratively adjusting the parameters of the image processing algorithm based on the comparison result includes: based on the detection result and the first The error between the annotation information of the sample image data is used to construct a target loss function, wherein the target loss function includes the parameters to be adjusted in the image processing algorithm; based on the target loss function, a back-propagation algorithm is used. and a gradient descent algorithm that iteratively adjusts the parameters of the image processing algorithm.

Based on the second aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, and image edge enhancement Or image noise reduction.

Based on the second aspect, in a possible implementation manner, the parameters of the image processing algorithm include at least one of the following: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the camera; The boundary coordinates of the color region in the image; the target brightness, target saturation and the filter kernel parameters used to generate the low-pass filtered image in the tone mapping algorithm; the contrast threshold in the contrast enhancement algorithm; the edge in the image edge enhancement algorithm enhancement factor; and the spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the image noise reduction algorithm.

Based on the second aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include: a neural network used to generate the image processing model. weight factor.

Based on the second aspect, in a possible implementation manner, the labeling information of the first sample image data is manually labelled; and the method further includes: converting the first sample image data into Human-annotated color images.

Based on the second aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting a confidence level, and predicting a motion trajectory of the target object .

In a third aspect, an embodiment of the present application provides an image detection device, the image detection device includes: a collection module configured to collect image data to be detected through a first camera device; a processing module configured to use an image processing algorithm to The image data to be detected is processed to generate a processed image; the detection module is configured to input the processed image into an image detection model to obtain a detection result; wherein, the parameters of the image processing algorithm are obtained through It is obtained by comparing the annotation information of the first sample image data collected by the first camera device with the detection result of the first sample image data by the image detection model, and adjusting based on the comparison result.

Based on the third aspect, in a possible implementation manner, the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.

Based on the third aspect, in a possible implementation manner, the parameters of the image processing algorithm are determined by a parameter adjustment module, and the parameter adjustment module includes: a comparison sub-module configured to The detection result of the image data is compared with the annotation information of the first sample image data to obtain the comparison result; the adjustment sub-module is configured to iteratively adjust the parameters of the image processing algorithm based on the comparison result; save the sub-module The module is configured to save the parameters of the image processing algorithm when the preset condition is satisfied.

Based on the third aspect, in a possible implementation manner, the comparison result is an error, and the adjustment sub-module is further configured to: based on the detection result of the first sample image data and the first sample The error between the annotation information of the image data is used to construct a target loss function, wherein the target loss function includes the parameters to be updated in the image processing algorithm; based on the target loss function, back-propagation algorithm and gradient descent are used. algorithm that iteratively updates the parameters of the image processing algorithm.

Based on the third aspect, in a possible implementation manner, the image processing algorithm includes at least one of the following image processing procedures: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image Edge enhancement and image noise reduction.

Based on the third aspect, in a possible implementation manner, the parameters of the image processing algorithm include at least one of the following: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the camera; The boundary coordinates of the color region in the image; the target brightness, target saturation and the filter kernel parameters used to generate the low-pass filtered image in the tone mapping algorithm; the contrast threshold in the contrast enhancement algorithm; the edge in the image edge enhancement algorithm enhancement factor; and the spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the image noise reduction algorithm.

Based on the third aspect, in a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include: a neural network used to generate the image processing model. weight factor.

Based on the third aspect, in a possible implementation manner, the annotation information of the first sample image data is manually annotated; and the method further includes: converting the first sample image data into Human-annotated color images.

Based on the third aspect, in a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling a detection frame, identifying a target object, predicting a confidence level, and predicting a motion trajectory of the target object .

In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device includes: a first camera device, configured to collect image data to be detected; an image signal processor, configured to use an image processing algorithm to detect the image data to be detected. The image data is processed to generate a processed image; an artificial intelligence processor is used to input the processed image into an image detection model to obtain a detection result; wherein, the parameters of the image processing algorithm are obtained by comparing the first It is obtained by adjusting the annotation information of the first sample image data collected by a camera device and the detection result of the first sample image data by the image detection model and based on the comparison result.

In a fifth aspect, an embodiment of the present application provides an image detection device, the image detection device includes one or more processors and a memory; the memory is coupled to the processor, and the memory is used to store one or more programs ; the one or more processors are configured to run the one or more programs to implement the method according to the first aspect.

In a sixth aspect, an embodiment of the present application provides a parameter adjustment apparatus for image processing, the parameter adjustment apparatus for image processing includes one or more processors and a memory; the memory is coupled to the processor, and the The memory is used to store one or more programs; the one or more processors are used to execute the one or more programs to implement the method according to the second aspect.

In a seventh aspect, embodiments of the present application provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by at least one processor, is used to implement the first aspect or the second aspect the method described.

In an eighth aspect, an embodiment of the present application provides a computer program product, which is used to implement the method according to the first aspect or the second aspect when the computer program product is executed by at least one processor.

It should be understood that the second to eighth aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding feasible implementation manner are similar, and will not be repeated here.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

1 is a schematic structural diagram of a terminal provided by an embodiment of the present application;

2 is a schematic structural diagram of an image processing process performed in combination with an ISP and an AI processor provided by an embodiment of the present application;

3 is a schematic structural diagram of a vehicle provided by an embodiment of the present application;

4 is a schematic diagram of a system architecture including an electronic device for parameter debugging of an image processing algorithm provided by an embodiment of the present application;

5 is a flowchart of a parameter debugging method of an image processing algorithm provided by an embodiment of the present application;

6 is a specific application flowchart of the parameter debugging method of the image processing algorithm provided by the embodiment of the present application;

7 is a flowchart of an image detection method provided by an embodiment of the present application;

8 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a parameter debugging apparatus provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the following description, reference is made to the accompanying drawings which form a part of this application and which illustrate, by way of illustration, specific aspects of embodiments of the application, or specific aspects of which embodiments of the application may be used. It should be understood that the embodiments of the present application may be utilized in other aspects and may include structural or logical changes not depicted in the accompanying drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the application is defined by the appended claims. For example, it should be understood that disclosures in connection with a described method may equally apply to a corresponding apparatus or system for performing the described method, and vice versa. For example, if one or more specific method steps are described, the corresponding apparatus may include one or more units, such as functional units, to perform one or more of the described method steps (eg, one unit performs one or more steps) , or units, each of which performs one or more of the steps), even if such unit or units are not explicitly described or illustrated in the figures. On the other hand, if, for example, a specific apparatus is described based on one or more units, such as functional units, the corresponding method may contain a step to perform the functionality of the one or more units (eg, a step to perform the one or more units) functionality, or steps, each of which performs the functionality of one or more of the plurality of units), even if such one or more steps are not explicitly described or illustrated in the figures. Further, it is to be understood that the features of the various exemplary embodiments and/or aspects described herein may be combined with each other unless expressly stated otherwise.

The image detection method described in the present application can be applied in the field of computer vision, in a scene where an image detection model obtained by training sample images collected by other photographing equipment needs to be combined with a new photographing equipment.

Please refer to FIG. 1 , which shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 1 , the electronic device 100 may be a user equipment (User Equipment, UE), such as various types of devices such as a mobile phone, a tablet computer, a smart screen, or an image capturing device. In addition, the electronic device 100 may also be a vehicle. A camera 101 may be provided in the electronic device 100 for capturing image data. In addition, the electronic device 100 may also include or be integrated into a module, chip, chip set, circuit board or component in the electronic device, and the chip or chip set or the circuit board equipped with the chip or chip set can be driven by necessary software Work. The electronic device 100 includes one or more processors, such as an image signal processor (ISP, Image Signal Processor) 102 and an AI processor 103 . Optionally, the one or more processors can be integrated in one or more chips, and the one or more chips can be regarded as a chipset, when one or more processors are integrated in the same chip The chip is also called a system on a chip (SOC). In addition to the one or more processors, the electronic device 100 also includes one or more other necessary components, such as memory and the like.

The camera device 101 shown in FIG. 1 may be a monocular camera. Alternatively, the camera device 101 may further include multi-camera cameras, and these cameras may be physically combined in one camera device, or may be physically separated into multiple camera devices. Multiple images are captured at the same time by a multi-eye camera, and can be processed according to these images to obtain an image to be detected. Of course, the camera 101 may also be in other situations, which are not specifically limited in the embodiments of the present application. In a specific implementation, the camera 101 can collect image data in real time, or collect image data periodically. The period is such as 3s, 5s, 10s and so on. The camera device 101 may also collect image data in other ways, which are not specifically limited in this embodiment of the present application. After the camera 101 collects the image data, it can transmit the image data to the ISP 102 .

The ISP 102 shown in FIG. 1 can set up a plurality of hardware modules or run necessary software programs to process the image data and communicate with the AI processor 103 . ISP102 can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU (Central Processing Unit, Central Processing Unit), GPU (Graphics Processing Unit, Graphics Processing Unit) or DSP (Digital Signal Processing Unit) processor, Digital Signal Processing). Illustratively, the CPU, GPU and DSP are all processors within a system-on-chip. ISP102 can perform multiple image processing processes, which may include but are not limited to: dark current correction, response nonlinearity correction, shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, edge enhancement, Noise reduction, color correction, and more.

It should be noted that the ISP 102 executes the above-mentioned multiple image processing processes by running the image processing algorithm. Each image processing process in the above-mentioned multiple image processing processes can be regarded as an independent image processing process, and thus, the image processing algorithm for executing each image processing process can be regarded as independent. Based on this, the ISP 102 may include multiple logic modules. For example, it includes, but is not limited to, a dark current correction module, a response nonlinearity correction module, a shading correction module, a demosaicing module, and the like. Each logic module is used to perform an image detection process. Each logic module may use its own specific hardware structure, and multiple logic modules may also share a set of hardware structures, which is not limited in this embodiment of the present application. Furthermore, the one or more image processing processes are typically performed sequentially. For example, after the image data acquired by the camera 101 is provided to the ISP, processing procedures such as dark current correction, response nonlinear correction, shading correction, demosaicing, white balance correction, etc. may be sequentially performed. It should be noted that this embodiment of the present application does not limit the sequence of the image processing processes performed by the ISP. For example, white balance correction may be performed first, and then demosaicing may be performed.

The AI processor 103 shown in FIG. 1 may include a special neural processor such as a neural network processor (Neural-network Processing Unit, NPU), including but not limited to a convolutional neural network processor, a tensor processor, or a neural processing unit. engine. The AI processor can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU, GPU or DSP. The AI processor 103 may run an image detection model, and the image detection model is obtained by training a deep neural network based on the sample image data set S1. This image detection model can perform specific detection tasks. The specific detection task may include, but is not limited to, labeling of detection frames, recognition of target objects, prediction of confidence levels, prediction of motion trajectories of target objects, or image segmentation, and the like. It should be noted that the image detection model is deployed in the AI processor 103 shown in FIG. 1 after the offline end training is completed. The offline end here can be regarded as a server device or a device for model training.

In a possible implementation manner, in addition to running an image detection model to perform image detection operations, the AI processor 103 may also perform one or more image processing operations, the one or more image processing operations Operations may include, but are not limited to: demosaicing, white balance correction, tone mapping, contrast enhancement, edge enhancement, noise reduction, color correction. At this time, the AI processor 103 may also run one or more image processing models, where each image processing model is used to execute a specific image processing process. In this possible implementation, the image data obtained from the camera 101 may undergo multiple image processing processes to generate a final image processing result, and the AI processor 103 may perform one or more of the above image processing processes, That is, corresponding to one or more of the above-mentioned image processing operations, the ISP 102 may also perform one or more of the above-mentioned image processing procedures. The AI processor 103 and the ISP 102 may perform different image processing processes. In addition, the AI processor 103 and the ISP 102 may also perform the same image processing process, such as performing further enhancement processing, which is not limited in this embodiment. In this possible implementation manner, each image processing model may be obtained by training a neural network by using a machine learning method based on the sample image data set S3. The sample image data set S3 includes: a plurality of sample image data H, and reference image data I corresponding to each sample image data H in the plurality of sample image data H. The reference image data I is used for image data comparison, and the image data to be compared with the reference image data I is: image data obtained after the neural network to be trained performs image processing on the sample image data H. It should be noted that the reference image data I and its corresponding sample image data H present the same scene. The same scene presented here can be understood as: the target objects presented by the reference image data I and the corresponding sample image data H and the positions of the target objects in the image are all the same. In addition, the image processing process performed by the image processing model is different, and the sample image dataset S3 used for training the image processing model is also different. For example, when the image processing model is used to perform the image processing flow of demosaicing, each sample image data H included in the sample image data set S3 is a single-channel raw image format of a*b*1 size (RAW, Raw Image Format) image data, each reference image data I corresponding to each sample image data is the RGB image data of a*b*3 size, wherein, a is the vertical pixel value of the image, and b is the horizontal pixel value of the image, 1 is single channel (eg R channel, G channel or B channel), 3 is three channel (RGB channel). For another example, when the image processing model is used to perform the image processing flow of white balance correction, each sample image data H included in the sample image data set S3 is single-channel RAW image data of a*b*1 size, which is the same as each sample image data H. Each reference image data I corresponding to one sample image data is single-channel image data of a*b*1 size, the sample image data set S3. The reference image data I and its corresponding sample image data H have the same scene but different white balance values. The following describes the training process of the image processing model by taking the image processing process in which the image processing model performs demosaicing as an example. Each sample image data H included in the sample image data set S3 is respectively input to the neural network to be trained to obtain processed image data. Based on the error between the processed image data and the reference image data I, a loss function is constructed. The loss function may include, but is not limited to, a mean absolute error (MAE) loss function or a mean square error (MSE) loss function, for example. The loss function includes the weight coefficients of each layer of the neural network to be trained. Based on the constructed loss function, the back-propagation algorithm and the gradient descent algorithm are used to iteratively adjust the weight coefficient values of each layer network in the neural network to be trained, until the processed image data output by the image processing model and the reference image data I The error between them is less than or equal to the preset threshold, or the number of iterations is less than the preset threshold, and the weight coefficient values of each layer of the neural network to be trained are saved. At this point, the neural network to be trained is an image processing model.

When the ISP102 and the AI processor 103 are combined to perform the image processing process, the ISP102 can be set with multiple ports, and the AI processor 103 can also be set with multiple ports. The ISP102 can pass the processed image data A through multiple ports. One of the ports is provided to the AI processor 103 , the AI processor 103 processes the image data A to generate image data B, and provides the image data B to the ISP 102 through one of the multiple ports. Taking the AI processor 103 performing demosaicing as an example, the combination of the ISP 102 and the AI processor 103 will be described with reference to FIG. 2 . In FIG. 2 , the ISP 102 acquires image data from the camera 101, performs the three image processing processes of dark current correction, response nonlinear correction, and shading correction on the acquired image data, and generates image data A, which is provided to AI through port Vio The input port Vai of the processor 103 . The image processing model run by the AI processor 103 performs demosaic processing on the image data A to generate image data B, and provides the image data B to the input port Vii of the ISP 102 through the output port Vao. The ISP 102 performs subsequent image processing procedures such as white balance correction and color correction on the image data B input through the input port Vii, and generates image data C that is input to the AI processor 103 . The image detection model run by the AI processor 103 can perform image detection processing on the image data C.

Based on the image processing method in which the ISP 102 and the AI processor 103 are combined, the AI processor 103 may include one or more. When the AI processor 103 includes one, the image processing model for performing image processing and the image processing model for performing image detection The image detection model can be set in the same AI processor 103. When there are multiple AI processors 103, the image processing model used to perform image processing and the image detection model used to perform image detection can be set to different AI processing models. in the device 103.

The parameters in the image processing algorithm run by the ISP 102 and the parameters in the image processing model run by the AI processor 103 described in the embodiments of the present application are based on the sample image data set S2 collected by the camera 101 and The image detection results of the image detection model are obtained by debugging. Wherein, for the image processing algorithm run by the ISP 102 and the debugging method of each parameter in the image processing model, refer to the embodiment shown in FIG. 5 below.

Usually, the sample image data set S1 used for training the image detection model running in the AI processor 103 is collected through big data, and the camera used for collecting the sample image data set S1 is the same as the camera shown in FIG. 1 . 101 is a different camera device. Since there are significant differences in the manufacturing process, photoelectric response function, noise level and other characteristics of different camera devices, the style of the sample image data D in the sample image data set S1 is different from the image data collected by the camera device 101. There are differences in the styles of the images obtained after processing, which in turn leads to significant differences in the feature distributions in the high-dimensional space between the image data collected by the camera device 101 and the sample image data D in the sample image data set S1, resulting in the deployment of image detection models. In the AI processor 103 , during the detection process of the image data collected by the camera 101 , the deviation between the detection result and the real result is relatively large, which reduces the detection accuracy of the image detection model deployed in the AI processor 103 .

Based on this, by keeping the parameters of the image detection model unchanged, the embodiment of the present application adjusts the parameters of the image processing algorithms used for executing multiple image processing processes (or adjusts the parameters of the image processing algorithms used for executing multiple image processing processes). parameters and parameters of the image processing model), so that the style of the image obtained by performing image processing on the image collected by the camera 101 is consistent with the style of the sample image data D in the sample image data set S1, thereby reducing the The difference between the image data collected by the camera 101 and the feature distribution of the sample image data D in the sample image data set S1 in the high-dimensional space is conducive to improving the accuracy of the image detection model inference.

In traditional computer vision technology, in order to adapt the trained image detection model to a new camera device and improve the accuracy of the detection results of the image detection model, it is usually necessary to use a new data set to retrain or retrain the trained image detection model. Fine tune. However, the process of collecting and labeling new data sets requires a lot of manpower and material resources, and in the process of retraining the image detection model, the image detection model must "forget" some of the knowledge it has learned, resulting in historical data cannot be used. Make the most of it. In this embodiment of the present application, by keeping the parameters of the trained image detection model unchanged, by adjusting the parameters of the image processing algorithms used for executing multiple image processing processes (or adjusting the parameters of the image processing algorithms used for executing multiple image processing processes) parameters and the parameters of the image processing model) to reduce the difference between the image data collected by the camera 101 and the feature distribution of the sample image data D in the high-dimensional space, which is different from the traditional technique of retraining the image detection model. Compared with other methods, the embodiment of the present application does not require any modification to the trained image detection model, which saves the time and computing power overhead required for retraining and fine-tuning the image detection model; Adjust the parameters of the image processing algorithm used to perform multiple image processing processes. Since the image processing algorithm does not perform the image detection process, the training can be completed with fewer training samples, thereby reducing the need for manual labeling of training samples. quantity, shortening the commissioning cycle when combining image detection models with new cameras.

Taking an automatic driving scenario as an example, the following describes the application scenario of the embodiment of the present application in more detail in conjunction with the schematic structural diagram of the electronic device 100 shown in FIG. 1 . Please refer to FIG. 3 , which shows a schematic structural diagram of a vehicle 300 provided by an embodiment of the present application.

Components coupled to or included in vehicle 300 may include control system 10 , propulsion system 20 , and sensor system 30 . It should be understood that the vehicle 300 may also include more systems, which will not be repeated here. The control system 10 may be configured to control the operation of the vehicle 300 and its components. The ISP 102 and the AI processor 103 shown in FIG. 1 can be set in the control system 10. In addition, the control system 10 can also include devices such as a central processing unit, a memory, and the like, and the memory is used to store the instructions and data required for the operation of each processor . Propulsion system 20 may be used for vehicle 300 to provide powered motion, which may include, but is not limited to, an engine/motor, energy source, transmission, and wheels. The sensor system 104 may include, but is not limited to, a global positioning system, an inertial measurement unit, a lidar sensor, or a millimeter-wave radar sensor. The camera device 101 shown in FIG. 1 may be provided in the sensor system 30 . The components and systems of vehicle 300 may be coupled together through a system bus, network, and/or other connection mechanism to operate in interconnection with other components within and/or outside of their respective systems. In specific work, various components in the vehicle 300 cooperate with each other to realize various automatic driving functions. The automatic driving function may include, but is not limited to, blind spot detection, parking assist or lane change assist, and the like.

In the process of realizing the above automatic driving function, the camera device 101 may periodically collect image data, and provide the collected image data to the ISP 102 . The ISP102 (or the image processing model in the ISP102 and the AI processor 103) processes the image data by performing multiple image processing processes, and converts it into image data that can be recognized or calculated by the image detection model running in the AI processor 103. The AI processor 103 enables the AI processor 103 to realize the reasoning or detection of a specific task, and generate the detection result. Other components in the control system 10 (for example, a CPU that executes decisions) control other devices or components to perform automatic driving functions based on the detection results of the AI processor 103 .

Usually, the manufacturer of the vehicle may not produce some parts by itself. The trained image detection model is ordered through manufacturer A, and the camera device is ordered through manufacturer B. In order to make the image detection model more accurate for the image data obtained by the camera device For detection, the training method described in the embodiments of the present application can be used to debug the parameters of the image processing algorithm or the image processing model used to execute the image processing flow. For another example, when a manufacturer upgrades certain models of vehicles, it is necessary to replace a camera device of a different model from the previously configured camera device. At this time, in order to match the replaced camera device with the image detection model, so that the detection result of the image detection model on the image data obtained by the new camera device is more accurate, the training method described in the embodiments of the present application can also be used to Debug the parameters of the image processing algorithm or image processing model used to execute the image processing flow.

It should be noted that the parameter debugging of the image processing algorithm or the image processing model may be completed at the offline end (or in other words, the training is completed in the server or the device used for model training). After the parameters of the image processing algorithm are debugged, the image processing algorithm can be deployed in the ISP of the terminal. When some image processing processes are performed by the image processing model, after the parameters in the image processing model are debugged, the image processing model can be deployed in the AI processor 103 . Please refer to FIG. 4 , which shows a schematic diagram 400 of a system architecture including an electronic device for parameter debugging of an image processing algorithm provided by an embodiment of the present application.

In FIG. 4 , the system architecture 400 includes a camera device 101 , a parameter debugging device 401 , a storage device 402 and a display device 403 . The camera 101 is used for collecting a plurality of sample image data E, and storing the collected sample image data E in the storage device 402 . The imaging device 101 and the imaging device 101 shown in FIG. 1 are the same (or the same) imaging device. The storage device 402 may include, but is not limited to, read-only memory or random access memory, and the like. It is used to store sample image data E. In addition, the storage device 402 may also store executable programs and data of an image processing algorithm for executing the image processing process, and executable programs and data of an image detection model for executing the image detection. The parameter debugging device 401 can run the image processing algorithm and the image detection model, and the parameter debugging device 401 can also call the sample image data E, the executable program and data of the image processing algorithm for executing the image processing process, and the data for the image processing from the storage device 101. The executable program and data of the image detection model that performs image detection to debug the parameters of the image processing algorithm. In addition, the parameter debugging device 401 may also store the data generated by the operation and the debugging result after each parameter debugging of the image processing algorithm into the storage device 402 . In addition, the parameter debugging device 401 and the storage device 402 may also be provided with I/O ports for data interaction with the display device 403 . The display device 403 may include a display device such as a screen to mark the sample image data E. Specifically, the parameter debugging device 401 may acquire sample image data E from the storage device 402 , perform image processing on the sample image data E, and provide the sample image data E to the display device 403 for presentation in the display device 403 . The user will mark the sample image data E through the display device 403 , and store the marking information of the sample image data E in the storage device 402 .

In the embodiment of the present application, in order to facilitate the processing of the sample image data E by the image processing algorithm and the detection of the image by the image detection model, the sample image data E output by the camera 101 is a high bit depth (for example, 16bit, 20bit or 24bit) single-channel Linear RAW image data, the dynamic range of which is much larger than the dynamic range that can be displayed by the monitor. In addition, the sample image data E is a color filter array (CFA, color filter array) image, which does not have color information, so it is difficult for the annotator to extract the image from the camera. Each target object is identified in the sample image data E output by the device 101 . Based on this, in the embodiment of the present application, the parameter debugging device 401 also runs an image processing algorithm T, and the image processing algorithm T is used to process the sample image data E to generate a color that can be presented on a display and has suitable brightness and color. An image, such as an RGB image, is convenient for the annotator to annotate the target object presented in the sample image data E. The image processing flow performed by the image processing algorithm T may include, but is not limited to: system error correction, global tone mapping, demosaicing or white balance correction. Each parameter in the image processing algorithm T does not need to be adjusted, which can be achieved by using traditional image processing algorithms. It should be noted that the image processing algorithm T described in the embodiments of the present application is used to process the sample image data E to generate image data that can be displayed on the display for annotators to mark; The image processing algorithm is used to process the sample image data E to generate image data for image detection by the image detection model, and its parameters need to be adjusted.

Based on the schematic structural diagram of the electronic device 100 shown in FIG. 1 , the application scenario of the vehicle shown in FIG. 3 , and the system architecture 400 shown in FIG. Details.

Please refer to FIG. 5 , which shows a flow 500 of a method for debugging parameters of an image processing algorithm provided by an embodiment of the present application. It should be noted that the execution body of the parameter debugging method for image processing described in the embodiments of the present application may be the parameter debugging device 401 shown in FIG. 4 . As shown in Figure 5, the parameter debugging method for image processing includes the following steps:

Step 501 , based on the sample image data set S2 , use an image processing algorithm to process each sample image data E in the sample image data set S2 to generate a plurality of image data F.

The sample image data set S2 includes a plurality of sample image data E and label information of each sample image data E. Wherein, each sample image data E in the sample image data set S2 is collected by the camera 101 as shown in FIG. 1 . The annotation information of the sample image data E is annotated based on the detection content performed by the image detection model. For example, when the image detection model is used to perform target detection, the annotation information of the sample image data E may include the target object and the position of the target object in the second sample image; when the image detection model is used to perform pedestrian intent detection, the sample image The annotation information of the data E may include the target object and the action information of the target object.

The image processing algorithms herein are used to perform one or more image processing procedures. The one or more image processing procedures include, but are not limited to, dark current correction, response nonlinearity correction, lens shading correction, demosaicing, white balance correction, tone mapping, noise reduction, contrast enhancement or edge enhancement, and the like. It should be noted that the one or more image processing processes are usually performed sequentially. This embodiment of the present application does not specifically limit the execution order of the image processing process.

In step 502, the image data F is detected by using the image detection model, and a detection result is generated.

Here, the image detection model may perform at least one of the following detections: object detection, lane line detection, or pedestrian intent detection, and the like. The image detection model is obtained by training a deep neural network based on the image dataset S1. It should be noted that the image data D in the image data set S1 is collected by other imaging devices different from the imaging device 101 . In addition, the training method of the image detection model is a traditional technology, and details are not described here.

Step 503 , based on the detection result and the label information of the sample image data E, adjust the parameters of the image processing algorithm.

In a possible implementation manner, the parameters of the image processing algorithm may be adjusted by using a machine learning method. The second possible implementation manner is described in detail below.

A loss function is constructed based on the error between the detection result of each sample image data E in the sample image data set S2 and the label information of the sample image data E. The loss function may include, but is not limited to, a cross-entropy function and the like. Then, the parameters of the image processing module for executing one or more image processing procedures in the image processing algorithm are adjusted by using the back-propagation algorithm and the gradient descent algorithm. The gradient descent algorithm may specifically include, but is not limited to, optimization algorithms such as SGD and Adam. When performing backpropagation based on the preset loss function, the chain rule can be used to calculate the gradient of the preset loss function with respect to each parameter in the image processing algorithm.

Specifically, the image processing algorithms used to execute each image processing flow are independent of each other. When the image processing flow performed by the image processing algorithm is dark current correction, response nonlinear correction, lens shading correction, demosaicing, white balance correction, noise reduction, contrast enhancement or edge enhancement, etc. When adjusting the parameters of the image processing algorithms in each process, the parameters that are propagated and adjusted first are the parameters of the image processing algorithm for performing edge enhancement, and then the parameters of the image processing algorithms that are adjusted sequentially are the image processing parameters for performing contrast enhancement. Parameters of an algorithm, parameters of an image processing algorithm for performing noise reduction, parameters of an image processing algorithm for performing white balance correction, and the like. It can be understood that the embodiments of the present application may include more or less image processing processes, and accordingly may include more or less parameters to be adjusted. In addition, since the embodiment of the present application does not limit the order of the image processing flow performed by the image processing algorithm, when using the backpropagation algorithm to adjust the parameters of the image processing algorithm, the first adjusted parameter and the last adjusted parameter The embodiments of the present application do not make specific limitations.

It should be noted that when the parameters of the image processing algorithm are adjusted by the back-propagation algorithm based on the preset loss function, the parameters in the image detection model are kept unchanged.

Step 504: Determine whether the loss value of the preset loss function is less than or equal to a preset threshold. If the loss value of the preset loss function is less than or equal to the preset threshold, the parameters of the image processing algorithm are saved; if the loss value of the preset loss function is greater than the preset threshold, step 505 is executed.

Step 505: Determine whether the number of times of iteratively adjusting the parameters of the image processing algorithm is greater than or equal to a preset threshold. If the number of times of iteratively adjusting the parameters of the image processing algorithm is greater than or equal to the preset threshold, the parameters of the image processing algorithm are saved, and if the number of times of iteratively adjusting the parameters of the image processing algorithm is less than the preset threshold, continue to execute steps 501-505.

In a possible implementation manner of the present application, the above-mentioned one or more image processing procedures may all be implemented by traditional image processing algorithms. When the above one or more image processing procedures are implemented by traditional image processing algorithms, it is assumed that the one or more image processing procedures include dark current correction, response nonlinearity correction, lens shading correction, demosaicing, white balance correction, tone mapping , Noise Reduction, Contrast Enhancement and Edge Enhancement. At this time, the parameters of the image processing algorithm to be adjusted may include, but are not limited to: parameters of an image processing algorithm for performing lens shading correction, parameters for an image processing algorithm for performing white balance correction, parameters for performing tone mapping Parameters of an image processing algorithm, parameters of an image processing algorithm for performing contrast enhancement, parameters for an image processing algorithm for performing edge enhancement, and parameters for an image processing algorithm for performing noise reduction. It should be noted that when a machine learning method is used to adjust the parameters of the image processing algorithm, the image detection algorithms described in the embodiments of the present application are all differentiable, so that they can be derived during backpropagation based on the chain rule. The parameters of each image processing algorithm to be adjusted are described in detail below.

Lens shading correction is used to correct the illuminance attenuation caused by the increase of the incident angle of the chief ray in the edge area of the image. It uses a polynomial to fit the illuminance attenuation surface, where the independent variable of the polynomial is the distance between each pixel of the image and the optical center of the camera device . Therefore, in the image processing algorithm for performing lens shading correction, the parameter to be adjusted is the distance between each pixel of the image and the optical center of the camera, that is, the value of the independent variable in the polynomial.

The execution process of white balance correction is as follows: first, the neutral color pixel search algorithm is used to screen the neutral color area in the image, and based on the screening result, the boundary coordinates of the neutral color area in the image are determined. Then, the pixel values in the filtered neutral regions are weighted using the luminance channel of the image to generate a binarized neutral pixel mask. The individual (near) neutral pixels are then weighted averaged using this neutral pixel mask to obtain an estimate of the color of the light source in the image. Finally, by calculating the ratio between the RGB channels of the light source color, the white balance correction coefficient corresponding to the image is obtained, and the white balance correction coefficient is applied to the original image, that is, the white balance corrected image is obtained. Therefore, in the image processing algorithm for performing white balance correction, the parameter to be adjusted is the boundary coordinates of the neutral color area in the image.

Tone mapping is used to receive a linear image with high bit depth, convert the linear image into a nonlinear image, and complete the compression of the image bit depth, and output an 8-bit image. When using the global gamma function as the tone mapping function, the trainable parameter is the γ parameter; when using the logarithmic transformation algorithm to compress the dynamic range of the linear image, the trainable parameter is the base of the logarithmic transformation; when using a more complex order When a tone mapping model, such as a retinax model based on the dynamic range response of the human eye, can be trained as a target luminance parameter (key), a target saturation parameter (saturation), and a filter kernel parameter for generating a low-pass filtered image.

Contrast Enhancement is used to enhance the contrast of an image. Specifically, the CLAHE (contrast limited adaptive histogram equalization, contrast limited adaptive histogram equalization) algorithm can be used to locally adjust the contrast of the image. The CLAHE algorithm contains two adjustable parameters: the contrast threshold parameter and the sub-patch size used for histogram statistics. In a possible implementation manner of the embodiment of the present application, the size of the sub-image block may be fixed, and only the contrast threshold parameter may be adjusted. Further, the size of the sub-image block can be fixed to the size of the input image.

In the edge enhancement of the image, the Y-channel image in the received image is first subjected to Gaussian filtering to obtain a low-pass Y-channel image Y _L ; the difference image between the original Y-channel image and the low-pass Y-channel image Y _L As the high-frequency signal in the image, that is, Y _HF =YY _L , the high-frequency signal usually corresponds to the edge area in the image; by amplifying the intensity of the high-frequency signal and superimposing it into the low-pass Y-channel image Y _L , it can be The image Y _E after edge enhancement is obtained, that is, Y _E =Y _L +α·(YY _L ), where α is an edge enhancement factor, and the degree of image edge enhancement increases with the increase of α. Among the parameters of the image processing algorithm for performing edge enhancement, the parameter to be adjusted is the edge enhancement factor α.

In image noise reduction, the bilateral filter noise reduction algorithm is usually used. In the bilateral filtering noise reduction algorithm, the trainable parameters can include: the spatial Gaussian kernel parameter σs used to control the relationship between the noise reduction intensity and the spatial distance, and the pixel used to control the relationship between the noise reduction intensity and the response value difference Value domain Gaussian kernel parameter σr.

In another possible implementation manner, part of the image processing flow in the above one or more image processing flows may be implemented by an image processing model. For example, the above-mentioned image processing processes such as demosaicing, white balance correction, tone mapping, noise reduction, contrast enhancement and edge enhancement are implemented by an image processing model. The image processing model may include a variety of image processing models, each of which is used to perform a specific image processing operation. For example, a noise-cancelling image processing model is used to perform a noise-cancelling image processing operation, and a demosaicing image processing model is used to perform a demosaicing image processing operation. Each image processing model may be obtained by using a traditional neural network training method, and using training samples to train a multi-layer neural network, and the training method will not be repeated in this embodiment of the present application. The work of each layer in each image processing model can be described by the mathematical expression y=a(W*x+b). Among them, W is the weight, x is the input vector (ie the input neuron), b is the bias data, y is the output vector (ie the output neuron), and a is a constant. From the physical level, the work of each layer in the deep neural network can be understood as completing the transformation from the input space to the output space (that is, the row space of the matrix to the column space) through five operations on the input space (set of input vectors). These five operations include: 1. Dimension up/down; 2. Zoom in/out; 3. Rotate; 4. Translation; 5. "Bend". Among them, the operations of 1, 2, and 3 are completed by W*x, the operation of 4 is completed by +b, and the operation of 5 is implemented by a(). The reason why the word "space" is used here is because the image to be processed is not a single thing, but a type of thing, and space refers to the collection of all individuals of this type of thing. Among them, W is the weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how the space is transformed. When one or more image processing models are used to process image data, the adjusted image processing algorithm is the weight matrix (the weight matrix formed by the vectors W of many layers) of all layers forming the image processing model. It should be noted that the above one or more image processing models are pre-trained at the offline end. When the parameters of the image processing algorithm are adjusted using the parameter debugging method described in this application, it is only necessary to fine-tune the parameters of the neural network used to form the image processing model, so that the image processing model can obtain the style of the image after processing the image. features, similar to the style features of the sample image data used to train the image detection model. Thereby, the detection accuracy of the image detection model is improved.

Based on the parameter debugging method of the image processing algorithm shown in FIG. 5 , the following takes the detection function performed by the image detection model as the target object recognition as an example to describe the parameter debugging method of the image processing algorithm described in the embodiment of the present application in more detail. illustrate. Please refer to FIG. 6 , which is a schematic diagram of a specific application of the parameter debugging method for image processing according to an embodiment of the present application. The execution subject of the parameter debugging method for image processing described in FIG. 6 may be the parameter debugging device 401 shown in FIG. 4 .

Step 601 , using the camera 101 to collect a plurality of sample image data E. The plurality of sample image data E are single-channel linear RAW image data of high bit depth (eg, 16 bits, 20 bits, or 24 bits).

Step 602 , using the image processing algorithm T to process the sample image data E, to generate the processed image data F for presentation on the display screen. The image data F is a color image, such as an RGB image.

Step 603: Manually label the image data F to obtain labeling information of each sample image data E. Since the detection performed by the image detection model is target detection, the annotation information of the sample image data includes the category of the target object presented by the sample image data and the position in the sample image data.

Step 604 , using an image processing algorithm to process the sample image data to generate image data D.

Step 605: Input the image data D into the image detection model to obtain an image detection result, wherein the image detection result includes the position area of the preset target object in the sample image data and the probability value of the preset target object.

Step 606: Construct a loss function based on the image detection result and the labeling information of the sample image data E.

Suppose N(·) is used to represent the image detection model, so we have

in

Represents the loss function of the image detection model. Y _out represents the image data input by the image detection model, that is, the image data finally output by the image processing algorithm by executing multiple image processing procedures.

Step 607: Determine whether the loss value of the loss function reaches a preset threshold. If the preset threshold is not reached, step 508 is performed, and if the preset threshold is reached, the parameters of the image processing algorithm are saved.

Step 608, using the back-propagation algorithm and the gradient descent algorithm to adjust the parameters of the image detection algorithm.

In the following, a more detailed description is given by taking as an example adjusting the parameters of the image processing algorithm for executing the edge enhancement process. The edge enhancement of the image is to amplify the high frequency signal in the image and superimpose it into the low-pass image to realize the edge sharpening of the image, namely: YE=YL+α·(Y-YL). For the meaning of each parameter in the formula, please refer to the relevant description in the above-mentioned edge enhancement of the image, which will not be repeated here. Among the parameters of the image processing algorithm for performing edge enhancement, the parameter to be adjusted is the edge enhancement factor α. Assume that all image operations after image edge enhancement are denoted by P(·), ie Y _out =P(Y _E ). The image obtained after all the image operations before edge enhancement is Y. Under the condition that all parameters in P( ) remain unchanged, the gradient of the objective function L with respect to the edge enhancement factor α can be obtained according to the chain rule:

According to the SGD algorithm, after one iteration, the updated value of α ^(t+1) is:

Among them, α ^(t+1) is the value of the current value of α ^(t) after stepping a distance in the opposite direction of its gradient, and δ represents the learning rate, which is used to control the step size of each iteration. Substitute formula (2) into formula (1) to obtain the updated value of edge enhancement factor α:

Assuming that the current value of the edge enhancement factor α is α ⁽⁰⁾ , the value of the edge enhancement factor α after one iteration is:

Similarly, the adjustment process of the parameters of the image detection algorithm for executing other image processing procedures may be similar to the adjustment process of the parameters of the image detection algorithm for edge enhancement, and details are not repeated here.

In the parameter debugging method shown in FIG. 6 , by repeatedly executing steps 604 to 608 , that is, performing multiple iterative adjustments to the parameters in the image processing algorithm, the minimum value in the image processing algorithm that makes the loss function L reach a minimum value can be obtained. best parameter value.

Based on the electronic device 100 shown in FIG. 1 and the parameter debugging method for image processing shown in FIG. 5 and FIG. 6 , an embodiment of the present application further provides an image detection method. Please refer to FIG. 7 , which shows a process 700 of an image detection method provided by an embodiment of the present application. Wherein, the execution subject of the image detection method described in FIG. 7 may be the ISP processor and the AI processor described in FIG. 1 . As shown in Figure 7, the image detection method includes the following steps:

Step 701 , the image data to be detected is collected by the camera 101 .

Step 702 , using an image processing algorithm to process the image data to be detected to generate a processed image.

Step 703: Input the processed image into the image detection model to obtain a detection result.

The parameters of the image processing algorithm described in step 702 may be obtained after debugging using the parameter debugging method for image processing as described in FIG. 5 .

In a possible implementation manner, the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence levels, and prediction of motion trajectories of target objects.

It can be understood that, in order to realize the above-mentioned functions, the electronic device includes corresponding hardware and/or software modules for executing each function. The present application can be implemented in hardware or in the form of a combination of hardware and computer software in conjunction with the algorithm steps of each example described in conjunction with the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functionality for each particular application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

In this embodiment, the above one or more processors may be divided into functional modules according to the foregoing method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. . The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that, the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division manners in actual implementation.

In the case where each functional module is divided according to each function, FIG. 8 shows a possible schematic diagram of the composition of the image detection apparatus 800 involved in the above embodiment. As shown in FIG. 8 , the image detection apparatus 800 may include: Collection module 801 , processing module 802 and detection module 803 . The acquisition module 801 is configured to collect image data to be detected through the first camera device; the processing module 802 is configured to process the image data to be detected by using an image processing algorithm to generate a processed image; the detection module 803 , which is configured to input the processed image into an image detection model to obtain a detection result; wherein, the parameters of the image processing algorithm are the annotations of the first sample image data collected by comparing the first camera device The information and the detection result of the image detection model on the first sample image data are obtained by adjusting based on the comparison result.

In a possible implementation manner, the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.

In a possible implementation manner, the parameters of the image processing algorithm are determined by a parameter adjustment module, and the parameter adjustment module (not shown in the figure) includes: a comparison sub-module (not shown in the figure), which is is configured to compare the detection result of the first sample image data with the annotation information of the first sample image data to obtain the comparison result; the adjustment sub-module (not shown in the figure) is configured to be based on The comparison result iteratively adjusts the parameters of the image processing algorithm; the saving sub-module (not shown in the figure) is configured to save the parameters of the image processing algorithm when a preset condition is satisfied.

In a possible implementation manner, the comparison result is an error, and the adjustment sub-module (not shown in the figure) is further configured to: based on the detection result of the first sample image data and the first The error between the annotation information of the sample image data is used to construct a target loss function, wherein the target loss function includes the parameters to be updated in the image processing algorithm; based on the target loss function, using the back propagation algorithm and A gradient descent algorithm that iteratively updates the parameters of the image processing algorithm.

In a possible implementation manner, the image processing algorithm includes at least one of the following: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, and image noise reduction.

In a possible implementation manner, the parameters of the image processing algorithm include at least one of the following: the distance between each pixel of the image in the lens shading correction algorithm and the optical center of the camera device; the neutral color area in the white balance correction algorithm is in the image the boundary coordinates in the tone mapping algorithm; the target brightness, target saturation, and filter kernel parameters used to generate the low-pass filtered image in the tone mapping algorithm; the contrast threshold in the contrast enhancement algorithm; the edge enhancement factor in the image edge enhancement algorithm; and the image The spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the noise reduction algorithm.

In a possible implementation manner, the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include: weight coefficients of the neural network used to generate the image processing model.

In a possible implementation manner, the annotation information of the first sample image data is manually annotated; and the apparatus further includes: a conversion module (not shown in the figure) configured to convert the first sample image data into The sample image data is converted into color images suitable for human annotation.

The image detection apparatus 800 provided in this embodiment is configured to execute the image detection method executed by the electronic device 100, and can achieve the same effect as the above-mentioned implementation method. The modules corresponding to FIG. 8 can be implemented in software, hardware or a combination of the two. For example, each module can be implemented in software to drive the ISP 102 and the AI processor 103 in the electronic device 100 shown in FIG. 1 . Alternatively, each module may include a corresponding processor and a corresponding driver software.

In the case where each functional module is divided according to each function, FIG. 9 shows a possible schematic diagram of the composition of the parameter debugging apparatus 900 for image processing involved in the above embodiment. As shown in FIG. The image processing parameter debugging apparatus 900 may include: a processing module 901 , a detection module 902 , a comparison module 903 and an adjustment module 904 . The processing module 901 is configured to perform image processing on the first sample image data by using an image processing algorithm to generate first image data, wherein the first sample image data is collected by a first camera; the detection module 902 , is configured to input the first image data into a pre-trained image detection model to obtain a detection result; a comparison module 903 is configured to compare the detection result with the annotation information of the first sample image data The error is obtained, and a comparison result is obtained; the adjustment module 904 is configured to adjust the parameters of the image processing algorithm based on the comparison result.

In a possible implementation manner, the comparison result is an error, and the adjustment module is configured to: construct a target based on the error between the detection result and the annotation information of the first sample image data A loss function, wherein the target loss function includes parameters to be adjusted in the image processing algorithm; based on the target loss function, the parameters of the image processing algorithm are iteratively adjusted by using a backpropagation algorithm and a gradient descent algorithm.

In a possible implementation manner, the image processing algorithm includes at least one of the following: dark current correction, lens shading correction, demosaicing, white balance correction, tone mapping, contrast enhancement, image edge enhancement, or image noise reduction.

Where an integrated unit is employed, the image detection apparatus 800 may include at least one processor and memory. Wherein, at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the electronic device 100, for example, it can be used to support the electronic device 100 to perform the steps performed by the above-mentioned modules. The memory may be used to support the execution of the electronic device 100 by storing program codes and data, and the like. The processor may implement or execute various exemplary logic modules described in conjunction with the present disclosure, which may be a combination of one or more microprocessors that implement computing functions, such as, but not limited to, the image signal shown in FIG. 1 . Processor 101 and AI Processor 103. In addition, the microprocessor combination may also include a central processing unit, a controller, and the like. In addition, the processor may include other programmable logic devices, transistor logic devices, or discrete hardware components in addition to the processors shown in FIG. 1 . The memory may include random access memory (RAM) and read only memory (ROM), among others. The random access memory may include volatile memory (such as SRAM, DRAM, DDR (Double Data Rate SDRAM, Double Data Rate SDRAM) or SDRAM, etc.) and non-volatile memory. The RAM can store data (such as image processing algorithms, etc.) and parameters required for the operation of the ISP102 and the AI processor 103, intermediate data generated by the operation of the ISP102 and the AI processor 103, image data processed by the ISP102, and the AI processor 103. The output after running, etc. Executable programs of the ISP 102 and the AI processor 103 may be stored in the read-only memory ROM. Each of the above components can perform their own work by loading an executable program. The executable program stored in the memory can execute the image detection method as described in FIG. 7 .

In the case of an integrated unit, the image detection apparatus 900 may include at least one processor and storage device. Wherein, at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the parameter debugging device 401 as shown in FIG. step. The memory may be used to support the execution of the parameter debugging device 401 to store program codes and data, and the like. The processor can implement or execute various exemplary logic modules described in conjunction with the disclosure of the present application, which can be one or more microprocessor combinations that implement computing functions, including but not limited to a central processing unit and a controller, etc. . In addition, the processor may also include other programmable logic devices, transistor logic devices, or discrete hardware components, or the like. The memory may include random access memory (RAM), read only memory ROM, and the like. The random access memory can include volatile memory (such as SRAM, DRAM, DDR (Double Data Rate SDRAM, Double Data Rate SDRAM) or SDRAM, etc.) and non-volatile memory. The RAM may store data (such as image processing algorithms) and parameters required for the operation of the parameter debugging device 401, intermediate data generated by the parameter debugging device 401, and output results after the parameter debugging device 401 runs. An executable program of the parameter debugging device 401 may be stored in the read-only memory ROM. Each of the above components can perform their own work by loading an executable program. The executable program stored in the memory can execute the parameter adjustment method described in FIG. 5 or FIG. 6 .

This embodiment further provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on the computer, the computer executes the above-mentioned related method steps to realize the image detection in the above-mentioned embodiment. The image detection method of the apparatus 800, or the parameter adjustment method of the parameter adjustment apparatus 900 in the above-mentioned embodiment is implemented.

This embodiment also provides a computer program product, when the computer program product is run on a computer, it causes the computer to execute the above-mentioned relevant steps, so as to realize the image detection method of the image detection apparatus 800 in the above-mentioned embodiment, or to realize the above-mentioned embodiment. The parameter adjustment method of the parameter adjustment device 900 in FIG.

Wherein, the computer-readable storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, for the beneficial effect that can be achieved, reference may be made to the corresponding method provided above. The beneficial effects will not be repeated here.

From the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated by different The function module is completed, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (which may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. that can store program codes. medium.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application. scope.

Claims

An image detection method, comprising:

Collect image data to be detected by the first camera device;

The image data to be detected is processed by using an image processing algorithm to generate a processed image, wherein the parameters of the image processing algorithm are the labels of the first sample image data collected by comparing the first camera device The information and the image detection model are obtained by adjusting the detection results of the first sample image data and based on the comparison results;

The processed image is input into the image detection model to obtain a detection result.
The image detection method according to claim 1, wherein the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.
The image detection method according to claim 1 or 2, wherein the parameters of the image processing algorithm are determined by the following steps:

comparing the detection result of the first sample image data with the annotation information of the first sample image data to obtain the comparison result;

Iteratively adjust parameters of the image processing algorithm based on the comparison results;

When the preset condition is satisfied, the parameters of the image processing algorithm are saved.
The image detection method according to claim 3, wherein the comparison result is an error, and the iteratively adjusting the parameters of the image processing algorithm based on the comparison result comprises:

Based on the error between the detection result of the first sample image data and the annotation information of the first sample image data, a target loss function is constructed, wherein the target loss function includes the target loss function to be used in the image processing algorithm. Adjusted parameters;

Based on the objective loss function, the parameters of the image processing algorithm are iteratively adjusted using a back-propagation algorithm and a gradient descent algorithm.
The image detection method according to any one of claims 1-4, wherein the image processing algorithm comprises at least one of the following: dark current correction algorithm, lens shading correction algorithm, demosaicing algorithm, white balance correction algorithm, Tone mapping algorithm, contrast enhancement algorithm, image edge enhancement algorithm and image noise reduction algorithm.
The image detection method according to claim 5, wherein the parameters of the image processing algorithm include at least one of the following:

The distance between each pixel of the image and the optical center of the camera in the lens shading correction algorithm;

The boundary coordinates of the neutral color area in the image in the white balance correction algorithm;

Target brightness, target saturation and filter kernel parameters used to generate low-pass filtered images in the tone mapping algorithm;

Contrast threshold in contrast enhancement algorithm;

the edge enhancement factor in the image edge enhancement algorithm; and

The spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the image noise reduction algorithm.
The image detection method according to claim 5, wherein the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include:

Weight coefficients for the neural network used to generate the image processing model.
The image detection method according to any one of claims 1-7, wherein the annotation information of the first sample image data is manually annotated; and

The method also includes:

Converting the first sample image data into a color image suitable for manual annotation.
The image detection method according to any one of claims 1-8, wherein the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence, Prediction of the motion trajectory of the target object.
An image detection device, characterized in that it includes:

an acquisition module, configured to acquire image data to be detected through the first camera device;

a processing module configured to process the image data to be detected by using an image processing algorithm to generate a processed image, wherein the parameters of the image processing algorithm are obtained by comparing the first image data collected by the first camera device The annotation information of the sample image data and the detection result of the image detection model on the first sample image data are obtained by adjusting based on the comparison result;

The detection module is configured to input the processed image to the image detection model to obtain a detection result.
The image detection device according to claim 10, wherein the image detection model is obtained by performing neural network training on the second sample image data collected by the second camera device.
The image detection device according to claim 10 or 11, wherein the parameters of the image processing algorithm are determined by a parameter adjustment module, and the parameter adjustment module comprises:

a comparison sub-module configured to compare the detection result of the first sample image data with the annotation information of the first sample image data to obtain the comparison result;

an adjustment submodule configured to iteratively adjust parameters of the image processing algorithm based on the comparison result;

The saving sub-module is configured to save the parameters of the image processing algorithm when the preset condition is satisfied.
The image detection device according to claim 12, wherein the comparison result is an error, and the adjustment sub-module is further configured to:

Based on the error between the detection result of the first sample image data and the annotation information of the first sample image data, a target loss function is constructed, wherein the target loss function includes the target loss function to be used in the image processing algorithm. updated parameters;

Based on the objective loss function, the parameters of the image processing algorithm are iteratively updated using a back-propagation algorithm and a gradient descent algorithm.
The image detection device according to any one of claims 10-13, wherein the image processing algorithm includes at least one of the following image processing procedures: dark current correction, lens shading correction, demosaicing, white balance correction, tone Mapping, contrast enhancement, image edge enhancement and image noise reduction.
The image detection device according to claim 14, wherein the parameters of the image processing algorithm include at least one of the following:

The distance between each pixel of the image and the optical center of the camera in the lens shading correction algorithm;

The boundary coordinates of the neutral color area in the image in the white balance correction algorithm;

Target brightness, target saturation and filter kernel parameters used to generate low-pass filtered images in the tone mapping algorithm;

Contrast threshold in contrast enhancement algorithm;

the edge enhancement factor in the image edge enhancement algorithm; and

The spatial domain Gaussian parameter and the pixel value domain Gaussian parameter in the image noise reduction algorithm.
The image detection device according to claim 14, wherein the image processing algorithm is executed by a trained image processing model; the parameters of the image processing algorithm include:

Weight coefficients for the neural network used to generate the image processing model.
The image detection apparatus according to any one of claims 10-16, wherein the labeling information of the first sample image data is manually labelled; and

The device also includes:

A conversion module configured to convert the first sample image data into a color image suitable for manual annotation.
The image detection device according to any one of claims 10-17, wherein the image detection model is used to perform at least one of the following detection tasks: labeling of detection frames, recognition of target objects, prediction of confidence levels, Prediction of the motion trajectory of the target object.
An electronic device, comprising:

a first camera device for collecting image data to be detected;

an image signal processor, configured to process the image data to be detected by using an image processing algorithm to generate a processed image;

an artificial intelligence processor for inputting the processed image into an image detection model to obtain a detection result;

The parameter of the image processing algorithm is a result of comparing the annotation information of the first sample image data collected by the first camera device with the detection result of the first sample image data by the image detection model, and Adjusted based on comparison results.
An image detection device, characterized in that it includes:

one or more processors and memories;

the memory is coupled to the processor, the memory for storing one or more programs;

The one or more processors are configured to run the one or more programs to implement the method of any one of claims 1-9.
A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by at least one processor, is used to implement the method according to any one of claims 1-9. method.
A computer program product, characterized in that, when the computer program product is executed by at least one processor, it is used to implement the method according to any one of claims 1-9.