WO2023035263A1 - 确定图像信号处理参数的方法、装置和感知系统 - Google Patents

确定图像信号处理参数的方法、装置和感知系统 Download PDF

Info

Publication number
WO2023035263A1
WO2023035263A1 PCT/CN2021/117953 CN2021117953W WO2023035263A1 WO 2023035263 A1 WO2023035263 A1 WO 2023035263A1 CN 2021117953 W CN2021117953 W CN 2021117953W WO 2023035263 A1 WO2023035263 A1 WO 2023035263A1
Authority
WO
WIPO (PCT)
Prior art keywords
simulator
image
parameter
signal processing
image signal
Prior art date
Application number
PCT/CN2021/117953
Other languages
English (en)
French (fr)
Inventor
刘劲松
柳海波
邱珏沁
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2021/117953 priority Critical patent/WO2023035263A1/zh
Priority to CN202180102253.8A priority patent/CN117940951A/zh
Publication of WO2023035263A1 publication Critical patent/WO2023035263A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining

Definitions

  • Embodiments of the present disclosure generally relate to the field of computers. More specifically, the embodiments of the present disclosure relate to methods, devices, devices, computer-readable storage media, and computer program products for determining image signal processing parameters.
  • ISP image signal processing
  • CV Computer vision
  • Embodiments of the present disclosure provide a solution for determining image signal processing parameters.
  • a method of determining image signal processing parameters includes: determining a simulator for simulating an image signal processing device, the simulator comprising operations having a differentiable representation; and updating the simulator by jointly training the simulator and a neural network associated with the image signal processing device A first parameter set for the trained neural network is used to perform a computer vision perception task on the image output by the image signal processing device; and based on the updated first parameter set, determine a second parameter set for the image signal processing device.
  • the end-to-end joint training with the CV perception network is realized by using a simulator that simulates the ISP device, so as to simultaneously optimize the parameters of the ISP and the CV perception network. In this way, the resulting ISP parameters are optimized for the CV perception task, thus helping to improve the performance of the CV perception task.
  • determining the simulator includes: generating an approximate version of the simulator based on the differentiability of operations in the operation set of the image signal processing device; processing the same image based on the image signal processing device and the approximate version diffing, verifying the approximate version; and identifying the approximate version as the emulator if the approximate version passes verification. In this way, the effectiveness of the simulator in simulating the operation of the image signal processing device can be ensured.
  • generating the approximate version of the simulator includes: if it is determined that the first operation in the operation set has a differentiable expression, retaining the first operation in the approximate version. In this way, an appropriate strategy can be taken for the characteristics of the original operation in the image signal processing device. This avoids an approximation to an otherwise differentiable operation, which can save the cost of generating a simulator.
  • generating the approximate version of the simulator includes: if it is determined that the second operation in the operation set has a non-differentiable representation, determining whether the importance of the second operation is higher than a threshold importance, important Indicating the effect of the second operation on the result or image quality of the computer vision perception task; and if the importance of the second operation is determined to be higher than a threshold importance, expressing the second operation in an approximate version by a differentiable function denoted by to approximate the second operation.
  • the differentiable function includes a piecewise linear function. In this way, the operation in the image signal processing device can be approximated simply and efficiently using a piecewise linear function.
  • verifying the approximate version includes: for the verification image in the verification image set captured by the image sensor, obtaining a first attribute value of an image attribute obtained by processing the verification image by the image signal processing device; obtaining a second attribute value of an image attribute resulting from processing the verification image by the approximate version; determining a difference based on the first attribute value and the second attribute value; and verifying the approximate version based on the respectively determined differences for each verification image in the verification image set .
  • similarity verification of the image signal processing device and the simulator can be performed simply and accurately using image attributes such as RGB values and the like.
  • verifying the approximate version based on the respectively determined differences includes: if the respectively determined differences are all smaller than a first threshold and the sum of the differences is smaller than a second threshold, determining that the approximate version passes the verification. In this way, by taking into account both the differences for individual verification images and the cumulative differences for the set of verification images, it is possible to verify the simulator with strict criteria. This further ensures the effectiveness of the simulator.
  • determining the second parameter set of the image signal processing device includes: if it is determined that the operation including the first parameter in the second parameter set is represented by a differentiable function in the simulator, based on the differentiable The value of the parameter of the function is used to determine the value of the first parameter; and if it is determined that the operation including the second parameter in the second parameter set is retained in the simulator and is not represented by a differentiable function, based on the second parameter stored in the image signal processing device
  • the specification of the register of the parameter converts the value of the third parameter corresponding to the second parameter in the simulator into the value of the second parameter.
  • the image signal processing parameters are determined according to how the operations to which the parameters belong are processed in the simulator. In this way, it can be ensured that the parameters of the optimized simulator can be accurately reflected back into the image signal processing device.
  • the method further includes: setting an image signal processing device based on the determined second parameter set. In this way, the image signal processing device is made to process data according to optimized parameters.
  • updating the first parameter set of the simulator includes: determining a processed image obtained by processing an original image by the simulator, and the original image is captured by an image sensor; performing the outcome of the computer perception task and the objective function, determining a task loss; and updating a second set of parameters of the simulator by minimizing the task loss. In this way, an end-to-end joint optimization with a CV-aware network is achieved.
  • an apparatus for determining image signal processing parameters includes: a simulator determining unit configured to determine a simulator for simulating an image signal processing device, the operations included in the simulator have differentiable expressions; a simulator parameter updating unit configured to simulate A device and a neural network associated with the image signal processing device to update the first parameter set of the simulator, the trained neural network is used to perform a computer vision perception task on an image output by the image signal processing device; and a processing parameter determination unit , configured to determine a second parameter set of the image signal processing device based on the updated first parameter set.
  • the simulator determining unit is further configured to: generate an approximate version of the simulator based on the differentiability of operations in the operation set of the image signal processing device; based on the image signal processing device and the approximate version processing a difference of the same image, verifying the approximate version; and identifying the approximate version as a simulator if the approximate version passes verification.
  • the simulator determination unit is further configured to: if it is determined that the first operation in the operation set has a differentiable expression, keep the first operation in the approximate version.
  • the simulator determination unit is further configured to: if it is determined that the second operation in the operation set has a non-differentiable expression, determine whether the importance of the second operation is higher than a threshold importance, Importance indicates the effect of the second operation on the result or image quality of the computer vision perception task; and if the importance of the second operation is determined to be higher than a threshold importance, the second operation is represented in an approximate version by a differentiable function, the differentiable function Used to approximate the second operation.
  • the differentiable function includes a piecewise linear function.
  • the simulator determining unit is further configured to: for the verification images in the verification image set captured by the image sensor, obtain the first image attribute obtained by processing the verification image by the image signal processing device an attribute value; obtaining a second attribute value of an image attribute resulting from processing the verification image by an approximate version; determining a difference based on the first attribute value and the second attribute value; and based on the respectively determined differences for each verification image in the verification image set , to verify the approximate version.
  • the simulator determination unit is further configured to: determine that the approximate version passes the verification if the respectively determined differences are less than the first threshold and the sum of the differences is less than the second threshold.
  • the processing parameter determination unit is further configured to: if it is determined that the operation including the first parameter in the second parameter set is represented by a differentiable function in the simulator, based on the parameters of the differentiable function to determine the value of the first parameter; and if it is determined that the operation including the second parameter in the second parameter set is retained in the simulator and is not represented by a differentiable function, based on a register storing the second parameter in the image signal processing device.
  • the specification of converts the value of the third parameter corresponding to the second parameter in the simulator to the value of the second parameter.
  • the device further includes: a setting unit configured to set the image signal processing device based on the determined second parameter set.
  • the simulator parameter update unit is further configured to: determine the processed image obtained by processing the original image by the simulator, the original image is captured by the image sensor; performing the outcome of the computer perception task and the objective function, determining a task loss; and updating a second set of parameters of the simulator by minimizing the task loss.
  • a perception system in a third aspect of the present disclosure, includes: an image signal processing device configured to process data from the image sensor to generate a processed image; and a neural network coupled to the image signal processing device and configured to perform a computer vision perception task on the processed image , to generate perceptual results.
  • the parameter set of the image signal processing device is determined by the method in any implementation manner of the first aspect.
  • an electronic device comprising: at least one processor; at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor.
  • the electronic device executes the method in any one implementation manner of the first aspect.
  • a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method in any one of the implementation manners of the first aspect is implemented.
  • a computer program product which is characterized by including computer-executable instructions, wherein the computer-executable instructions implement the method in any one of the implementation manners of the first aspect when executed by a processor .
  • the apparatus of the second aspect, the perception system of the third aspect, the electronic device of the fourth aspect, the computer storage medium of the fifth aspect, or the computer program product of the sixth aspect provided above are all used to execute the provided method. Therefore, the explanations or descriptions about the first aspect are also applicable to the second aspect, the third aspect, the fourth aspect, the fifth aspect and the sixth aspect.
  • the beneficial effects achieved by the second aspect, the third aspect, the fourth aspect, the fifth aspect, and the sixth aspect can refer to the beneficial effects in the corresponding methods, and will not be repeated here.
  • Figure 1 shows a schematic diagram of an example system to which various embodiments of the present disclosure can be applied
  • FIG. 2 shows a schematic diagram of a system architecture for determining ISP parameters according to some embodiments of the present disclosure
  • Fig. 3 shows a schematic diagram of a process of determining a simulator according to some embodiments of the present disclosure
  • FIG. 4 shows a schematic diagram of a process of generating an approximate version of a simulator according to some embodiments of the present disclosure
  • Fig. 5 shows a schematic diagram of approximating operations with piecewise linear functions according to some embodiments of the present disclosure
  • FIG. 6 shows a flowchart of a process of verifying an approximate version according to some embodiments of the present disclosure
  • FIG. 7 shows a schematic diagram of jointly training a simulator and a neural network according to some embodiments of the present disclosure
  • Fig. 8 shows the schematic diagram of training neural network in traditional scheme
  • FIG. 9 shows a schematic diagram of a deployment awareness system according to some embodiments of the present disclosure.
  • FIG. 10 shows a schematic diagram of a process of configuring ISP parameters according to some embodiments of the present disclosure
  • Figure 11 shows a flowchart of an example method of determining ISP parameters according to some embodiments of the present disclosure
  • FIG. 12 shows a schematic block diagram of an apparatus for determining ISP parameters according to some embodiments of the present disclosure.
  • Figure 13 shows a block diagram of a computing device capable of implementing various embodiments of the present disclosure.
  • the term “comprising” and its similar expressions should be interpreted as an open inclusion, that is, “including but not limited to”.
  • the term “based on” should be understood as “based at least in part on”.
  • the term “one embodiment” or “the embodiment” should be read as “at least one embodiment”.
  • the terms “first”, “second”, etc. may refer to different or the same object.
  • the term “and/or” means at least one of the two items associated with it. For example "A and/or B" means A, B, or A and B. Other definitions, both express and implied, may also be included below.
  • a "neural network” is capable of processing input and providing a corresponding output, which generally includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer.
  • Neural networks used in deep learning applications often include many hidden layers, extending the depth of the network. The layers of the neural network are connected in sequence so that the output of the previous layer is provided as the input of the subsequent layer, where the input layer receives the input of the neural network, and the output of the output layer serves as the final output of the neural network.
  • the terms "neural network”, “network”, “neural network model” and “model” are used interchangeably.
  • determining a parameter refers to determining the value of a parameter, including a specific value, a set of values, or a range of values.
  • Determining a parameter set refers to determining the value of a parameter in a parameter set, including a specific value, a set of values, or a range of values, etc.
  • FIG. 1 shows a schematic diagram of an example perception system 100 to which various embodiments of the present disclosure can be applied.
  • the perception system 100 includes an ISP device 110 and a neural network 120 .
  • the ISP device 110 in the form of hardware is used to process sensor data 101 captured by the image sensor of the front-end to generate a processed image 102 .
  • ISP device 110 may be, for example, an image signal processor, or any other suitable ISP device.
  • the processing of the sensor data 101 by the ISP device 110 may include, but is not limited to, black level correction (BLC), demosaic, sharpening, lens shading correction, bad pixel correction, and the like.
  • BLC black level correction
  • demosaic demosaic
  • sharpening lens shading correction
  • bad pixel correction bad pixel correction
  • Each of the above functions may include one or more operations to implement corresponding processing. Each operation may in turn include one or more parameters, also referred to herein as "ISP parameters.”
  • the ISP parameters of the individual functions constitute a parameter set of the ISP device 110, which is also referred to herein as an "ISP parameter set”.
  • the values of these ISP parameters are stored in registers of the ISP device 110 .
  • neural network 120 is trained.
  • the trained neural network 120 is used to perform a CV perception task on the processed image 102 output by the ISP device 110 to output a perception result 103 .
  • the neural network 120 can be used to detect and recognize target objects from the processed image 120 .
  • the neural network 120 may also be called a perceptual network 120 .
  • neural network 120 may be implemented using any suitable type of network, such as convolutional neural network, recurrent neural network, regional convolutional neural network (R-CNN), fast R-CNN, and the like.
  • the neural network 120 also includes one or more parameters, also referred to as "network parameters.”
  • a parameter set of these parameters (also referred to as a “network parameter set”) is determined during training of the neural network 120 and remains constant during deployment and use of the perception system 100 .
  • ISP parameters and network parameters have an impact on the accuracy of the perception result 103 .
  • the debugging of ISP parameters is carried out manually.
  • debugging experts analyze and debug according to subjective and objective aspects such as human eye preference brightness and color accuracy.
  • the debugged set of ISP parameters is solidified into the ISP device, and an image set is output using the set of ISP parameters. This image set is provided to the subsequent perception network to fine-tune or retrain the perception network.
  • This traditional solution has many drawbacks.
  • the debugging of ISP parameters is aimed at the pleasure of human eyes, and has nothing to do with perceived performance.
  • the manual debugging of ISP parameters takes a long time, the cost of manpower and time is high, and different people have different debugging solutions, resulting in extremely low debugging efficiency.
  • the optimization of the perception network does not take into account the variables of the ISP, and does not form an optimization link between the ISP and the CV perception task. As a result, the optimization order and the optimization goal are separated, which cannot improve the performance of the CV perception task end-to-end.
  • This ISP with network structure mainly includes a Visual Local Modulation Map (VLTM) part and a Trainable Visual Scaling (TVS) part.
  • VLTM Visual Local Modulation Map
  • TVS Trainable Visual Scaling
  • the VLTM part is composed of several traditional simple calculation modules. Since the calculations are all simple continuous functions, it maintains the ability of automatic derivation.
  • the TVS part is a convolutional neural network structure, which is used to complete functions such as feature extraction and scaling.
  • the TVS part can be coupled with the perception network of the subsequent stage.
  • the output image is presented in the form of a feature map, making it invisible to human eyes.
  • the non-network structure part is a simple image processing module, which does not meet the high requirements of consumer-grade ISPs.
  • the input is an RGB image instead of a raw (raw) image, so it is not a complete end-to-end optimization of the ISP and perception network.
  • a simulator for simulating an image signal processing device is first determined.
  • the operations included in the simulator have a differentiable representation.
  • the parameter set of the simulator is updated by jointly training the simulator and the neural network associated with the image processing signal device.
  • the trained neural network is used to perform a computer vision perception task on the image output by the image signal processing device.
  • the parameter set of the image signal processing device is determined.
  • the end-to-end joint training with the CV perception network is realized by using a simulator that simulates the ISP device, so as to simultaneously optimize the parameters of the ISP and the CV perception network.
  • a simulator that simulates the ISP device, so as to simultaneously optimize the parameters of the ISP and the CV perception network.
  • This solves the problem of separate optimization of ISP and CV perception networks.
  • the resulting ISP parameters are optimized for the CV perception task, thus helping to improve the performance of the CV perception task.
  • this also shortens the labor and time costs of ISP parameter debugging and improves efficiency.
  • FIGS. 2 to 13 Various example embodiments of the present disclosure are described below with reference to FIGS. 2 to 13 .
  • FIG. 2 shows a schematic diagram of a system architecture 200 for determining ISP parameters according to some embodiments of the present disclosure.
  • the system architecture 200 includes an inheritance process 250 , an optimization process 260 and a deployment process 270 .
  • the inheritance process 250 is used to determine the ISP simulator 210 that simulates the ISP device 110 .
  • the operations included in the ISP simulator 210 have differentiable representations.
  • the differentiable ISP simulator 210 simulates the ISP device 110 by inheriting all or most of the operations of the ISP device 110 and has similar parameter behavior to the ISP device 110 . It can be understood that since the ISP simulator 210 inherits the operation of the ISP device 110 , the ISP simulator 210 also has corresponding parameters of the inherited operation.
  • the parameters of the ISP simulator 210 are also referred to as “simulation parameters”, and the parameter sets of the ISP simulator 210 are also referred to as “simulation parameter sets”.
  • the ISP simulator 210 can be regarded as a soft proxy of the hardware ISP, and thus can also be called a "soft proxy ISP”.
  • the ISP device 110 includes a processing pipeline composed of different functions.
  • hardware functions 201 - 1 , 201 - 2 , 201 - 3 are shown in FIG. 2 , which may be individually or collectively referred to as hardware functions 201 .
  • the ISP simulator 210 includes differentiable functions 203-1, 203-2, and 203-3 respectively corresponding to the hardware functions 201-1, 201-2, and 201-3, which may be individually or collectively referred to as differentiable functions 203.
  • Each differentiable function 203 is the same as or similar to the corresponding hardware function 201 .
  • ISP simulator 210 may be verified to ensure that ISP simulator 210 is sufficiently similar to ISP device 110 .
  • the inheritance process 250 may include an approximation process 251 and a verification process 252 , as shown in FIG. 2 .
  • the approximation process 251 is used to generate an approximate version 205 of the ISP simulator based on the differentiability of each operation in the set of operations of the ISP device. That is, the approximation process 251 is used to approximate a differentiable ISP simulator to be verified from the perspective of the ISP device 110 .
  • Approximate version 205 includes differentiable functions 202-1, 202-2, 202-3 corresponding to hardware functions 201-1, 201-2, 201-3 respectively, which may be individually or collectively referred to as differentiable functions 202. Each differentiable function 202 is derived by approximating the corresponding hardware function 201 . An example process for generating an approximate version will be described below with reference to FIG. 4 .
  • the approximate version 205 is verified based on the difference of the same image processed by the ISP device 110 and the approximate version 205 . That is, the verification process 252 is used to verify the similarity of the approximate version 205 to the ISP device 110 . If the approximation 205 fails validation, the approximation process 251 needs to be re-executed, and the approximation process 251 may need to be adjusted. If the approximate version 205 passes the verification, the approximate version 205 is determined as the ISP simulator 210 to be used subsequently. In such an embodiment, the validity of the approximation process and ISP simulator 210 can be ensured using a verification process.
  • the determined ISP simulator 210 will be used in the next optimization process 260 .
  • the ISP simulator 210 and the neural network 120 are jointly trained based on the objective function 220 .
  • the simulation parameter set of the ISP simulator 210 and the network parameter set of the neural network 120 are updated.
  • the differentiable ISP simulator 210 performs forward propagation together with the neural network 120 and the objective function 220, thereby optimizing the parameters of the ISP simulator 210 and the parameters of the neural network 120 end-to-end . In this way, the updated simulation parameter set is optimized for the perception task.
  • the ISP device 110 is configured with the updated simulation parameter set of the ISP simulator 210 .
  • the ISP simulator 210 is an approximation to the ISP device 110
  • some operating parameters may be different from the corresponding parameters in the ISP device 110 in terms of value ranges, value requirements, quantization intervals, and the like. Therefore, in order to configure the ISP device 110, the ISP parameter set may first be determined based on the simulated parameter set, ie, the value of the ISP parameter is determined. Then, the ISP device 110 is set based on the determined set of ISP parameters, eg, values stored in various registers of the ISP device 110 are set. The ISP device 110 thus configured together with the trained neural network 120 achieves CV perception.
  • FIG. 3 shows a schematic diagram of a process 300 of determining an ISP simulator according to some embodiments of the present disclosure. Process 300 may be considered an example implementation that inherits process 250 .
  • a code describing the ISP device 110 also referred to as a history code, is obtained.
  • codes may include, but are not limited to, various suitable chip design codes, such as C-model, verilog, and the like.
  • the approximation process 251 is performed to generate the approximation version 205 .
  • the operations in the ISP simulator 210 are determined based on the differentiability of each operation of the ISP device 110 described by the design code. In this way, the differentiable function 202 corresponding to the hardware function 201 is generated.
  • the process 400 described below may be performed for each operation in the hardware function 201, as will be described in detail below.
  • the approximate version 205 is verified. If the approximate version 205 fails the verification, the process 300 returns to block 320, and the approximation process 251 needs to be re-executed to generate a new approximate version. In such cases, the approximation process 251 may need to be adjusted. If the approximate version 205 passes verification, the process 300 proceeds to block 340 . At block 340, the approximate version 205 is determined as the ISP simulator 210 to be used subsequently.
  • the process 300 may also include other actions not shown.
  • the approximate version 205 in code form may be converted into a programming language for subsequent training with the neural network 120 .
  • the approximate version 205 can be programmed under a deep learning framework such as pytorch.
  • the ISP emulator is determined 340 using the historical code. Since ISP devices generally have complex operations, in this way an ISP simulator can be obtained in a simple manner, reducing the cost of developing the ISP simulator.
  • the perception-oriented pipeline can be redeveloped for the functions to be realized by the ISP device 110, so that each operation of the pipeline is differentiable . Then, the redeveloped pipeline can be translated into a procedural language.
  • the hardware ISP device with complex operation can be inherited by using the inheritance process, which solves the problem that the traditional hardware ISP device is difficult to continuously obtain derivatives.
  • the obtained ISP simulator has the characteristics of interpretability, controllable expression space and easy convergence.
  • Process 400 may be considered as a specific implementation of block 320 .
  • Process 400 may be performed for each operation in hardware function 201 until process 400 is complete for all operations, thereby generating approximate version 205 .
  • the operation currently under consideration also referred to as the current operation
  • process 400 proceeds to block 460 .
  • the operation is removed from the approximation 205 . This operation will not be preserved in approximate version 460. By removing unimportant operations, the complexity of the approximation process can be reduced without compromising subsequent optimization for perception tasks.
  • process 400 proceeds to block 430 .
  • process 400 proceeds to block 440 .
  • process 400 may proceed to block 440 if the original expression of the operation is not differentiable everywhere or only a small part is differentiable.
  • the operation is represented in an approximate version by a differentiable function. That is, the operation is replaced by a differentiable function.
  • a differentiable function for approximating the operation may be determined first, and then the operation is replaced by the determined differentiable function.
  • the differentiable function may be a piecewise linear (PWL) function.
  • the PWL function is an efficient approximation and fitting of piecewise functions that have discontinuities and are not differentiable.
  • the ISP device 110 there is an operation of reading a lookup table.
  • the key (Key) is used to determine the corresponding value (Value).
  • the keys and values in a lookup table are usually discrete and not contiguous. Therefore, the operation of reading the lookup table is non-differentiable and suitable to be approximated by the PWL function.
  • FIG. 5 shows a schematic diagram of an approximate operation using a PWL function according to some embodiments of the present disclosure.
  • Points shown in Figure 5, such as point 510, represent key-value pairs in the lookup table.
  • the PWL function 520 can approximate the relationship of the key-value pairs in the lookup table, so the operation of reading the lookup table can be represented by the PWL function 520 in the approximation version 205 .
  • process 400 is exemplary only, and is not intended to limit the scope of the present disclosure. Variations can be made to process 400 . In some embodiments, blocks 420 and 460 may not be included. That is, all operations of the ISP device 110 will be preserved or approximated in the approximated version.
  • the process 400 can be viewed as generating a concrete implementation of the ISP simulator 210 for subsequent use. That is, the process 400 may be performed for each operation in the hardware function 201 until the process 400 is completed for all operations, thereby generating the ISP simulator 210 .
  • a verification process 252 may be performed on the approximate version 205 in some embodiments.
  • the approximate version 205 is verified based on the differences in how the ISP device 110 and the approximate version 205 process the same image.
  • differences in image properties resulting from processing the same image by the ISP device 110 and the approximate version 205 may be used as a verification criterion.
  • a set of verification images captured by an image sensor can be obtained.
  • the validation images in the validation image set are raw data captured by the image sensor.
  • the attribute value also referred to as "first attribute value”
  • the attribute value also called "second attribute value”
  • Differences in processing of the same image by ISP device 110 and approximate version 205 may be determined based on the first property value and the second property value.
  • the approximate version 205 may be verified based on differences determined respectively for each verification image in the verification image set.
  • the approximate version 205 fails validation if the respective determined differences exceeding a threshold number of differences are greater than a threshold; otherwise, the approximate version 205 passes validation.
  • the threshold number may eg be 1, or depend on the number of validation images in the validation image set.
  • the sum of the individual differences determined separately, ie the cumulative difference may also be considered. Where the cumulative difference is less than the corresponding threshold, the approximate version 205 passes validation.
  • FIG. 6 shows a flowchart of a process 600 of verifying an approximate version, according to some embodiments of the present disclosure.
  • Process 600 of FIG. 6 uses RGB values as an example of an image attribute. It should be understood that a similar process can be applied to other image attributes.
  • a verification image in a verification image set is obtained. For example, verification images may be sequentially retrieved from the verification image set.
  • the RGB values resulting from processing the verification image by the ISP device 110 are obtained.
  • a verification image may be input to the ISP device 110, and RGB values at respective pixels of the output image may be determined.
  • the RGB values resulting from processing the verification image by the approximation version 205 are obtained.
  • a verification image may be input to the approximation version 205 and the RGB values at each pixel of the output image determined.
  • a difference D in RGB values is determined for the current verification image, as the ISP device 110 and approximation 205 process differences for the same image.
  • the difference D thus determined may be regarded as the distance in RGB space between the image processed by the ISP device 110 and the image processed by the approximation version 205 .
  • the difference D can be calculated using a predefined metric function. Metric functions may include, for example, mean square error (MSE), difference in image statistics, etc., as the scope of the present disclosure is not limited in this respect.
  • MSE mean square error
  • After determining the difference D it is also possible to calculate the cumulative difference D_t for the set of validation images, ie the sum of the differences D determined for the considered validation images.
  • process 600 determines whether the difference D for the current verification image is less than a first threshold. If the difference D is not less than the first threshold, process 600 proceeds to block 660 . At block 660, it is determined that the approximate version 205 fails validation. If the difference D is less than the first threshold, process 600 proceeds to block 670 . At block 670, it is determined whether the current verification image is the last verification image in the set of verification images. If the current verification image is not the last verification image in the verification image set, process 600 returns to block 610 to perform blocks 610 through 650 on the next image in the verification image set.
  • process 600 proceeds to block 680 .
  • it is determined that the approximate version 205 fails validation. If it is determined that the cumulative difference D_t is less than the second threshold, process 600 proceeds to block 690 . At block 690, it is determined that the approximate version 205 passes validation.
  • block 650 may be omitted by not determining for each verification image whether the difference D is less than the first threshold.
  • block 680 may be omitted, ie, cumulative differences may not be considered.
  • it may be determined whether the number of verification images whose difference D is smaller than the first threshold exceeds the threshold number or whether the ratio exceeds the threshold ratio, as the verification standard.
  • the approximate version 205 is verified using differences in image attributes as verification criteria.
  • other verification criteria may be selected, for example, the first image and the second image obtained by processing the verification images respectively by the ISP device 110 and the approximation version 205 may be obtained. Then, feature extraction is performed on the first image and the second image, and the distance of these two images is calculated in the feature space, as the ISP device 110 and the approximation version 205 process the difference of the same image.
  • An ISP simulator with approximate parameter behavior can be obtained through the above inheritance process 251 .
  • This ISP simulator has identified differentiable properties in the approximation process. Utilizing this property on the classic deep learning framework, the derivation function and backpropagation function can be completed. This allows the ISP simulator to be optimized simultaneously with the perception network on a single framework, enabling end-to-end automatic optimization of perception performance.
  • FIG. 7 shows a schematic diagram of co-training the ISP simulator 210 and the neural network 120 according to some embodiments of the present disclosure.
  • the ISP simulator 210 and the neural network 120 can be trained and updated simultaneously in the software environment of the same deep learning framework.
  • the ISP simulator 210 is propagated to the objective function 220 to realize the forward calculation of the gradient descent method.
  • sensor data ie, raw images
  • the ISP simulator 210 processes sensor data and outputs processed training images.
  • the training images are input to the neural network 120 .
  • a task loss may be determined.
  • the reverse calculation 702 the reverse calculation of the gradient descent method is implemented, and the reverse gradient is propagated to the ISP simulator 210 .
  • the parameter set of the ISP simulator 210 and the parameter set of the neural network 120 are updated simultaneously.
  • FIG. 8 shows a schematic diagram of training a neural network in a traditional solution.
  • the forward calculation 801 in the traditional scheme propagates from the ISP device 810 of the hardware to the objective function 830, but in the reverse calculation 802, the backward gradient propagation stops at the output of the ISP device 810 and cannot propagate to ISP device 810 . Therefore, as indicated by arrow 803, only the parameter set of neural network 820 is updated.
  • end-to-end joint optimization with the perception network solves the problems of separation of traditional optimization and inconsistency of goals.
  • the tuning of ISP parameters is oriented towards perception tasks, not towards human eye pleasing. This helps to improve the performance of CV perception tasks and get more accurate perception results.
  • the ISP simulator can automatically learn parameters optimized (eg, optimal) for perceived performance.
  • the ISP device of the hardware utilizes the learned optimization parameters to provide images to the perception network in real time.
  • FIG. 9 shows a schematic diagram of a deployment awareness system according to some embodiments of the present disclosure.
  • the ISP device 110 is configured using the learned simulation parameter set 910 of the ISP simulator 210 .
  • the configured ISP device 110 processes sensor data based on optimized parameters and outputs images to the trained neural network 120 .
  • the neural network 120 utilizes images from the ISP device 110 to generate perception results for the CV perception task.
  • the approximation process 251 makes the configuration of the parameters of the ISP simulator 210 may not correspond to the hardware ISP device one by one, and some operating parameters may be different from the corresponding parameters in the ISP device 110 in terms of value range, value requirements, quantization interval, etc. .
  • an optimized ISP parameter set based on the optimized simulated parameter set is first required.
  • FIG. 10 shows a schematic diagram of a process 1000 of determining ISP parameters according to some embodiments of the present disclosure.
  • Process 1000 can be performed for each ISP parameter.
  • the original value of the parameter in the ISP device 110 is used. Since it has been determined in the verification process 252 that such manipulations have no significant impact (eg, by ablation tests), using the original value for this parameter will not have a large impact on the perceived effect.
  • process 1000 proceeds to block 1020 .
  • a value of a parameter of the differentiable function is determined based on the value of the parameter. Specifically, based on the value of the parameter of the differentiable function, the value of the differentiable function under each independent variable can be determined. Then, the differentiable function can be sampled according to the characteristics of the parameter (for example, quantization interval, etc.), and the value of the parameter can be determined based on the value at the sampling point.
  • the following uses the approximation of the PWL function shown in FIG. 5 as an example for illustration.
  • the PWL function 520 is used to approximate the operation of reading a lookup table. Assuming that the lookup table has 33 indexes, 33 sampling points are required.
  • the value range of the independent variable (that is, the abscissa) of the PWL function 520 is 0 to 1, so a sampling point can be set every 1/33 step from 0 to 1, that is, the PWL function 520 is sampled every 1/33 .
  • the value (ie, the ordinate) of the PWL function 520 at the sampling point can be used as the output value of the lookup table corresponding to the index, or the value can be scaled and used as the output value of the lookup table corresponding to the index.
  • process 1000 proceeds to block 1030 .
  • the value of the analog parameter corresponding to the parameter is converted to the value of the parameter based on the specification of the register storing the parameter in the ISP device 110.
  • the specifications of the register may include, for example, the register's value range, value requirements, quantization interval, and the like. Therefore, according to the value range, value requirement, quantization interval, etc. of the register storing the ISP parameter, the corresponding analog parameter can be quantized as the value of the ISP parameter.
  • the simulation parameters usually have a value range of [0,1], and the corresponding ISP parameter registers may have a value range of [0,16].
  • the value of the simulation parameter can be converted into the value of the ISP parameter according to the correspondence between [0,1] and [0,16].
  • the analog parameter is scaled according to the value range of the register as the value of the ISP parameter.
  • the determined set of ISP parameters is subsequently used to configure the ISP device 110 , ie, write the values of these ISP parameters into registers of the ISP device 110 .
  • the determined ISP parameter set may be loaded into the memory of the ISP device 110 in the form of a file. Then, in the ISP device 110, the values of the respective ISP parameters are read and written into the corresponding registers.
  • Example implementations of determining ISP parameters according to the present disclosure are described above with reference to FIGS. 2 to 10 . From the above description, some beneficial effects of the present disclosure can be seen.
  • the optimization achieved by the embodiments of the present disclosure is an end-to-end joint optimization associated with perceived performance. This can greatly improve perceptual performance, and optimization can be performed by high-efficiency or special-purpose computing devices (such as graphics processing units GPU or artificial intelligence chips), orders of magnitude faster than human manual optimization time.
  • the embodiments of the present disclosure Compared with the above-mentioned ISP scheme with a network structure, in the embodiments of the present disclosure, complex ISP operations can be retained and inherited, there is no need to choose a simple processing algorithm for differentiability, and the reverse backhaul can Arriving at the original graph input. In addition, there is no network structure in the embodiment of the present disclosure, and the image processed by the image signal can be viewed and recorded. Compared with the above-mentioned scheme of simulating an ISP device using an offline network, the embodiments of the present disclosure are based on differentiable approximation of hardware ISP operation, thus inheriting the interpretability of hardware ISP. This leads to better convergence with respect to both image quality and perceptual optimization.
  • FIG. 11 shows a flowchart of an example method 1100 of determining ISP parameters according to some embodiments of the present disclosure.
  • Method 1100 can be implemented by any suitable computing device or computing system. It should be understood that method 1100 may also include additional actions not shown and/or illustrated actions may be omitted. The scope of the present disclosure is not limited in this respect.
  • a simulator for simulating the image signal processing device is determined.
  • the operations included in the simulator have a differentiable representation.
  • a first set of parameters of the simulator is updated by jointly training the simulator and a neural network associated with the image signal processing device.
  • the trained neural network is used to perform a computer vision perception task on the image output by the image signal processing device.
  • a second set of parameters of the image signal processing device is determined based on the updated first set of parameters.
  • an approximate version of the simulator can be generated based on the differentiability of operations in the operation set of the image signal processing device; based on the difference between the image signal processing device and the approximate version processing the same image, verify the approximate version; and, if the approximate version passes verification, identifying the approximate version as an emulator.
  • the first operation in the operation set has a differentiable expression, the first operation is retained in the approximate version.
  • the second operation in the operation set has a non-differentiable expression
  • the first attribute value of the image attribute obtained by processing the verification image by the image signal processing device A second attribute value of the image attribute resulting from the verification image is processed; based on the first attribute value and the second attribute value, a difference is determined.
  • the approximate version may be verified based on the differences respectively determined for each verification image in the verification image set.
  • the approximate version when the approximate version is verified based on the respectively determined differences, if the respectively determined differences are all smaller than a first threshold and the sum of the differences is smaller than a second threshold, it may be determined that the approximate version passes the verification.
  • the operation including the first parameter in the second parameter set when determining the second parameter set of the image signal processing device, if it is determined that the operation including the first parameter in the second parameter set is represented by a differentiable function in the simulator, it may be based on the parameters of the differentiable function value to determine the value of the first parameter. If it is determined that the operation including the second parameter in the second parameter set is reserved in the simulator and is not represented by a differentiable function, it may be combined in the simulator with the second parameter corresponding to the value of the third parameter converted to the value of the second parameter.
  • the method 1100 further includes: setting an image signal processing device based on the determined second parameter set.
  • the first parameter set of the simulator when updating the first parameter set of the simulator, it is possible to determine a processed image obtained by processing the raw image captured by the image sensor by the simulator; performing a computer perception task on the processed image based on a neural network The result and the objective function of , determine the task loss; and update the second parameter set of the simulator by minimizing the task loss.
  • Fig. 12 shows a schematic block diagram of an apparatus 1200 for determining ISP parameters according to some embodiments of the present disclosure.
  • Apparatus 1200 may be used to implement or be included in a computing device or computing system implementing method 1100 .
  • the device 1200 includes a simulator determination unit 1210 configured to determine a simulator for simulating an image signal processing device, and the operations included in the simulator have differentiable expressions.
  • the device 1200 also includes a simulator parameter update unit 1220 configured to update the first parameter set of the simulator by jointly training the simulator and the neural network associated with the image signal processing device, the trained neural network is used to image The image output by the signal processing device performs a computer vision perception task.
  • the device 1200 also includes a processing parameter determining unit 1230 configured to determine a second parameter set of the image signal processing device based on the updated first parameter set.
  • the simulator determination unit 1210 is further configured to: generate an approximate version of the simulator based on the differentiability of operations in the operation set of the image signal processing device; process the same image based on the image signal processing device and the approximate version diffing, verifying the approximate version; and identifying the approximate version as the emulator if the approximate version passes verification.
  • the simulator determination unit 1210 is further configured to: if it is determined that the first operation in the operation set has a differentiable expression, keep the first operation in the approximate version.
  • the simulator determining unit 1210 is further configured to: if it is determined that the second operation in the operation set has a non-differentiable expression, determine whether the importance of the second operation is higher than a threshold importance, the importance indicating the second the effect of the operation on the outcome or image quality of the computer vision perception task; and if the importance of the second operation is determined to be above a threshold importance, the second operation is represented in an approximate version by a differentiable function, which is used to approximate the second operate.
  • the differentiable function includes a piecewise linear function.
  • the simulator determination unit 1210 is further configured to: for the verification image in the verification image set captured by the image sensor, obtain the first attribute value of the image attribute obtained by processing the verification image by the image signal processing device; obtain processing a second attribute value of an image attribute resulting from the verification image by the approximate version; determining a difference based on the first attribute value and the second attribute value; and verifying the approximate version based on the respective determined differences for each verification image in the verification image set.
  • the simulator determination unit 1210 is further configured to: if the respectively determined differences are all smaller than the first threshold and the sum of the differences is smaller than the second threshold, determine that the approximate version passes the verification.
  • the processing parameter determination unit 1230 is further configured to: if it is determined that the operation including the first parameter in the second parameter set is represented by a differentiable function in the simulator, determine the second parameter based on the value of the parameter of the differentiable function. the value of a parameter; and if it is determined that the operation including the second parameter in the second parameter set is reserved in the simulator and is not represented by a differentiable function, based on the specifications of the register storing the second parameter in the image signal processing device, the simulated The value of the third parameter corresponding to the second parameter in the register is converted into the value of the second parameter.
  • the apparatus 1200 further includes: a setting unit configured to set the image signal processing apparatus based on the determined second parameter set.
  • the simulator parameter update unit 1220 is further configured to: determine the processed image obtained by processing the original image captured by the image sensor by the simulator; perform computer perception tasks on the processed image based on the neural network determining a task loss as a result and an objective function; and updating a second set of parameters of the simulator by minimizing the task loss.
  • the present application also provides a perception system, such as the perception system 100 shown in FIG. 1 .
  • a perception system such as the perception system 100 shown in FIG. 1 .
  • such perception systems are included in automated driving systems or security systems.
  • Fig. 13 shows a schematic block diagram of a device 1300 capable of implementing multiple embodiments of the present application.
  • the device 1300 may be used to implement the method for determining ISP parameters according to the present disclosure.
  • device 1300 includes computing unit 1301, which may be loaded into RAM and/or ROM according to computer program instructions stored in random access memory (RAM) and/or read only memory (ROM) 1302 or from storage unit 1307 1302 to perform various appropriate actions and processes.
  • RAM and/or ROM 1302 various programs and data necessary for the operation of the device 1300 may also be stored.
  • the computing unit 1301 and the RAM and/or ROM 1302 are connected to each other via a bus 1303.
  • An input/output (I/O) interface 1304 is also connected to the bus 1303 .
  • I/O input/output
  • the I/O interface 1304 includes: an input unit 1305, such as a keyboard, a mouse, etc.; an output unit 1306, such as various types of displays, speakers, etc.; a storage unit 1307, such as a magnetic disk, an optical disk, etc. and a communication unit 1308, such as a network card, a modem, a wireless communication transceiver, and the like.
  • the communication unit 1308 allows the device 1300 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.
  • the computing unit 1301 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 1301 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc.
  • the computing unit 1301 executes various methods and processes described above, such as any one of the processes 300 , 400 , 600 , 1000 and the method 1100 . For example, in some embodiments, any of processes 300 , 400 , 600 , 1000 and method 1100 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1307 .
  • part or all of the computer program may be loaded and/or installed onto device 1300 via RAM and/or ROM and/or communication unit 1308 .
  • the computer program When the computer program is loaded into RAM and/or ROM and executed by computing unit 1301, one or more steps of any of the processes 300, 400, 600, 1000 and method 1100 described above may be performed.
  • the computing unit 1301 may be configured to execute any one of the processes 300 , 400 , 600 , 1000 and the method 1100 in any other suitable manner (for example, by means of firmware).
  • Program codes for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented.
  • the program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • EPROM Erasable Programmable Read Only Memory
  • CD-ROM compact disk read only memory
  • magnetic storage or any suitable combination of the foregoing.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

本公开涉及一种确定图像信号处理参数的方法、装置和感知系统。该方法包括：确定用于模拟图像信号处理装置的模拟器，模拟器所包括的操作具有可微的表达方式。该方法还包括：通过共同训练模拟器和与图像信号处理装置相关联的神经网络，来更新模拟器的第一参数集，经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务。该方法进一步包括：基于经更新的第一参数集，确定图像信号处理装置的第二参数集。以此方式，所得到的ISP参数是针对CV感知任务而优化的，从而有助于提高CV感知任务的性能。

Description

确定图像信号处理参数的方法、装置和感知系统 技术领域
本公开的实施例主要涉及计算机领域。更具体地，本公开的实施例涉及用于确定图像信号处理参数的方法、装置、设备、计算机可读存储介质以及计算机程序产品。
背景技术
随着计算机技术的发展，计算机视觉感知技术在诸多应用和产品中变得越来越重要。例如，在自动驾驶中，计算机视觉感知技术可以用于检测和识别道路上的各种目标，诸如需要躲避的行人。又如，在智慧安防中，计算机视觉感知技术可以用于检测和识别潜在的危险目标。为了实现诸如此类的计算机视觉感知任务，图像信号处理(ISP)装置(例如，图像信号处理器)需要处理图像传感器捕获的原始数据以生成图像，基于深度学习的感知网络需要对ISP装置输出的图像执行计算机视觉(CV)感知任务，以检测和识别目标对象。感知网络的参数是通过训练而学习到的，而现有的ISP装置以硬件为载体，其参数通常是人工调试好的。
发明内容
本公开的实施例提供了一种用于确定图像信号处理参数的方案。
在本公开的第一方面，提供了一种确定图像信号处理参数的方法。该方法包括：确定用于模拟图像信号处理装置的模拟器，模拟器所包括的操作具有可微的表达方式；通过共同训练模拟器和与图像信号处理装置相关联的神经网络，来更新模拟器的第一参数集，经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务；以及基于经更新的第一参数集，确定图像信号处理装置的第二参数集。
在本公开中，利用模拟ISP装置的模拟器实现了与CV感知网络的端到端联合训练，从而同时优化ISP与CV感知网络的参数。以此方式，所得到的ISP参数是针对CV感知任务而优化的，从而有助于提高CV感知任务的性能。
在第一方面的一种实现方式中，确定模拟器包括：基于图像信号处理装置的操作集中的操作的可微性，生成模拟器的近似版本；基于图像信号处理装置和近似版本处理相同图像的差异，验证近似版本；以及如果近似版本通过验证，将近似版本确定为模拟器。以此方式，可以确保模拟器在模拟图像信号处理装置的操作方面的有效性。
在第一方面的又一种实现方式中，生成模拟器的近似版本包括：如果确定操作集中的第一操作具有可微的表达方式，在近似版本中保留第一操作。以此方式，可以针对图像信号处理装置中的原本操作的特性，来采取适当的策略。这避免了对原本可微操作的近似，可以节省生成模拟器的成本。
在第一方面的又一种实现方式中，生成模拟器的近似版本包括：如果确定操作集中的第二操作具有不可微的表达方式，确定第二操作的重要性是否高于阈值重要性，重要性指示第二操作对计算机视觉感知任务的结果或图像质量的影响；以及如果确定第二操作的重要性高于阈值重要性，在近似版本中用可微函数表示第二操作，可微函数用于近似第二操作。以此方式，可以针对图像信号处理装置中的原本操作的特性，来采取适当的策略。通过仅针对重要操作进行可微函数近似，可以减少生成模拟器的复杂度，而不损害后续面向感知任务的优 化效果。
在第一方面的又一种实现方式中，可微函数包括分段线性函数。以此方式，利用分段线性函数能够简单且高效近似图像信号处理装置中的操作。
在第一方面的又一种实现方式中，验证近似版本包括：针对由图像传感器捕获的验证图像集中的验证图像，获得由图像信号处理装置处理验证图像而得到的图像属性的第一属性值；获得由近似版本处理验证图像而得到的图像属性的第二属性值；基于第一属性值和第二属性值，确定差异；以及基于针对验证图像集中的各个验证图像分别确定的差异，验证近似版本。以此方式，利用图像属性(诸如，RGB值等)可以简单且准确地进行图像信号处理装置和模拟器的相似性验证。
在第一方面的又一种实现方式中，基于分别确定的差异验证近似版本包括：如果分别确定的差异均小于第一阈值并且差异之和小于第二阈值，确定近似版本通过验证。以此方式，通过同时考虑针对各个验证图像的差异和针对验证图像集的累积差异，能够以严格的标准验证模拟器。这进一步确保了模拟器的有效性。
在第一方面的又一种实现方式中，确定图像信号处理装置的第二参数集包括：如果确定包括第二参数集中的第一参数的操作在模拟器中由可微函数表示，基于可微函数的参数的值来确定第一参数的值；以及如果确定包括第二参数集中的第二参数的操作在模拟器中被保留并且未由可微函数表示，基于图像信号处理装置中存储第二参数的寄存器的规格，将模拟器中与第二参数相对应的第三参数的值转换成第二参数的值。在这种实现方式中，根据参数所属于的操作在模拟器中的处理方式，来确定图像信号处理参数。以此方式，可以确保经优化的模拟器的参数能够准确地反映回图像信号处理装置中。
在第一方面的又一种实现方式中，该方法还包括：基于所确定的第二参数集，设置图像信号处理装置。以此方式，使图像信号处理装置按照优化的参数来处理数据。
在第一方面的又一种实现方式中，更新模拟器的第一参数集包括：确定由模拟器处理原始图像而得到的经处理图像，原始图像由图像传感器捕获；基于神经网络对经处理图像执行计算机感知任务的结果和目标函数，确定任务损失；以及通过最小化任务损失，来更新模拟器的第二参数集。以此方式，实现了与CV感知网络的端到端联合优化。
在本公开的第二方面，提供了一种用于确定图像信号处理参数的装置。该装置包括：模拟器确定单元，被配置为确定用于模拟图像信号处理装置的模拟器，模拟器所包括的操作具有可微的表达方式；模拟器参数更新单元，被配置为通过共同训练模拟器和与图像信号处理装置相关联的神经网络，来更新模拟器的第一参数集，经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务；以及处理参数确定单元，被配置为基于经更新的第一参数集，确定图像信号处理装置的第二参数集。
在第二方面的一种实现方式中，模拟器确定单元进一步被配置为：基于图像信号处理装置的操作集中的操作的可微性，生成模拟器的近似版本；基于图像信号处理装置和近似版本处理相同图像的差异，验证近似版本；以及如果近似版本通过验证，将近似版本确定为模拟器。
在第二方面的又一种实现方式中，模拟器确定单元进一步被配置为：如果确定操作集中的第一操作具有可微的表达方式，在近似版本中保留第一操作。
在第二方面的又一种实现方式中，模拟器确定单元进一步被配置为：如果确定操作集中的第二操作具有不可微的表达方式，确定第二操作的重要性是否高于阈值重要性，重要性指 示第二操作对计算机视觉感知任务的结果或图像质量的影响；以及如果确定第二操作的重要性高于阈值重要性，在近似版本中用可微函数表示第二操作，可微函数用于近似第二操作。
在第二方面的又一种实现方式中，可微函数包括分段线性函数。
在第二方面的又一种实现方式中，模拟器确定单元进一步被配置为：针对由图像传感器捕获的验证图像集中的验证图像，获得由图像信号处理装置处理验证图像而得到的图像属性的第一属性值；获得由近似版本处理验证图像而得到的图像属性的第二属性值；基于第一属性值和第二属性值，确定差异；以及基于针对验证图像集中的各个验证图像分别确定的差异，验证近似版本。
在第二方面的又一种实现方式中，模拟器确定单元进一步被配置为：如果分别确定的差异均小于第一阈值并且差异之和小于第二阈值，确定近似版本通过验证。
在第二方面的又一种实现方式中，处理参数确定单元进一步被配置为：如果确定包括第二参数集中的第一参数的操作在模拟器中由可微函数表示，基于可微函数的参数的值来确定第一参数的值；以及如果确定包括第二参数集中的第二参数的操作在模拟器中被保留并且未由可微函数表示，基于图像信号处理装置中存储第二参数的寄存器的规格，将模拟器中与第二参数相对应的第三参数的值转换成第二参数的值。
在第二方面的又一种实现方式中，该装置还包括：设置单元，被配置为基于所确定的第二参数集，设置图像信号处理装置。
在第二方面的又一种实现方式中，模拟器参数更新单元进一步被配置为：确定由模拟器处理原始图像而得到的经处理图像，原始图像由图像传感器捕获；基于神经网络对经处理图像执行计算机感知任务的结果和目标函数，确定任务损失；以及通过最小化任务损失，来更新模拟器的第二参数集。
在本公开的第三方面，提供了一种感知系统。该感知系统包括：图像信号处理装置，被配置为处理来自图像传感器的数据，以生成经处理图像；以及神经网络，耦合到图像信号处理装置，并且被配置为对经处理图像执行计算机视觉感知任务，以生成感知结果。其中图像信号处理装置的参数集是通过第一方面的任意一种实现方式中的方法确定的。
在本公开的第四方面，提供了一种电子设备，包括：至少一个处理器；至少一个存储器，至少一个存储器被耦合到至少一个处理器并且存储用于由至少一个处理器执行的指令。指令当由至少一个处理器执行时，使电子设备执行第一方面的任意一种实现方式中的方法。
在本公开的第五方面，提供了一种计算机可读存储介质，其上存储有计算机程序，程序被处理器执行时实现第一方面的任意一种实现方式中的方法。
在本公开的第六方面，提供了一种计算机程序产品，其特征在于，包括计算机可执行指令，其中计算机可执行指令在被处理器执行时实现第一方面的任意一种实现方式中的方法。
可以理解地，上述提供的第二方面的装置、第三方面的感知系统、第四方面的电子设备、第五方面的计算机存储介质或者第六方面的计算机程序产品均用于执行第一方面所提供的方法。因此，关于第一方面的解释或者说明同样适用于第二方面、第三方面、第四方面、第五方面和第六方面。此外，第二方面、第三方面、第四方面、第五方面和第六方面所能达到的有益效果可参考对应方法中的有益效果，此处不再赘述。
应当理解，发明内容部分中所描述的内容并非旨在限定本公开的实施例的关键或重要特征，亦非用于限制本公开的范围。本公开的其它特征将通过以下的描述变得容易理解。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了能够应用本公开的多个实施例的示例系统的示意图；
图2示出了根据本公开的一些实施例的确定ISP参数的系统架构的示意图；
图3示出了根据本公开的一些实施例的确定模拟器的过程的示意图；
图4示出了根据本公开的一些实施例的生成模拟器的近似版本的过程的示意图；
图5示出了根据本公开的一些实施例的用分段线性函数来近似操作的示意图；
图6示出了根据本公开的一些实施例的验证近似版本的过程的流程图；
图7示出了根据本公开的一些实施例的共同训练模拟器和神经网络的示意图；
图8示出了传统方案中训练神经网络的示意图；
图9示出了根据本公开的一些实施例的部署感知系统的示意图；
图10示出了根据本公开的一些实施例的配置ISP参数的过程的示意图；
图11示出了根据本公开的一些实施例的确定ISP参数的示例方法的流程图；
图12示出了根据本公开的一些实施例的用于确定ISP参数的装置的示意性框图；以及
图13示出了能够实施本公开的多个实施例的计算设备的框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实施例的描述中，术语“包括”及其类似用语应当理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实施例”或“该实施例”应当理解为“至少一个实施例”。术语“第一”、“第二”等等可以指代不同的或相同的对象。术语“和/或”表示由其关联的两项的至少一项。例如“A和/或B”表示A、B、或者A和B。下文还可能包括其他明确的和隐含的定义。
如本文所使用的，“神经网络”能够处理输入并且提供相应输出，其通常包括输入层和输出层以及在输入层与输出层之间的一个或多个隐藏层。在深度学习应用中使用的神经网络通常包括许多隐藏层，从而延长网络的深度。神经网络的各个层按顺序相连，从而前一层的输出被提供作为后一层的输入，其中输入层接收神经网络的输入，而输出层的输出作为神经网络的最终输出。在本文中，术语“神经网络”、“网络”、“神经网络模型”和“模型”可替换地使用。如本文所适用的，“确定参数”或类似表达是指确定参数的值，包括具体值、取值集合或取值范围等。“确定参数集”或类似表达是指确定参数集中的参数的值，包括具体值、取值集合或取值范围等。
示例系统
图1示出了能够应用本公开的多个实施例的示例感知系统100的示意图。总体上，感知系统100包括ISP装置110和神经网络120。以硬件形式的ISP装置110用于处理由前端的图像传感器捕获的传感器数据101，以生成经处理图像102。ISP装置110可以例如是图像信号 处理器，或其他任何合适的ISP装置。ISP装置110对传感器数据101的处理可以包括但不限于黑电平校正(BLC)、去马赛克(demosaic)、锐化、镜头阴影校正、坏像素校正等。
以上每个功能又可以包括一个或多个操作来实现对应的处理。每个操作又可以包括一个或多个参数，其在本文中也称为“ISP参数”。各个功能的ISP参数构成了ISP装置110的参数集，其在本文中也称为“ISP参数集”。这些ISP参数的值被存储在ISP装置110的寄存器中。
在感知系统100的部署和使用中，神经网络120是经训练的。经训练的神经网络120用于对ISP装置110输出的经处理图像102执行CV感知任务，以输出感知结果103。例如，在自动驾驶和智慧安防的应用中，神经网络120可以用于从经处理图像120中检测和识别目标对象。相应地，神经网络120也可以称为感知网络120。在本公开的实施例中，可以使用任何合适类型的网络来实现神经网络120，例如卷积神经网络、递归神经网络、区域卷积神经网络(R-CNN)、快速R-CNN等。
为了实现CV感知任务，神经网络120也包括一个或多个参数，其也称为“网络参数”。由这些参数构成的参数集(也称为“网络参数集”)在神经网络120的训练中被确定，并且在感知系统100的部署和使用中保持不变。ISP参数和网络参数对感知结果103的准确性具有影响。
传统上，ISP参数的调试由人工进行。通常，调试专家根据人眼喜好亮度、颜色准确性等主客观来分析和调试。所调试的一组ISP参数被固化到ISP装置中，并利用该组ISP参数输出图像集。该图像集被提供给后级的感知网络，以对感知网络进行微调或再训练。这种传统方案具有诸多缺陷。首先，ISP参数的调试是针对人眼愉悦的，与感知性能完全没有关联。其次，ISP参数的人工调试时间长、人力和时间成本大，并且不同人有不同调试方案，造成调试效率极低。此外，感知网络的优化没有考虑到ISP的变量，没有形成ISP连同CV感知任务的优化链路。导致优化顺序与优化目标是分割的，不能端到端地提升CV感知任务的性能。
在另一种方案中，提出了利用带有网络结构的ISP联合优化后级的感知网络。这种带网络结构的ISP主要包括视觉局部调节映射(VLTM)部分和可训练视觉缩放(TVS)部分。VLTM部分由若干传统的简单计算模块构成，由于计算都是简单的连续函数，所以保持了可以自动求导的能力。TVS部分是一种卷积神经网络结构，用于完成特征提取和缩放等功能。TVS部分可以与后级的感知网络耦合起来。然而，由于TVS部分存在网络结构，输出的图像以特征图的形式呈现，使得人眼不可看。这导致输出的图像无法用于显示，难以在实际应用中使用。非网路结构部分是简单的图像处理模块，不满足消费级ISP的高要求。此外，该方案中，输入是RGB图，而不是原始(raw)图，因而不是完全端到端优化ISP和感知网络。
在又一种方案中，提出了利用离线的网络模拟ISP装置的参数行为。在该方案中，需要配置硬件ISP装置的不同参数组合，生成包括大量参数与图像对的训练集。用该训练集来训练类U-net的网络结构，以模拟硬件ISP装置的参数调节行为，从而可以与后级的感知网络进行端到端的优化。然而，该方案需要采集所有参数组合下的训练图像。在参数空间变大时，这易造成某一参数行为数据不均衡问题。同时，参数拼接(concatenate)的方式使网络体量变大，网络难收敛，因而无法精确模拟ISP装置的参数行为。
为了至少部分地解决上述问题以及其他潜在问题，本公开的各种实施例提供了一种用于确定ISP参数的方案。总体而言，根据在此描述的各种实施例，首先确定用于模拟图像信号 处理装置的模拟器。该模拟器所包括的操作具有可微的表达方式。通过共同训练模拟器和与图像处理信号装置相关联的神经网络，来更新模拟器的参数集。经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务。然后，基于模拟器的经更新的参数集，确定图像信号处理装置的参数集。
在本公开的实施例中，利用模拟ISP装置的模拟器实现了与CV感知网络的端到端联合训练，从而同时优化ISP与CV感知网络的参数。这解决了ISP与CV感知网络分离优化的问题。以此方式，所得到的ISP参数是针对CV感知任务而优化的，从而有助于提高CV感知任务的性能。此外，与传统方案相比，这也缩短了ISP参数调试的人力和时间成本，提高了效率。
以下参考图2至图13来描述本公开的各种示例实施例。
示例架构和总体流程
图2示出了根据本公开的一些实施例的确定ISP参数的系统架构200的示意图。总体而言，系统架构200包括继承流程250、优化流程260和部署流程270。继承流程250用于确定模拟ISP装置110的ISP模拟器210。ISP模拟器210所包括的操作具有可微的表达方式。换言之，可微的ISP模拟器210通过继承ISP装置110的全部或大部分操作，来模拟ISP装置110，并且具有与ISP装置110相似的参数行为。可以理解的是，由于ISP模拟器210继承了ISP装置110的操作，因此ISP模拟器210也具有所继承的操作的对应参数。在下文中，ISP模拟器210的参数也称为“模拟参数”，ISP模拟器210的参数集也称为“模拟参数集”。ISP模拟器210可以视为硬件ISP的软代理，因而也可以称为“软代理ISP”。
ISP装置110包括由不同功能构成的处理流水线(pipeline)。作为示例，图2中示出了硬件功能201-1、201-2、201-3，其可以单独或统称为硬件功能201。相应地，ISP模拟器210包括分别与硬件功能201-1、201-2、201-3相对应的可微功能203-1、203-2、203-3，其可以单独或统称为可微功能203。每个可微功能203与对应的硬件功能201相同或相近。
在一些实施例中，可以对ISP模拟器210进行验证，以确保ISP模拟器210与ISP装置110足够相似。在这种实施例中，继承流程250可以包括近似流程251和验证流程252，如图2所示。近似流程251用于基于ISP装置的操作集中各个操作的可微性，生成ISP模拟器的近似版本205。也即，近似流程251用于从ISP装置110的角度近似出可微的待验证ISP模拟器。近似版本205包括分别与硬件功能201-1、201-2、201-3相对应的可微功能202-1、202-2、202-3，其可以单独或统称为可微功能202。每个可微功能202通过近似对应的硬件功能201而得出。下文将参考图4描述生成近似版本的示例过程。
在接下来的验证流程252中，基于ISP装置110和近似版本205处理相同图像的差异，验证近似版本205。也即，验证流程252用于认证近似版本205与ISP装置110的相似性。如果近似版本205未通过验证，则需要重新执行近似流程251，并且可能需要调整近似流程251。如果近似版本205通过验证，则将近似版本205确定为后续要使用的ISP模拟器210。在这种实施例中，利用验证流程可以确保近似流程和ISP模拟器210的有效性。
所确定的ISP模拟器210将用于接下来的优化流程260。在优化流程260中，基于目标函数220来共同训练ISP模拟器210和神经网络120。通过训练，ISP模拟器210的模拟参数集合神经网络120的网络参数集被更新。具体来说，在深度学习的框架上，可微的ISP模拟器210与神经网络120和目标函数220一起进行正向传播，从而端到端地优化ISP模拟器210 的参数和神经网络120的参数。以此方式，经更新的模拟参数集是面向感知任务而优化的。
在部署流程270中，利用ISP模拟器210的经更新的模拟参数集来配置ISP装置110。考虑到ISP模拟器210是对ISP装置110的近似，一些操作的参数可能在取值范围、取值要求、量化间隔等方面与ISP装置110中对应的参数不同。因此，为了配置ISP装置110，可以首先基于模拟参数集来确定ISP参数集，即，确定ISP参数的值。然后，基于所确定的ISP参数集来设置ISP装置110，例如设置ISP装置110的各个寄存器中所存储的值。如此配置的ISP装置110与经训练的神经网络120一起实现CV感知。
继承流程的示例
下面参考图3至图6来详细描述继承流程250的一些示例。通常，ISP装置110并不是重新开发的。有鉴于此，可以利用描述ISP装置110的代码来实现继承流程250。图3示出了根据本公开的一些实施例的确定ISP模拟器的过程300的示意图。过程300可以视为继承流程250的示例实现。
在框310，获取描述ISP装置110的代码，也称为历史代码。这种代码可以包括但不限于各种合适的芯片设计代码，诸如C-model、verilog等。在框320，执行近似流程251，以生成近似版本205。具体地，基于设计代码所描述的ISP装置110的各个操作的可微性，来确定ISP模拟器210中的操作。如此，生成了与硬件功能201相对应的可微功能202。在一些实施例中，可以对硬件功能201中的每个操作执行下文描述的过程400，如下文将详细描述的。
在框330，验证近似版本205。如果近似版本205未通过验证，则过程300返回框320，需要重新执行近似流程251，以生成新的近似版本。在这种情况下，可能需要调整近似流程251。如果近似版本205通过验证，则过程300进行到框340。在框340，将近似版本205确定为后续要使用的ISP模拟器210。
应当理解，过程300还可以包括未示出的其他动作。在一些实施例中，在框320之后，可以将代码形式的近似版本205转换为程序语言，以便于后续与神经网络120一起训练。例如，可以在深度学习框架(诸如，pytorch)下对近似版本205进行编程。
在图3所示的过程200中，利用历史代码来确定ISP模拟器340。由于ISP装置通常具有复杂的操作，以此方式可以用简单地方式获得ISP模拟器，减少了开发ISP模拟器的成本。备选地，在一些实施例中，在没有描述ISP装置110的代码的情况下，可以针对ISP装置110要实现的功能，重新开发面向感知的流水线，使得该流水线的每个操作是可微的。然后，可以将重新开发的流水线转换为程序语言。
利用继承流程可以继承操作复杂的硬件ISP装置，解决了传统硬件ISP装置难连续求导问题。所获得的ISP模拟器具有可解释性、表达空间可控、容易收敛等特性。
近似流程的示例
现在参考图4，其示出了根据本公开的一些实施例的生成ISP模拟器的近似版本的过程400的示意图。过程400可以视为框320的具体实现。可以针对硬件功能201中的每个操作执行过程400，直到针对所有操作完成过程400，从而生成近似版本205。
针对当前所考虑的操作(也称为当前操作)，在框410，确定该操作是否可微。换言之，确定该操作是否具有可微的表达方式。如果该操作可微，则过程400进行到框450。在框450， 在近似版本205中保留该操作。如果该操作不可微，则过程400进行到框460。
在框420，确定该操作的重要性是否高于重要性阈值。该重要性指示操作对CV感知任务的结果或图像质量的影响。换言之，在框420，确定该操作对CV感知任务或图像质量是否具有重要影响。例如，可以通过消融测试来确定该操作是否具有重要影响。如果在框420确定该操作的重要性小于重要性阈值，也即该操作对CV感知任务或图像质量不具有重要影响，则过程400进行到框460。在框460，从近似版本205中去除该操作。近似版本460中将不会保留该操作。通过去除不重要的操作，可以减少近似流程的复杂度，而不损害后续面向感知任务的优化效果。
如果在框420确定该操作的重要性大于重要性阈值，也即该操作对CV感知任务或图像质量具有重要影响，则过程400进行到框430。在框430，确定是否使用可微函数来近似该操作。是否使用可微函数可以取决于该操作原本的表达方式是否便于或能够用可微函数来近似。以此方式，可以针对不同特点的操作使用最合适的近似方式。如果在框430确定不使用可微函数，则过程400进行到框470。例如，如果该操作原本的表达方式局部不可微，则过程400可以进行到框470。在框470，定义不可微处的左微分和右微分。
如果在框430确定使用可微函数，则过程400进行到框440。例如，如果该操作原本的表达方式处处不可微或仅一小部分可微，则过程400可以进行到框440。在框440，在近似版本中用可微函数表示该操作。也即，该操作被可微函数替换。具体地，可以首先确定用于近似该操作的可微函数，然后再用所确定的可微函数替换该操作。
在一些实施例中，可微函数可以是分段线性(PWL)函数。PWL函数是对有间断点而不可导的分段函数的一种高效近似和拟合。在ISP装置110中，存在读取查找表的操作。在查找表中，通过键(Key)来确定确定对应的值(Value)。查找表中的键和值通常是离散、不连续的。因此，读取查找表的操作是不可微的，并且合适于用PWL函数来近似。图5示出了根据本公开的一些实施例的用PWL函数来近似操作的示意图。图5中所示的点(诸如点510)表示查找表中的键值对。PWL函数520可以近似查找表中的键值对的关系，从而可以在近似版本205中用PWL函数520来表示读取该查找表的操作。
继续参考图4。应当理解，过程400仅是示例性的，而无意限制本公开的范围。可以对过程400进行变化。在一些实施例中，可以不包括框420和框460。也即，ISP装置110的所有操作都将在近似版本中被保留或者被近似。
此外，在无需验证的实施例中，过程400可以视为生成后续使用的ISP模拟器210的具体实现。也即，可以针对硬件功能201中的每个操作执行过程400，直到针对所有操作完成过程400，从而生成ISP模拟器210。
验证流程的示例
在如前所述的近似流程251中，存在近似和删除。为了保证这样的近似和删除不影响感知性能，并保证ISP模拟器对于感知性能有近似性和可代理性，在一些实施例中可以对近似版本205执行验证流程252。在验证流程252中，基于ISP装置110和近似版本205处理相同图像的差异，验证近似版本205。
在一些实施例中，可以将ISP装置110和近似版本205处理相同图像而产生的图像属性的差异作为验证标准。具体地，可以获得由图像传感器捕获的验证图像集。验证图像集中的验证图像是由图像传感器捕获的原始数据。然后，针对每个验证图像，可以获得由ISP装置 110处理该验证图像而得到的图像属性的属性值(也称为“第一属性值”)，并且获得由近似版本205处理该验证图像而得到的图像属性的属性值(也称为“第二属性值”)。可以基于第一属性值和第二属性值，确定ISP装置110和近似版本205处理相同图像的差异。可以基于针对验证图像集中的各个验证图像分别确定的差异，验证近似版本205。
在一些实施例中，如果分别确定的各个差异中超过阈值数目的差异大于阈值，则近似版本205未通过验证；否则近似版本205通过验证。阈值数目可以例如是1，或者取决于验证图像集中的验证图像的数目。在一些实施例中，还可以考虑分别确定的各个差异之和，即累积差异。在累积差异小于对应的阈值的情况下，近似版本205通过验证。
所考虑的图像属性可以包括任何合适类型的属性，例如RGB值、灰度值、色调、亮度、对比度等。图6示出了根据本公开的一些实施例的验证近似版本的过程600的流程图。图6的过程600以RGB值作为图像属性的示例。应当理解，类似的过程可以应用于其他的图像属性。
在框610，获得验证图像集中的验证图像。例如，可以依次从验证图像集中取回验证图像。在框620，获得由ISP装置110处理验证图像所得到的RGB值。例如，可以将验证图像输入到ISP装置110，并且确定输出的图像在各个像素处的RGB值。在框630，获得由近似版本205处理验证图像所得到的RGB值。例如，可以将验证图像输入到近似版本205，并且确定输出的图像在各个像素处的RGB值。
在框640，针对当前的验证图像确定RGB值的差异D，作为ISP装置110和近似版本205处理相同图像的差异。如此确定的差异D可以视为由ISP装置110处理的图像与由近似版本205处理的图像在RGB空间中的距离。可以利用预定义的度量函数来计算差异D。度量函数例如可以包括均方误差(MSE)、图像统计值的差等，本公开的范围在此方面不受限制。在确定差异D之后，还可以计算针对验证图像集的累积差异D_t，即，针对已考虑的验证图像而确定的差异D之和。
在框650，确定针对当前验证图像的差异D是否小于第一阈值。如果差异D不小于第一阈值，则过程600进行到框660。在框660，确定近似版本205未通过验证。如果差异D小于第一阈值，则过程600进行到框670。在框670，确定当前的验证图像是否是验证图像集中的最后一个验证图像。如果当前的验证图像不是验证图像集中的最后一个验证图像，则过程600返回到框610，以对验证图像集中的下一图像执行框610至框650。
如果当前的验证图像是验证图像集中的最后一个验证图像，则过程600进行到框680。在框680，确定累积差异D_t是否小于第二阈值。也即，确定针对所有验证图像的差异D之和是否小于第二阈值。如果确定累积差异D_t不小于第二阈值，则过程600进行到框660。在框660，确定近似版本205未通过验证。如果确定累积差异D_t小于第二阈值，则过程600进行到框690。在框690，确定近似版本205通过验证。
应当理解，图6所示的过程600仅是示例性的，而无意限制本公开的范围，还可以在过程600的基础上进行适当变化。例如，可以省略框650，即不针对每个验证图像都确定差异D是否小于第一阈值。又如，可以省略框680，即可以不考虑累积差异。再如，可以确定差异D小于第一阈值的验证图像的数目是否超过阈值数目或比例是否超过阈值比例，作为验证标准。
在以上描述的实施例中，将图像属性的差异作为验证标准，来验证近似版本205。备选地，在其他的一些实施例中，可以选择其他的验证标准，例如，可以获得由ISP装置110和近似版本205分别处理验证图像而得到的第一图像和第二图像。然后，对第一图像和第二图 像进行特征提取，并且在特征空间中计算这两个图像的距离，作为ISP装置110和近似版本205处理相同图像的差异。
优化流程的示例
通过上述的继承流程251可以获得具有参数行为近似的ISP模拟器。这种ISP模拟器在近似流程中已经确认了可微的属性。利用这种属性在经典的深度学习框架上，可以完成求导功能和反向传播功能。这使得ISP模拟器与感知网络同时在一个框架上优化，从而实现端到端的感知性能自动优化。
图7示出了根据本公开的一些实施例的共同训练ISP模拟器210和神经网络120的示意图。如图7所示，可以在同一深度学习框架的软件环境中同时训练并更新ISP模拟器210和神经网络120。在正向计算701中，从ISP模拟器210传播至目标函数220，实现梯度下降法的正向计算。具体地，将作为训练数据的传感器数据(即，原始图像)输入到ISP模拟器210。ISP模拟器210处理传感器数据，并输出经处理的训练图像。训练图像被输入到神经网络120。基于神经网络120对训练图像执行训练图像的结果和目标函数，可以确定任务损失。在反向计算702中，实现梯度下降法的反向计算，反向梯度传播到ISP模拟器210。如箭头703和704所示，ISP模拟器210的参数集和神经网络120的参数集同时被更新。
作为对比，图8示出了传统方案中训练神经网络的示意图。如图8所示，传统方案中的正向计算801从硬件的ISP装置810传播到目标函数830，但在反向计算802中，反向梯度传播止步于ISP装置810的输出，而不能传播到ISP装置810。因此，如箭头803所示，仅神经网络820的参数集被更新。
由以上描述可以看出，在本公开的实施例中，与感知网络端到端联合优化解决了传统优化分开、目标不一致等问题。ISP参数的调试是面向感知任务的，而不是面向人眼愉悦的。这有助于提升CV感知任务的性能，得到更准确的感知结果。
部署流程的示例
通过以上描述的端到端的优化流程，ISP模拟器能够自动学习出针对感知性能而优化(例如，最优)的参数。在实际部署中，硬件的ISP装置利用已学习的优化参数，向感知网络实时提供图像。
图9示出了根据本公开的一些实施例的部署感知系统的示意图。在部署流程270中，利用ISP模拟器210的已学习的模拟参数集910来配置ISP装置110。经配置的ISP装置110基于优化参数来处理传感器数据，并向经训练的神经网络120输出图像。神经网络120利用来自ISP装置110的图像生成针对CV感知任务的感知结果。
近似流程251使得ISP模拟器210的参数的配置可能不能一一对应于硬件的ISP装置，一些操作的参数可能在取值范围、取值要求、量化间隔等方面与ISP装置110中对应的参数不同。有鉴于此，为了用优化参数来配置ISP装置110，需要首先基于优化的模拟参数集来优化的ISP参数集。
图10示出了根据本公开的一些实施例的确定ISP参数的过程1000的示意图。可以针对每个ISP参数执行过程1000。在框1010，确定该参数是否属于从ISP模拟器210中去除的操作。换言之，确定包括该参数的操作是否被保留在ISP模拟器210中。如果在框1010确定该参数属于去除的操作，则过程1000进行到框1040。在框1040，使用ISP装置110中该参数 原本的值。由于已经在验证流程252中确定这样的操作不具有重要影响(例如，通过消融测试)，因此针对该参数使用原本的值对感知效果不会产生大的影响。
如果在框1010确定该参数不属于去除的操作，即确定包括该参数的操作被保留在ISP模拟器210中，则过程1000进行到框1020。在框1020，确定该参数是否属于可微函数近似。换言之，确定包括该参数的操作在ISP模拟器210中是否由可微函数表示。如果在框1020，确定该参数属于可微函数近似，即确定包括该参数的操作在ISP模拟器210中由可微函数表示，则过程1000进行到框1050。
在框1050，基于可微函数的参数的值来确定该参数的值。具体地，基于可微函数的参数的值可以确定可微函数在每个自变量下的取值。然后，可以根据该参数的特点(例如，量化间隔等)对可微函数采样，基于采样处的取值确定该参数的值。下面以图5所示的PWL函数近似为例进行说明。PWL函数520用于近似读取查找表的操作。假设该查找表具有33个索引，那么需要33个采样点。PWL函数520的自变量(即，横坐标)的取值范围是0到1，那么可以从0到1每1/33步长设置一个采样点，即每个1/33对PWL函数520进行采样。采样点处PWL函数520的取值(即，纵坐标)即可作为查找表对应于该索引的输出值，或者对该取值进行缩放后作为查找表对应于该索引的输出值。
继续参考图10。如果在框1020，确定该参数不属于可微函数近似，则过程1000进行到框1030。在框1030，基于ISP装置110中存储该参数的寄存器的规格，将与该参数对应的模拟参数的值转换为该参数的值。寄存器的规格可以包括例如寄存器的值域、取值要求、量化间隔等。因此，可以根据存储ISP参数的寄存器的值域、取值要求、量化间隔等，量化对应的模拟参数，以作为该ISP参数的值。仅作为示例，在深度学习的框架下，模拟参数通常具有[0,1]的取值范围，而对应的ISP参数的寄存器可能具有[0,16]的值域。相应地，可以根据[0,1]和[0,16]的对应性，来将模拟参数的值转换为ISP参数的值。在这种情况下，根据寄存器的值域对模拟参数进行了缩放，作为ISP参数的值。
所确定的ISP参数集后续用于设置ISP装置110，即，将这些ISP参数的值写入ISP装置110的寄存器中。例如，可以将所确定的ISP参数集以文件形式加载到ISP装置110的存储器中。然后，在ISP装置110中，各个ISP参数的值被读出，并且写入对应的寄存器中。
以上参考图2至图10描述了根据本公开的确定ISP参数的示例实现。从以上描述中，可以看出本公开的一些有益效果。与传统的人工调试方案相比，本公开的实施例所实现的优化是与感知性能相关联的端到端联合优化。这能够大大提高感知性能，并且优化可以由高效或专用计算装置(诸如，图形处理单元GPU或人工智能芯片)执行，比人工手动优化的时间快几个数量级。
与上文提及的带有网络结构的ISP方案相比，在本公开的实施例中，可以保留和继承复杂的ISP操作，无需为可微性选择简单的处理算法，并且反向回传可以到达原始图输入。此外，本公开的实施例不存在网络结构，经图像信号处理的图像是可观看、可记录的。与上文提及的利用离线的网络模拟ISP装置的方案相比，本公开的实施例基于硬件ISP操作的可微化近似，因而继承了硬件ISP的可解释性。这相对于图像质量与感知优化而言都可以更好收敛。
示例方法、装置和设备
图11示出了根据本公开的一些实施例的确定ISP参数的示例方法1100的流程图。方法 1100可以由任何合适的计算设备或计算系统实施。应当理解，方法1100还可以包括未示出的附加动作和/或可以省略所示出的动作。本公开的范围在此方面不受限制。
在框1110，确定用于模拟图像信号处理装置的模拟器。模拟器所包括的操作具有可微的表达方式。在框1120，通过共同训练模拟器和与图像信号处理装置相关联的神经网络，来更新模拟器的第一参数集。经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务。在框1130，基于经更新的第一参数集，确定图像信号处理装置的第二参数集。
在一些实施例中，在确定模拟器时，可以基于图像信号处理装置的操作集中的操作的可微性，生成模拟器的近似版本；基于图像信号处理装置和近似版本处理相同图像的差异，验证近似版本；以及如果近似版本通过验证，将近似版本确定为模拟器。
在一些实施例中，在生成模拟器的近似版本时，如果确定操作集中的第一操作具有可微的表达方式，在近似版本中保留第一操作。
在一些实施例中，在生成模拟器的近似版本时，如果确定操作集中的第二操作具有不可微的表达方式，可以确定第二操作的重要性是否高于阈值重要性。重要性指示第二操作对计算机视觉感知任务的结果或图像质量的影响。如果确定第二操作的重要性高于阈值重要性，可以在近似版本中用可微函数表示第二操作。可微函数用于近似第二操作。在一些实施例中，可微函数包括分段线性函数。
在一些实施例中，在验证近似版本时，可以针对由图像传感器捕获的验证图像集中的验证图像，获得由图像信号处理装置处理验证图像而得到的图像属性的第一属性值；获得由近似版本处理验证图像而得到的图像属性的第二属性值；基于第一属性值和第二属性值，确定差异。可以基于针对验证图像集中的各个验证图像分别确定的差异，验证近似版本。
在一些实施例中，在基于分别确定的差异验证近似版本时，如果分别确定的差异均小于第一阈值并且差异之和小于第二阈值，可以确定近似版本通过验证。
在一些实施例中，在确定图像信号处理装置的第二参数集时，如果确定包括第二参数集中的第一参数的操作在模拟器中由可微函数表示，可以基于可微函数的参数的值来确定第一参数的值。如果确定包括第二参数集中的第二参数的操作在模拟器中被保留并且未由可微函数表示，可以基于图像信号处理装置中存储第二参数的寄存器的规格，将模拟器中与第二参数相对应的第三参数的值转换成第二参数的值。
在一些实施例中，方法1100还包括：基于所确定的第二参数集，设置图像信号处理装置。
在一些实施例中，在更新模拟器的第一参数集时，可以确定由模拟器处理原始图像而得到的经处理图像，原始图像由图像传感器捕获；基于神经网络对经处理图像执行计算机感知任务的结果和目标函数，确定任务损失；以及通过最小化任务损失，来更新模拟器的第二参数集。
图12示出了根据本公开的一些实施例的用于确定ISP参数的装置1200的示意性框图。装置1200可以用于实现或被包括在实施方法1100的计算设备或计算系统。如图12所示，装置1200包括模拟器确定单元1210，被配置为确定用于模拟图像信号处理装置的模拟器，模拟器所包括的操作具有可微的表达方式。装置1200还包括模拟器参数更新单元1220，被配置为通过共同训练模拟器和与图像信号处理装置相关联的神经网络，来更新模拟器的第一参数集，经训练的神经网络用于对图像信号处理装置所输出的图像执行计算机视觉感知任务。装置1200还包括处理参数确定单元1230，被配置为基于经更新的第一参数集，确定图像信 号处理装置的第二参数集。
在一些实施例中，模拟器确定单元1210进一步被配置为：基于图像信号处理装置的操作集中的操作的可微性，生成模拟器的近似版本；基于图像信号处理装置和近似版本处理相同图像的差异，验证近似版本；以及如果近似版本通过验证，将近似版本确定为模拟器。
在一些实施例中，模拟器确定单元1210进一步被配置为：如果确定操作集中的第一操作具有可微的表达方式，在近似版本中保留第一操作。
在一些实施例中，模拟器确定单元1210进一步被配置为：如果确定操作集中的第二操作具有不可微的表达方式，确定第二操作的重要性是否高于阈值重要性，重要性指示第二操作对计算机视觉感知任务的结果或图像质量的影响；以及如果确定第二操作的重要性高于阈值重要性，在近似版本中用可微函数表示第二操作，可微函数用于近似第二操作。在一些实施例中，可微函数包括分段线性函数。
在一些实施例中，模拟器确定单元1210进一步被配置为：针对由图像传感器捕获的验证图像集中的验证图像，获得由图像信号处理装置处理验证图像而得到的图像属性的第一属性值；获得由近似版本处理验证图像而得到的图像属性的第二属性值；基于第一属性值和第二属性值，确定差异；以及基于针对验证图像集中的各个验证图像分别确定的差异，验证近似版本。
在一些实施例中，模拟器确定单元1210进一步被配置为：如果分别确定的差异均小于第一阈值并且差异之和小于第二阈值，确定近似版本通过验证。
在一些实施例中，处理参数确定单元1230进一步被配置为：如果确定包括第二参数集中的第一参数的操作在模拟器中由可微函数表示，基于可微函数的参数的值来确定第一参数的值；以及如果确定包括第二参数集中的第二参数的操作在模拟器中被保留并且未由可微函数表示，基于图像信号处理装置中存储第二参数的寄存器的规格，将模拟器中与第二参数相对应的第三参数的值转换成第二参数的值。
在一些实施例中，装置1200还包括：设置单元，被配置为基于所确定的第二参数集，设置图像信号处理装置。
在一些实施例中，模拟器参数更新单元1220进一步被配置为：确定由模拟器处理原始图像而得到的经处理图像，原始图像由图像传感器捕获；基于神经网络对经处理图像执行计算机感知任务的结果和目标函数，确定任务损失；以及通过最小化任务损失，来更新模拟器的第二参数集。
此外，本申请还提供了一种感知系统，诸如图1所示的感知系统100。在一些实施例中，这样的感知系统被包括在自动驾驶系统或安防系统中。
图13示出了能够实施本申请的多个实施例的设备1300的示意性框图。设备1300可以用于实施根据本公开的确定ISP参数的方法。如图所示，设备1300包括计算单元1301，其可以根据存储在随机存取存储器(RAM)和/或只读存储器(ROM)1302的计算机程序指令或者从存储单元1307加载到RAM和/或ROM 1302中的计算机程序指令，来执行各种适当的动作和处理。在RAM和/或ROM 1302中，还可存储设备1300操作所需的各种程序和数据。计算单元1301和RAM和/或ROM 1302通过总线1303彼此相连。输入/输出(I/O)接口1304也连接至总线1303。
设备1300中的多个部件连接至I/O接口1304，包括：输入单元1305，例如键盘、鼠标等；输出单元1306，例如各种类型的显示器、扬声器等；存储单元1307，例如磁盘、光盘等； 以及通信单元1308，例如网卡、调制解调器、无线通信收发机等。通信单元1308允许设备1300通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。
计算单元1301可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元1301的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元1301执行上文所描述的各个方法和处理，例如过程300、400、600、1000和方法1100中的任一个。例如，在一些实施例中，过程300、400、600、1000和方法1100中的任一个可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元1307。在一些实施例中，计算机程序的部分或者全部可以经由RAM和/或ROM和/或通信单元1308而被载入和/或安装到设备1300上。当计算机程序加载到RAM和/或ROM并由计算单元1301执行时，可以执行上文描述的过程300、400、600、1000和方法1100中的任一个的一个或多个步骤。备选地，在其他实施例中，计算单元1301可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行过程300、400、600、1000和方法1100中的任一个。
用于实施本申请的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本申请的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
此外，虽然采用特定次序描绘了各操作，但是这应当理解为要求这样操作以所示出的特定次序或以顺序次序执行，或者要求所有图示的操作应被执行以取得期望的结果。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本申请的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实现中。相反地，在单个实现的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实现中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (24)

  1. 一种确定图像信号处理参数的方法，其特征在于，所述方法包括：
    确定用于模拟图像信号处理装置的模拟器，所述模拟器所包括的操作具有可微的表达方式；
    通过共同训练所述模拟器和与所述图像信号处理装置相关联的神经网络，来更新所述模拟器的第一参数集，经训练的所述神经网络用于对所述图像信号处理装置所输出的图像执行计算机视觉感知任务；以及
    基于经更新的所述第一参数集，确定所述图像信号处理装置的第二参数集。
  2. 根据权利要求1所述的方法，其特征在于，确定所述模拟器包括：
    基于所述图像信号处理装置的操作集中的操作的可微性，生成所述模拟器的近似版本；
    基于所述图像信号处理装置和所述近似版本处理相同图像的差异，验证所述近似版本；以及
    如果所述近似版本通过验证，将所述近似版本确定为所述模拟器。
  3. 根据权利要求2所述的方法，其特征在于，生成所述模拟器的所述近似版本包括：
    如果确定所述操作集中的第一操作具有可微的表达方式，在所述近似版本中保留所述第一操作。
  4. 根据权利要求2所述的方法，其特征在于，生成所述模拟器的所述近似版本包括：
    如果确定所述操作集中的第二操作具有不可微的表达方式，确定所述第二操作的重要性是否高于阈值重要性，所述重要性指示所述第二操作对所述计算机视觉感知任务的结果或图像质量的影响；以及
    如果确定所述第二操作的重要性高于所述阈值重要性，在所述近似版本中用可微函数表示所述第二操作，所述可微函数用于近似所述第二操作。
  5. 根据权利要求4所述的方法，其特征在于，所述可微函数包括分段线性函数。
  6. 根据权利要求2-5中的任一项所述的方法，其特征在于，验证所述近似版本包括：
    针对由图像传感器捕获的验证图像集中的验证图像，
    获得由所述图像信号处理装置处理所述验证图像而得到的图像属性的第一属性值；
    获得由所述近似版本处理所述验证图像而得到的所述图像属性的第二属性值；
    基于所述第一属性值和所述第二属性值，确定所述差异；以及
    基于针对所述验证图像集中的各个验证图像分别确定的所述差异，验证所述近似版本。
  7. 根据权利要求6所述的方法，其特征在于，基于分别确定的所述差异验证所述近似版本包括：
    如果分别确定的所述差异均小于第一阈值并且所述差异之和小于第二阈值，确定所述近似版本通过验证。
  8. 根据权利要求1-7中的任一项所述的方法，其特征在于，确定所述图像信号处理装置的所述第二参数集包括：
    如果确定包括所述第二参数集中的第一参数的操作在所述模拟器中由可微函数表示，基于所述可微函数的参数的值来确定所述第一参数的值；以及
    如果确定包括所述第二参数集中的第二参数的操作在所述模拟器中被保留并且未由可微函数表示，基于所述图像信号处理装置中存储所述第二参数的寄存器的规格，将所述模拟器 中与所述第二参数相对应的第三参数的值转换成所述第二参数的值。
  9. 根据权利要求1-8中的任一项所述的方法，其特征在于，所述方法还包括：
    基于所确定的所述第二参数集，设置所述图像信号处理装置。
  10. 根据权利要求1-9中的任一项所述的方法，其特征在于，更新所述模拟器的所述第一参数集包括：
    确定由所述模拟器处理原始图像而得到的经处理图像，所述原始图像由图像传感器捕获；
    基于所述神经网络对所述经处理图像执行所述计算机感知任务的结果和目标函数，确定任务损失；以及
    通过最小化所述任务损失，来更新所述模拟器的所述第二参数集。
  11. 一种用于确定图像信号处理参数的装置，其特征在于，所述装置包括：
    模拟器确定单元，被配置为确定用于模拟图像信号处理装置的模拟器，所述模拟器所包括的操作具有可微的表达方式；
    模拟器参数更新单元，被配置为通过共同训练所述模拟器和与所述图像信号处理装置相关联的神经网络，来更新所述模拟器的第一参数集，经训练的所述神经网络用于对所述图像信号处理装置所输出的图像执行计算机视觉感知任务；以及
    处理参数确定单元，被配置为基于经更新的所述第一参数集，确定所述图像信号处理装置的第二参数集。
  12. 根据权利要求11所述的装置，其特征在于，所述模拟器确定单元进一步被配置为：
    基于所述图像信号处理装置的操作集中的操作的可微性，生成所述模拟器的近似版本；
    基于所述图像信号处理装置和所述近似版本处理相同图像的差异，验证所述近似版本；以及
    如果所述近似版本通过验证，将所述近似版本确定为所述模拟器。
  13. 根据权利要求12所述的装置，其特征在于，所述模拟器确定单元进一步被配置为：
    如果确定所述操作集中的第一操作具有可微的表达方式，在所述近似版本中保留所述第一操作。
  14. 根据权利要求12所述的装置，其特征在于，所述模拟器确定单元进一步被配置为：
    如果确定所述操作集中的第二操作具有不可微的表达方式，确定所述第二操作的重要性是否高于阈值重要性，所述重要性指示所述第二操作对所述计算机视觉感知任务的结果或图像质量的影响；以及
    如果确定所述第二操作的重要性高于所述阈值重要性，在所述近似版本中用可微函数表示所述第二操作，所述可微函数用于近似所述第二操作。
  15. 根据权利要求14所述的装置，其特征在于，所述可微函数包括分段线性函数。
  16. 根据权利要求12-15中的任一项所述的装置，其特征在于，所述模拟器确定单元进一步被配置为：
    针对由图像传感器捕获的验证图像集中的验证图像，
    获得由所述图像信号处理装置处理所述验证图像而得到的图像属性的第一属性值；
    获得由所述近似版本处理所述验证图像而得到的所述图像属性的第二属性值；
    基于所述第一属性值和所述第二属性值，确定所述差异；以及
    基于针对所述验证图像集中的各个验证图像分别确定的所述差异，验证所述近似版本。
  17. 根据权利要求16所述的装置，其特征在于，所述模拟器确定单元进一步被配置为：
    如果分别确定的所述差异均小于第一阈值并且所述差异之和小于第二阈值，确定所述近似版本通过验证。
  18. 根据权利要求11-17中的任一项所述的装置，其特征在于，所述处理参数确定单元进一步被配置为：
    如果确定包括所述第二参数集中的第一参数的操作在所述模拟器中由可微函数表示，基于所述可微函数的参数的值来确定所述第一参数的值；以及
    如果确定包括所述第二参数集中的第二参数的操作在所述模拟器中被保留并且未由可微函数表示，基于所述图像信号处理装置中存储所述第二参数的寄存器的规格，将所述模拟器中与所述第二参数相对应的第三参数的值转换成所述第二参数的值。
  19. 根据权利要求11-18中的任一项所述的装置，其特征在于，所述装置还包括：
    设置单元，被配置为基于所确定的所述第二参数集，设置所述图像信号处理装置。
  20. 根据权利要求11-19中的任一项所述的装置，其特征在于，所述模拟器参数更新单元进一步被配置为：
    确定由所述模拟器处理原始图像而得到的经处理图像，所述原始图像由图像传感器捕获；
    基于所述神经网络对所述经处理图像执行所述计算机感知任务的结果和目标函数，确定任务损失；以及
    通过最小化所述任务损失，来更新所述模拟器的所述第二参数集。
  21. 一种感知系统，其特征在于，所述感知系统包括：
    图像信号处理装置，被配置为处理来自图像传感器的数据，以生成经处理图像；以及
    神经网络，耦合到所述图像信号处理装置，并且被配置为对所述经处理图像执行计算机视觉感知任务，以生成感知结果，并且
    其中所述图像信号处理装置的参数集是通过根据权利要求1-10中任一项所述的方法确定的。
  22. 一种电子设备，其特征在于，包括：
    至少一个处理器；
    至少一个存储器，所述至少一个存储器被耦合到所述至少一个处理器并且存储用于由所述至少一个处理器执行的指令，所述指令当由所述至少一个处理器执行时，使所述电子设备执行根据权利要求1-10中任一项所述的方法。
  23. 一种计算机可读存储介质，其特征在于，其上存储有计算机程序，所述程序被处理器执行时实现根据权利要求1-10中任一项所述的方法。
  24. 一种计算机程序产品，其特征在于，包括计算机可执行指令，其中所述计算机可执行指令在被处理器执行时实现根据权利要求1-10中任一项所述的方法。
PCT/CN2021/117953 2021-09-13 2021-09-13 确定图像信号处理参数的方法、装置和感知系统 WO2023035263A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2021/117953 WO2023035263A1 (zh) 2021-09-13 2021-09-13 确定图像信号处理参数的方法、装置和感知系统
CN202180102253.8A CN117940951A (zh) 2021-09-13 2021-09-13 确定图像信号处理参数的方法、装置和感知系统

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/117953 WO2023035263A1 (zh) 2021-09-13 2021-09-13 确定图像信号处理参数的方法、装置和感知系统

Publications (1)

Publication Number Publication Date
WO2023035263A1 true WO2023035263A1 (zh) 2023-03-16

Family

ID=85507153

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/117953 WO2023035263A1 (zh) 2021-09-13 2021-09-13 确定图像信号处理参数的方法、装置和感知系统

Country Status (2)

Country Link
CN (1) CN117940951A (zh)
WO (1) WO2023035263A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2747026A1 (en) * 2012-12-20 2014-06-25 Valeo Schalter und Sensoren GmbH Method for determining the visibility of objects in a field of view of a driver of a vehicle, taking into account a contrast sensitivity function, driver assistance system, and motor vehicle
WO2019053052A1 (en) * 2017-09-12 2019-03-21 Itu Business Development A/S METHOD FOR (RE-) TRAINING MACHINE LEARNING COMPONENT
CN110414546A (zh) * 2018-04-27 2019-11-05 英特尔公司 使用中间损失函数训练图像信号处理器
CN111630854A (zh) * 2018-01-30 2020-09-04 高通股份有限公司 用于图像信号处理器调谐的系统和方法
CN111988544A (zh) * 2019-05-21 2020-11-24 三星电子株式会社 使用机器学习预测参数的最优值
CN112700378A (zh) * 2019-10-23 2021-04-23 顶级公司 图像信号处理

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2747026A1 (en) * 2012-12-20 2014-06-25 Valeo Schalter und Sensoren GmbH Method for determining the visibility of objects in a field of view of a driver of a vehicle, taking into account a contrast sensitivity function, driver assistance system, and motor vehicle
WO2019053052A1 (en) * 2017-09-12 2019-03-21 Itu Business Development A/S METHOD FOR (RE-) TRAINING MACHINE LEARNING COMPONENT
CN111630854A (zh) * 2018-01-30 2020-09-04 高通股份有限公司 用于图像信号处理器调谐的系统和方法
CN110414546A (zh) * 2018-04-27 2019-11-05 英特尔公司 使用中间损失函数训练图像信号处理器
CN111988544A (zh) * 2019-05-21 2020-11-24 三星电子株式会社 使用机器学习预测参数的最优值
CN112700378A (zh) * 2019-10-23 2021-04-23 顶级公司 图像信号处理

Also Published As

Publication number Publication date
CN117940951A (zh) 2024-04-26

Similar Documents

Publication Publication Date Title
CN111754596B (zh) 编辑模型生成、人脸图像编辑方法、装置、设备及介质
CN109583501B (zh) 图片分类、分类识别模型的生成方法、装置、设备及介质
US10049307B2 (en) Visual object recognition
WO2019169688A1 (zh) 车辆定损方法、装置、电子设备及存储介质
US10600171B2 (en) Image-blending via alignment or photometric adjustments computed by a neural network
CN108230278B (zh) 一种基于生成对抗网络的图像去雨滴方法
CN108520503A (zh) 一种基于自编码器和生成对抗网络修复人脸缺损图像的方法
JP2020502665A (ja) ターゲット・ドメイン画像へのソース・ドメイン画像の変換
JP2017224027A (ja) データのラベリングモデルに係る機械学習方法、コンピュータおよびプログラム
US10832149B2 (en) Automatic segmentation of data derived from learned features of a predictive statistical model
US11341034B2 (en) Analysis of verification parameters for training reduction
CN112241784A (zh) 训练生成模型和判别模型
US20210390667A1 (en) Model generation
US10936938B2 (en) Method for visualizing neural network models
CN110310247A (zh) 图像处理方法、装置、终端及计算机可读存储介质
CN112465709B (zh) 一种图像增强方法、装置、存储介质及设备
DE102022106057A1 (de) Authentifikatorintegriertes generatives adversariales netzwerk (gan) zur sicheren deepfake-erzeugung
CN113505854A (zh) 一种人脸图像质量评价模型构建方法、装置、设备及介质
US11335128B2 (en) Methods and systems for evaluating a face recognition system using a face mountable device
CN107729885B (zh) 一种基于多重残差学习的人脸增强方法
WO2020216286A1 (zh) 教师风格预测模型的训练方法及计算机存储介质
WO2023035263A1 (zh) 确定图像信号处理参数的方法、装置和感知系统
Hagn et al. Optimized data synthesis for DNN training and validation by sensor artifact simulation
CN115170919B (zh) 图像处理模型训练及图像处理方法、装置、设备和存储介质
KR20230068989A (ko) 멀티-태스크 모델의 학습을 수행하는 방법 및 전자 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21956442

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180102253.8

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE