CN114885094A

CN114885094A - Image processing method, image processor, image processing module and equipment

Info

Publication number: CN114885094A
Application number: CN202210307243.4A
Authority: CN
Inventors: 吴凡
Original assignee: Beijing Kuangshi Technology Co Ltd; Beijing Megvii Technology Co Ltd
Current assignee: Beijing Kuangshi Technology Co Ltd; Beijing Megvii Technology Co Ltd
Priority date: 2022-03-25
Filing date: 2022-03-25
Publication date: 2022-08-09
Anticipated expiration: 2042-03-25
Also published as: CN114885094B

Abstract

The embodiment of the application provides an image processing method, an image processor, an image processing module and equipment, wherein the image processing method is applied to the image processor and comprises the following steps: acquiring an image to be processed; and obtaining a target output image corresponding to the image to be processed at least by inquiring a lookup table, wherein the lookup table is used for storing output data respectively corresponding to each traversal image block in a plurality of traversal image blocks, and the traversal image blocks are image blocks determined by the receptive field of the target neural network. Compared with an image processor scheme realized by adopting a neural network, the image processing method of the embodiment of the application is greatly improved in operation speed, and consumed logic and hardware resources are obviously reduced.

Description

Image processing method, image processor, image processing module and equipment

Technical Field

The present application relates to the field of image processing, and in particular, to an image processing method, an image processor, an image processing module, and an image processing device.

Background

An image processor isp (image Signal process) is a processing pipeline that converts a RAW image (e.g., RAW data in Bayer mosaic format) output by an image sensor (e.g., CMOS sensor and CCD sensor) into an RGB format or YUV format image. For example, for camera products, it is also necessary to further convert the RGB format or YUV image output by the image processor into JPEG format for convenient storage.

With the rise of Deep Learning technology (Deep Learning), it has been proposed to use a neural network to replace an image processor ISP in the related art and achieve a better effect, but due to the complexity of a neural network model, such a technical scheme has no advantages in time and chip area, and cannot be widely applied in practical scenes.

Disclosure of Invention

The image processing method of the embodiment of the application adopts a table look-up mode to obtain a target output image corresponding to an image to be processed, so that compared with the technical scheme of adopting a neural network to realize the image processor, the operation speed is greatly improved, and consumed logic and hardware resources are obviously reduced.

In a first aspect, some embodiments of the present application provide an image processing method applied to an image processor, the image processing method including: acquiring an image to be processed; and obtaining a target output image corresponding to the image to be processed at least by inquiring a lookup table, wherein the lookup table is used for storing output data respectively corresponding to each traversal image block in a plurality of traversal image blocks, and the traversal image blocks are image blocks determined by the receptive field of the target neural network.

Some embodiments of the application obtain the target output image corresponding to the image to be processed by replacing the neural network with a table look-up mode, thereby improving the data processing speed and further saving the resource consumption.

In some embodiments, the obtaining, at least by querying a lookup table, a target output image corresponding to the image to be processed includes: traversing the image to be processed to obtain all image blocks to be inquired, wherein the size of the image blocks to be inquired is the same as that of the traversed image blocks; obtaining output data corresponding to at least part of the image blocks to be inquired in all the image blocks to be inquired by inquiring the lookup table; and obtaining the target output image according to the output data.

Some embodiments of the application obtain the image block to be queried in a traversal mode and obtain output data corresponding to at least part of the image block to be queried through table lookup, so that the data processing speed can be increased.

In some embodiments, before the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table, the image processing method further includes: obtaining all image blocks capable of being input into the target neural network according to the size of the receptive field of the target neural network and the pixel value bit width of the pixel points on the image to be processed, wherein the size of each image block capable of being input into the target neural network is the same as the size of the receptive field of the target neural network, the maximum value of the pixel value of each pixel point on each image block is determined by the pixel value bit width, and the image blocks capable of being input into the target neural network are the plurality of traversal image blocks or the image blocks capable of being input into the target neural network comprise the plurality of traversal image blocks; and acquiring the lookup table according to all image blocks capable of inputting the target neural network.

According to some embodiments of the application, the target neural network acquires the output data corresponding to the image blocks capable of being input into the target neural network in advance to obtain the lookup table, and the image blocks to be inquired included in the image to be processed can all obtain the corresponding target output image by inquiring the lookup table, so that the universality of the technical scheme can be improved.

In some embodiments, the obtaining the look-up table according to all image blocks inputtable to the target neural network comprises: sampling all image blocks which can be input into the target neural network according to a target sampling rate to obtain a target image block set; acquiring output data corresponding to each image block in the target image block set through the target neural network to obtain the lookup table; the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table includes: and obtaining the target output image by inquiring the lookup table and an interpolation algorithm.

Some embodiments of the present application sample all image blocks that can be input into the target neural network and obtain output data corresponding to a part of the sampled image blocks through the target neural network, and the lookup table is only used for storing the part of the output data, so that the data processing amount can be effectively reduced and the size of the lookup table can be reduced.

In some embodiments, after the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table, the image processing method further includes: and carrying out multi-frame noise reduction processing and distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

Some embodiments of the present application further include a method for post-processing a target output image, so that an image obtained by using the image processor provided by the embodiments of the present application can be directly used for display or target recognition purposes.

In some embodiments, the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, and automatic white balance processing on an original image acquired by an image sensor.

According to some embodiments of the application, black level correction, vignetting correction, dead pixel correction and automatic white balance processing of an original image acquired by an image sensor are not realized in a table look-up mode but are realized by software and hardware of modules designed by a related technology, so that on one hand, the workload for acquiring the look-up table can be reduced, the data processing speed is increased, and on the other hand, the time for learning the functions of the modules by adopting a neural network can be reduced, and the speed for acquiring a target neural network is increased.

In some embodiments, the number of the lookup tables is multiple, wherein one lookup table is obtained according to one target neural network.

Some embodiments of the present application increase the receptive field by increasing the number of look-up tables to improve the quality of the target output image obtained by the look-up table approach.

In some embodiments, the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, automatic white balance processing, and bit compression processing on an original image acquired by an image sensor.

Some embodiments of the present application use the image obtained by the bit compression processing as the basis for the lookup table, which can further increase the speed of data processing and reduce the size of the lookup table.

In some embodiments, the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, automatic white balance, and multi-frame noise reduction on an original image acquired by an image sensor, wherein after the target output image corresponding to the image to be processed is obtained at least by querying a lookup table, the image processing method further includes: and carrying out distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

According to some embodiments of the application, a single-frame image output by the multi-frame noise reduction module is used as an image to be processed input into the table look-up module, so that the query speed can be increased, and a target output image corresponding to the image to be processed can be obtained more quickly.

In a second aspect, some embodiments of the present application provide an image processor configured to obtain a target output image corresponding to an image to be processed at least by querying a lookup table, wherein the lookup table is used to store output data corresponding to a plurality of traversed image blocks, the traversed image blocks being image blocks determined by a receptive field of a target neural network.

In some embodiments, the image processor comprises: the table look-up module is configured to obtain output data corresponding to a target image block to be queried through a look-up table, wherein the target image block to be queried is a whole or part of image blocks to be queried included in the image to be queried, the image block to be queried is obtained by traversing the image to be processed, and the target output image is obtained according to the output data.

In some embodiments, the target image block to be queried is the partial image block to be queried, the all image blocks to be queried include the partial image block to be queried and remaining image blocks to be queried, and the image processor further includes: and the interpolation calculation module is configured to calculate the output data of the residual image blocks to be inquired through an interpolation algorithm.

In some embodiments, the image processor further comprises: the image preprocessing module is configured to preprocess an original image acquired by an image sensor to obtain the image to be processed; and/or an image post-processing module configured to process the target output image to obtain a target image; wherein the target image may be used for display or target recognition.

In some embodiments, the image pre-processing module comprises: the device comprises a black level correction module, a lens vignetting correction module, a dead pixel correction module and an automatic white balance module; the image post-processing module includes: the device comprises a multi-frame noise reduction module and a distortion removal module.

In some embodiments, the image pre-processing module further comprises a data compression module.

In some embodiments, the image pre-processing module comprises: the device comprises a black level correction module, a lens vignetting correction module, a dead pixel correction module, an automatic white balance module and a multi-frame noise reduction module; the image post-processing module includes: and a distortion removal module.

In some embodiments, the number of the table lookup module is multiple, and one table lookup module is used for storing output data obtained according to one target neural network.

In a third aspect, some embodiments of the present application provide an image sensor module, including: an image sensor; and an image processor as described in any embodiment of the second aspect; wherein the output data of the image sensor is used as the input data of the image processor.

In a fourth aspect, some embodiments of the present application provide an electronic device comprising: a camera and an image processor as in any embodiment of the second aspect or an image sensor module as in the third aspect.

In a fifth aspect, some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, is adapted to implement the method according to any of the embodiments of the first aspect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

FIG. 1 is a diagram of an image processing system architecture provided in the related art;

fig. 2 is an image processor provided in the related art;

FIG. 3 is a diagram of an image processor according to an embodiment of the present application;

FIG. 4 is a second image processor according to an embodiment of the present application;

FIG. 5 is a third exemplary embodiment of an image processor;

FIG. 6 is a fourth embodiment of an image processor;

FIG. 7 is a flowchart of an image processing method according to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of input data included in intercepted training data provided by an embodiment of the application;

FIG. 9 is a schematic diagram of ideal output data included in truncated training data according to an embodiment of the present application;

FIG. 10 is a schematic diagram of the preprocessing of the input data obtained in FIG. 8 according to an embodiment of the present disclosure;

FIG. 11 is a diagram of a neural network architecture provided in an embodiment of the present application;

fig. 12 is a schematic diagram of obtaining a lookup table according to a plurality of traversal image blocks and a target neural network provided in an embodiment of the present application;

fig. 13 is a block diagram of an image sensor module according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.

In recent years, technical research based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition, has been actively developed. Artificial Intelligence (AI) is an emerging scientific technology for studying and developing theories, methods, techniques and application systems for simulating and extending human Intelligence. The artificial intelligence subject is a comprehensive subject and relates to various technical categories such as chips, big data, cloud computing, internet of things, distributed storage, deep learning, machine learning and neural networks. Computer vision is used as an important branch of artificial intelligence, particularly a machine is used for identifying the world, and the computer vision technology generally comprises the technologies of face identification, living body detection, fingerprint identification and anti-counterfeiting verification, biological feature identification, face detection, pedestrian detection, target detection, pedestrian identification, image processing, image identification, image semantic understanding, image retrieval, character identification, video processing, video content identification, behavior identification, three-dimensional reconstruction, virtual reality, augmented reality, synchronous positioning and map construction (SLAM), computational photography, robot navigation and positioning and the like. With the research and development of artificial intelligence technology, the technology is applied to many fields, such as security protection, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone images, cloud services, smart homes, wearable equipment, unmanned driving, automatic driving, intelligent medical treatment, face payment, face unlocking, fingerprint unlocking, human evidence verification, smart screens, smart televisions, cameras, mobile internet, network, beauty, makeup, medical beauty, intelligent temperature measurement and the like.

As described in the background section, the use of a neural network instead of the entire image processor ISP in the related art often faces the problem of excessive resource consumption, resulting in a large area size and cost of the final chip.

At least to solve the above technical problem, some embodiments of the present application utilize a target neural network (i.e., a trained neural network) to obtain output data corresponding to various image blocks that can be input into the target neural network, or obtain output data corresponding to various image blocks that can be input into the target neural network (e.g., sampling all of the image blocks that can be input into the target neural network to obtain various image blocks that can be input into the target neural network), and construct a lookup table based on the output data. It is understood that the image processor of the embodiment of the present application may obtain the target output image corresponding to the image to be processed directly by querying the constructed lookup table after the lookup table is constructed. The operation speed of the table lookup operation can be increased sharply by directly operating the table lookup operation on the FPGA, and resources consumed by converting most of traditional image processor algorithms (namely ISP algorithms) into hardware implementation can be saved.

That is, some embodiments of the present application provide an image processor based on a lookup table, and the embodiments of the present application can at least effectively overcome the technical problem that hardware resources and logic resources required by a deep learning network are too large.

Referring to fig. 1, fig. 1 is a diagram illustrating an image processing system according to the related art, the image processing system including: a lens 100, an image sensor 200, and an image processor 300. The process of acquiring a display image is illustratively set forth below in conjunction with fig. 1.

The sunlight of fig. 1 irradiates the cup, and the light reflected by the cup passes through the lens 100 to reach the image sensor 200, which converts the light signal reflected by the cup into an electrical signal to obtain an original image (i.e., RAW image data or simply RAW data). Since the original image captured by the image sensor cannot be directly used for display or object segmentation or recognition, as shown in fig. 1, the original image captured by the image sensor 200 needs to be processed by the image processor 300 to obtain an object image that can be directly used for display or for object recognition or classification.

The function of the image processor 300 will be described below in connection with a CMOS type image sensor.

The image sensor 200 is a CMOS image sensor, which tends to have a problem of insufficient dynamic range, and thus, overexposure and underexposure may occur. In addition, this type of image sensor has relatively high noise and may not meet the high quality output requirements of real-world devices. In addition, the raw image collected by this type of image sensor is a bayer format (e.g., bayer) image, which cannot be directly used for display or target recognition, etc. (since each pixel point has only a pixel value of a certain color). These problems need to be solved by using an image processor isp (image Signal processor).

It can be understood that the image processor 300 needs to perform processes such as noise reduction, detail enhancement, interpolation process, visibility enhancement, etc. on the original image collected by the image sensor, and a target image is obtained after the processing by the image processor 300, wherein a part of the target image needs to be output to a high definition interface such as a terminal device such as a monitor, a mobile phone, and a computer display, and a part of the target image is to be used in the fields of image recognition and target positioning, for example, target positioning of an intelligent robot, etc. Embodiments of the present application do not limit the specific use of the target image.

The image output by the image sensor of fig. 1 is a RAW image, which is RAW data in which a CMOS or CCD image sensor converts a captured light source signal into a digital signal. The RAW file is a file in which RAW information of a digital camera sensor is recorded, and at the same time, some Metadata (Metadata such as ISO setting, shutter speed, aperture value, white balance, etc.) generated by camera shooting is recorded. Each pixel of the image sensor corresponds to a color filter, the filters are distributed according to Bayer patterns, and the corresponding image output by the image sensor is a Bayer format image. The image processor of fig. 1 is at least used to obtain an RGB image by color interpolation from data of each pixel (i.e., RAW RGB dataRaw data (RAW RGB)) acquired from the image sensor. In some embodiments, the image processor may further convert the RGB image to a YUV format image (Y stands for luminance and U and V in combination may represent chrominance).

One specific image processor disclosed by the related art is exemplarily set forth below in conjunction with fig. 2.

The image processor 300 of fig. 2 includes:

a black level correction processing module 301 (labeled as BLC in the figure) configured to perform black level correction processing on the raw image captured by the image sensor, resulting in a black level corrected image.

A lens vignetting correction processing module 302 (labeled LSC in the figure) configured to perform lens vignetting correction on the black level corrected image to obtain a lens vignetting corrected image.

And a dead pixel correction processing module 303 (marked as BPC in the figure), configured to perform dead pixel correction (divided into static dead pixels and dynamic dead pixels) on the image after the lens shading correction, to obtain an image after the dead pixel correction.

An automatic white balance processing module 304 (labeled AWB in the figure) configured to perform automatic white balance processing on the dead-pixel corrected image, resulting in an automatic white balance processed image.

A high dynamic range image processing module 305 (labeled HDR in the figure) configured to perform high dynamic range imaging processing on the white balance processed image (each time image includes a long exposure image, a medium exposure image, and a short exposure image) to obtain a high dynamic range image.

A Bit compression processing module 306 (labeled DRC in the figure) configured to Bit compress the resulting high dynamic range image (high-Bit RAW data into relatively low-Bit RAW data) to obtain a low-Bit RAW image. It is understood that the bit width of the data is reduced while preserving as much dynamic range of the image as possible. The aim is to reduce memory space and logic computation resources and thereby reduce chip area.

And a Bayer-domain noise reduction processing module 307 (labeled Bayer NR in the figure), configured to perform Bayer-domain noise reduction processing on the low-order RAW image to obtain a noise-reduced image.

A demosaicing processing module 308 (labeled DMS in the figure) configured to demosaic the noise-reduced image, thereby converting the RAW domain image into an RGB format image.

And a color correction processing module 309 (marked as CCM in the figure), which performs color correction on the RGB format image according to the color correction matrix to obtain a color-corrected image.

A format conversion processing module 310 (denoted as RYC in the figure) having a function of converting RGB data into YUV data, i.e., converting a color-corrected image into a YUV format image.

The multi-frame noise reduction processing module 311 (marked as MFNR in the figure) is configured to perform multi-frame noise reduction processing on the YUV format image, so as to obtain a second noise-reduced image.

A local tone mapping processing module 312 (denoted LTM in the figure) is configured to perform local tone mapping on the second noise reduced image to obtain a mapped image.

And a single-frame denoising processing module 313 (labeled as YNR in the figure) configured to perform single-frame denoising in the YUV domain on the mapped image, resulting in a third denoised image.

And a sharpening module 314 (marked as LSH in the figure) configured to perform sharpening on the third noise-reduced image to obtain a sharpened image.

A color noise reduction processing module 315 (marked as CNR in the figure), configured to perform color noise reduction on the sharpened image to obtain a color noise reduced image; and

a distortion removal processing module 316 (labeled as DeWarp) configured to perform distortion removal processing on the color-reduced and noise-removed image to obtain a target image.

As can be seen from the above description, the target image can be directly used for display, and can also be used for the purpose of target classification or identification.

It should be noted that fig. 2 is only used to exemplarily illustrate various processing modules that may be included in the image processor, and it is understood that a person skilled in the art may adaptively change the processing modules included in the image processor of fig. 2 according to the scene difference, the purpose of acquiring the target image, and the like, for example, the front and back positions between some processing modules included in the image processor of fig. 2 may be adjusted, unnecessary processing modules of fig. 2 may be deleted, or other processing modules may be added to the image processor of fig. 2 to achieve a specific purpose. Embodiments of the present application do not limit the type of processor modules included in the image processor and the front-to-back location of each processing module in the image processor.

As described in the background section, the related art solution uses a neural network to implement the functions of the image processor 300, but such a solution is resource-consuming and time-consuming. At least to solve the technical problem, some embodiments of the present application implement the image processing algorithm of one or more processing modules included in the image processor 300 by using a table lookup.

Some embodiments of the present application provide an image processor configured to obtain a target output image corresponding to an image to be processed at least by querying a lookup table, wherein the lookup table is used to store output data corresponding to a plurality of traversed image blocks, the traversed image blocks being image blocks determined by a receptive field of a target neural network.

It should be noted that the target neural network is obtained by training a neural network, and the size of the traversal image block is determined according to the receptive field of the target neural network.

In some embodiments of the present application, one lookup table module is used to replace all image processing modules on the existing image processor ISP, and it is understood that, because the image processors in the prior art include different image processing modules according to differences of application scenarios and application purposes, the specific functional modules that the lookup table module can replace may also have differences in different embodiments. For example, in some embodiments of the present application, the whole image processor 300 in fig. 2 is replaced by a lookup table module, which finds the output data corresponding to each pixel point on the image to be processed according to a constructed lookup table to obtain the target image, and the target image can be directly used for displaying or used for tasks such as target identification and classification.

In other embodiments of the present application, the lookup module is used to replace one or more image processing modules in the image processor, for example, the lookup module of the embodiment of the present application is used to replace the HDR image processing module, and it is understood that the corresponding image processor necessarily needs to include other image processing modules, for example, a black level correction module, an automatic white balance module, and the like. For example, in some embodiments of the present application, some of the processing modules included in the image processor 300 of fig. 2 are replaced with a look-up table module, as shown in fig. 3, 4, 5, and 6. It can be understood that, because a part of the processing modules included in the image processor is replaced by one search module, the remaining processing modules may be implemented based on the technical solutions disclosed in the related art, and details of specific algorithms used by the processing modules or related hardware circuits are not repeated.

It should be noted that the image processors shown in fig. 3, 4, 5, and 6 are modifications of the image processor shown in fig. 2, and since the present application may further include an image processor having a structure different from that of fig. 2, the technical solution of replacing one or more processing modules included in the processor having a different structure with a look-up table module should also be covered in the scope of the present application. The image processors of fig. 3-6 and the design concept of these processors are described below one by one.

The image processor of fig. 3 differs from the image processor of fig. 2 in that all processing modules included in the image processor provided in fig. 2, except for BLC, LSC, BPC, AWB, MFNR, and Dewarp, are replaced with a first look-up table module 321 (i.e., the LUT of fig. 3). In fig. 3, the input image (i.e. the image to be processed) of the table lookup module is the image output by the AWB module, and the target output image of fig. 3 needs to be input with MFNR and Dewarp for multi-frame denoising and deskew processing to obtain the target image.

Since the algorithms of BLC, LSC, BPC, AWB, MFNR, and DeWarp are simpler to implement or are related to calibration (i.e., parameters associated with the image sensor itself), in some embodiments of the present application, the functions of these processing modules are not implemented using a deep learning network, and therefore the look-up table module is not used to implement the functions of these processing modules.

The image processor of fig. 4 differs from the image processor of fig. 2 in that all processing modules included in the image processor provided in fig. 2, except BLC, LSC, BPC, AWB, DRC, MFNR, and Dewarp, are replaced with a second look-up table module 322 (i.e., the LUT of fig. 4). In fig. 4, the input image (i.e. the image to be processed) of the table lookup module is the image output by the DRC module, and the target output image of fig. 4 needs to be input with MFNR and Dewarp for multi-frame denoising and de-distortion processing to obtain the target image.

It should be noted that the image processor of fig. 4 and the image processor of fig. 3 have an advantage in processing speed. That is, when the range of RAW data supported by the image sensor 200 is too large (i.e., the data bit width corresponding to each pixel point is too large) (e.g., 16bit/20 bit), the data processing speed can be further increased by using the image processor of fig. 4. This is because, in fig. 4, DRC first performs data compression on the image output by the AWB module (for example, compresses each pixel bit width to 10bit/12bit), and it is because the bit width of each pixel on the image to be processed is reduced, the total number of image blocks that can be input into the target neural network is reduced, and it can be understood that the speed of increasing the table lookup speed to obtain the corresponding target output image is also increased because the table lookup table is reduced.

The image processor of fig. 5 differs from the image processor of fig. 2 in that all processing modules included in the image processor provided in fig. 2, except for BLC, LSC, BPC, AWB, MFNR and Dewarp, are replaced with a third look-up table module 325 (i.e. the LUT of fig. 4). In fig. 5, the input image (i.e. the image to be processed) of the table lookup module is the image output by the MFNR module, and the target output image of fig. 5 needs to be input into Dewarp for performing the distortion removal processing to obtain the target image. It will be appreciated that the image processor of fig. 5 differs from the image processor of fig. 3 in that fig. 5 places the MFNR before the table look-up module.

It should be noted that, fig. 5 places the multi-frame noise reduction module of fig. 3 before the table lookup module, so that compared with the image processor of fig. 3, only one frame of image of the image processor of fig. 5 needs to obtain the target output image by querying the lookup table, and therefore, compared with the technical scheme in which all the frames need to perform table lookup to obtain the corresponding target output image, the image processor of fig. 5 obviously increases the data processing speed.

In some embodiments of the present application, a plurality of table lookup modules are used to replace a plurality of image processing units on the image processor, respectively, where one table lookup module may correspond to one image processing module or may correspond to a plurality of image processing modules. For example, the image processor of fig. 6 is different from the image processor of fig. 3 in that, by replacing one table lookup module of fig. 3 with three adjacent table lookup modules, i.e., the fourth table lookup module 326, the fifth table lookup module 327, and the sixth table lookup module 328 of fig. 6, the quality of the target output image obtained by the solution using a plurality of table lookup modules is better than that of the solution using a single table lookup module of fig. 3. This is because the plurality of table lookup modules correspond to the plurality of target neural networks (one table lookup module corresponds to one lookup table obtained from one target neural network), and thus the plurality of table lookup modules equivalently improve the receptive field of the neural networks. It will be appreciated that in some embodiments of the present application, some of the processing modules included in the image processor 300 may be replaced with a plurality of look-up tables, as shown in FIG. 6.

It is understood that the look-up tables corresponding to the look-up modules of fig. 3-6 are different because the processing modules replaced by the look-up modules are different. In addition, the input data of the query module in the embodiment of the present application is the image to be processed (i.e. the data corresponding to the image to be processed), and it cannot be understood that the types of the image to be processed at least include: an image acquired by an image sensor is taken as an image to be processed at the position, or an image processed by black level correction, lens halo correction, dead pixel correction and white balance is taken as an image to be processed at the position (for example, fig. 3), or an image processed by black level correction, lens halo correction, dead pixel correction, white balance processing and bit width compression is taken as an image to be processed at the position (for example, fig. 4), or an image processed by black level correction, lens halo correction, dead pixel correction, white balance processing and multi-frame noise reduction is taken as an image to be processed at the position (for example, fig. 5). The format of the target output image may include: the RAW format image, the RGB format image or the YUV format image, it is understood that the specific format of the target output image is related to the kind of processing module replaced by the table look-up module.

That is, in some embodiments of the present application, the image processor includes: the table look-up module is configured to obtain output data corresponding to a target image block to be queried by querying a lookup table (a method for obtaining the lookup table will be described below), where the target image block to be queried is all or part of image blocks to be queried included in the image to be queried, the image block to be queried is obtained by traversing the image to be processed (a process for obtaining the image block to be queried will be described below), and the target output image is obtained according to the output data.

It will be appreciated that in order to reduce the amount of data processing, in some embodiments of the present application, the image processor comprises, in addition to at least one look-up module: the image preprocessing module is configured to preprocess an original image acquired by an image sensor to obtain the image to be processed; and/or an image post-processing module configured to process the target output image to obtain a target image; wherein the target image may be used for display or target recognition. That is, in some embodiments of the present application, the image processor includes an image pre-processing module and at least one look-up table module. In some embodiments of the present application, the image processor comprises at least one lookup module and an image post-processing module. In some embodiments of the present application, an image processor comprises: the device comprises an image preprocessing module, at least one table look-up module and an image post-processing module.

Some examples of image pre-processing modules and image post-processing modules are set forth below in connection with fig. 3-6.

As shown in fig. 3, in some embodiments of the present application, the image pre-processing module comprises: black level correction module, camera lens vignetting correction module, dead pixel correction module and automatic white balance module, image post-processing module includes: the device comprises a multi-frame noise reduction module and a distortion removal module.

As shown in fig. 4, in some embodiments of the present application, the image pre-processing module comprises: black level correction module, camera lens vignetting correction module, dead pixel correction module, automatic white balance module and data compression module, image post-processing module includes: the device comprises a multi-frame noise reduction module and a distortion removal module.

As shown in fig. 5, in some embodiments of the present application, the image pre-processing module comprises: a black level correction module, a lens vignetting correction module, a dead pixel correction module, an automatic white balance module, and a multi-frame noise reduction module, where in fig. 5 the image post-processing module includes: and a distortion removal module.

In some embodiments of the present application, the at least one table lookup module includes a plurality of table lookup modules, and one table lookup module is configured to store output data obtained according to one target neural network. As shown in fig. 6, three look-up tables are included.

It will be understood that the output image of AWB is taken as the image to be processed in fig. 3, but the position of the image processor block included in the image pre-processing block in fig. 3 may be adjusted, and in these embodiments, the output of other image processing blocks may be taken as the image to be processed. Similar embodiments can be generalized to fig. 4-6, and are not described herein in detail to avoid repetition.

The image processor method performed by any of the image processors described above is exemplarily set forth below.

As shown in fig. 7, some embodiments of the present application provide an image processing method applied to an image processor, where the image processing method includes: s101, acquiring an image to be processed; and S102, obtaining a target output image corresponding to the image to be processed at least by inquiring a lookup table, wherein the lookup table is used for storing output data corresponding to each traversal image block in a plurality of traversal image blocks, and the traversal image blocks are image blocks determined by the receptive field of the target neural network.

It can be understood that, in order to perform the query operation of S102, the corresponding lookup table needs to be obtained in advance, and therefore, the image processing method of the present application further includes a step of obtaining the lookup table.

The above steps are exemplarily set forth below.

In some embodiments of the present application, the to-be-processed image referred to in S101 is an image obtained by performing black level correction, vignetting correction, dead pixel correction, and automatic white balance processing on an original image acquired by the image sensor 200, for example, the to-be-processed image corresponding to fig. 3.

In some embodiments of the present application, the to-be-processed image referred to in S101 is an image obtained by performing black level correction, vignetting correction, dead pixel correction, automatic white balance processing, and bit compression processing on an original image acquired by an image sensor, for example, the to-be-processed image corresponding to fig. 4.

In some embodiments of the present application, the to-be-processed image S101 is an image obtained by performing black level correction, vignetting correction, dead pixel correction, automatic white balance, and multi-frame noise reduction on an original image acquired by an image sensor, for example, the to-be-processed image corresponding to fig. 5.

In some embodiments of the present application, the number of the lookup tables is multiple, it is understood that the number of the corresponding query modules is also one more, and one query module corresponds to one lookup table, for example, the structure of three lookup tables is shown in fig. 6.

The following exemplarily sets forth the implementation of S102.

Since the lookup table stores the output data corresponding to the traversal image block, it can be understood that the image block to be queried needs to be acquired in order to obtain the target output image corresponding to the image to be processed. In some embodiments of the present application, each image block to be queried is obtained in a traversal manner.

For example, S102 illustratively includes:

the method comprises the steps of firstly, traversing the image to be processed to obtain all image blocks to be inquired, wherein the size of the image blocks to be inquired is the same as that of the traversed image blocks. For example, reading the image to be processed, setting a sliding window with the same size as the traversal image block (or the same size as the image block capable of inputting the target neural network), and gradually sliding the sliding window from the upper left corner of the image to be processed to obtain each image block to be queried.

And secondly, obtaining output data corresponding to at least part of the image blocks to be inquired in all the image blocks to be inquired by inquiring the lookup table.

The output data obtained by the table lookup may be RGB format data or YUV format data, and it may be understood that the specific type of the output data is the same as the output data corresponding to the processing module replaced by the query module. For example, if the target output image is a YUV format image, the output data may be specific values of Y, U and V, which correspond to one pixel point.

It is understood that, if the lookup table includes output data corresponding to all image blocks that can be input into the target neural network, the second step obtains output data corresponding to all image blocks to be queried by querying the lookup table. If the lookup table only comprises output data corresponding to part of the image blocks which can be input into the target neural network, the second step obtains the output data corresponding to part of the image blocks to be inquired by inquiring the lookup table, and the rest of the output data corresponding to the image blocks to be inquired are obtained by an interpolation algorithm.

For example, in some embodiments of the present application, a total number of the all to-be-queried image blocks is N, a total number of the at least part of the to-be-queried image blocks is M, and remaining to-be-queried image blocks are image blocks, except for the at least part of the to-be-queried image blocks, included in the all to-be-queried image blocks, where M and N are integers, and N is greater than M, where querying the lookup table and an interpolation algorithm to obtain the target output image includes: obtaining first output data corresponding to the M image blocks to be inquired by inquiring the lookup table; obtaining second output data corresponding to each image block to be inquired in the rest image blocks to be inquired through an interpolation algorithm; and obtaining the target output image according to the first output data and the second output data. For example, the first output data and the second output data are respectively used as the pixel values of the pixels on the target output image. It can be understood that, if the target output image is an image in RGB format, the first output data is the RGB values of the corresponding pixel points obtained by the query, the second output data is the RGB values of the corresponding pixel points obtained according to the interpolation algorithm, and the target output image is an image composed of the RGB values of all the pixel points. For example, if the target output image is an image in YUV format, the first output data is the YUV value of the corresponding pixel point obtained by the query, the second output data is the YUV value of the corresponding pixel point obtained according to the interpolation algorithm, and the target output image is an image composed of the YUV values of all the pixel points.

It should be noted that the description defined by the first and second pairs of output data is used to distinguish one type of data from another type of data (which are not stored in the lookup table) that is obtained by looking up the lookup table.

And thirdly, obtaining the target output image according to the output data. For example, in some embodiments of the present application, the lookup table is used to store output data corresponding to all image blocks that can be input into the target neural network, and then output data corresponding to all image blocks to be queried can be obtained through querying, and the output data is used as a pixel value of each pixel point on the target output image to obtain the target output image. In some embodiments of the present application, the lookup table is used to store output data corresponding to a part of image blocks that can be input into the target neural network, and then the output data obtained by querying and the output data obtained by interpolation calculation are used as pixel values of each pixel point on the target output image, so as to obtain the target output image.

It is understood that, before executing S102, the image processing method of some embodiments of the present application further includes the following step of obtaining a look-up table:

the method comprises the steps of firstly, obtaining all image blocks capable of being input into a target neural network according to the size of a receptive field of the neural network and the pixel value bit width of pixel points on an image to be processed, wherein the size of each image block capable of being input into the target neural network is the same as the size of the receptive field, and the maximum value of the pixel value of each pixel point on each image block capable of being input into the target neural network is determined by the pixel value bit width. For example, when the bit width of the pixel value of each pixel point on the image to be processed is a bit, the minimum value of the pixel value of each pixel point on each image block capable of being input into the target neural network is 0, and the maximum value is 2 ^A -1. Specifically, the bit width of the pixel value of each pixel point on the image to be processed is 12bits, and when the minimum value of the pixel value of each pixel point on each image block capable of being input into the target neural network is 0, the maximum value is 2 ¹² -1。

It should be noted that the target neural network is obtained by training a neural network, and the size of the traversal image block is determined according to the receptive field of the target neural network. The image patches of all inputtable target neural networks are a plurality of traversal image patches or the image patches of all inputtable target neural networks include the plurality of traversal image patches (e.g., the image patches of all inputtable target neural networks are employed to derive the plurality of traversal image patches).

And secondly, acquiring the lookup table according to all the image blocks capable of being input into the target neural network.

In some embodiments of the present application, the second step illustratively comprises: and inputting all image blocks capable of being input into the target neural network into the trained neural network to obtain output data corresponding to each image block, and obtaining a lookup table. It will be appreciated that the output image corresponding to the image to be processed can be obtained directly by looking up the look-up table if based on this look-up table. In some embodiments of the present application, the second step illustratively comprises: and inputting part of the image blocks which can be input into the target neural network into the trained neural network to obtain output data corresponding to the image blocks so as to obtain a lookup table. It can be understood that, in this case, if the image to be processed is to be processed, the table lookup and the interpolation algorithm are combined to obtain the target output image corresponding to the input image to be processed. For example, this step illustratively includes: sampling all image blocks which can be input into the target neural network according to a target sampling rate to obtain a target image block set (namely obtaining part of the image blocks which can be input into the target neural network); and acquiring output data corresponding to each image block in the target image block set according to the target neural network to obtain the lookup table. It is understood that S102 exemplarily includes, for such a lookup table: and obtaining the target output image by inquiring the lookup table and an interpolation algorithm. In some embodiments of the present application, the pixel value of each pixel point on an image block in the target image block set needs to cover the minimum value and the maximum value of the pixel point, for example, part of the image blocks inputtable to the target neural network may be obtained by uniformly sampling all the image blocks inputtable to the target neural network according to a sampling rate.

The following describes, with reference to fig. 8 to fig. 12, an exemplary process for obtaining a lookup table provided in some embodiments of the present application, it should be noted that, as shown in fig. 3, an architecture of an image sensor corresponding to these embodiments is illustrated, it is understood that a target neural network to be replaced by the lookup module of fig. 3 needs to have a function of obtaining a YUV format image with an HDR effect according to an input RAW image, the corresponding lookup module of fig. 3 can convert multiple frames of RAW data into YUV data, and the lookup module can implement the function according to a pre-stored lookup table.

For example, the process of obtaining the lookup table corresponding to the query module LUT of fig. 3 exemplarily includes: firstly, training a neural network according to the collected RAW data and YUV data to obtain a target neural network (it can be understood that the target neural network has a function of obtaining YUV data meeting requirements according to the input RAW data), then obtaining output data corresponding to various image blocks capable of being input into the network through the target neural network, and then establishing a lookup table based on the output data. When the method is applied specifically, a module to be queried corresponding to the image to be processed is obtained, the constructed lookup table is searched to obtain output data, and then the pixel value of each pixel point on the target output image is obtained.

It will be appreciated that in some embodiments of the present application, to implement the image processing method, the following steps need to be performed: the image sensor related parameters are obtained by calibration (for example, the related parameters include Again (analog gain), Dgain (digital gain), DRCgain, lens shaing related parameters and dewarp related parameters). Training data for training a neural network is intercepted from a prior art image processor, the training data comprising: input data (e.g., RAW data stream) and ideal output data (e.g., YUV data stream). And training the neural network according to the intercepted input data and the ideal output data to obtain the target neural network. A plurality of traversal patches (e.g., various patches that may be input into the target neural network) are input into the trained neural network (i.e., the target neural network) to obtain corresponding output data, which is stored as a look-up table. In specific application, the target output image corresponding to the image to be processed can be obtained by directly searching the constructed table, the running speed is greatly improved, and consumed logic and hardware resources are obviously reduced.

As can be known from the above method for obtaining the lookup table, in order to obtain the lookup table, the neural network needs to be trained to obtain the target neural network, and the target neural network may implement the functions of all the remaining processing modules in fig. 3 except BLC, LSC, BPC, AWB, MFNR, and Dewarp.

The following first exemplarily explains a process of acquiring a target neural network with reference to fig. 8, 9, 10, and 11, and the process exemplarily includes: training data is acquired (as shown in fig. 8 and 9), the training data is preprocessed (as shown in fig. 10), and the constructed neural network (as shown in fig. 11) is trained for multiple times based on the training data to obtain a target neural network (corresponding to the small-receptive-field convolutional neural network 400 of fig. 12).

Fig. 8 is input data for training a neural network, which is taken from an image processor of the related art (e.g., the image processor of fig. 2). For example, the image processor shown in fig. 2 is configured to intercept a RAW domain data stream (e.g., 16 bits) of high Bit of RAW output from the RAW output, and to have multiple frames of continuous RAW images at a fixed time interval, and if the target neural network is also required to have HDR function, it is also required to intercept images with three different exposures, namely long exposure, medium exposure, and short exposure. Specifically, the intercepted frame for training the neural network is shown in fig. 8, and includes a long exposure image, a medium exposure image and a short exposure image corresponding to the time t-1, a long exposure image, a medium exposure image and a short exposure image corresponding to the time t, and a long exposure image, a medium exposure image and a short exposure image corresponding to the time t-1.

Fig. 9 is ideal output data for training a neural network, taken from an image processor of the related art. For example, the YUV format image shown in fig. 2 is intercepted as training data, and it is possible to determine whether the training process for the neural network can be ended or not. The specific intercepted output data is as shown in fig. 9, that is, three consecutive frames of output images corresponding to the YUV format image are intercepted, and the three frames of output images respectively correspond to the three frames of consecutive images in fig. 8 one to one. It is to be understood that for one training process, it comprises: inputting the input stream intercepted in the figure 8 into a neural network, predicting three frames of output images through the neural network, then comparing the three frames of output images with three frames of ideal output images corresponding to the intercepted figure 9, determining whether the training process of the neural network can be finished according to a loss function, and obtaining a target neural network when the training of the neural network is finished according to the loss function.

It can be understood that a large amount of training data can be collected in order to improve the application scenario of the target neural network, the application scenarios corresponding to the training data can be enriched as much as possible, and the collected data volume is as large as possible. In order to make the size of the input image and the output image after the processing by the neural network the same, in some embodiments of the present application, a filling operation is further performed on the image corresponding to the input stream included in the training data set. In some embodiments of the present application, the truncated long-exposure, short-exposure, and mid-exposure images of frames similar to those of fig. 8 are stitched (e.g., fig. 10) and input to a neural network, which is trained. Fig. 10 is a diagram for illustrating an example that a long exposure, a medium exposure, and a short exposure corresponding to three moments are spliced, and then the data spliced in fig. 10 is input into a neural network to train the neural network.

While the neural network 400 shown in fig. 11 is also provided as an example in some embodiments of the present application, the network architecture employed in other embodiments of the present application may be different from that of fig. 11, for example, by adding several layers to fig. 11 or changing the receptive field size of fig. 11. It will be appreciated that the neural network's receptive field may be controlled at 5 x 5 or less for controlling the look-up table scale. The following example sets forth a process for training the neural network of fig. 11.

1) Inputting a gray scale image with the size of (H +1, W +1,1) (namely, an image obtained by carrying out zero padding and splicing on one line of an input image collected similarly to the image in FIG. 8), carrying out convolution on 16 convolution kernels with the size of (2,2,1) by taking 1 as a step length to obtain a feature map with the size of (H, W,16), carrying out relu activation function on the feature map, and then carrying out BN layer to obtain a feature map with the size of (H, W, 16).

2) And (3) after 32 convolution kernels with the size of (1,1,16) are carried out on the feature map with the size of (H, W,16) in the last step, and the feature map with the size of (H, W,32) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 32).

3) And (4) after 64 convolution kernels with the size of (1,1,32) are carried out on the feature map with the size of (H, W,32) in the last step, and the feature map with the size of (H, W,64) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 64).

4) And (4) after the feature map with the size of (H, W,64) in the previous step is subjected to 128 convolution kernels with the size of (1,1,64), and the feature map with the size of (H, W,128) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 128).

5) And (5) after the feature map with the size of (H, W,128) in the previous step is subjected to 64 convolution kernels with the size of (1, 128), and the feature map with the size of (H, W,64) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 64).

6) And (4) after 32 convolution kernels with the size of (1,1,64) are carried out on the feature map with the size of (H, W,64) in the last step, and the feature map with the size of (H, W,32) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 32).

7) And (3) after 16 convolution kernels with the size of (1,1,32) are carried out on the feature map with the size of (H, W,32) in the last step, and the feature map with the size of (H, W,16) is obtained by performing convolution with the step size of 1. The feature map passes through a relu activation function and then through the BN layer to obtain a feature map with the size of (H, W, 16).

8) And (3) carrying out convolution on the feature map with the size (H, W,16) in the previous step by taking 1 as a step size through 3 convolution kernels with the size (1,1,16) to obtain a result map with the size (H, W, 3).

For example, the loss function of the neural network corresponding to fig. 11 may be mse (mean Squared error), the activation function may be ReLu, and the Rotation engineer phase-change may be used to increase the receptive field to 3 × 3 in the training process.

It is to be understood that the neural network of fig. 11 is trained according to the truncated training data to obtain a target neural network (e.g., the small-receptive-field convolutional neural network Net _0 shown in fig. 12), and how to obtain the look-up table according to the target neural network is exemplarily described below with reference to fig. 12.

The network architecture in conjunction with FIG. 11 maySince the receptive field of the neural network is 2 x 2, the size of all the image blocks that can be input into the target neural network is 2 x 2 (both are equal in size). Because the bit width of each pixel point pixel value on the image to be processed is 12bits, when the minimum value of the pixel points on the image block which can be input into the target neural network is 0, the maximum value of the pixel point is 2 ¹² -1 ═ 4095. It is understood that all image blocks that can be input into the target neural network in fig. 12 include all data blocks in which each pixel point on the image block is increased from 0 to 4095 step by step.

As described above, in some embodiments of the present application, all image blocks that can be input into the target neural network (i.e., the input image blocks of fig. 12) included in fig. 12 may be input into the target neural network to obtain output data corresponding to each image block, so as to obtain the lookup table.

In order to reduce the size of the lookup table and reduce the data processing amount, in some embodiments of the present application, all image blocks that can be input into the target neural network shown in fig. 12 may also be sampled according to the target sampling rate, and a part of the sampled image blocks is input into the target neural network, so that only output data corresponding to the part of the image blocks is obtained. It can be understood that, in specific applications, since the image block to be queried can obtain the output data in a table lookup manner, the output data of the image block needs to be obtained by an interpolation algorithm, so as to obtain the target output image. It should be noted that the output data is the pixel value of each pixel point on the target output image. For example, the sampling rate is 32, and the pixel values corresponding to the sampled image blocks need to cover 0 to 4095. For example, the interpolation algorithm may use a trilinear interpolation (or a nearest neighbor interpolation algorithm). It will be appreciated that the sampling rate may also be 64, etc.

The following exemplary explanation further processes the target output image obtained in S102 to obtain a better quality target image for display or target recognition.

In some embodiments of the present application, after S102, in order to obtain a higher quality display image or an image for object recognition or classification, the image processing method further includes: and carrying out multi-frame noise reduction processing and distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

As described above, in some embodiments of the present application, in S101, the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, auto white balance, and multi-frame noise reduction processing on an image captured by an image sensor, and in accordance with these embodiments, after S102 is executed, the image processing method in some embodiments of the present application further includes: and carrying out distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

As shown in fig. 13, some embodiments of the present application provide an image sensor module 700, where the image sensor module 700 includes: the image processing device comprises an image sensor and an image processor provided by the embodiment of the application and described in any one of the above embodiments, wherein output data of the image sensor is used as input data of the image processor.

Some embodiments of the present application provide an electronic device, comprising: the camera, and the image processor or the image sensor module that this application embodiment provided above. For example, the electronic device is a mobile terminal such as a mobile phone.

Some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program when executed by a processor may implement the method according to the embodiments corresponding to fig. 7.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. An image processing method applied to an image processor, the image processing method comprising:

acquiring an image to be processed;

and obtaining a target output image corresponding to the image to be processed at least by inquiring a lookup table, wherein the lookup table is used for storing output data respectively corresponding to each traversal image block in a plurality of traversal image blocks, and the traversal image blocks are image blocks determined by the receptive field of the target neural network.

2. The image processing method according to claim 1, wherein said obtaining the target output image corresponding to the image to be processed by at least querying a lookup table comprises:

traversing the image to be processed to obtain all image blocks to be inquired, wherein the size of the image blocks to be inquired is the same as that of the traversed image blocks;

obtaining output data corresponding to at least part of the image blocks to be inquired in all the image blocks to be inquired by inquiring the lookup table;

and obtaining the target output image according to the output data.

3. The image processing method according to any one of claims 1-2,

before the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table, the image processing method further includes:

obtaining all image blocks capable of being input into the target neural network according to the size of the receptive field of the target neural network and the bit width of the pixel values of the pixel points on the image to be processed, wherein the image blocks capable of being input into the target neural network are the plurality of traversal image blocks or the image blocks capable of being input into the target neural network comprise the plurality of traversal image blocks, the size of each image block capable of being input into the target neural network is the same as the size of the receptive field of the target neural network, and the maximum value of the pixel values of each pixel point on each image block capable of being input into the target neural network is determined by the bit width of the pixel values;

and acquiring the lookup table according to all image blocks capable of inputting the target neural network.

4. The image processing method according to claim 3, wherein the obtaining the look-up table according to all image blocks inputtable to the target neural network comprises:

sampling all image blocks which can be input into the target neural network according to a target sampling rate to obtain a target image block set;

acquiring output data corresponding to each image block in the target image block set through the target neural network to obtain the lookup table;

the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table includes:

and obtaining the target output image by inquiring the lookup table and an interpolation algorithm.

5. The image processing method according to any one of claims 1 to 4, wherein after said obtaining at least a target output image corresponding to the image to be processed by looking up a look-up table, the image processing method further comprises:

and carrying out multi-frame noise reduction processing and distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

6. The image processing method according to claim 5, wherein the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, and automatic white balance processing on an original image acquired by an image sensor;

alternatively, the first and second electrodes may be,

the image to be processed is an image obtained by performing black level correction, shading correction, dead pixel correction, automatic white balance processing and bit compression processing on an original image acquired by an image sensor.

7. The image processing method according to claim 6, wherein the number of the look-up tables is plural, and one look-up table is obtained based on one target neural network.

8. The image processing method according to any one of claims 1 to 4,

the image to be processed is an image obtained by performing black level correction, vignetting correction, dead pixel correction, automatic white balance and multi-frame noise reduction on an original image acquired by an image sensor, wherein,

after the obtaining of the target output image corresponding to the image to be processed at least by querying the lookup table, the image processing method further includes:

and carrying out distortion removal processing on the target output image to obtain a target image, wherein the target image is used for display or target identification.

9. An image processor, characterized in that the image processor is configured to obtain a target output image corresponding to an image to be processed at least by querying a look-up table, wherein the look-up table is used to store output data corresponding to a plurality of traversed image blocks, the traversed image blocks being image blocks determined by a receptive field of a target neural network.

10. The image processor of claim 9, wherein the image processor comprises:

the table look-up module is configured to obtain output data corresponding to a target image block to be queried through a look-up table, wherein the target image block to be queried is a whole or part of image blocks to be queried included in the image to be queried, the image block to be queried is obtained by traversing the image to be processed, and the target output image is obtained according to the output data.

11. The image processor of claim 10, wherein the target image block to be queried is the partial image block to be queried, the all image blocks to be queried include the partial image block to be queried and remaining image blocks to be queried, the image processor further comprising:

and the interpolation calculation module is configured to calculate the output data of the residual image blocks to be inquired through an interpolation algorithm.

12. The image processor of any one of claims 10-11, further comprising:

the image preprocessing module is configured to preprocess an original image acquired by an image sensor to obtain the image to be processed; and/or

The image post-processing module is configured to process the target output image to obtain a target image;

wherein the target image may be used for display or target recognition.

13. An image sensor module, comprising:

an image sensor; and

the image processor of any one of claims 9-12;

wherein the output data of the image sensor is used as the input data of the image processor.

14. An electronic device, characterized in that the electronic device comprises: a camera and an image processor as claimed in any one of claims 9 to 12 or an image sensor module as claimed in claim 13.

15. A computer program product, characterized in that the computer program product comprises a computer program, wherein the computer program when executed by a processor is adapted to perform the method of any of claims 1-8.