CN115103173A

CN115103173A - Method, device and chip for realizing automatic white balance of image

Info

Publication number: CN115103173A
Application number: CN202210663161.3A
Authority: CN
Inventors: 叶红磊; 温建新; 叶红波; 蒋亮亮; 姚清志
Original assignee: Shanghai IC R&D Center Co Ltd; Shanghai IC Equipment Material Industry Innovation Center Co Ltd
Current assignee: Shanghai IC R&D Center Co Ltd; Shanghai IC Equipment Material Industry Innovation Center Co Ltd
Priority date: 2022-06-13
Filing date: 2022-06-13
Publication date: 2022-09-23
Anticipated expiration: 2042-06-13
Also published as: CN115103173B

Abstract

The invention provides a method, a device and a chip for realizing automatic white balance of an image, wherein the method comprises the following steps: a programmable logic module of the chip acquires a multi-channel image corresponding to an original image from an image sensor; the programmable logic module carries out pixel value statistics on the multi-channel image to obtain a characteristic value; a processing system module of the chip obtains a white balance gain value according to the characteristic value of the multi-channel image; and the programmable logic module performs white balance processing on the original image based on the multi-channel image and the gain value to obtain a target image. The method has the characteristics of high integration level, high speed and low delay, and can meet the real-time requirement of image data statistics.

Description

Method, device and chip for realizing automatic white balance of image

Technical Field

The invention relates to the field of semiconductor integrated circuit design, in particular to a method, a device and a chip for realizing automatic white balance of an image.

Background

The High Level Synthesis (HLS) includes the steps of compiling, optimizing, scheduling, resource allocation, controller synthesis, etc. The high-level language is compiled and optimized, then translated into a low-level intermediate expression and used as the input of high-level comprehensive steps such as subsequent scheduling and resource allocation.

At present, research hotspots of high-level synthesis items are mainly put on scheduling, resource allocation and controller synthesis, and the steps of combinatorial logic optimization, process mapping and the like are completed by a subsequent Field Programmable Gate Array (FPGA) logic synthesis tool, so that deep optimization aiming at combinatorial logic operation in design is not involved in the high-level synthesis process, and hardware bottom layer information below a register transmission level can be ignored. This causes two problems: 1. the combinational logic operation in the high-level synthesis process is not deeply optimized, so that redundant combinational logic operation exists, and excessive clock cycles are distributed to the hardware design in the scheduling process, so that the time sequence performance of the generated hardware is influenced; 2. because the hardware bottom layer information is not introduced in the high-level synthesis process, in order to avoid that the hardware generated by the high-level synthesis tool does not meet the time sequence and resource requirements of the design, the time delay of the combinational logic operation and the resource overhead must be estimated in a pessimistic way, which results in that unnecessary clock cycles are distributed to the design in the scheduling process to influence the data throughput of the generated hardware, and less hardware resources are distributed to the design in the resource distribution process to reduce the parallelism of the hardware design and influence the running speed of the hardware.

The automatic white balance algorithm needs to carry out a large amount of data statistics, and the obtained statistical data is subjected to related algorithm calculation to finally complete the automatic white balance of the picture. The problems encountered by the automatic white balance algorithm realized by utilizing a high-level comprehensive language are as follows: high-level synthesis is realized by converting a high-level language through a tool, the execution efficiency is low, and the statistical result is incorrect due to the fact that the statistical error is generated when a large number of images are counted.

Therefore, a method for implementing automatic white balance of an image is needed to improve the above problems.

Disclosure of Invention

The invention provides a method, a device and a chip for realizing automatic white balance of an image, which are used for realizing the automatic white balance of the image based on the cooperation of software and hardware.

In a first aspect, the present invention provides a method for implementing automatic white balance of an image, in which a programmable logic module of a chip may obtain a multi-channel image corresponding to an original image from an image sensor; the programmable logic module carries out pixel statistics on the multi-channel image to obtain a characteristic value; a processing system module of the chip obtains a white balance gain value according to the characteristic value of the multi-channel image; and the programmable logic module carries out white balance processing on the multi-channel image based on the multi-channel image and the gain value to obtain a target image.

The method for realizing the automatic white balance of the image provided by the embodiment of the invention has the following beneficial effects: the method uses an AXI bus to interconnect the PS module and the PL module, designs an IP core for automatic image white balance calculation, adopts a processor to drive the IP core to realize real-time data interaction of the PS module and the PL module, integrates functions of image acquisition, storage, target detection, image data statistics, display output and the like on a single chip, has high integration level, high speed and low delay, can meet the real-time requirement of image data statistics, is realized through the FPGA platform which completes the AWB algorithm, and can run on the FPGA platform, calculate and modify the AWB algorithm in real time.

In one possible implementation, the programmable logic module is implemented using Verilog or VHDL and the processing system module is implemented using HLS.

In one possible implementation, the processing system module converts serial data corresponding to the target image into parallel data.

In a possible implementation, before the processing system module of the chip obtains the gain value of white balance according to the feature value of the multi-channel image, the method further includes: and after completing the statistical calculation of one frame of image, the programmable logic module sends an interrupt signal to the processing system module.

In a possible implementation, the obtaining, by a processing system module of the chip, a gain value of white balance according to the feature value of the multi-channel image includes:

and the processing system module of the chip inputs the characteristic value of the multi-channel image into a neural network model to obtain a white balance gain value, wherein the neural network model comprises one or more of a deep neural network model, a convolutional neural network, a long-short term memory network module or a cyclic neural network module.

In a second aspect, an embodiment of the present invention further provides an apparatus for implementing automatic white balance of an image, where the apparatus includes a module/unit for performing any one of the possible design methods of the first aspect. These modules/units may be implemented by hardware or by hardware executing corresponding software.

In a third aspect, an embodiment of the present invention provides a chip, where the memory is used to store one or more programs; the one or more programs stored in the memory, when executed by the processor, enable the chip to implement the method of any of the possible designs of the first aspect described above.

In a fourth aspect, an embodiment of the present invention further provides a readable storage medium, where the readable storage medium includes a program, and when the program runs on a chip, the program causes the chip to perform any one of the possible design methods of any one of the above aspects.

In a fifth aspect, an embodiment of the present invention further provides a program product, which, when running on a chip, causes the chip to perform any one of the possible design methods of any one of the above aspects.

For the beneficial effects of the second to fifth aspects, reference may be made to the description of the first aspect, and the description is not repeated.

Drawings

Fig. 1 is a schematic structural diagram of a terminal device according to an embodiment of the present invention;

fig. 2 is a schematic flowchart of a method for implementing automatic white balance of an image according to an embodiment of the present invention.

Detailed Description

The technical solution of the present invention will be described with reference to the accompanying drawings.

Hereinafter, some terms referred to hereinafter will be explained to facilitate understanding by those skilled in the art.

(1) The Mobile Industry Processor Interface (MIPI) alliance is an open standard and a specification established for mobile application processors by the MIPI alliance.

That is, MIPI is not just a federation but also the name of a set of interface protocol specifications. It governs different Work groups (Work groups) to meet different requirements of each subsystem of the mobile terminal. Different working groups are responsible for formulating specific protocols, and define a set of internal interface standards of the mobile device, such as a camera serial interface, a display serial interface, a radio frequency interface and the like, respectively for different hardware devices.

(2) The RAW image is RAW data obtained by converting a captured light source signal into a digital signal by a CMOS or CCD image sensor, and is generally a RAW format file. The RAW format file is a file in which RAW information of a digital camera sensor is recorded, and also some Metadata (Metadata) generated by camera shooting, such as setting of sensitivity, shutter speed, aperture value, white balance, and the like, are recorded.

(3) A Field Programmable Gate Array (FPGA) is a logic device composed of many logic units, wherein the logic units include gates, lookup tables and flip-flops, and the FPGA has rich hardware resources, strong parallel processing capability and flexible reconfigurable capability, and is increasingly widely applied in many fields such as data processing, communication, network, and the like. The FPGA appears as a semi-custom circuit in the field of Application Specific Integrated Circuits (ASICs), which not only solves the disadvantages of custom circuits, but also overcomes the drawback of limited gate circuits of the original programmable devices.

With the rapid development of mobile phone chips, the photographing function of mobile phones becomes more and more abundant, and users have made higher requirements on the basic quality (color, definition, etc.) of pictures photographed by mobile phones. The color is one of the important factors for evaluating the quality of the mobile phone photo, and the Automatic White Balance (AWB) is an important ring for forming the color of the picture.

The human visual system has the characteristic of color constancy that the human visual system can resist such light source color changes, thereby constantly perceiving the color of an object. However, in an Image Sensor (Image Sensor), objects appear in different colors under different light, for example, in a natural environment, the same object will appear in different colors under different light, for example, green leaves will turn yellow under morning light, and turn blue in the evening. In order to eliminate the influence of the light source on the imaging of the image sensor, simulate the color constancy of the human visual system and ensure that the white seen in any scene is a true white, an automatic white balance technology needs to be introduced.

The white balance is an index for describing the accuracy of white color generated by mixing red, green and blue three primary colors in a display screen, and the automatic white balance technology is mainly used for solving the problem of color cast of images under different light sources, so that the image of a scene in the image accords with the color vision habit of human eyes. The problem is solved by computational color constancy in automatic white balance, which mainly aims to calculate the color of an unknown light source represented by any one image, and then to perform color correction on the input image by using the light source color, so as to realize display under standard white light.

White balance, which means that in the process of image processing, the image of an object which is originally white is subjected to color restoration, and the influence of the color temperature of an external light source is removed, so that the object also displays white on a picture. The current AWB (Automatic white balance) algorithm is various, and usually, a gray decision point and a white decision point are counted in a current frame by a gray world method and a white world method, and then a decision point of the current frame is calculated according to a decision point of a previous frame to determine R _ gain and B _ gain of the current frame.

According to the method, on one hand, a traditional Verilog or VHDL (hardware description language) can be used for designing and realizing a Programmable Logic (PL) module in a chip, and the PL module is used for receiving, carrying out statistical operation processing and outputting image information; on the other hand, the HLS is utilized to design and realize a PS (processing system) module in the chip, the PS module is used for receiving the statistic value of the pixel from the PL module and finishing the calculation of R _ gain, B _ gain and G _ gain in the image automatic white balance algorithm and the adjustment of the related algorithm, on one hand, the design scheme can ensure the correctness of image statistics and avoid the statistics error when a large amount of images are counted, and on the other hand, the development efficiency of the automatic white balance algorithm can be improved.

The technical solution in the embodiment of the present invention is described below with reference to the drawings in the embodiment of the present invention. In describing embodiments of the present invention, the terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present invention, "at least one", "one or more" means one or more than two (including two). The term "and/or" is used to describe the association relationship of the associated objects, indicating that three relationships may exist; for example, a and/or B, may represent: a alone, both A and B, and B alone, where A, B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present invention. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather mean "one or more, but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically stated. The term "coupled" includes both direct and indirect connections, unless otherwise noted. "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

In embodiments of the present invention, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described as "exemplary" or "such as" in an embodiment of the present invention is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

As shown in fig. 1, the present invention provides an image processing terminal, which mainly includes an image sensor 10, a chip 20, and a display 30, wherein the chip 20 integrates a PL (programmable logic) module 201 and a PS (processing system) module 202, and these components can communicate on one or more communication buses. The development board of the chip 20 is provided with an LVDS interface and an HDMI interface, and supports various extensible devices, the image sensor 10 is connected with the chip 20 through an LDVS interface, and the display 30 is connected with the chip 20 through an HDMI interface, so as to realize the acquisition, input, display and output of image data.

The image sensor 10 is used for acquiring image information, converting the acquired image into MIPI data, and sending the MIPI data to the PL module through the MIPI interface, and the PL module 201 is used for receiving, performing statistical operation processing and outputting the image information; the PS module 202 is used for receiving the statistics of the pixels from the PL module, and completing the calculation of R _ gain, B _ gain, and G _ gain in the auto white balance algorithm and the adjustment of the related algorithms. The display 30 is used to display the processed image.

It should be noted that the MIPI data may be dual-channel or multi-channel, and the MIPI data further includes MIPI CLK (clock) information. The MIPI data may be a digital-to-speech interpolation (DSI) signal from a display screen interface or a CMOS Image Sensor (CIS) signal from a camera interface, or may be a signal from other MIPI interfaces of the mobile terminal, and the MIPI interface 410 may be an interface such as GTX or GTH.

The terminal may further include a Memory 40, and the Memory 40 may include a Volatile Memory (Volatile Memory), such as a Random Access Memory (RAM), a cache (cache); the Memory may also include a Non-Volatile Memory (Non-Volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (Hard Disk Drive, HDD), or a Solid-State Drive (SSD); the memory 40 may also comprise a combination of memories of the kind described above. The memory 40 may be configured to store data such as an original image, a target image after white balance processing, previous and subsequent frame image information, a shooting parameter, scene semantic information, and the like; the memory 40 may also be used to store program instructions for the processor to invoke and execute the method of automatic white balancing of images described herein.

Based on the various components of the terminal described above, automatic white balancing of images can be achieved by the following process: when a camera of the terminal performs photographing, an optical image of an object (person, object, scene, etc.) in an external environment is projected onto the surface of the image sensor 10 through a camera, and is converted into an electrical signal, and the electrical signal is converted into a digital image signal after analog-to-digital conversion (a/D conversion), and the digital image signal is an original image. The image sensor 10 sends the raw image to the chip 20 for image optimization. When the chip 20 needs to perform Auto White Balance (AWB), the PL module 201 in the chip 20 may calculate a neural network model using the input information, and obtain light source color information corresponding to the image. And then, the light source color information is fed back to the PS module 202, and the PS module 202 completes AWB according to the light source color information and performs other aspects of image processing to obtain a target image, for example, an image in YUV or RGB format. The chip 20 then sends the target image to the display 30 for display.

The terminal can be a terminal device containing an ASIC chip or a terminal device containing an FPGA, or an embedded system containing an ASIC/FPGA. A terminal device including an ASIC/FPGA of the present invention may also be referred to as a User Equipment (UE), an access terminal, a subscriber unit, a subscriber station, a mobile station, a remote terminal, a mobile device, a user terminal, a wireless communication device, a user agent, or a user equipment. The terminal device in the embodiment of the present invention may be a mobile phone (mobile phone), a tablet computer (Pad), a smart printer, a train detector, a gas station detector, a computer with wireless transceiving function, a Virtual Reality (VR) terminal device, an Augmented Reality (AR) terminal device, a wireless terminal in industrial control (industrial control), a wireless terminal in unmanned driving (self driving), a wireless terminal in remote medical (remote medical), a wireless terminal in smart grid (smart grid), a wireless terminal in transportation security (transportation security), a wireless terminal in city (smart city), a wireless terminal in home (smart home), and the like. The embodiment of the present invention does not limit the application scenario.

It should be understood that fig. 1 is a simplified schematic diagram illustrating only for the sake of understanding, and an image signal processor or other modules may be further included in the terminal, which are not listed in fig. 1 one by one.

The method for implementing automatic white balance of an image provided by the embodiment of the invention can be executed by the chip 20 in the terminal shown in fig. 1. It should be understood that, in the present invention, the steps performed by the chip 20 may also be specifically performed by one or more modules or components of the chip 20. As shown in fig. 2, a method for implementing automatic white balance of an image according to an embodiment of the present invention is shown, where the method includes:

s201, acquiring a multi-channel image corresponding to the original image.

In this step, the original image is a RAW format file, and after the chip 20 acquires the original image from the image sensor 10, the original image may be processed into a multi-channel image, where the multi-channel image is an image in which each pixel point can be represented by values (or color components) of multiple image channels. The image channel refers to the individual red (R), green (G), and blue (B) portions in the RGB color mode. For example, in one example, the multi-channel image may specifically be a color three-channel image, such as an RGB three-channel image.

S202, the PL module 201 conducts pixel value statistics operation on the multi-channel image corresponding to the original image to obtain a characteristic value.

In this step, the PL module 201 may implement feature extraction through an arithmetic circuit, for example, implement statistical operation on pixels of a channel image through a feature extraction network, so as to obtain a feature value. The feature extraction network may include one or more convolution kernels, and statistical operations on pixels of the channel image are performed through convolution processing to obtain the feature values.

S203, the PS module 202 obtains the characteristic value of the multi-channel image and obtains a gain value of white balance according to the characteristic value of the multi-channel image.

Alternatively, the PS module 202 inputs the feature values of the multi-channel image into a neural network model to obtain a gain value of white balance, or obtains light source color information required in white balance processing, such as an image light source value (r/g,1, b/g), from the shooting parameters and the multi-channel image, and calculates the gain value of white balance. Before this step is performed, the PL module 201 may send an interrupt signal to the PS module 202 after completing the statistical calculation of one frame of image, thereby triggering the PS module to perform the above operation.

In this step, the PS module 202 may complete the calculation of R _ gain, B _ gain, and G _ gain in the automatic white balance algorithm based on the feature values.

S204, the PL module 201 performs white balance processing on the original image based on the multi-channel image and the gain value to obtain a target image.

The neural network model described in this embodiment may be a single neural network model in terms of type, or may be a combination of two or more neural network models. For example, the Neural Network model may be a model constructed based on a deep learning manner, and may be, for example, one of or a combination of a Deep Neural Network (DNN) model, a Convolutional Neural Network (CNN), a Long Short-Term Memory Network (LSTM) or a Recurrent Neural Network (RN), and the like.

The neural network model provided by this embodiment can obtain light source color information required in white balance processing, for example, image light source values (r/g,1, b/g), according to the shooting parameters and the multi-channel image, and after outputting the light source color information, the terminal can perform white balance processing on the channel image by using the light source color information through the chip 20 configured by the terminal, thereby realizing correction of image color difference caused by light source color temperature, so that the color of an object in the image is close to the original color thereof, and the overall image presenting effect conforms to the visual habit and cognitive habit of human eyes.

Generally, a frame of data consists of 1080 lines, and a line consists of 1920 pixels, so that after receiving 1080 lines of serial valid data, the chip 20 can obtain a frame of data, and then sequentially convert the serial valid data into parallel data signals in a form similar to a Digital Video Port (DVP) according to the above method, so that the display 30 can correctly display images.

The chip 20 is used as a carrier, the AXI bus is used for interconnecting the PS module and the PL module, an IP core for image automatic white balance calculation is designed, the processor is adopted to drive the IP core to realize real-time data interaction of the PS module and the PL module, functions of image acquisition, storage, target detection, image data statistics, display output and the like are integrated on a single chip, the integration level is high, the real-time requirements of image data statistics can be met by high-speed and low-delay digital image processing and data transmission, the AWB algorithm is realized through the FPGA platform which finishes the AWB algorithm, the FPGA platform can operate, and the AWB algorithm can be calculated and modified in real time.

The embodiment of the invention also provides a readable medium, wherein a program is stored on the readable medium, and the program is used for realizing the method of any one of the method embodiments when being executed by a chip.

The embodiment of the invention also provides a program product, and the program product realizes the method of any method embodiment when being executed by a chip.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

In short, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for realizing automatic white balance of an image is characterized by comprising the following steps:

a programmable logic module of the chip acquires a multi-channel image corresponding to an original image from an image sensor;

the programmable logic module carries out pixel value statistics on the multi-channel image to obtain a characteristic value;

a processing system module of the chip obtains a white balance gain value according to the characteristic value of the multi-channel image;

and the programmable logic module performs white balance processing on the original image based on the multi-channel image and the gain value to obtain a target image.

2. The method of claim 1, wherein the programmable logic module is implemented using Verilog or VHDL and the processing system module is implemented using high-level synthesis HLS.

3. The method of claim 2, further comprising:

and the processing system module converts the serial data corresponding to the target image into parallel data.

4. The method according to any one of claims 1 to 3, wherein before the processing system module of the chip obtains the gain value of white balance according to the feature value of the multi-channel image, the method further comprises:

and after the programmable logic module completes the statistical calculation of one frame of image, an interrupt signal is sent to the processing system module.

5. The method according to any one of claims 1 to 3, wherein the obtaining of the white balance gain value by the processing system module of the chip according to the feature value of the multi-channel image comprises:

6. An apparatus for implementing automatic white balance of an image, the apparatus comprising:

the programmable logic module is used for acquiring a multi-channel image corresponding to the original image from the image sensor;

the programmable logic module is also used for carrying out pixel value statistics on the multi-channel image to obtain a characteristic value;

the processing system module is used for obtaining a white balance gain value according to the characteristic value of the multi-channel image;

and the programmable logic module is further used for carrying out white balance processing on the original image based on the multi-channel image and the gain value to obtain a target image.

7. The apparatus of claim 6, wherein the programmable logic module is implemented using a Verilog or VHDL design and the processing system module is implemented using a high-level synthesis HLS design.

8. The apparatus of claim 7, wherein the processing system module is further configured to:

and converting the serial data corresponding to the target image into parallel data.

9. The apparatus according to any one of claims 6 to 8, wherein before the processing system module of the apparatus obtains the gain value of white balance according to the feature value of the multi-channel image, the apparatus further comprises:

10. A chip for performing the method of any one of claims 1 to 5.