WO2022174460A1

WO2022174460A1 - Sensor, electrical device, and non-transitory computer readable medium

Info

Publication number: WO2022174460A1
Application number: PCT/CN2021/077306
Authority: WO
Inventors: Toshihiko Arai
Original assignee: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date: 2021-02-22
Filing date: 2021-02-22
Publication date: 2022-08-25
Also published as: CN116803096A

Abstract

An imaging sensor (60) includes a first circuit (62) and a second circuit (64). The first circuit (62) obtains a first captured pixel data which is a data of a regular gain pixel and outputs the first captured pixel data as a regular processing image data from the imaging sensor (60). The second circuit (64) obtains a second captured pixel data which is a data of a low gain pixel and regularizes the second captured pixel data to obtain a first regularized pixel data, outputs the first regularized pixel data as the regular processing image data from the imaging sensor (60) if the regular processing image data is not saturated, and clips a second regularized pixel data from the first regularized pixel data and outputs the second regularized pixel data as the regular processing image data from the imaging sensor (60) if the regular processing image data is saturated.

Description

SENSOR, ELECTRICAL DEVICE, AND NON-TRANSITORY COMPUTER READABLE MEDIUM

TECHNICAL FIELD

The present disclosure relates to a sensor, an electrical device, and a non-transitory computer readable medium.

BACKGROUND

Electrical devices such as smartphones and tablet terminals are widely used in our daily life. Nowadays, many of the electrical devices are equipped with a camera assembly for capturing images. Some of the electrical devices are portable and are thus easy to carry. Therefore, a user of the electrical device can easily take a picture of an object by using the camera assembly of the electrical device anytime, anywhere.

Many electrical devices can produce an HDR (High Dynamic Range) image data which can express a wide range of brightness from dark pixels to bright pixels. In this case, sensitivity levels of pixel data captured by an imaging sensor of the camera assembly are different between regular gain pixels and low gain pixels. Therefore, a special process to regularize the captured pixel data of the low gain pixel from the imaging sensor is needed. This special process is necessary even if the HDR image data is not required. Also, a specific hardware structure and/or a specific software process to execute the special process is required. As a result, costs for developing the specific hardware structure and/or the specific software are increased.

SUMMARY

The present disclosure aims to solve at least one of the technical problems mentioned above. Accordingly, the present disclosure needs to provide an imaging sensor, an electrical device and a non-transitory computer readable medium.

In accordance with the present disclosure, an imaging sensor may include:

a first circuit configured to obtain a first captured pixel data which is a data of a regular gain pixel and to output the first captured pixel data as a regular processing image data from the imaging sensor; and

a second circuit configured to obtain a second captured pixel data which is a data of a low gain pixel and to regularize the second captured pixel data to obtain a first regularized pixel data,

wherein the second circuit is further configured to output the first regularized pixel data as the regular processing image data from the imaging sensor if the regular processing image data is not saturated when the regular processing image data is inputted into a regular image pipeline; and

the second circuit is further configured to clip a second regularized pixel data from the first regularized pixel data and to output the second regularized pixel data as the regular processing image data from the imaging sensor if the regular processing image data is saturated when the regular processing image data is inputted into the regular image pipeline.

In accordance with the present disclosure, an electrical device may include:

the imaging sensor mentioned above; and

a third circuit configured to gather the plurality of the data pieces embedded in the spare space of the regular processing image data of the low gain pixel itself and the spare space of the another regular processing image data of the another pixel to obtain the compressed data.

In accordance with the present disclosure, a non-transitory computer readable medium may include program instructions stored thereon for performing, at least, the following:

gathering a plurality of data pieces embedded in a spare space of a regular processing image data of a low gain pixel itself and a spare space of another regular processing image data of another pixel, wherein the regular processing image data is outputted from an imaging sensor;

obtaining a compressed data based on the gathered data pieces; and

expanding the compressed data to obtain a reconstructed first regularized pixel data.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of embodiments of the present disclosure will become apparent and more readily appreciated from the following descriptions made with reference to the drawings, in which:

FIG. 1 illustrates a plan view of a first side of an electrical device according to an embodiment of the present disclosure;

FIG. 2 illustrates a plan view of a second side of the electrical device according to the embodiment of the present disclosure;

FIG. 3 illustrates a block diagram of the electrical device according to the embodiment of the present disclosure;

FIG. 4 illustrates one example of a pixel array of an imaging sensor in a camera assembly of the electrical device according to the embodiment of the present disclosure;

FIG. 5 illustrates another example of a pixel array of the imaging sensor in the camera assembly of the electrical device according to the embodiment of the present disclosure;

FIG. 6 illustrates a structure of the imaging sensor of the camera assembly and a process executed in the imaging sensor to generate a regular processing image data for a regular image pipeline of the electrical device according to the embodiment of the present disclosure;

FIG. 7 illustrates a flowchart of a regular processing image data generation process for generating a regular processing image data for the regular image pipeline of the electrical device according to the embodiment of the present disclosure;

FIG. 8 illustrates the structure of the imaging sensor of the camera assembly and a process executed in the imaging sensor to generate the regular processing image data for an HDR image pipeline of the electrical device according to the embodiment of the present disclosure;

FIG. 9 illustrates a flowchart of a regular processing image data generation process for generating the regular processing image data for the HDR image pipeline of the electrical device according to the embodiment of the present disclosure;

FIG. 10 illustrates one example of a compression tone curve for compressing a first regularized pixel data by reducing a number of bits of the first regularized pixel data;

FIG. 11 illustrates a relationship between the first regularized pixel data and its output from the imaging sensor if the output is linear;

FIG. 12 illustrates a bit format of the regular processing image data in the electrical device according to the present disclosure;

FIG. 13 illustrates one example of embedding four data pieces into the regular processing image data in a certain pixel array;

FIG. 14 illustrates a first regularized pixel data recovery process for reconstructing the first regularized pixel data based on the regular processing image data outputted from the second circuit;

FIG. 15 illustrates one example of gathering the four data pieces from the regular processing image data in the pixel array shown in FIG. 13;

FIG. 16 illustrates the reconstruction of the first regularized pixel data to be inputted to the HDR image pipeline;

FIG. 17 illustrates an expansion tone curve for expanding the compressed data obtained by gathering a plurality of the data pieces from the regular processing image data; and

FIG. 18 illustrates the manner to utilize the regular image pipeline and the HDR image pipeline in the electrical device according to the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in detail and examples of the embodiments will be illustrated in the accompanying drawings. The same or similar elements and the elements having same or similar functions are denoted by like reference numerals throughout the descriptions. The embodiments described herein with reference to the drawings are explanatory, which aim to illustrate the present disclosure, but shall not be construed to limit the present disclosure.

FIG. 1 is a plan view of a first side of an electrical device 10 according to an embodiment of the present disclosure and FIG. 2 is a plan view of a second side of the electrical device 10 according to the embodiment of the present disclosure. The first side may be referred to as a back side of the electrical device 10 whereas the second side may be referred to as a front side of the electrical device 10.

As shown in FIG. 1 and FIG. 2, the electrical device 10 may include a display 20 and a camera assembly 30. In the present embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34 and a sub camera 36. The first main camera 32 and the second main camera 34 can capture an image in the first side of the electrical device 10 and the sub camera 36 can capture an image in the second side of the electrical device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called out-cameras whereas the sub camera 36 is a so-called in-camera. As an example, the electrical device 10 can be a mobile phone, a tablet computer, a personal digital assistant, and so on.

Each of the first main camera 32, the second main camera 34 and the sub camera 36 has an imaging sensor which converts a light which has passed a color filter to an electrical signal. A signal value of the electrical signal depends on an amount of the light which has passed the color filter.

Although the electrical device 10 according to the present embodiment has three cameras, the electrical device 10 may have less than three cameras or more than three cameras. For example, the electrical device 10 may have two, four, five, and so on, cameras.

FIG. 3 is a block diagram of the electrical device 10 according to the present embodiment. As shown in FIG. 3, in addition to the display 20 and the camera assembly 30, the electrical device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46 and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power supply circuit 46 and the communication circuit 48 are connected with each other via a bus 50.

The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various applications and data processing of the electrical device 10 by executing the program instructions. The main processor 40 may be one or more computer processors. The main processor 40 is not limited to one CPU core, but it may have a plurality of CPU cores. The main processor 40 may be a main CPU of the electrical device 10, an image processing unit (IPU) or a DSP provided with the camera assembly 30.

The image signal processor 42 controls the camera assembly 30 and processes various kinds of image data captured by the camera assembly 30 to generate a target image data. For example, the image signal processor 42 can apply a demosaicing process, a noise reduction process, an auto exposure process, an auto focus process, an auto white balance process, a high dynamic range process and so on, to the image data captured by the camera assembly 30.

In the present embodiment, the main processor 40 and the image signal processor 42 collaborate with each other to generate a target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture the image of the object by means of the camera assembly 30 and apply various kinds of image processing to the captured image data.

The memory 44 stores program instructions to be executed by the main processor 40, and various kinds of data. For example, data of the captured image are also stored in the memory 44.

The memory 44 may include a high-speed RAM memory, and/or a non-volatile memory such as a flash memory and a magnetic disk memory. That is, the memory 44 may include a non-transitory computer readable medium in which the program instructions are stored.

The power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

The communication circuit 48 is configured to receive and transmit data to communicate with base stations of the telecommunication network system, the Internet or other devices via wireless communication. The wireless communication may adopt any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication) , CDMA (Code Division Multiple Access) , LTE (Long Term Evolution) , LTE-Advanced, 5th generation (5G) . The communication circuit 48 may include an antenna and an RF (radio frequency) circuit.

FIG. 4 illustrates one example of a pixel array of an imaging sensor 60 in the camera assembly 30. In other words, the camera assembly 30 has the imaging sensor 60 to capture an image of an object. The pixel array shown in FIG. 4 includes green pixels, red pixels and blue pixels. There are two types of the green pixels, i.e., regular gain green pixels GR and low gain green pixels GL. A sensitivity of the low gain green pixels GL is lower than a sensitivity of the regular gain green pixels GR. For example, an analog gain of the low gain green pixels GL is lower than an analog gain of the regular gain green pixels GR. In another example, an exposure time of the low gain green pixels GL is shorter than an exposure time of the regular gain green pixels GR.

Similarly, there are two types of the red pixels, i.e., regular gain red pixels RR and low gain red pixels RL. A sensitivity of the low gain red pixels RL is lower than a sensitivity of the regular gain red pixels RR. In addition, there are two types of the blue pixels, i.e., regular gain blue pixels BR and low gain blue pixels BL. A sensitivity of the low gain blue pixels BL is lower than a sensitivity of the regular gain blue pixels BR.

FIG. 5 illustrates another example of a pixel array of the imaging sensor 60 in the camera assembly 30. In this example as well, the imaging sensor 60 has the regular gain green pixels GR, the low gain green pixels GL, the regular gain red pixels RR, the low gain red pixels RL, the regular gain blue pixels BR, and the low gain blue pixels BL. However, an arrangement of the pixels in the pixel array in FIG. 5 is different from an arrangement of the pixels in the pixel array in FIG. 4.

Incidentally, the arrangement of the pixel array is optional. Therefore, any kind of the arrangement of the pixel array can be applied to the imaging sensor 60. Moreover, colors of the pixel array of the imaging sensor 60 are not limited to green, red and blue. For example, the colors of the pixel array of the imaging sensor 60 may include red, yellow and blue (RYB) . More specifically, the imaging sensor may have regular gain red pixels, low gain red pixels, regular gain yellow pixels, low gain yellow pixels, regular gain blue pixels, and low gain blue pixels. Therefore, hereinafter, the pixel or the pixels indicate any color of the pixel or the pixels.

FIG. 6 illustrates a structure of the imaging sensor 60 of the camera assembly 30 and a process executed in the imaging sensor 60 to generate a regular processing image data for a regular image pipeline 70. The imaging sensor 60 in FIG. 6 generates the regular processing image data and outputs the regular processing image data to the regular image pipeline 70. For example, the regular image pipeline 70 is included in the image signal processor 42 to process a regular image data of the regular processing image data. In the present embodiment, the regular image pipeline 70 can process 12 bits regular image data, which is a normal bit width for the regular image pipeline 70.

As shown in FIG. 6, the imaging processor 60 has a first circuit 62 for regular gain pixels of the pixel array and a second circuit 64 for low gain pixels of the pixel array. Hereinafter, functions of the first circuit 62 and the second circuit 64 will be explained by taking one pixel as an example.

The first circuit 62 obtains a first captured pixel data which is a data of a regular gain pixel. A value of the first captured pixel data depends on the amount of light which has passed the color filter. In other words, an amount of electrons accumulated in the regular gain pixel is proportional to a strength of the light which has passed the color filter. The first circuit 62 converts the accumulated electrons to a digital value of the first captured pixel data.

Thereafter, the first circuit 62 outputs the first captured pixel data as a regular processing image data from the imaging sensor 60 to the regular image pipeline 70. That is, the first circuit 62 does not change the first captured pixel data because the first captured pixel data is obtained from the regular gain pixel, the sensitivity of which is regular. Therefore, the first circuit 62 outputs the first captured pixel data itself as the regular processing image data from the imaging sensor 60 to the regular image pipeline 70. The regular image processing data is compatible with the regular image pipeline 70.

In the present embodiment, for example, the regular processing image data is composed of 16 bits, and the first regularized pixel data is composed of 12 bits. Therefore, the first regularized pixel data can be accommodated in the regular processing image data. In other words, the regular image pipeline 70 accepts 12 bits regular image data which is the maximum bits to be processed in the regular image pipeline 70. That is, the normal bit width of the regular image pipeline 70 in the present embodiment is 12 bits.

On the other hand, the second circuit 64 obtains a second captured pixel data, which is a data of a low gain pixel, and processes the second captured pixel data in order to generate the regular processing image data. FIG. 7 illustrates a flowchart of a regular processing image data generation process for generating the regular processing image data for the regular image pipeline 70. In the present embodiment, the regular processing image data generation process is executed in the second circuit 64 in the imaging sensor 60 which is constructed with the hardware structure.

As shown in FIG. 7, at first, the second circuit 64 obtains the second captured pixel data from the pixel array (Step S10) . A value of the second captured pixel data depends on the amount of light which has passed the color filter. In other words, an amount of electrons accumulated in the low gain pixel is proportional to a strength of the light which has passed the color filter. The second circuit 64 converts the accumulated electrons to a digital value of the second captured pixel data.

Next, as shown in FIG. 7, the second circuit 64 regularizes the second captured pixel data to obtain a first regularized pixel data (Step S12) . As explained above, the sensitivity of the low gain pixel is lower than the sensitivity of the regular gain pixel. Therefore, it is necessary to regularize the value of the second captured pixel data to obtain the first regularized pixel data.

As shown in FIG. 7, the value of the second captured pixel data is converted to a regular gain level in Step S12. In other words, the value of the first regularized pixel data becomes substantially equal to the value in a case where the sensitivity of the low gain pixel is equal to the sensitivity of the regular gain pixel. Therefore, the regularizing process is a sort of a compensation process to compensate the low sensitivity of the low gain pixel.

Next, as shown FIG. 7, the second circuit 64 judges whether the value of the regular processing pixel data is saturated to determine whether the first regularized pixel data should be outputted as the regular processing image data to be inputted into the regular image pipe line 70 (Step S14) .

In the present embodiment, for example, the first regularized pixel data is composed of 18 bits, and the regular image pipeline 70 can accept 12 bits image data. Therefore, if the value of the first regularized pixel data is equal to or less than 4095 (2 ¹² can express equal to or more than 0 through equal to or less than 4095) , the regular processing data is not saturated when the regular processing data is inputted into the regular image pipeline 70. In this case, the second circuit 64 judges whether the value of the first regularized pixel data is equal to or less than 4095 (2 ¹² can express from 0 to 4095) .

If the regular processing image data is not saturated (Step S14: No) , the second circuit 64 outputs the first regularized pixel data as the regular processing image data from the imaging sensor 60 (Step S16) . That is, since the first regularized pixel data is equal to or less than 4095 and thus the regular processing image data is not saturated, the second circuit 64 outputs the first regularized pixel data itself as the regular processing image data from the imaging sensor 60 to the regular image pipeline 70.

Since the first regularized pixel data is equal to or less than 4095, it can be expressed by 12 bits. Therefore, the first regularized pixel data composed of 12 bits can be accommodated in the regular processing image data composed of 16 bits.

The 12 bits image data of the regular processing image data which corresponds to the first regularized pixel data is inputted into the regular image pipeline 70. That is, the regular image pipeline 70 processes 12 bits image data of the regular processing image data outputted from the second circuit 64 in the same manner as the regular image data of the regular processing image data outputted from the first circuit 62.

On the other hand, if the regular processing image data is saturated (Step S14: Yes) , the second circuit 64 clips a second regularized pixel data from the first regularized pixel data (Step S18) . In the present embodiment, for example, the first regularized pixel data is composed of 18 bits. In this case, the second circuit 64 clips the lower 12 bits of the first regularized pixel data composed of 18 bits as the second regularized pixel data. That is, the second regularized pixel data has the maximum value of 12 bits data, and thus the value of the second regularized pixel data is saturated at 4095.

Next, the second circuit 64 outputs the second regularized pixel data as the regular processing image data from the imaging sensor 60 to the regular image pipeline 70 (Step S20) . In the present embodiment, for example, the second regularized pixel data is at 4095 (2 ¹²) and the 12 bits of the regular processing image data are inputted into the regular image pipeline 70. That is, the regular image pipeline 70 processes the 12 bits image data of the regular processing image data outputted from the second circuit 64 in the same manner as the regular image data of the regular processing image data outputted from the first circuit 62.

The regular processing image data generation process is completed after the Step S20 is executed. However, the regular processing image data generation process shown in FIG. 7 is directed to one pixel. Therefore, the second circuit 64 of the imaging sensor 60 executes the regular processing image data generation process in FIG. 7 against every low gain pixel.

The process of generating the regular processing image data for the regular image pipeline 70 has been explained based on FIG. 6 and FIG. 7. However, the electrical device 10 according to the present embodiment also has an HDR (High Dynamic Range) image pipeline. Therefore, the imaging sensor 60 also needs to capture and output the HDR image data for the HDR image pipeline.

Since the image signal processor 42 has both the regular image pipeline 70 and the HDR image pipeline, the regular image pipeline 70 can also be referred to as a non-HDR image pipeline for processing the non-HDR image data.

FIG. 8 illustrates a process executed in the imaging sensor 60 to generate the regular processing image data for the HDR image pipeline 72. The drawing of FIG. 8 for the HDR image pipeline 72 corresponds to the drawing of FIG. 6 for the regular image pipeline 70.

The function of the first circuit 62 in FIG. 8 for the HDR image pipeline 72 is the same as the function of the first circuit 62 in FIG. 8. Therefore, the first circuit 62 obtains the first captured pixel data and outputs the first captured pixel data as the regular processing image data in the same manner as that represented in FIG. 6.

On the other hand, the function of the second circuit 64 in FIG. 8 for the HDR image pipeline 72 has an additional function as compared to the function of the second circuit 64 in FIG. 6. That is, in the second circuit 64 in FIG. 8, a saturated information on the first regularized pixel data composed of 18 bits is embedded into the regular processing image data so as to be able to recover the saturated information. That is, the second circuit 64 embeds the saturated information on the first regularized pixel data into the regular processing image data itself and into another regular processing image data such that the first regularized pixel data is able to be reconstructed later.

FIG. 9 illustrates a flowchart of a regular processing image data generation process for generating the regular processing image data for the HDR image pipeline 72. In the present embodiment, the regular processing image data generation process is executed in the second circuit 64 in the imaging sensor 60 which is constructed with the hardware structure.

The processes from Step S10 to Step S20 in FIG. 9 are the same as those in FIG. 7. Therefore, the detailed explanation therefor is omitted. After the Step S20, the second circuit 64 compresses the first regularized pixel data in order to reduce a data amount and to obtain a compressed data (Step S30) . The first regularized pixel data is a data which has been generated in Step S12. In the present embodiment, for example, the first regularized pixel data composed of 18 bits is compressed into the compressed data composed of 16 bits.

There are various methods to compress the first regularized pixel data. In the present embodiment, for example, the second circuit 64 compresses the first regularized pixel data based on a compression tone curve. In the compression tone curve, when the first regularized pixel data is larger, the compression rate is higher. That is, when the first regularized pixel data is smaller, the compression rate is lower.

FIG. 10 illustrates one example of the compression tone curve for compressing the first regularized pixel data by reducing a number of bits of the first regularized pixel data, FIG. 11 illustrates a relationship between the first regularized pixel data and the linear output of the first regularized data, and FIG. 12 illustrates a bit format of the regular processing image data in the electrical device 10 of the present embodiment.

As shown in FIG. 12, in the present embodiment, the regular processing image data is composed of 16 bits, but the effective bits for the image data are 12 bits because the regular image pipeline 70 accepts 12 bits image data. The first regularized pixel data is composed of 18 bits, and the second regularized pixel data is composed of 12 bits. That is, the second circuit 64 regularizes the second captured pixel data to obtain the first regularized pixel data composed of 18 bits in Step S12.

The remaining bits of the regular processing image data composed of 16 bits are 4 bits which constitute a spare space. In the present embodiment, the spare space is used for accommodating the saturated information on the first regularized pixel data. Since the spare space is composed of 4 bits only, the saturated information on the first regularized pixel data is inlaid sparsely and embedded into a plurality of the spare spaces of the regular processing image data.

As shown in FIG. 11, the first regularized pixel data composed of 18 bits can express values equal to or more than 0 through equal to or less than 262143. If the value of the first regularized pixel data is equal to or more than 0 and equal to or less than 4095, the value can be expressed by 12 bits and thus the value can be accommodated in the second regularized pixel data which can be accommodated in the regular processing image data composed of 16 bits.

On the other hand, if the value of the first regularized pixel data is equal to or more than 4096 and equal to or less than 262143, the value cannot be expressed by 12 bits and thus the value cannot be accommodated in the second regularized pixel data which can be accommodated in the regular processing image data. Therefore, the second circuit 64 embeds the saturated information on the first regularized image data into the regular processing image data.

In Step 30, the second circuit 64 compresses the first regularized pixel data in order to obtain the compressed data and to reduce the data amount by using the compression tone curve shown in FIG. 10.

For example, based on the compression tone curve in FIG. 10, when the first regularized pixel data is 0, the compressed data assigned is also 0. On the other hand, based on the compression tone curve in FIG. 10, when the first regularized pixel data is 262143, the compressed data assigned is 65535. The higher the value of the first regularized pixel data is, the more the compression rate is increased.

If the regular processing image data is saturated, it means the pixel is very bright. The brighter the pixel is, the harder it is for the human eye to distinguish between pixels of different brightness. Therefore, in the present embodiment, as the value of the first regularized pixel data increases, the compression rate also increases.

As a result of the compression process, the first regularized pixel data composed of 18 bits can be converted into the compressed data composed of 16 bits which can express values equal to or more than 0 and equal to or less than 65535.

Next, as shown in FIG. 9, the second circuit 64 divides the compressed data into a plurality of data pieces and the plurality of the data pieces are embedded into the spare spaces of the regular processing image data of the low gain pixel itself and another regular processing image data of another pixel (Step S32) .

In the present embodiment, since the compressed data is composed of 16 bits, the compressed data is divided into 4 data pieces, each of which is composed of 4 bits. As shown in FIG. 12, the spare space of the regular processing image data composed of 16 bits is composed of 4 bits. Therefore, the second circuit 64 needs to dispersively embed the 4 data pieces, each composed of 4 bits, into the spare spaces of the 4 regular processing image data.

FIG. 13 illustrates one example of embedding the 4 data pieces into the regular processing image data in a certain pixel array. In this example, the second circuit 64 embeds a first data piece into the spare space of the regular processing image data of the low gain red pixel RL itself that the compressed data has been generated from.

In addition, the second circuit 64 embeds a second data piece into the spare space of the regular processing image data of a regular gain red pixel RR neighbor to the low gain red pixel RL to the right side. Moreover, the second circuit 64 embeds a third data piece into the spare space of the regular processing image data of a regular gain red pixel RR neighbor to the low gain red pixel RL to the bottom right side. Furthermore, the second circuit 64 embeds a fourth data piece into the spare space of the regular processing image data of a regular gain red pixel RR neighbor to the low gain red pixel RL to the bottom side.

However, the method of embedding the data pieces into the spare spaces of the regular processing image data is not limited to the example of FIG. 13. There are various methods to sparsely inlay the data pieces of the compressed data. Furthermore, the data pieces may be embedded into the regular processing image data of the same color pixel as, or the different color pixel from, the original pixel that the compressed data has been generated from.

Next, as shown in FIG. 9, the second circuit 64 outputs the second regularized pixel data and the data piece embedded into the spare space as the regular processing image data from the imaging sensor 60 (Step S34) .

As shown in FIG. 12, the second circuit 64 stores the second regularized pixel data composed of 12 bits in the effective bits of the regular processing image data and the data piece composed of 4 bits in the spare space of the regular processing image data. As a result, the regular processing image data composed of 16 bits is outputted from the imaging sensor 60 to the regular image pipeline 70 and the HDR image pipeline. That is, in the present embodiment, the regular processing image data includes the second regularized pixel data and the data piece of the compressed data.

The regular processing image data generation process is completed after the Step S34 is executed. However, the regular processing image data generation process shown in FIG. 9 is directed to one pixel. Therefore, the second circuit 64 of the imaging sensor 60 executes the regular processing image data generation process in FIG. 9 against every low gain pixel.

As shown in FIG. 6, the regular processing image data based on the first captured pixel data from the first circuit 62, the regular processing image data based on the first regularized pixel data from the second circuit 64 and the regular processing image data based on the second regularized pixel data can be inputted directly into the regular image pipeline 70.

On the other hand, as shown in FIG. 8, in order to obtain the first regularized pixel data of the HDR image data inputted into the HDR image pipeline 72, the electrical device 10 has to reconstruct the first regularized pixel data based on the plurality of the regular processing image data.

FIG. 14 illustrates a first regularized pixel data recovery process for reconstructing the first regularized pixel data based on the regular processing image data outputted from the second circuit 64.

The first regularized pixel data recovery process is executed by a third circuit 66 as shown in FIG. 8. For example, the third circuit 66 may be composed of the main processor 40 which executes a software program to implement the first regularized pixel data recovery process. The instructions of the software program may be stored on a non-transitory computer readable medium, and the main processor 40 reads out the instructions of the software program from the non-transitory computer readable medium and executes them to perform the first regularized pixel data recovery process.

Otherwise, the third circuit 66 may be composed of a combination of the main processor 40 and the image signal processor 42, or the third circuit 66 may be implemented with an ASIC (Application Specific Integrated Circuit) to implement the first regularized pixel data recovery process.

As shown in FIG. 14, the third circuit 66 gathers the plurality of the data pieces in the spare space of the regular processing image data of the low gain pixel itself and the spare space of another regular processing image data of another pixel to obtain the compressed data (Step S40) .

FIG. 15 illustrates one example of gathering the 4 data pieces from the regular processing image data in the pixel array shown in FIG. 13. FIG. 16 illustrates the reconstruction of the first regularized pixel data of the HDR image data to be inputted to the HDR image pipeline 70.

As shown in FIG. 15 and FIG. 16, in the present embodiment, the third circuit 66 gathers the first data piece from the spare space of the regular processing image data of the low gain red pixel RL, the second data piece from the spare space of the regular processing image data of the regular gain red pixel RR to the right side, the third data piece from the spare space of the regular processing image data of the regular gain red pixel RR to the bottom right side, and the fourth data piece from the spare space of the regular processing image data of the regular gain red pixel RR to the bottom side.

In the present embodiment, the spare space is composed of 4 bits, and thus the compressed data is composed of 16 bits (4 times 4 bits) . In other words, the third circuit 66 joins the gathered 4 data pieces together to reconstruct the compressed data.

Next, as shown in FIG. 14, the third circuit 66 expands the compressed data, which has been gathered in Step S40, to obtain a reconstructed first regularized pixel data (Step S42) . In the present embodiment, the second circuit 64 has compressed the first regularized pixel data by using the compression tone curve shown in FIG. 10 in Step S30. Therefore, the third circuit 66 expands the compression data by using an expansion tone curve shown in FIG. 17. The expansion tone curve in FIG. 17 is a mirror image of the compression tone curve in FIG. 10. By using the expansion tone curve in FIG. 17, the third circuit 66 inversely converts the compressed data into the reconstructed first regularized pixel data which is nearly equal to the first regularized pixel data in the imaging sensor 60.

As shown in FIG. 17, the value of the compressed data is equal to or more than 0 and equal to or less than 65535 (2 ¹⁶) , but the value of the first regularized pixel data is equal to or more than 0 and equal to or less than 262143 (2 ¹⁸) . Therefore, the value of the first regularized pixel data cannot be precisely recovered. However, it is not a major problem because the brightness of the first regularized pixel data having the large value is almost saturated and it is impossible for the human eye to perceive errors of the compression.

Next, as shown in FIG. 14, the third circuit 66 outputs the reconstructed first regularized pixel data to the HDR image pipeline 72 (Step S44) . In the present embodiment, the reconstructed first regularized pixel data is composed of 18 bits. Therefore, the reconstructed first regularized pixel data can be inputted into the HDR image pipeline 72 to process the HDR image data.

The first regularized pixel data recovery process is completed after the Step S44 is executed. However, the first regularized pixel data recovery process shown in FIG. 14 is directed to one pixel. Therefore, the third circuit 66 executes the first regularized pixel data recovery process shown in FIG. 14 against every low gain pixel.

FIG. 18 illustrates the manner to utilize the regular image pipeline 70 and the HDR image pipeline 72. The regular image pipeline 70 can process the regular image data speedily but the quality of the regular image data is no so high. On the other hand, the HDR image pipeline 72 can generate a high quality image from the HDR image data but the process is not so speedy.

In the present embodiment, for example, the electrical device 10 generates a moving image (video) by using the clipped second regularized pixel data with the regular image pipeline 70. In this case, the electrical device 10 needs to process many images in a certain time due to characteristics of the moving image.

On the other hand, the electrical device 10 generates a still image by using the reconstructed first regularized pixel data with the HDR image pipeline 72. In this case, the electrical device 10 can have enough processing time to reconstruct and process the reconstructed first regularized pixel data.

In another example, the electrical device 10 may switch the regular image pipeline 70 and the HDR image pipeline 72 based on a frame rate of the moving image. For example, if the frame rate of the moving image is equal to or more than a certain threshold (for example, 60 images per second) , the electrical device 10 generates the moving image based on the clipped second regularized pixel data with the regular image pipeline 70.

On the other hand, if the frame rate of the moving image is less than the certain threshold, the electrical device 10 generates the moving image based on the reconstructed first regularized pixel data with the HDR image pipeline 72.

Of course, the electrical device 10 may select one of the regular image pipeline 70 and the HDR image pipeline 72 based on various factors in order to process the image data smoothly and to obtain high quality images for the users.

As described above, in accordance with the electrical device 10 according to the present embodiment, as shown in FIG. 6, the electrical device 10 can handle and process the HDR image data as if it is the regular image data. Therefore, the electrical device 10 can process the regular image data smoothly and speedily.

In addition, as shown in FIG. 8, the electrical device 10 can reconstruct the HDR image data based on the embedded data pieces of the compressed data based on the first regularized pixel data. Therefore, the electrical device 10 can handle and process the HDR image data to obtain high quality images for the users. As a result, the user can obtain unsaturated image and enjoy the HDR image.

Incidentally, in the embodiment mentioned above, although the first regularized pixel data is compressed in order to reduce the data amount, it is not necessarily to compress the first regularized pixel data. In such case, the second circuit 64 of the imaging sensor 60 divides the first regularized pixel data into a plurality of data pieces and embeds them into the regular processing image data of the low gain pixel itself and another regular processing image data of another pixel.

Moreover, if the data length of the compressed data is equal to or shorter than the data length of the spare space of the regular processing image data, the second circuit 64 does not need to divide the compressed data into the plurality of data pieces. In such case, the second circuit 64 embeds the compressed data into the spare space of the regular processing image data of the low gain pixel itself.

In the description of embodiments of the present disclosure, it is to be understood that terms such as "central" , "longitudinal" , "transverse" , "length" , "width" , "thickness" , "upper" , "lower" , "front" , "rear" , "back" , "left" , "right" , "vertical" , "horizontal" , "top" , "bottom" , "inner" , "outer" , "clockwise" and "counterclockwise" should be construed to refer to the orientation or the position as described or as shown in the drawings under discussion. These relative terms are only used to simplify description of the present disclosure, and do not indicate or imply that the device or element referred to must have a particular orientation, or constructed or operated in a particular orientation. Thus, these terms cannot be constructed to limit the present disclosure.

In addition, terms such as "first" and "second" are used herein for purposes of description and are not intended to indicate or imply relative importance or significance or to imply the number of indicated technical features. Thus, the feature defined with "first" and "second" may comprise one or more of this feature. In the description of the present disclosure, "a plurality of" means two or more than two, unless specified otherwise.

In the description of embodiments of the present disclosure, unless specified or limited otherwise, the terms "mounted" , "connected" , "coupled" and the like are used broadly, and may be, for example, fixed connections, detachable connections, or integral connections; may also be mechanical or electrical connections; may also be direct connections or indirect connections via intervening structures; may also be inner communications of two elements, which can be understood by those skilled in the art according to specific situations.

In the embodiments of the present disclosure, unless specified or limited otherwise, a structure in which a first feature is "on" or "below" a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are contacted via an additional feature formed therebetween. Furthermore, a first feature "on" , "above" or "on top of" a second feature may include an embodiment in which the first feature is right or obliquely "on" , "above" or "on top of" the second feature, or just means that the first feature is at a height higher than that of the second feature; while a first feature "below" , "under" or "on bottom of" a second feature may include an embodiment in which the first feature is right or obliquely "below" , "under" or "on bottom of" the second feature, or just means that the first feature is at a height lower than that of the second feature.

Various embodiments and examples are provided in the above description to implement different structures of the present disclosure. In order to simplify the present disclosure, certain elements and settings are described in the above. However, these elements and settings are only by way of example and are not intended to limit the present disclosure. In addition, reference numbers and/or reference letters may be repeated in different examples in the present disclosure. This repetition is for the purpose of simplification and clarity and does not refer to relations between different embodiments and/or settings. Furthermore, examples of different processes and materials are provided in the present disclosure. However, it would be appreciated by those skilled in the art that other processes and/or materials may be also applied.

Reference throughout this specification to "an embodiment" , "some embodiments" , "an exemplary embodiment" , "an example" , "a specific example" or "some examples" means that a particular feature, structure, material, or characteristics described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Thus, the appearances of the above phrases throughout this specification are not necessarily referring to the same embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

Any process or method described in a flow chart or described herein in other ways may be understood to include one or more modules, segments or portions of codes of executable instructions for achieving specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, in which it should be understood by those skilled in the art that functions may be implemented in a sequence other than the sequences shown or discussed, including in a substantially identical sequence or in an opposite sequence.

The logic and/or step described in other manners herein or shown in the flow chart, for example, a particular sequence table of executable instructions for realizing the logical function, may be specifically achieved in any computer readable medium to be used by the instruction execution system, device or equipment (such as the system based on computers, the system comprising processors or other systems capable of obtaining the instruction from the instruction execution system, device and equipment and executing the instruction) , or to be used in combination with the instruction execution system, device and equipment. As to the specification, "the computer readable medium" may be any device adaptive for including, storing, communicating, propagating or transferring programs to be used by or in combination with the instruction execution system, device or equipment. More specific examples of the computer readable medium comprise but are not limited to: an electronic connection (an electronic device) with one or more wires, a portable computer enclosure (a magnetic device) , a random access memory (RAM) , a read only memory (ROM) , an erasable programmable read-only memory (EPROM or a flash memory) , an optical fiber device and a portable compact disk read-only memory (CDROM) . In addition, the computer readable medium may even be a paper or other appropriate medium capable of printing programs thereon, this is because, for example, the paper or other appropriate medium may be optically scanned and then edited, decrypted or processed with other appropriate methods when necessary to obtain the programs in an electric manner, and then the programs may be stored in the computer memories.

It should be understood that each part of the present disclosure may be realized by the hardware, software, firmware or their combination. In the above embodiments, a plurality of steps or methods may be realized by the software or firmware stored in the memory and executed by the appropriate instruction execution system. For example, if it is realized by the hardware, likewise in another embodiment, the steps or methods may be realized by one or a combination of the following techniques known in the art: a discrete logic circuit having a logic gate circuit for realizing a logic function of a data signal, an application-specific integrated circuit having an appropriate combination logic gate circuit, a programmable gate array (PGA) , a field programmable gate array (FPGA) , etc.

Those skilled in the art shall understand that all or parts of the steps in the above exemplifying method of the present disclosure may be achieved by commanding the related hardware with programs. The programs may be stored in a computer readable storage medium, and the programs comprise one or a combination of the steps in the method embodiments of the present disclosure when run on a computer.

In addition, each function cell of the embodiments of the present disclosure may be integrated in a processing module, or these cells may be separate physical existence, or two or more cells are integrated in a processing module. The integrated module may be realized in a form of hardware or in a form of software function modules. When the integrated module is realized in a form of software function module and is sold or used as a standalone product, the integrated module may be stored in a computer readable storage medium.

The storage medium mentioned above may be read-only memories, magnetic disks, CD, etc.

Although embodiments of the present disclosure have been shown and described, it would be appreciated by those skilled in the art that the embodiments are explanatory and cannot be construed to limit the present disclosure, and changes, modifications, alternatives and variations can be made in the embodiments without departing from the scope of the present disclosure.

Claims

An imaging sensor comprising:

a first circuit configured to obtain a first captured pixel data which is a data of a regular gain pixel and to output the first captured pixel data as a regular processing image data from the imaging sensor; and

a second circuit configured to obtain a second captured pixel data which is a data of a low gain pixel and to regularize the second captured pixel data to obtain a first regularized pixel data,

wherein the second circuit is further configured to output the first regularized pixel data as the regular processing image data from the imaging sensor if the regular processing image data is not saturated when the regular processing image data is inputted into a regular image pipeline; and

the second circuit is further configured to clip a second regularized pixel data from the first regularized pixel data and to output the second regularized pixel data as the regular processing image data from the imaging sensor if the regular processing image data is saturated when the regular processing image data is inputted into the regular image pipeline.
The imaging sensor according to claim 1, wherein the regular processing image data is inputted into the regular image pipeline which processes a non-HDR (High Dynamic Range) image data.
The imaging sensor according to claim 1, wherein the second circuit is further configured to embed a saturated information on the first regularized pixel data into the regular processing image data itself and another regular processing image data.
The imaging sensor according to claim 3, wherein the second circuit is further configured to compress the first regularized pixel data in order to reduce a data amount and to obtain a compressed data.
The imaging sensor according to claim 4, wherein the second circuit is configured to divide the compressed data into a plurality of data pieces and the plurality of the data pieces are embedded into a spare space of the regular processing image data of the low gain pixel itself and another regular processing image data of another pixel, the spare space being a remaining space of the regular processing image data after the second regularized pixel data has been accommodated in the regular processing image data.
The imaging sensor according to claim 5, wherein the second circuit is further configured to compress the first regularized pixel data by reducing a number of bits of the first regularized pixel data based on a compression tone curve.
The imaging sensor according to claim 6, wherein the first regularized pixel data is compressed based on the compression tone curve, wherein the larger the first regularized pixel data is, the higher a compression rate of the compression tone curve is.
The imaging sensor according to claim 7, wherein the plurality of the data pieces are embedded into the spare space of the regular processing image data of the low gain pixel itself and the spare space of the regular processing image data of a pixel neighbor to the low gain pixel.
The imaging sensor according to claim 8, wherein the second circuit is further configured to output the regular processing image data including the second regularized pixel data and the data piece of the compressed data from the imaging sensor.
An electrical device comprising:

the imaging sensor according to any one of claims 5-9; and

a third circuit configured to gather the plurality of the data pieces embedded in the spare space of the regular processing image data of the low gain pixel itself and the spare space of the another regular processing image data of the another pixel to obtain the compressed data.
The electrical device according to claim 10, wherein the third circuit is further configured to expand the compressed data, which has been obtained by gathering the data pieces, to obtain a reconstructed first regularized pixel data.
The electrical device according to claim 11, wherein the third circuit is further configured to output the reconstructed first regularized pixel data.
The electrical device according to claim 12, wherein the reconstructed first regularized pixel data is inputted into an HDR image pipeline to process the reconstructed first regularized pixel data.
A non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following:

gathering a plurality of data pieces embedded in a spare space of a regular processing image data of a low gain pixel itself and a spare space of another regular processing image data of another pixel, wherein the regular processing image data is outputted from an imaging sensor;

obtaining a compressed data based on the gathered data pieces; and

expanding the compressed data to obtain a reconstructed first regularized pixel data.