WO2022183437A1

WO2022183437A1 - Method of generating embedded image data, image sensor, electrical device and non-transitory computer readable medium

Info

Publication number: WO2022183437A1
Application number: PCT/CN2021/079085
Authority: WO
Inventors: Toshihiko Arai
Original assignee: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date: 2021-03-04
Filing date: 2021-03-04
Publication date: 2022-09-09
Also published as: CN116711307A

Abstract

A method of generating an embedded image data according to the embodiment of the present disclosure includes: obtaining a piece of subject pixel data from first image data; obtaining a plurality of pieces of neighboring pixel data from second image data; calculating residual data based on the value of the subject pixel data, a smallest value among the values of the pieces of the neighboring pixel data, and a largest value among the values of the pieces of the neighboring pixel data; compressing the residual data by using a compression curve to generate compressed data; splitting the compressed data into a plurality of pieces of split data; and embedding the plurality of the pieces of the split data into the second image data to generate embedded image data.

Description

METHOD OF GENERATING EMBEDDED IMAGE DATA, IMAGE SENSOR, ELECTRICAL DEVICE AND NON-TRANSITORY COMPUTER READABLE MEDIUM

TECHNICAL FIELD

The present disclosure relates to a method of generating embedded image data, an image sensor, an electrical device and a non-transitory computer readable medium.

BACKGROUND

Electrical devices such as smartphones and tablet terminals are widely used in our daily life. Nowadays, many of the electrical devices are equipped with a camera assembly for capturing images. Some of the electrical devices are portable and are thus easy to carry. Therefore, a user of the electrical device can easily take a picture of an object by using the camera assembly of the electrical device anytime, anywhere.

There are many color filter formats of an image sensor. Among the color filter formats, the Bayer format is one of the most popular formats. In order to obtain a more precise image, the image sensor also captures an all green image in addition to the Bayer format because the human eye is very sensitive to the green color. The electrical device executes a fusion process against image data in the Bayer format and image data of the all green image to generate more precise images.

However, storing the image data of the all green image requires, not only extra transfer bandwidth from the image sensor to a memory, but also other resources. It also requires a more complicated hardware structure and higher development costs, thus also increasing manufacturing costs. The same problem exists in the other color filter formats.

SUMMARY

The present disclosure aims to solve at least one of the technical problems mentioned above. Accordingly, the present disclosure needs to provide a method of generating embedded image data, an image sensor, an electrical device and a non-transitory computer readable medium.

In accordance with the present disclosure, a method of generating an embedded image data may include:

obtaining a piece of subject pixel data from first image data, wherein the first image data is composed of first color pixels and the subject pixel data contains a value of a subject pixel;

obtaining a plurality of pieces of neighboring pixel data from second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data;

calculating residual data based on the value of the subject pixel data, a smallest value among the values of the plurality of the pieces of the neighboring pixel data, and a largest value among the values of the plurality of the pieces of the neighboring pixel data;

compressing the residual data by using a compression curve to generate compressed data, wherein the more a value of the residual data is increased, the more a compression ratio is increased on the compression curve;

splitting the compressed data into a plurality of pieces of split data; and

embedding the plurality of the pieces of the split data into the second image data to generate embedded image data.

In accordance with the present disclosure, a method of reconstructing a value of a subject pixel of subject pixel data of first image data composed of first color pixels may include:

extracting a plurality of pieces of the split data from an embedded image data;

combining the plurality of the pieces of the split data to generate a compressed data including residual compression data and an information bit;

expanding the residual compression data of the compressed data to obtain an expansion data by using a compression curve, wherein the more a value of the residual compression data is increased, the more a compression ratio is increased on the compression curve; and

reconstructing the value of the subject pixel of the subject pixel data based on the expansion data, the information bit, and a smallest value and a largest value among values of a plurality of pieces of neighboring pixel data of second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data, wherein the information bit indicates how to reconstruct the value of the subject pixel of the subject pixel data based on the expansion data and the plurality of the pieces of the neighboring pixel data of the second image data.

In accordance with the present disclosure, an image sensor may include:

a first obtainer configured to obtain a piece of subject pixel data from first image data, wherein the first image data is composed of first color pixels and the subject pixel data contains a value of a subject pixel;

a second obtainer configured to obtain a plurality of pieces of neighboring pixel data from second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data;

a calculator configured to calculate residual data based on the value of the subject pixel data, a smallest value among the values of the plurality of the pieces of the neighboring pixel data, and a largest value among the values of the plurality of the pieces of the neighboring pixel data;

a compressor configured to compress the residual data by using a compression curve to generate compressed data, wherein the more a value of the residual data is increased, the more a compression ratio is increased on the compression curve;

a splitter configured to split the compressed data into a plurality of pieces of split data; and

an embedder configured to embed the plurality of the pieces of the split data into the second image data to generate the embedded image data.

In accordance with the present disclosure, an electrical device may include:

the image sensor mentioned above; and

a memory configured to store the embedded image data output from the image sensor

In accordance with the present disclosure, a non-transitory computer readable medium may include program instructions stored thereon to reconstruct a value of a subject pixel of subject pixel data of first image data composed of first color pixels and to perform at least the following:

extracting a plurality of pieces of the split data from an embedded image data;

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of embodiments of the present disclosure will become apparent and more readily appreciated from the following descriptions made with reference to the drawings, in which:

FIG. 1 is a plan view of a first side of an electrical device according to an embodiment of the present disclosure;

FIG. 2 is a plan view of a second side of the electrical device according to the embodiment of the present disclosure;

FIG. 3 is a block diagram of the electrical device according to the embodiment of the present disclosure;

FIG. 4 illustrates an example of an arrangement of color filters in an image sensor of the camera assembly of the electrical device according to the embodiment of the present disclosure;

FIG. 5 illustrates a process flow of generating an embedded image data and a target image data in the electrical device of the embodiment of the present disclosure;

FIG. 6 illustrates a process flow of embedding and reconstructing image data of all green image;

FIG. 7 is a flowchart of an embedded image data generation process for generating the embedded image data based on image data of the Bayer format and image data of the all green image;

FIG. 8 illustrates one example of data formats of data processed by the electrical device 10 according to the embodiment of the present disclosure;

FIG. 9 illustrates example patterns of brightness of a subject green pixel and neighboring green pixels;

FIG. 10 is an explanatory drawing of a first option for reducing a value of residual data;

FIG. 11 is an explanatory drawing of the first option for reducing the value of the residual data;

FIG. 12 is an explanatory drawing of a second option for reducing the value of the residual data;

FIG. 13 illustrates one example of a compression curve to compress the residual data;

FIG. 14 illustrates one example of a compression curve modified from the compression curve shown in FIG. 13;

FIG. 15 illustrates the embedded image data in which a plurality of pieces of the split data are embedded into a spare space of a data space of each of corresponding pixels and a spare space of a data space of each of coupling pixels;

FIG. 16 is a flowchart of an all green image reconstruction process for reconstructing the image data of the all green image; and

FIG. 17 illustrates a process flow of reconstructing the image data of the all green image.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in detail and examples of the embodiments will be illustrated in the accompanying drawings. The same or similar elements and the elements having same or similar functions are denoted by like reference numerals throughout the descriptions. The embodiments described herein with reference to the drawings are explanatory and aim to illustrate the present disclosure, but shall not be construed to limit the present disclosure.

FIG. 1 is a plan view of a first side of an electrical device 10 according to an embodiment of the present disclosure and FIG. 2 is a plan view of a second side of the electrical device 10 according to the embodiment of the present disclosure. The first side may be referred to as a back side of the electrical device 10 whereas the second side may be referred to as a front side of the electrical device 10.

As shown in FIG. 1 and FIG. 2, the electrical device 10 may include a display 20 and a camera assembly 30. In the present embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34 and a sub camera 36. The first main camera 32 and the second main camera 34 can capture an image in the first side of the electrical device 10 and the sub camera 36 can capture an image in the second side of the electrical device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called out-cameras whereas the sub camera 36 is a so-called in-camera. As an example, the electrical device 10 can be a mobile phone, a tablet computer, a personal digital assistant, and so on.

Each of the first main camera 32, the second main camera 34 and the sub camera 36 has an imaging sensor which converts a light which has passed a color filter to an electrical signal. A signal value of the electrical signal depends on an amount of the light which has passed the color filter.

Although the electrical device 10 according to the present embodiment has three cameras, the electrical device 10 may have less than three cameras or more than three cameras. For example, the electrical device 10 may have two, four, five, and so on, cameras.

FIG. 3 is a block diagram of the electrical device 10 according to the present embodiment. As shown in FIG. 3, in addition to the display 20 and the camera assembly 30, the electrical device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46 and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power supply circuit 46 and the communication circuit 48 are connected with each other via a bus 50.

The main processor 40 executes one or more program instructions stored in the memory 44. The main processor 40 implements various applications and data processing of the electrical device 10 by executing the program instructions. The main processor 40 may be one or more computer processors. The main processor 40 is not limited to one CPU core, but it may have a plurality of CPU cores. The main processor 40 may be a main CPU of the electrical device 10, an image processing unit (IPU) or a DSP provided with the camera assembly 30.

The image signal processor 42 controls the camera assembly 30 and processes various kinds of image data captured by the camera assembly 30 to generate a target image data. For example, the image signal processor 42 can apply a demosaicing process, a noise reduction process, an auto exposure process, an auto focus process, an auto white balance process, a high dynamic range process and so on, to the image data captured by the camera assembly 30.

In the present embodiment, the main processor 40 and the image signal processor 42 collaborate with each other to generate a target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture the image of the object by means of the camera assembly 30 and apply various kinds of image processing to the captured image data.

The memory 44 stores program instructions to be executed by the main processor 40, and various kinds of data. For example, data of the captured image are also stored in the memory 44.

The memory 44 may include a high-speed RAM memory, and/or a non-volatile memory such as a flash memory and a magnetic disk memory. That is, the memory 44 may include a non-transitory computer readable medium in which the program instructions are stored.

The power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.

The communication circuit 48 is configured to receive and transmit data to communicate with base stations of the telecommunication network system, the Internet or other devices via wireless communication. The wireless communication may adopt any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication) , CDMA (Code Division Multiple Access) , LTE (Long Term Evolution) , LTE-Advanced, 5th generation (5G) . The communication circuit 48 may include an antenna and an RF (radio frequency) circuit.

FIG. 4 illustrates an example of an arrangement of color filters in an image sensor of the camera assembly 30. In the example shown in FIG. 4, the arrangement of the color filters is in compliance with the Bayer format to output image data of the Bayer format. In addition, the image sensor captures and outputs image data of an all green image to generate a more precise image.

In order to simultaneously capture the image data of the Bayer format and the image data of the all green image, the image sensor has pixels including red and green filters, pixels including blue and green filters, and pixels including all green filters. Therefore, the image sensor can output the image data of the Bayer format and the image data of the all green image at one time.

In the image data of the Bayer format, according to the present embodiment, a density of the green pixels is twice more than a density of the red pixels or a density of the blue pixels. The density of the red pixels and the density of the blue pixels are substantially the same. However, in the image data of the Bayer format, if a density of the green pixels is equal to or more than twice than a density of the red pixels or a density of the blue pixels, the present embodiment can be implemented.

On the other hand, in the image data of the all green image, according to the present embodiment, all pixels are composed of the green pixels. A density of the green pixels of the image data of the all green image is substantially equal to the density of the pixels including the red, blue and green pixels of the image data of the Bayer format.

Next, a process flow of generating an embedded image data and a target image data according to the embodiment of the present disclosure will be explained. FIG. 5 illustrates a process flow of generating the embedded image data and the target image data in the electrical device 10 of the embodiment of the present disclosure, and FIG. 6 illustrates a process flow of embedding and reconstructing the image data of the all green image.

As shown in FIG. 5 and FIG. 6, according to the electrical device 10 of the embodiment of the present disclosure, a first obtainer 60a of an image sensor 60 obtains the image data of the all green image and a second obtainer 60b of the image sensor 60 obtains the image data of the Bayer format. In the present embodiment, the image sensor 60 is included in the camera assembly 30.

The image data of the Bayer format includes the green pixels G1 and the image data of the all green image also includes the green pixels G1. Values of the green pixels G1 in the image data of the Bayer format are substantially equal to values of the green pixels G1 in the image data of the all green image. In other words, the values of the green pixels G1 in the image data of the all green image can also be obtained from the image data of the Bayer format later. As a result, in the present embodiment, information on the values of the green pixels G1 in the image data of the all green image is discarded.

On the other hand, values of the green pixels G2 in the image data of the all green image cannot be obtained from the image data of the Bayer format. Therefore, in the present embodiment, information on the values of the green pixels G2 is embedded into the image data of the Bayer format.

More specifically, as shown in FIG. 6, a calculator 60c of the image sensor 60 calculates a residual data based on a value of a subject pixel data of a subject green pixel G2x of the all green image and values of a plurality of pieces of neighboring pixel data of the neighboring green pixels G1a, G1b, G1c and G1d. The subject green pixel G2x of the image data of the all green image is subject to embedding into the image data of the Bayer format so that the information of the value of the subject pixel data of the subject green pixel G2x can be reconstructed later. The neighboring green pixels G1a, G1b, G1c and G1d in the image data of the Bayer format are pixels neighbor to the subject green pixel G2x in image data of the all green image because the neighboring green pixels G1a, G1b, G1c and G1d in the image data of the Bayer format can be regarded as the neighboring green pixels G1a, G1b, G1c and G1d in the image data of the all green image.

Under normal circumstances, the value of the subject green pixel G2x is close to the values of the neighboring green pixels G1a, G1b, G1c and G1d neighbor to the subject green pixel G2x. Therefore, the value of the residual data is very small.

Thereafter, a compressor 60d of the image sensor 60 compresses the residual data and a splitter 60e of the image sensor 60 splits the residual data into a plurality of pieces of the split data. Then, an embedder 60f of the image sensor 60 embeds the plurality of the pieces of the split data into the image data of the Bayer format to generate an embedded image data.

That is, as shown in FIG. 5, the image sensor 60 outputs the embedded image data, and the embedded image data is stored in the electrical device 10, for example, in the memory 44 of the electrical device 10. In other words, the image sensor 60 according to the present embodiment does not output the image data of the all green image. As a result, a mechanism to transfer the image data of the all green image from the image sensor 60 to the memory 44 can be omitted.

Thereafter, the electrical device 10 reads out the imbedded image data from the memory 44 and extracts the information on the subject green pixels G2 from the embedded image data of the Bayer format and reconstructs the image data of the all green image based on the information of the neighboring green pixels G1 of the image data of the Bayer format and the information on the reconstructed subject green pixels G2 from the image data of the Bayer format.

Then, the electrical device 10 executes a Bayer image process against the image data of the Bayer format and an all green image process against the image data of the all green image to generate the target image. For example, the target image is displayed on the display 20, stored in the memory 44, or transmitted to another electrical device via the communication circuit 48. The data formats of the target image data are, for example, JPG, PNG, GIF, TIFF and so on.

Next, processes for generating the target image executed in the electrical device 10 according to the embodiment of the present disclosure will be explained in more detail.

FIG. 7 is a flowchart of an embedded image data generation process for generating an embedded image data based on the image data of the Bayer format and the image data of the all green image.

As shown in FIG. 5 and FIG. 7, the first obtainer 60a of the image sensor 60 of the electrical device 10 obtains a subject pixel data from the image data of the all green image (Step S10) . The subject pixel data contains the value of the subject green pixel G2x to be processed in the image data of the all green image.

FIG. 8 illustrates one example of data formats of data processed by the electrical device 10 according to the embodiment of the present disclosure. As shown in FIG. 8, the value of the subject green pixel G2x in the image data of the all green image can be expressed by 12 bits. However, a data space of the subject pixel data is composed of 16 bits, and thus the data space of the subject pixel data has a spare space composed of 4 bits. Since the value of the subject green pixel G2x is expressed by 12 bits in the present embodiment, a data range of the subject green pixel G2x is between 0 and 4095.

Next, as shown in FIG. 5 and FIG. 7, the second obtainer 60b of the image sensor 60 of the electrical device 10 obtains the plurality of the pieces of the neighboring pixel data of the neighboring green pixels G1a, G1b, G1c and G1d neighbor to the subject green pixel G2x (Step S12) . That is, four pieces of the neighboring pixel data are obtained. The four pieces of the neighboring pixel data contain the value of the green pixel G1a, G1b, G1c and G1d, respectively. The neighboring pixel data may be obtained from the image data of the Bayer format or the image data of the all green image because the values of the green pixels G1a, G1b, G1c and G1d in the image data of the Bayer format are substantially equal to the values of the green pixels G1a, G1b, G1c and G1d in the image data of the all green image.

As shown in FIG. 8, each value of the neighboring green pixels G1a, G1b, G1c and G1d can be expressed by 12 bits. However, a data space of the neighboring pixel data is composed of 16 bits, and thus the data space of the neighboring pixel data has a spare space composed of 4 bits. Since the value of the neighboring green pixel is expressed by 12 bits, a data range of the neighboring green pixel is also between 0 and 4095.

Next, as shown in FIG. 5 and FIG. 7, the calculator 60c of the image sensor 60 of the electrical device 10 calculates a residual data based on the subject pixel data and the neighboring pixel data (Step S14) . In the present embodiment, the residual data is calculated based on the subject pixel data and the four pieces of the neighboring pixel data.

FIG. 9 illustrates example patterns of brightness of the green pixels G2x, G1a, G1b, G1c and G1d. For example, the subject green pixel G2x might be located in a plane of the brightness, in a boundary of the brightness, in a gradation of the brightness, in a diagonal boundary of the brightness, in a diagonal gradation of the brightness, or in a narrow line of the brightness. That is, there are many patterns of the brightness of the green pixels G2x, G1a, G1b, G1c and G1d.

Therefore, in the present embodiment, the image sensor 60 of the electrical device 10 employs a special method to calculate the residual data based on the four pieces of the neighboring pixel data in order to reduce the value of the residual data.

FIG. 10 and FIG. 11 are explanatory drawings of a first option for reducing the value of the residual data. As shown in FIG. 10 and FIG. 11, i ₁ through i ₄ indicate the values of the neighboring pixel data of the neighboring green pixels G1a, G1b, G1c and G1d, respectively. In the present embodiment, the four pieces of the neighboring pixel data is sorted in order of the values. That is, the value i ₁ is the smallest, i.e., the darkest. The value i ₄ is the largest, i.e., the brightest.

The image sensor 60 of the electrical device 10 calculates (i ₁+i ₄) /2 and judges whether the value x of the subject pixel data is less than (i ₁+i ₄) /2 or the value x of the subject pixel data is equal to or more than (i ₁+i ₄) /2. If the value x of the subject pixel data is less than (i ₁+i ₄) /2, 0 is assigned to an information bit D, which is a first logical value in the present embodiment. Then, the image sensor 60 calculates the residual data by subtracting the value i ₁ from the value x of the subject pixel data. That is, x-i ₁ is calculated. The information bit D is added to a compressed data later.

On the other hand, if the value x of the subject pixel data is equal to or more than (i ₁+i ₄) /2, 1 is assigned to the information bit D, which is a second logical value in the present embodiment. Then, the image sensor 60 calculates the residual data by subtracting the value x of the subject pixel data from the value i ₄. That is, i ₄-x is calculated. The information bit D is added to the compressed data later.

The first option shown in FIG. 10 and FIG. 11 is suitable for an image which includes many edges between black and white. For example, the first option is suitable for an image including many characters.

FIG. 12 is an explanatory drawing of a second option for reducing the value of the residual data. As shown in FIG. 12, the information bit D indicates whether the value x of the subject pixel data is equal to or between the smallest value i ₁ and the largest value i ₄. If the value x of the subject pixel data is equal to or larger than the smallest value i ₁ and equal to or smaller than the largest value i ₄, 0 is assigned to the information bit D, which is the first logical value in the present embodiment. On the other hand, if the value x of the subject pixel data is less than the smallest value i ₁ or larger than the largest value i ₄, 1 is assigned to the information bit D, which is the second logical value.

If the value x of the subject pixel data is less than the value i ₁, the image sensor 60 calculates the residual data by subtracting the value i ₁ from the value x of the subject pixel data. That is, x-i ₁ is calculated. Here, a sign of the residual data calculated by x-i ₁ is always negative. The information bit D=1 is added to the compressed data later.

If the value x of the subject pixel data is equal to or larger than the smallest value i ₁ and equal to or smaller than the largest value i ₄, the image sensor 60 calculates the residual data by subtracting the value of (i ₂+i ₃) /2 from the value x of the subject pixel data. That is, x- (i ₂+i ₃) /2 is calculated. Here, the value i ₂ is the second smallest value and the value i ₃ is the third smallest value among the values of the plurality of the pieces of the neighboring pixel data. The information bit D=0 is added to the compressed data later.

If the value of the subject pixel data is larger than the value i ₄, the image sensor 60 calculates the residual data by subtracting the value i ₄ from the value x of the subject pixel data. That is, x-i ₄ is calculated. Here, a sign of the residual data calculated by x-i ₄ is always positive. The information bit D=1 is added to the compressed data later.

It should be understood that, in both the first option and the second option, the information bit D indicates how to calculate the residual data. Although the first logical value is 0 and the second logical value is 1 in the present embodiment mentioned above, the first logical value may be 1 and the second logical value may be 0.

Moreover, in the second option, the sign of the residual data also indicates how to calculate the residual data. That is, if the information bit D is 1 and the sign of the residual data is negative, it indicates that x-i ₁ has been calculated to generate the residual data. On the contrary, if the information bit D is 1 and the sign of the residual data is positive, it indicates that x-i ₄ has been calculated to generate the residual data.

Furthermore, which of the first option and the second option should be used is decided based on a brightness profile of the image data of the Bayer format and/or the image data of the all green image. For example, if the brightness profile of the image data of the green pixels G1 of the Bayer format indicates that the brightness is polarized to a bright area and a dark area, the first option should be selected. On the other hand, if the brightness profile of the image data of the green pixels G1 of the Bayer format indicates that the brightness is spread from the bright area to the dark area, the second option should be selected. An evaluation value of the brightness profile of the image data of the green pixels G1 of the Bayer format can be calculated. Therefore, which of the first option and the second option has been used can be identified later based on the evaluation value of the green pixels G1 of the image data of the Bayer format.

In the present embodiment, the evaluation value of the brightness profile of the image data of the green pixels G1 of the Bayer format is calculated by the calculator 60c of the image sensor 60. Therefore, the image sensor 60 may automatically decide which of the first option and the second option should be used to generate the residual data based on the evaluation value.

As shown in FIG. 8, the residual data can be expressed by 13 bits because the values of the subject green pixel G2x and the neighboring green pixels G1a, G1b, G1c and G1d are expressed by 12 bits. That is, the residual data is composed of signed 13 bits. However, in the present embodiment, it is expected that the value of the residual data is normally very small because of the special method to calculate the residual data mentioned above.

Next, as shown in FIG. 5 and FIG. 7, the compressor 60d of the image sensor 60 of the electrical device 10 according to the embodiment of the present disclosure generates the compressed data based on the residual data (Step S16) . The compressed data includes a residual compression data and the information bit D. The residual compression data is generated by compressing the residual data. That is, the number of the bits of the residual compression data is smaller than the number of the bits of the residual data. In the present embodiment, as shown in FIG. 8, the residual compression data is composed of 7 bits.

There are various methods to compress the residual data. In the present embodiment, the image sensor 60 of the electrical device 10 uses a compression curve which is a kind of function defining an input of the residual data and an output of the residual compression data.

FIG. 13 illustrates one example of the compression curve. In this example, the compression curve is expressed by Gamma 2.2. On the compression curve, the more the value of the residual data is increased, the more the compression ratio is increased.

However, in the present embodiment, it can be expected that the value of the residual data is very small under normal circumstances. Therefore, it is also possible to modify the compression curve to lower the compression ratio for a range in which the value of the residual data is small. In this case, the compression ratio for a range in which the value of the residual data is large becomes high. However, since the possibility that the value of the residual data is large is very low, this is not a big problem.

In other words, if the value of the residual data is large, it means that the brightness of the subject green pixel G2x is significantly different from the brightness of the neighboring green pixels G1a, G1b, G1c and G1d. However, even if the brightness includes some amount of error after the residual data is expanded later, such errors are not distinguishable to the human eye.

FIG. 14 illustrates one example of a compression curve modified from the compression curve shown in FIG. 13. In the example shown in FIG. 14, the compression ratio for the range in which the value of the residual data is small is smaller than that of the compression curve shown in FIG. 13. Therefore, it is possible to expand the residual compression data more precisely as compared to the compression curve shown in FIG. 13.

In the present embodiment, as shown in FIG. 8, the compressed data is composed of 8 bits, and it includes the information bit D composed of 1 bit and the residual compression data composed of 7 bits.

Next, as shown in FIG. 5 and FIG. 7, the splitter 60e of the image sensor 60 of the electrical device 10 according to the embodiment of the present disclosure generates split data based on the compressed data (Step S18) . That is, the compressed data is split into at least two pieces of the split data. In the present embodiment, as shown in FIG. 8, the compressed data composed of 8 bits is split into two pieces of the split data composed of 4 bits.

Next, as shown in FIG. 5 and FIG. 7, the embedder 60f of the image sensor 60 of the electrical device 10 according to the embodiment of the present disclosure embeds the split data into the image data of the Bayer format to generate an embedded image data (Step S20) . For example, as shown in FIG. 8, since one piece of the split data is composed of 4 bits and the spare space of the pixel data of the image data of the Bayer format is composed of 4 bits, two spare spaces are needed for embedding the two pieces of the split data.

In general, the two pieces of the split data may be embedded into any spare space of the data space of the pixel data in the image data of the Bayer format. However, in order to efficiently execute processes for embedding and extracting the two pieces of the split data, two spare spaces in the image data of the Bayer format should be close.

Therefore, as shown in FIG. 15, the image sensor 60 of the electrical device according to the embodiment of the present disclosure embeds the two pieces of the split data into the spare space of the data space of pixel data of a corresponding pixel R or B and the spare space of the data space of the pixel data of a coupling pixel G1.

For example, when the subject green pixel G2x is a subject pixel to be embedded, a corresponding pixel R or B of the subject green pixel G2x is the red pixel Rc in the image data of the Bayer format. A position of the corresponding pixel Rc in the image data of the Bayer format corresponds to a position of the subject green pixel G2x in the all green image. A coupling pixel G1 of the corresponding pixel Rc in the image data of the Bayer format is the coupling pixel G1d in the image data of the Bayer format. A position of the coupling pixel G1d is next to the position of the corresponding pixel Rc. The value of the neighboring green pixel G1d in the all green image is substantially the same as the value of the coupling pixel G1d in the Bayer format.

Therefore, for example, a first piece of the split data generated from the subject pixel data of the subject green pixel G2x is embedded into the spare space of the data space of the pixel data of the corresponding pixel Rc. In addition, a second piece of the split data generated from the subject pixel data of the subject green pixel G2x is embedded into the spare space of the data space of the pixel data of the coupling pixel G1d.

Similarly, when the subject green pixel G2y is a subject pixel to be embedded, a corresponding pixel R or B of the subject green pixel G2y is the blue pixel Bc in the image data of the Bayer format. A position of the corresponding pixel Bc in the image data of the Bayer format corresponds to a position of the subject green pixel G2y in the all green image. A coupling pixel G1 of the corresponding pixel Bc in the image data of the Bayer format is the green pixel G1b in the image data of the Bayer format. A position of the coupling pixel G1b is next to the position of the corresponding pixel Bc. The value of the neighboring green pixel G1b in the all green image is substantially the same as the value of the coupling pixel G1b in the Bayer format.

Therefore, for example, a first piece of the split data generated from the subject pixel data of the subject green pixel G2y is embedded into the spare space of the data space of the pixel data of the corresponding pixel Bc. In addition, a second piece of the split data generated from the subject pixel data of the subject green pixel G2y is embedded into the spare space of the data space of the pixel data of the coupling pixel G1b.

That is, the two pieces of the split data are embedded into the spare space of the data space of the pixel data of the corresponding pixel R or B corresponding to the subject green pixel G2 and the spare space of the data space of the pixel data of the coupling pixel G1 next to the corresponding pixel R or B in the Bayer format.

Next, as shown in FIG. 5 and FIG. 7, the image sensor 60 of the electrical device 10 according to the embodiment of the present disclosure outputs the embedded image data (Step S22) . In other words, the image sensor 60 does not output the image data of the all green image because the information for reconstructing the image data of the all green image has been embedded into the image data of the Bayer format which is referred to as the embedded image data.

After the image sensor 60 outputs the embedded image data in Step S22, the embedded image data generation process is completed. However, the embedded image data generation process is repeatedly executed until all of the subject image data of the subject green pixels G2 have been processed.

As shown in FIG. 3 and FIG. 5, the embedded image data is outputted from the image sensor 60 and stored, for example, in the memory 44. The embedded image data is also referred to as the raw image data of the Bayer format. Therefore, the embedded image data should be subjected to a target image generation process to generate the target image data.

In the electrical device 10 according to the embodiment of the present disclosure, the target image generation process is executed by the image signal processor 42. The image signal processor 42 executes the Bayer image process against the image data of the Bayer format and the all green image process against the image data of the all green image. Thereafter, the image signal processor 42 fuses the image data of the Bayer format and the image data of the all green image and finally outputs the target image data.

Therefore, in the present embodiment, the electrical device 10 has to reconstruct the image data of the all green image from the embedded image data because the information for reconstructing the image data of the all green image has been embedded into the embedded image data by the embedded image data generation process shown in FIG. 7.

In order to reconstruct the image data of the all green image, the electrical device 10 according to the embodiment of the present disclosure executes an all green image reconstruction process shown in FIG. 16.

That is, FIG. 16 is a flowchart of the all green image reconstruction process for reconstructing the image data of the all green image. FIG. 17 illustrates a process flow of reconstructing the image data of the all green image. Basically, the all green image reconstruction process is an inverse process of the embedded image data generation process mentioned above.

In the present embodiment, the all green image reconstruction process is executed by the main processor 40. That is, the all green image reconstruction process is implemented as a software program including program instructions. The program instructions may be stored on a non-transitory computer readable medium, which is, for example, the memory 44 in the electrical device 10. Also, the program instructions may be stored on a non-transitory computer readable medium outside the electrical device 10. In this case, for example, the electrical device 10 may obtain the program instructions via the communication circuit 48 through the Internet from a server storing the program instructions.

However, the all green image reconstruction process may be implemented by a piece of hardware such as an application-specific integrated circuit.

As shown in FIG. 16 and FIG. 17, the electrical device 10 according to the embodiment of the present disclosure extracts the split data from the embedded image data (Step S30) . For example, the electrical device 10 obtains the embedded image data from the memory 44 and extracts two pieces of the split data from the spare space of the data space of the pixel data of the corresponding pixel R or B and the spare space of the data space of the pixel data of the coupling pixel G1.

Next, as shown in FIG. 16 and FIG. 17, the electrical device 10 according to the embodiment of the present disclosure combines the split data obtained in Step S30 to generate the compressed data (Step S32) . For example, the electrical device 10 combines the two pieces of the split data composed of 4 bits each and generates the compressed data composed of 8 bits.

Next, as shown in FIG. 16 and FIG. 17, the electrical device 10 according to the embodiment of the present disclosure expands the compressed data to generate an expansion data (Step S34) . As shown in FIG. 8, in the present embodiment, the compressed data composed of 8 bits incudes the information bit D composed of 1 bit and the residual compression data composed of 7 bits.

Therefore, the electrical device 10 expands the residual compression data by using the compression curve shown in FIG. 13 or FIG. 14 to obtain the expansion data. In other words, the expansion data composed of 13 bits is obtained by inversely converting the residual compression data based on the compression curve which has been used to generate the residual compression data in Step S14.

As mentioned above, the value of the expansion data is not precisely equal to the value of the residual data. That is, as mentioned above, if the value of the expansion data is small, the value of the expansion data is almost precisely equal to the value of the residual data because the compression ratio is very low. On the other hand, if the value of the expansion data is large, the value of the expansion data is not precisely equal to the value of the residual data because the compression ratio is very high. However, this is not a problem as explained above.

Next, as shown in FIG. 16 and FIG. 17, the electrical device 10 according to the embodiment of the present disclosure reconstructs the subject pixel data based on the expansion data and the information bit D (Step S36) . The process of the Step S36 is an inverse process of the Step S14 in the embedded image data generation process shown in FIG. 7.

Therefore, if the compressed data has been generated based on the first option mentioned above in Step S14, the subject pixel data is reconstructed based on the first option by an inverse process of calculating the residual data in Step S14.

On the other hand, if the compression data has been generated based on the second option mentioned above in Step S14, the subject pixel data is reconstructed based on the second option by an inverse process of calculating the residual data in Step S14. As explained in Step S14, which of the inverse processes should be used is decided based on the brightness profile of the image data of the Bayer format, i.e., based on the evaluation value of the green pixels G1 of the image data of the Bayer format.

Next, as shown in FIG. 16 and FIG. 17, the electrical device 10 according to the embodiment of the present disclosure introduces the subject pixel data of the subject green pixel G2x into the image data of the neighboring green pixels G1 to reconstruct the image data of the all green image (Step S38) . That is, the reconstructed subject pixel data of the subject green pixel G2x is introduced between the four pieces of the neighboring pixel data of the neighboring green pixels G1a, G1b, G1c and G1d.

After the reconstructed subject pixel data of the subject green pixel G2 is introduced in Step 38, the all green image reconstruction process is completed. However, the all green image reconstruction process is repeatedly executed until all of the split data embedded in the embedded image data are processed in order to reconstruct the image data of the all green image including the neighboring green pixels G1 and the subject green pixels G2.

After the image data of the all green image has been reconstructed, the image data of the all green image is subjected to the all green image process. Also, the image data of the Bayer format is subjected to the Bayer image process. Thereafter, a fusion process is executed to fuse the image data of the all green image and the image data of the Bayer format to generate the target image data.

In the present embodiment, these processes of the all green image process, the Bayer image process and the fusion process are executed in the image signal processor 42. Thus, the image data of the all green image and the image data of the Bayer format are inputted into the image signal processor 42 from the main processor 40. However, these processes of the all green image process, the Bayer image process and the fusion process may be implemented as a software program including program instructions which is operated by, for example, the main processor 40.

As described above, in accordance with the electrical device 10 according to the embodiment of the present disclosure, as shown in FIG. 8, the compression data includes the residual compression data and the information bit D which indicates how to calculate the residual data. Therefore, the electrical device 10 can decide how to reconstruct the subject pixel data from the expansion data and the neighboring pixel data.

Moreover, since the information bit D indicates how to calculate the residual data, a more suitable method for reducing the value of the residual data can be selected among a plurality of methods when calculating the residual data. Therefore, it is possible to modify the compression curve to lower the compression ratio for the range in which the value of the residual data is small instead of raising the compression ratio for the range in which the value of the residual data is large. As a result, the target image data can be generated more precisely because the value of the residual data is usually very small.

Incidentally, in the embodiment described above, the image data of the all green image is one of the examples of the first image data composed of first color pixels, which are the green pixels G1 and G2. The first color pixels are not limited to the green pixels G1 and G2, and another color can be applied to the first color pixels.

Also, the image data of the Bayer format is one of the examples of the second image data composed of second color pixels, third color pixels and fourth color pixels. The format of the second image data is not limited to the Bayer format, that is, another format can be applied to the second image data.

In the description of embodiments of the present disclosure, it is to be understood that terms such as "central" , "longitudinal" , "transverse" , "length" , "width" , "thickness" , "upper" , "lower" , "front" , "rear" , "back" , "left" , "right" , "vertical" , "horizontal" , "top" , "bottom" , "inner" , "outer" , "clockwise" and "counterclockwise" should be construed to refer to the orientation or the position as described or as shown in the drawings under discussion. These relative terms are only used to simplify description of the present disclosure, and do not indicate or imply that the device or element referred to must have a particular orientation, or constructed or operated in a particular orientation. Thus, these terms cannot be constructed to limit the present disclosure.

In addition, terms such as "first" and "second" are used herein for purposes of description and are not intended to indicate or imply relative importance or significance or to imply the number of indicated technical features. Thus, the feature defined with "first" and "second" may comprise one or more of this feature. In the description of the present disclosure, "a plurality of" means two or more than two, unless specified otherwise.

In the description of embodiments of the present disclosure, unless specified or limited otherwise, the terms "mounted" , "connected" , "coupled" and the like are used broadly, and may be, for example, fixed connections, detachable connections, or integral connections; may also be mechanical or electrical connections; may also be direct connections or indirect connections via intervening structures; may also be inner communications of two elements, which can be understood by those skilled in the art according to specific situations.

In the embodiments of the present disclosure, unless specified or limited otherwise, a structure in which a first feature is "on" or "below" a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are contacted via an additional feature formed therebetween. Furthermore, a first feature "on" , "above" or "on top of" a second feature may include an embodiment in which the first feature is right or obliquely "on" , "above" or "on top of" the second feature, or just means that the first feature is at a height higher than that of the second feature; while a first feature "below" , "under" or "on bottom of" a second feature may include an embodiment in which the first feature is right or obliquely "below" , "under" or "on bottom of" the second feature, or just means that the first feature is at a height lower than that of the second feature.

Various embodiments and examples are provided in the above description to implement different structures of the present disclosure. In order to simplify the present disclosure, certain elements and settings are described in the above. However, these elements and settings are only by way of example and are not intended to limit the present disclosure. In addition, reference numbers and/or reference letters may be repeated in different examples in the present disclosure. This repetition is for the purpose of simplification and clarity and does not refer to relations between different embodiments and/or settings. Furthermore, examples of different processes and materials are provided in the present disclosure. However, it would be appreciated by those skilled in the art that other processes and/or materials may be also applied.

Reference throughout this specification to "an embodiment" , "some embodiments" , "an exemplary embodiment" , "an example" , "a specific example" or "some examples" means that a particular feature, structure, material, or characteristics described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Thus, the appearances of the above phrases throughout this specification are not necessarily referring to the same embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

Any process or method described in a flow chart or described herein in other ways may be understood to include one or more modules, segments or portions of codes of executable instructions for achieving specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, in which it should be understood by those skilled in the art that functions may be implemented in a sequence other than the sequences shown or discussed, including in a substantially identical sequence or in an opposite sequence.

The logic and/or step described in other manners herein or shown in the flow chart, for example, a particular sequence table of executable instructions for realizing the logical function, may be specifically achieved in any computer readable medium to be used by the instruction execution system, device or equipment (such as the system based on computers, the system comprising processors or other systems capable of obtaining the instruction from the instruction execution system, device and equipment and executing the instruction) , or to be used in combination with the instruction execution system, device and equipment. As to the specification, "the computer readable medium" may be any device adaptive for including, storing, communicating, propagating or transferring programs to be used by or in combination with the instruction execution system, device or equipment. More specific examples of the computer readable medium comprise but are not limited to: an electronic connection (an electronic device) with one or more wires, a portable computer enclosure (a magnetic device) , a random access memory (RAM) , a read only memory (ROM) , an erasable programmable read-only memory (EPROM or a flash memory) , an optical fiber device and a portable compact disk read-only memory (CDROM) . In addition, the computer readable medium may even be a paper or other appropriate medium capable of printing programs thereon, this is because, for example, the paper or other appropriate medium may be optically scanned and then edited, decrypted or processed with other appropriate methods when necessary to obtain the programs in an electric manner, and then the programs may be stored in the computer memories.

It should be understood that each part of the present disclosure may be realized by the hardware, software, firmware or their combination. In the above embodiments, a plurality of steps or methods may be realized by the software or firmware stored in the memory and executed by the appropriate instruction execution system. For example, if it is realized by the hardware, likewise in another embodiment, the steps or methods may be realized by one or a combination of the following techniques known in the art: a discrete logic circuit having a logic gate circuit for realizing a logic function of a data signal, an application-specific integrated circuit having an appropriate combination logic gate circuit, a programmable gate array (PGA) , a field programmable gate array (FPGA) , etc.

Those skilled in the art shall understand that all or parts of the steps in the above exemplifying method of the present disclosure may be achieved by commanding the related hardware with programs. The programs may be stored in a computer readable storage medium, and the programs comprise one or a combination of the steps in the method embodiments of the present disclosure when run on a computer.

In addition, each function cell of the embodiments of the present disclosure may be integrated in a processing module, or these cells may be separate physical existence, or two or more cells are integrated in a processing module. The integrated module may be realized in a form of hardware or in a form of software function modules. When the integrated module is realized in a form of software function module and is sold or used as a standalone product, the integrated module may be stored in a computer readable storage medium.

The storage medium mentioned above may be read-only memories, magnetic disks, CD, etc.

Although embodiments of the present disclosure have been shown and described, it would be appreciated by those skilled in the art that the embodiments are explanatory and cannot be construed to limit the present disclosure, and changes, modifications, alternatives and variations can be made in the embodiments without departing from the scope of the present disclosure.

Claims

A method of generating an embedded image data, comprising:

obtaining a piece of subject pixel data from first image data, wherein the first image data is composed of first color pixels and the subject pixel data contains a value of a subject pixel;

obtaining a plurality of pieces of neighboring pixel data from second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data;

calculating residual data based on the value of the subject pixel data, a smallest value among the values of the plurality of the pieces of the neighboring pixel data, and a largest value among the values of the plurality of the pieces of the neighboring pixel data;

compressing the residual data by using a compression curve to generate compressed data, wherein the more a value of the residual data is increased, the more a compression ratio is increased on the compression curve;

splitting the compressed data into a plurality of pieces of split data; and

embedding the plurality of the pieces of the split data into the second image data to generate embedded image data.
The method according to claim 1, wherein the compression data includes a residual compression data and an information bit, the residual compression data is generated by compressing the residual data, and the information bit indicates how to calculate the residual data.
The method according to claim 2, wherein the plurality of the pieces of the neighboring pixel data are four pieces.
The method according to claim 3, wherein the calculating the residual data comprises:

calculating (i ₁+i ₄) /2, where i ₁ is the smallest value among the values of the plurality of the pieces of the neighboring pixel data, and i ₄ is the largest value among the values of the plurality of the pieces of the neighboring pixel data;

assigning a first logical value to the information bit if the value of the subject pixel data is less than (i ₁+i ₄) /2; and

assigning a second logical value to the information bit if the value of the subject pixel data is equal to or more than (i ₁+i ₄) /2.
The method according to claim 4, wherein the calculating the residual data further comprises:

calculating the residual data by subtracting the smallest value i ₁ from the value of the subject pixel data if the value of the subject pixel data is less than (i ₁+i ₄) /2; and

calculating the residual data by subtracting the value of the subject pixel data from the largest value i ₄ if the value of the subject pixel data is equal to or more than (i ₁+i ₄) /2.
The method according to claim 3, wherein the calculating the residual data comprises:

assigning a first logical value to the information bit if the value of the subject pixel data is equal to or larger than the smallest value i ₁ and equal to or smaller than the largest value i ₄; and

assigning a second logical value to the information bit if the value of the subject pixel data is less than the smallest value i ₁ or larger than the largest value i ₄.
The method according to claim 6, wherein the calculating the residual data further comprises:

calculating the residual data by subtracting the smallest value i ₁ from the value of the subject pixel data if the value of the subject pixel data is less than the smallest value i ₁;

calculating the residual data by subtracting the value of (i ₂+i ₃) /2 from the value of the subject pixel data if the value of the subject pixel data is equal to or larger than the smallest value i ₁ and equal to or smaller than the largest value i ₄, where the value i ₂ is the second smallest value and the value i ₃ is the third smallest value among the values of the plurality of the pieces of the neighboring pixel data; and

calculating the residual data by subtracting the largest value i ₄ from the value of the subject pixel data if the value of the subject pixel data is larger than the largest value i ₄.
The method according to claim 1, wherein the compressed data is split into two pieces of the split data.
The method according to claim 8, wherein the two pieces of the split data are embedded into a spare space of a data space of a corresponding pixel of the second image data and a spare space of a data space of a coupling pixel of the second image data,

a position of the corresponding pixel in the second image data corresponds to a position of the subject pixel in the first image data, and

a position of the coupling pixel is next to the position of the corresponding pixel in the second image data.
The method according to claim 1, wherein the second image data is image data of the

Bayer format.
The method according to claim 1, wherein the first color pixels are green pixels, the second color pixels are green pixels, the third color pixels are red color pixels, and the fourth color pixels are blue color pixels.
A method of reconstructing a value of a subject pixel of subject pixel data of first image data composed of first color pixels, comprising:

extracting a plurality of pieces of split data from an embedded image data;

combining the plurality of the pieces of the split data to generate a compressed data including residual compression data and an information bit;

expanding the residual compression data of the compressed data to obtain an expansion data by using a compression curve, wherein the more a value of the residual compression data is increased, the more a compression ratio is increased on the compression curve; and

reconstructing the value of the subject pixel of the subject pixel data based on the expansion data, the information bit, and a smallest value and a largest value among values of a plurality of pieces of neighboring pixel data of second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data, wherein the information bit indicates how to reconstruct the value of the subject pixel of the subject pixel data based on the expansion data and the plurality of the pieces of the neighboring pixel data of the second image data.
An image sensor, comprising:

a first obtainer configured to obtain a piece of subject pixel data from first image data, wherein the first image data is composed of first color pixels and the subject pixel data contains a value of a subject pixel;

a second obtainer configured to obtain a plurality of pieces of neighboring pixel data from second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data;

a calculator configured to calculate residual data based on the value of the subject pixel data, a smallest value among the values of the plurality of the pieces of the neighboring pixel data, and a largest value among the values of the plurality of the pieces of the neighboring pixel data;

a compressor configured to compress the residual data by using a compression curve to generate compressed data, wherein the more a value of the residual data is increased, the more a compression ratio is increased on the compression curve;

a splitter configured to split the compressed data into a plurality of pieces of split data; and

an embedder configured to embed the plurality of the pieces of the split data into the second image data to generate embedded image data.
An electrical device comprising:

the image sensor according to claim 13; and

a memory configured to store the embedded image data output from the image sensor.
A non-transitory computer readable medium comprising program instructions stored thereon to reconstruct a value of a subject pixel of subject pixel data of first image data composed of first color pixels and to perform at least the following:

extracting a plurality of pieces of split data from an embedded image data;

combining the plurality of the pieces of the split data to generate a compressed data including residual compression data and an information bit;

expanding the residual compression data of the compressed data to obtain an expansion data by using a compression curve, wherein the more a value of the residual compression data is increased, the more a compression ratio is increased on the compression curve; and

reconstructing the value of the subject pixel of the subject pixel data based on the expansion data, the information bit, and a smallest value and a largest value among values of a plurality of pieces of neighboring pixel data of second image data, wherein the plurality of the pieces of the neighboring pixel data contain values of neighboring pixels neighbor to the subject pixel, the second image data is composed of second color pixels, third color pixels and fourth color pixels, a density of the second color pixels is equal to or more than twice than a density of the third color pixels or a density of the fourth color pixels, and a density of the first color pixels of the first image data is substantially equal to a density of pixels including the second, third and fourth color pixels of the second image data, wherein the information bit indicates how to reconstruct the value of the subject pixel of the subject pixel data based on the expansion data and the plurality of the pieces of the neighboring pixel data of the second image data.