WO2022193244A1

WO2022193244A1 - Method of embedding additional data onto raw pixel data and electronic device

Info

Publication number: WO2022193244A1
Application number: PCT/CN2021/081605
Authority: WO
Inventors: Toshihiko Arai
Original assignee: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date: 2021-03-18
Filing date: 2021-03-18
Publication date: 2022-09-22
Also published as: CN116848545A

Abstract

Disclosed is a method of embedding additional data onto RAW pixel data read out from a pixel of an image sensor. The method includes acquiring additional data, a size of the additional data being less than a size of the RAW pixel data, masking the additional data with a random pattern to generate random data, and embedding the random data on the LSB side of the RAW pixel data.

Description

METHOD OF EMBEDDING ADDITIONAL DATA ONTO RAW PIXEL DATA AND ELECTRONIC DEVICE

TECHNICAL FIELD

The present disclosure relates to a method of embedding additional data onto RAW pixel data read out from a pixel of an image sensor, and an electronic device implementing such method.

BACKGROUND

There are some techniques of embedding data onto an image captured by an image sensor. For example, it has been known to add a monochrome image or depth information onto the Bayer RAW data.

However, an image signal processor must have additional resources or perform additional data processing in order to perform image processing on the image with the additional data/information embedded.

SUMMARY

The present disclosure aims to solve at least one of the technical problems mentioned above. Accordingly, the present disclosure needs to provide a method of embedding additional data onto RAW pixel data read out from a pixel of an image sensor, and an electrical device implementing such method.

In accordance with the present disclosure, the method may include:

acquiring additional data, a size of the additional data being less than a size of the RAW pixel data;

masking the additional data with a random pattern to generate random data; and

embedding the random data on the LSB side of the RAW pixel data.

In some embodiments, the method may further include, before embedding the random data on the LSB side of the RAW pixel data, shifting the RAW pixel data to the MSB side to create a bit container, wherein the random data is embedded in the bit container.

In some embodiments, a size of the bit container may be the same as a size of the random data.

In some embodiments, at least one bit of the RAW pixel data may be overwritten by the random data.

In some embodiments, the masking the additional data with a random pattern to generate random data may include performing an XOR operation on the additional data and the random pattern.

In some embodiments, the random pattern may be different for each pixel of the image sensor.

In some embodiments, the random pattern may be a bit sequence generated based on a location of the pixel in the image sensor.

In some embodiments, the random pattern may be a bit sequence generated based on a value of the pixel.

In some embodiments, the additional data may be depth data or exposure time.

In some embodiments, the method may further include performing image processing on the RAW pixel data with the random data embedded by treating the random data as noise.

In accordance with the present disclosure, a method of obtaining the additional data embedded to the RAW pixel data, the method may include:

extracting the random data from the RAW pixel data; and

masking the random data with the random pattern to restore the additional data.

In some embodiments, the masking the random data with the random pattern to restore the additional data may include performing an XOR operation on the random data and the random pattern.

In accordance with the present disclosure, an electric device may include a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, cause the processor to perform the method according to the present disclosure.

In accordance with the present disclosure, a non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program is executed by a computer to implement the method according to the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of embodiments of the present disclosure will become apparent and more readily appreciated from the following descriptions made with reference to the drawings, in which:

FIG. 1 is a functional block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 2 is a flowchart illustrating a method of embedding additional data onto RAW pixel data read out from a pixel of an image sensor according to an embodiment of the present disclosure;

FIG. 3 is a diagram for illustrating a step of shifting the RAW pixel data;

FIG. 4 is a diagram for illustrating a step of masking the additional data with a random pattern to generate random data;

FIG. 5 is a diagram for illustrating a step of embedding the random data on the LSB side of the RAW pixel data;

FIG. 6 is a diagram for illustrating an image processing on the RAW pixel data with embedded data performed by an image signal processor;

FIG. 7 is a diagram for illustrating a step of restoring the embedded data;

FIG. 8 is an example of the Gamma curve;

FIG. 9 is a diagram for illustrating a step of shifting the RAW pixel data in which some bits on the LSB side are buried under noise;

FIG. 10 is a diagram for illustrating a step of masking the additional data with a random pattern to generate random data which is larger than a bit shift amount of the RAW pixel data;

FIG. 11 is a diagram for illustrating a step of embedding the random data on the LSB side of the RAW pixel data so that 2 bits on the LSB side are overwritten;

FIG. 12 is a diagram for illustrating a modification example in which the step of shifting the RAW pixel data is not performed.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in detail and examples of the embodiments will be illustrated in the accompanying drawings. The same or similar elements and elements having same or similar functions are denoted by like reference numerals throughout the descriptions. The embodiments described herein with reference to the drawings are explanatory, which aim to illustrate the present disclosure, but shall not be construed to limit the present disclosure.

An electronic device 100 will be described with reference to FIG. 1. FIG. 1 is a circuit diagram illustrating an example of a configuration of the electronic device 100 according to an embodiment of the present disclosure.

The electronic device 100 is a mobile device such as a smartphone in this embodiment, but may be other types of electronic device equipped with one or more camera modules.

As shown in FIG. 1, the electronic device 100 includes a camera module 10, a range sensor module 20, and an image signal processor (ISP) 30 that controls the camera module 10 and the range sensor module 20. The image signal processor 30 is also configured to perform image processing based on an image acquired from the camera module 10 and a depth image acquired from the range sensor module 20.

The camera module 10 includes a master camera module 11 and a slave camera module 12 as shown in FIG. 1. The master camera module 11 includes a first lens 11a that is capable of focusing on a subject, a first image sensor 11b that detects an image inputted via the first lens 11a, and a first image sensor driver 11c that drives the first image sensor 11b, as shown in FIG. 1. The slave camera module 12 includes a second lens 12a that is capable of focusing on a subject, a second image sensor 12b that detects an image inputted via the second lens 12a, and a second image sensor driver 12c that drives the second image sensor 12b, as shown in FIG. 1.

The master camera module 11 captures a master camera image. Similarly, the slave camera module 12 captures a slave camera image. The master camera image and the slave camera image are used for binocular stereo viewing. The master camera image and the slave camera image may be a color image such as an RGB image, or a monochrome image.

The image sensor 11b (12b) sends image data in RAW pixel data units to the ISP 30. A RAW pixel data unit is 12 bits, for example. A plurality of RAW pixel data for each pixel of the image sensor 11b (12b) constitute a regular RAW image.

The

image sensors

11b, 12b have a circuit which performs bit operations on the RAW pixel data. The

image sensors

11b, 12b may also have another circuit to generate a random pattern. The

image sensors

11b, 12b send the bit operated RAW pixel data to the image signal processor 30.

The range sensor module 20 captures a depth image. Specifically, the range sensor module 20 acquires time of flight (ToF) data by emitting pulsed light toward a subject and detecting light reflected from the subject. The ToF data indicates an actual distance between the electronic device 100 and the subject.

The image signal processor 30 controls the master camera module 11, the slave camera module 12, and the range sensor module 20 to capture image and ToF data. The image signal processor 30 performs image processing such as bokeh processing based on the RAW pixel data received from the

image sensors

11b, 12b.

As shown in FIG. 1, the electronic device 100 includes a global navigation satellite system (GNSS) module 40, a wireless communication module 41, a CODEC 42, a speaker 43, a microphone 44, a display module 45, an input module 46, an inertial measurement unit (IMU) 47, a main processor 48, and a memory 49.

The GNSS module 40 measures a current position of the electronic device 100. The wireless communication module 41 performs wireless communications with the Internet. The CODEC 42 bi-directionally performs encoding and decoding, using a predetermined encoding/decoding method. The speaker 43 outputs a sound in accordance with sound data decoded by the CODEC 42. The microphone 44 outputs sound data to the CODEC 42 based on inputted sound.

The display module 45 displays various information such as a camera image so that the user can check it in real time. The display module 45 can display the image with bokeh after performing image processing.

The input module 46 inputs information via a user’s operation. For example, the input module 46 inputs an instruction to capture and store an image displayed on the display module 45.

An IMU 47 detects the angular velocity and the acceleration of the electronic device 100. A posture of the electronic device 100 can be grasped by a measurement result of the IMU 47. The display module 45 may display an image according to the posture of the electronic device 100.

The main processor 48 controls the global navigation satellite system (GNSS) module 40, the wireless communication module 41, the CODEC 42, the speaker 43, the microphone 44, the display module 45, the input module 46, and the IMU 47.

The memory 49 stores data of image captured by the camera module 10, data of depth image captured by the range sensor module 20, and a program which runs on the image signal processor 30 and/or the main processor 48.

Regarding the configuration of the electronic device 100, one of the master camera module 11 and the slave camera module 12 may be omitted. The range sensor module 20 may also be omitted.

A method of embedding additional data onto RAW pixel data will be described in detail. FIG. 2 is a flowchart which illustrates the method according to an embodiment of the present disclosure.

In the step S1, the image sensor 11b (12b) acquires RAW pixel data. As shown in FIG. 3, in this example, a size of the RAW pixel data is 12 bits and a unit of RAW image processed by the ISP 30 is 16 bits. Of course, the size of the RAW pixel data and the size of the unit processed by the ISP 30 are not limited to this example.

In the step S2, the image sensor 11b (12b) shifts the RAW pixel data to the MSB (Most Significant Bit) side by a predetermined bit amount to create a bit container, as shown in FIG. 3. In this example, the image sensor 11b (12b) shifts the RAW pixel data to the MSB side by a number of bits of the random data generated in the step S4. In the example, the RAW pixel data is shifted to the MSB side by 4 bits so that the MSB of the RAW pixel data is located on the MSB of the unit processed by the ISP 30. As a result, 4 bits on the LSB (Least Significant Bit) side become free bits to store additional data (i.e., the bit container) .

Please note that the step S2 can be performed at any timing before the step S5.

In the step S3, the image sensor 11b (12b) acquires additional data. For example, the additional data is depth data obtained from the range sensor module 20, or exposure time of HDR (High Dynamic Range) pixel in the image sensor 11b (12b) . A size of the additional data is less than a size of the RAW pixel data. In this example, the size of the additional data is 4 bits.

In the step S4, the image sensor 11b (12b) masks the additional data with a random pattern to generate random data. A size of the random pattern is the same as that of the additional data. Specifically, as shown in FIG. 4, the image sensor 11b (12b) performs an XOR operation on the additional data and the random pattern. For example, performing the XOR operation between the additional data “0010” and the random pattern mask “1110” produces the data to be embedded (random data) “1100” .

The random pattern is generated by the image sensor 11b (12b) , but it may be generated by another circuit or processing unit.

It is preferable that the random pattern is different for each pixel so that the effect of data randomization can be securely ensured. The random pattern may be a bit sequence generated based on a location of the pixel in the image sensor. Alternatively, the random pattern may be a bit sequence generated based on a value of the pixel. Generating a random pattern based on a location or a value of the pixel can give an appropriate random distribution between pixels of the image sensor.

The bit operation in the step S4 is not limited to the XOR operation, any restorable operation can be used.

In the step S5, the image sensor 11b (12b) embeds the random data on the LSB side of the RAW pixel data as shown in FIG. 5. That is to say, the random data is embedded in the bit container created in the step S2.

Regarding the above steps, a processor other than the circuit of the image sensor (e.g., the image signal processor 30, the main processor 48) may perform at least one of the steps S1 to S5.

The RAW pixel data with embedded data can be obtained by implementing the above steps. The RAW pixel data with embedded data is sent to the image signal processor 30 which performs image processing on the RAW pixel data with the random data embedded as it is. That is to say, the ISP 30 treats the embedded data as noise.

It is not necessary for the ISP 30 to perform any additional data processing for extracting the additional data from the RAW pixel data with embedded data because the additional data is randomized and embedded in the LSB side of processing bits of the ISP 30 and thus it can be treated as noise.

According to the present disclosure, the image signal processor 30 can perform image processing on the RAW pixel data with embedded data even if the electronic device 100 does not perform any additional steps or use any additional resources.

A method of restoring the additional data embedded onto the RAW pixel data will be described with reference to FIG. 7.

First, the main processor 48 extracts the random data from the RAW pixel data with embedded data. As shown in FIG. 7, the random data “1100” is extracted. After that, the main processor 48 masks the extracted random data with the random pattern which is used in the step S4. Specifically, the main processor 48 performs an XOR operation on the random data “1100” and the random pattern “1110” to obtain the additional data “0010” . The bit operation is not limited to the XOR operation, any restorable operation can be used.

When the additional data is depth data, the ISP 30 or the main processor 48 may perform bokeh processing based on the RAW pixel data with embedded data and the restored additional data.

In the above description, the number of bits for storing additional data (i.e., size of the bit container) is 4 bits. The number can be increased when a pixel value is high and some bits on the LSB side can be rounded in an output image such as JPEG image (8-bit pixel data) .

As is clear from the Gamma curve shown in FIG. 8, a slope of the Gamma curve is well below 1 in the bright range. As a result, a pixel value is rounded in the bright range after converting the RAW image to a small size format such as the JPEG. Some bits on the LSB side can be ignored because of being rounded in the output image in the final format (e.g., JPEG format) .

The size of the bit container can also be increased when a captured image is dark or the ISO sensitivity is high and thus the image contains a lot of noise. Some bits on the LSB side of the RAW pixel data can be ignored because data stored in these bits are buried under noise as shown in FIG. 9.

In the cases mentioned above, some bits on the LSB side of the RAW pixel data can be used as a bit container to store the additional data. In the example shown in FIG. 10, a size of the additional data is 6 bits. The data to be embedded onto RAW pixel data (12 bits) is generated in the same way as described above. Specifically, the data to be embedded “101100” is generated by performing the XOR operation between the additional data “000010” and the random pattern mask “101110” .

The generated random data is embedded onto the shifted RAW pixel data as shown in FIG. 11. In this example, 2 bits on the LSB side are overwritten by the data to be embedded since a size of the additional data (6 bits) is larger than a difference between a size of a unit processed by the ISP 30 (16 bits) and a size of the RAW pixel data (12 bits) . At least one bit of the RAW pixel data is overwritten by the random data. However, as mentioned, the overwritten bits are not important and thus the ISP 30 can perform regular/usual image processing on the RAW pixel data with embedded data.

Optionally, the noise data may be erased before embedding the random data onto the shifted RAW pixel data.

Finally, the modification example will be described with reference to FIG. 12. FIG. 12 shows a modification example in which the RAW pixel data is not shifted.

In this example, a size of the RAW pixel data is 16 bits which is the same size of a unit processed by the ISP 30. A size of additional data is 4 bits. The RAW pixel data is not shifted, and the bits for the number of the additional data (i.e., 4 bits) on the LSB side of the RAW pixel data are overwritten.

As described above, according to the present disclosure, the additional data is randomized and embedded in the bit container at the LSB side of the RAW pixel data and thus it can be treated as noise. Therefore, a processor can perform image processing on the RAW pixel data with embedded data as it is. Specifically, the ISP 30 does not need any additional resources such as software or hardware and any additional data processing such as signal transferring in order to perform image processing on an image with the additional data embedded.

In other words, the image data generated by the method according to the present disclosure can be handled with the regular/usual image processing framework although it contains the additional data. The method does not require any additional special or complicated calculation for image processing and thus it does not increase the processing load. Therefore, the method is excellent in real-time image processing.

Further, according to the present disclosure, the additional data can be extracted from the RAW pixel data and handled to improve picture quality (e.g., bokeh processing)

Still further, according to the present disclosure, unimportant bits of the RAW pixel data can be effectively used as a bit container to contain additional data.

In the description of embodiments of the present disclosure, it is to be understood that terms such as "central" , "longitudinal" , "transverse" , "length" , "width" , "thickness" , "upper" , "lower" , "front" , "rear" , "back" , "left" , "right" , "vertical" , "horizontal" , "top" , "bottom" , "inner" , "outer" , "clockwise" and "counterclockwise" should be construed to refer to the orientation or the position as described or as shown in the drawings in discussion. These relative terms are only used to simplify the description of the present disclosure, and do not indicate or imply that the device or element referred to must have a particular orientation, or must be constructed or operated in a particular orientation. Thus, these terms cannot be constructed to limit the present disclosure.

In addition, terms such as "first" and "second" are used herein for purposes of description and are not intended to indicate or imply relative importance or significance or to imply the number of indicated technical features. Thus, a feature defined as "first" and "second" may comprise one or more of this feature. In the description of the present disclosure, "a plurality of" means “two or more than two” , unless otherwise specified.

In the description of embodiments of the present disclosure, unless specified or limited otherwise, the terms "mounted" , "connected" , "coupled" and the like are used broadly, and may be, for example, fixed connections, detachable connections, or integral connections; may also be mechanical or electrical connections; may also be direct connections or indirect connections via intervening structures; may also be inner communications of two elements which can be understood by those skilled in the art according to specific situations.

In the embodiments of the present disclosure, unless specified or limited otherwise, a structure in which a first feature is "on" or "below" a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are in contact via an additional feature formed therebetween. Furthermore, a first feature "on" , "above" or "on top of" a second feature may include an embodiment in which the first feature is orthogonally or obliquely "on" , "above" or "on top of" the second feature, or just means that the first feature is at a height higher than that of the second feature; while a first feature "below" , "under" or "on bottom of" a second feature may include an embodiment in which the first feature is orthogonally or obliquely "below" , "under" or "on bottom of" the second feature, or just means that the first feature is at a height lower than that of the second feature.

Various embodiments and examples are provided in the above description to implement different structures of the present disclosure. In order to simplify the present disclosure, certain elements and settings are described in the above. However, these elements and settings are only by way of example and are not intended to limit the present disclosure. In addition, reference numbers and/or reference letters may be repeated in different examples in the present disclosure. This repetition is for the purpose of simplification and clarity and does not refer to relations between different embodiments and/or settings. Furthermore, examples of different processes and materials are provided in the present disclosure. However, it would be appreciated by those skilled in the art that other processes and/or materials may also be applied.

Reference throughout this specification to "an embodiment" , "some embodiments" , "an exemplary embodiment" , "an example" , "a specific example" or "some examples" means that a particular feature, structure, material, or characteristics described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Thus, the appearances of the above phrases throughout this specification are not necessarily referring to the same embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

Any process or method described in a flow chart or described herein in other ways may be understood to include one or more modules, segments or portions of codes of executable instructions for achieving specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, in which it should be understood by those skilled in the art that functions may be implemented in a sequence other than the sequences shown or discussed, including in a substantially identical sequence or in an opposite sequence.

The logic and/or step described in other manners herein or shown in the flow chart, for example, a particular sequence table of executable instructions for realizing the logical function, may be specifically achieved in any computer readable medium to be used by the instructions execution system, device or equipment (such as a system based on computers, a system comprising processors or other systems capable of obtaining instructions from the instructions execution system, device and equipment executing the instructions) , or to be used in combination with the instructions execution system, device and equipment. As to the specification, "the computer readable medium" may be any device adaptive for including, storing, communicating, propagating or transferring programs to be used by or in combination with the instruction execution system, device or equipment. More specific examples of the computer readable medium comprise but are not limited to: an electronic connection (an electronic device) with one or more wires, a portable computer enclosure (a magnetic device) , a random access memory (RAM) , a read only memory (ROM) , an erasable programmable read-only memory (EPROM or a flash memory) , an optical fiber device and a portable compact disk read-only memory (CDROM) . In addition, the computer readable medium may even be a paper or other appropriate medium capable of printing programs thereon, this is because, for example, the paper or other appropriate medium may be optically scanned and then edited, decrypted or processed with other appropriate methods when necessary to obtain the programs in an electric manner, and then the programs may be stored in the computer memories.

It should be understood that each part of the present disclosure may be realized by the hardware, software, firmware or their combination. In the above embodiments, a plurality of steps or methods may be realized by the software or firmware stored in the memory and executed by the appropriate instructions execution system. For example, if it is realized by the hardware, likewise in another embodiment, the steps or methods may be realized by one or a combination of the following techniques known in the art: a discrete logic circuit having a logic gate circuit for realizing a logic function of a data signal, an application-specific integrated circuit having an appropriate combination logic gate circuit, a programmable gate array (PGA) , a field programmable gate array (FPGA) , etc.

Those skilled in the art shall understand that all or parts of the steps in the above exemplifying method of the present disclosure may be achieved by commanding the related hardware with programs. The programs may be stored in a computer readable storage medium, and the programs comprise one or a combination of the steps in the method embodiments of the present disclosure when run on a computer.

In addition, each function cell of the embodiments of the present disclosure may be integrated in a processing module, or these cells may be separate physical existence, or two or more cells are integrated in a processing module. The integrated module may be realized in a form of hardware or in a form of software function modules. When the integrated module is realized in a form of software function module and is sold or used as a standalone product, the integrated module may be stored in a computer readable storage medium.

The storage medium mentioned above may be read-only memories, magnetic disks, CD, etc.

Although embodiments of the present disclosure have been shown and described, it would be appreciated by those skilled in the art that the embodiments are explanatory and cannot be construed to limit the present disclosure, and changes, modifications, alternatives and variations can be made in the embodiments without departing from the scope of the present disclosure.

Claims

A method of embedding additional data onto RAW pixel data read out from a pixel of an image sensor, comprising:

acquiring additional data, a size of the additional data being less than a size of the RAW pixel data;

masking the additional data with a random pattern to generate random data; and

embedding the random data on the LSB side of the RAW pixel data.
The method of claim 1, further comprising, before embedding the random data on the LSB side of the RAW pixel data, shifting the RAW pixel data to the MSB side to create a bit container,

wherein the random data is embedded in the bit container.
The method of claim 2, wherein a size of the bit container is the same as a size of the random data.
The method of claim 1 or 2, wherein at least one bit of the RAW pixel data is overwritten by the random data.
The method of any one of claims 1 to 4, wherein the masking the additional data with a random pattern to generate random data comprises performing an XOR operation on the additional data and the random pattern.
The method of any one of claims 1 to 5, wherein the random pattern is different for each pixel of the image sensor.
The method of claim 6, wherein the random pattern is a bit sequence generated based on a location of the pixel in the image sensor.
The method of claim 6, wherein the random pattern is a bit sequence generated based on a value of the pixel.
The method of any one of claims 1 to 8, wherein the additional data is depth data or exposure time.
The method of any one of claims 1 to 9, further comprising performing image processing on the RAW pixel data with the random data embedded by treating the random data as noise.
A method of obtaining the additional data embedded to the RAW pixel data according to the method of any one of claims 1 to 10, comprising:

extracting the random data from the RAW pixel data; and

masking the random data with the random pattern to restore the additional data.
The method of claim 11, wherein the masking the random data with the random pattern to restore the additional data comprises performing an XOR operation on the random data and the random pattern.
An electronic device for image processing, comprising a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 12.
A non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program is executed by a computer to implement the method according to any of claims 1 to 12.