WO2020047860A1

WO2020047860A1 - Electronic device and image processing method

Info

Publication number: WO2020047860A1
Application number: PCT/CN2018/104682
Authority: WO
Inventors: Hajime Numata
Original assignee: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date: 2018-09-07
Filing date: 2018-09-07
Publication date: 2020-03-12
Also published as: CN112567723A; CN112567723B

Abstract

An electronic device includes: a light-transmissive display provided on a front surface of the electronic device; a first camera provided on the front surface and outside a display screen of the display, and configured to capture a first image in the visible spectrum; a filter covering at least a part of a back surface of the display and configured to cut light in the visible spectrum and pass light outside the visible spectrum; a second camera provided behind the filter, and configured to capture a second image with light outside the visible spectrum passed through the display and the filter; and a processor configured to process the first image and second image.

Description

ELECTRONIC DEVICE AND IMAGE PROCESSING METHOD

FIELD

This application relates generally to an electronic device and an image processing method.

BACKGROUND

Recently, multiple sensors are incorporated in a mobile terminal with introduction of high functionality and multi-functions. For example, a mobile terminal with two front-facing cameras provided on a top part of a front surface of the terminal is known, which enables high-quality image processing for selfies. On the other hand, in arranging sensors in the mobile terminal, an area where front-facing cameras are provided is preferably made as narrow as possible in order to maximize a display area of the mobile terminal. One method for maximizing the display area of a mobile terminal with front-facing cameras is to provide the front-facing cameras on a back of a transparent display of the mobile terminal. However, when both of the display and the cameras are active at the same time, images captured by the cameras are damaged due to light emitted by the display.

The present disclosure has been created in view of the foregoing circumstances, and an objective of the disclosure is to provide an electronic device that includes a plurality of cameras provided on a front surface of the electronic device and can secure a large display area.

SUMMARY

To achieve the above-described objective, an electronic device according to a first aspect of the present disclosure includes: a light-transmissive display provided on a front surface of the electronic device; a first camera provided on the front surface and outside a display screen of the display, and configured to capture a first image in the visible spectrum; a filter covering at least a part of a back surface of the display and configured to cut light in the visible spectrum and pass light outside the visible spectrum; a second camera provided behind the filter, and configured to capture a second image with light outside the visible spectrum passed through the display and the filter; and a processor configured to process the first image and second image..

In the electronic device of the aforementioned aspect, the electronic device may further include a mirror provided on the back surface of the display to reflect light in the visible spectrum and including an aperture, wherein the aperture is provided in front of the filter and configured to pass light both in and outside the visible spectrum; the second camera configured to capture the second image with light outside the visible spectrum passed through the display, the aperture, and the filter.

In the electronic device of the aforementioned aspect, the filter may cover the back surface of the display to reflect light in the visible spectrum emitted from the display.

In the electronic device of the aforementioned aspect, the first camera may be configured to capture an image of a face as the first image, the second camera may be configured to capture an image of the face as the second image, and the processor may be configured to correct a gaze of the face in the first image based on a gaze of the face in the second image.

In the electronic device of the aforementioned aspect, the processor may be configured to reduce noise in the first image based on the second image.

In the electronic device of the aforementioned aspect, the first camera may be configured to capture an image of a face as the first image, the second camera may be configured to capture an image of the face in the infrared spectrum as the second image, and the processor may be configured to reduce at least one of stains, speckles and wrinkles on the face in the first image based on the face in the second image.

In the electronic device of the aforementioned aspect, the processor may be configured to estimate depth information based on a parallax between the first image and the second image, and to process the first image based on the depth information.

To achieve the above-described objective, an image processing method according to a second aspect of the present disclosure is an image processing method for an electronic device including a light-transmissive display provided on a front surface of the electronic device, a first camera provided on the front surface and outside a display screen of the display, a filter covering at least a part of a back surface of the display and configured to cut light in the visible spectrum and pass light outside the visible spectrum, and a second camera provided behind the filter. The method includes: capturing a first image in the visible spectrum by the first camera; capturing a second image with light outside the visible spectrum passed through the display and the filter by the second camera; and processing the first image and second image.

According to the present disclosure, the electronic device can include a plurality of cameras provided on a front surface of the electronic device and secure a large display area.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained when the following detailed description is considered in conjunction with the following drawings, in which:

FIG. 1 is an illustration showing a front view of the electronic device according to an embodiment of the present disclosure;

FIG. 2 is a schematic cross-sectional view illustrating the electronic device taken along line A-A of FIG. 1;

FIGS. 3 is a schematic block diagram illustrating a hardware configuration of the electronic device;

FIG. 4A is an example of a first image;

FIG. 4B is an example of a second image;

FIG. 5 is an illustration for explaining an example of a method of correcting gaze based on the first image and the second image;

FIG. 6 is a flowchart of the image processing procedure according to the embodiment;

FIG. 7 is a schematic cross-sectional view illustrating an electronic device according to another embodiment of the present disclosure;

FIG. 8 is an illustration showing a front view of the electronic device according to another embodiment of the present disclosure; and

FIG. 9 is a schematic cross-sectional view illustrating the electronic device taken along line B-B of FIG. 8.

DETAILED DESCRIPTION

An embodiment of the present disclosure is described below with reference to the drawings.

FIG. 1 is a front view illustrating an electronic device 1 according to an embodiment of the present disclosure. FIG. 2 is a schematic cross-sectional view illustrating the electronic device 1 taken along line A-A of FIG. 1. In this embodiment, the electronic device 1 is described as a smartphone. However, the electronic device of the present disclosure is not limited to a smartphone. The electronic device of the present disclosure may be an electronic device such as a mobile phone, a laptop, a tablet, or a camera. As illustrated in FIGS. 1 and 2, the electronic device 1 includes a housing 10, a display 20, a mirror 30, an IR pass filter 40, a first camera 50, and a second camera 60.

The housing 10 has a substantially rectangular outer shape and accommodates electronic components of the electronic device 1.

The display 20 is provided on a front surface of the housing 10 and display various information. In this embodiment, the display 20 is a light-transmissive display so that light from the outside both in and outside the visible spectrum is transmitted to through the display 20. For example, the display 20 may include a Transparent Organic Light Emitting Diode (TOLED) display, a transparent liquid crystal display (LCD) , and the like. The display 20 includes a display screen 21 for display of information. The display screen 21 includes a notch 22 formed on an upper edge of the display screen 21 in a portrait orientation of the electronic device 1.

The mirror 30 is provided on the back surface of the display 20. The mirror 30 reflects the visible light emitted by the display 20 to maintain good visibility thereof. In this embodiment, the mirror 30 includes an aperture 31 positioned nearly at the center of the display screen 21 as viewed from the front.

The IR pass filter 40 is an infrared pass filter to cut light in the visible spectrum and pass light in the infrared spectrum. The IR pass filter 40 may include a cold mirror. The IR pass filter 40 is provided on the back surface of the mirror 30 to cover an area of the back surface of the display 20 exposed by the aperture 31. The IR pass filter 40 cuts the visible light, and transmits the infrared light from the outside which passes through the display 20 and the aperture 31. For example, the IR pass filter 40 reflects the visible light emitted by the display 20 to maintain good visibility.

The first camera 50 is a so-called front-facing camera, provided in the notch 22 on the front surface of the electronic device 1 to capture a first image in the visible spectrum. The first camera 50 includes a lens and an image sensor such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor with sensitivity in the visible spectrum. In this embodiment, when the user holds and operates the electronic device 1, the first camera 50 captures a color image of a user's face in the field of view thereof as the first image and outputs the first image to the processor for gaze correction as described later. The first image may be a still image or a moving image.

The second camera 60 is provided behind the IR pass filter 40 to capture a second image made with the light in the infrared spectrum passed through the display 20, the aperture 31 and the IR pass filter 40. The second camera 60 includes a lens and an image sensor such as a CCD image sensor or a CMOS image sensor with sensitivity in the infrared spectrum. In this embodiment, the second camera 50 captures an infrared image of the user's face in the field of view of the second camera 50 as the second image when the user holds and operates the electronic device 1, and the second camera 50 outputs the captured image to the processor for gaze correction as described later. The second image may be a still image or a moving image.

FIG. 3 is a schematic diagram showing the hardware configuration of the electronic device 1. As shown in FIG. 3, the electronic device 1 includes a processor 101, a memory 102, a communicator 103, an inputter 104, an outputter 105, and an imager 106. These elements are connected by a bus 107.

The processor 101 includes a central processing unit (CPU) to execute an overall control of the electronic device1. The memory 102 stores an operating system, various programs, temporary data generated and used during execution of various programs, as well as other information. The memory 102 may include a read only memory (ROM) , a random access memory (RAM) or any other volatile or non-volatile memories. The processor 101 executes programs stored in the memory 102 (for example, a program regarding an image processing procedure described later) to realize the functions of a camera controller 111 and an image processor 112 described later.

The communicator 103 includes a communication interface for communicably connecting an external device to the electronic device 1. The communicator may include a communication module for connecting the electronic device 1 to one or more wireless networks.

The inputter 104 includes an input device such as buttons and a touchpad. The inputter 104 receives operation input from the user and outputs to the processor 101 a signal corresponding to the received operation input. For example, the display 20 and a touchpad may be stacked on one another other such that the display 20 can be configured to function as a touch screen so as to allow a user to perform touch-type input of information. The inputter 104 may also include a microphone for voice input.

The outputter 105 includes the display 20. The outputter 105 may also include a speaker to output voice and sound. The imager 106 includes the first camera 50 and the second camera 60.

The functional configuration of the processor 101 of the electronic device 1 is described next. As shown in FIG. 3, the processor 101 functions as a camera controller 111 and an image processor 112.

The camera controller 111 controls the first camera 50 and the second camera 60 to acquire the first image and the second image, respectively. Specifically, the camera controller 111 transmits to the first camera 50 an instruction to capture the first image and transmits to the second camera 60 an instruction to capture the second image. The camera controller 111 then receives the first image from the first camera 50 and receives the second image from the second camera 60. The camera controller may receive a still image or a moving image from the first camera 50 and the second camera 60.

The image processor 112 processes the first image and second image. In this embodiment, the image processor 112 corrects a gaze of the user in the first image based on a gaze of the user in the second image. FIG. 4A is an example of the first image and FIG. 4B is an example of the second image according to this embodiment. As shown in FIG. 4A, the user 2 in the first image 200 is looking down because the user 2 is usually looking at the display 20 when taking a selfie or using a video chat, and the first camera 50 is positioned at the top of the display 20. On the other hand, as shown in FIG. 4B, the user 2 in the second image 300 is looking at the center position because the second camera 60 is provided behind the display. Therefore, the image processor 112 processes the first image 200 and the second image 300 so that the gaze of the user 2 in the first image 200 is the same as the gaze of the user 2 in the second image 300.

An example of a method of correcting gaze based on the first image and the second image is described with reference to FIG. 5. In this example, the first camera 50 captures a first image 400, and the second camera 60 captures a second image 500. First, the image processor 112 extracts luminance information 401 and chroma information 402 from the first image 400. For example, the chroma information 402 includes blue-difference chroma component Cb and red-difference chroma component Cr. Then, the image processor 112 applies a low pass filter to the luminance information 401 to obtain global intensity information 403. The image processor 112 also applies a low pass filter to the chroma information 402 to obtain filtered chroma information 404. Furthermore, the image processor 112 extracts texture information 501 from the second image 500. The image processor 112 combines the texture information 501 with the global intensity information 403 to obtain a final luminance image 601. Finally, the image processor 112 combines the filtered chroma information 403 with the final luminance image 601 to obtain a gaze-corrected image 600.

The image processing procedure executed by the processor 101 of the electronic device 1 according to this embodiment is described next. FIG. 6 is a flowchart of the image processing procedure according to this embodiment. For example, triggered by reception of operation input giving an order via the inputter 104 to start this procedure, this image processing procedure starts.

First, the processor 101 transmits to the first camera 50 an instruction to capture the first image and transmits to the second camera 60 an instruction to capture the second image (step S101) .

Then, the processor 101 receives the first image from the first camera 50 and the second image from the second camera 60 (step S102) .

The processor 101 processes the first image and the second image to obtain the gaze-corrected image (step S103) . Finally, the processor 101 outputs the gaze-corrected image to the display (step S104) , and ends this procedure. Although the procedure explained above is for displaying a static image, the procedure can be applied to a moving image by repeating the steps S101 to S104.

As described above, the electronic device 1 according to this embodiment includes the first camera 50 for capturing the first image of a user and the second camera 60 for capturing the second image of the user. Since the second camera 60 is provided behind the display 20 and only the first camera 50 is provided on the front surface of the electronic device 1, the electronic device 1 can secure larger display area comparing to an electronic device which includes front-facing cameras at an upper edge of a display screen of the electronic device.

Furthermore, since the user is generally looking at the display 20 when holding and operating the electronic device 1, the second camera 60 can capture the second image in which the user is looking forward. Thus, the processor 101 can process the first image and the second image to obtain a gaze-corrected image based on the gaze of the user in the second image, thereby enabling the electronic device 1 to provide a better user experience. For example, when a user is making a video call to a recipient on the electronic device 1, the recipient feels like the user is looking at the recipient and that the user and recipient have eye contact.

Moreover, the second camera 60 captures the second image in the infrared spectrum. Thus, the electronic device 1 can prevent light emitted by the display 20 from damaging the second image.

The above embodiment is described by way of example only, and the specific configuration of the electronic device 1 and the image processing procedure of the processor 101 are not confined to those described in the above embodiment and can be modified as follows.

In the above-described embodiment, the IR pass filter 40 is provided behind the mirror 30 to cover an area of the back surface of the display 20 exposed by the aperture 31 as shown in FIG. 2. However, the area where the IR pass filter 40 is provided is not confined to the above-mentioned area. The IR pass filter 40 may cover the whole back surface of the display 20. For example, when visible light reflectivity of the IR pass filter 40 is sufficient for display quality, the IR pass filter 40, instead of the mirror 30, may be provided on the back surface of the display 20 to reflect the visible light. FIG. 7 is a schematic cross-sectional view illustrating an electronic device 1A according to another embodiment of the present disclosure. As shown in FIG. 7, the IR pass filter 40A, instead of the mirror 30 shown in FIG. 2, covers the back surface of the display 20 to reflect light emitted from the display 20. Thus, the electronic device 1A without the mirror 30 can prevent light emitted by the display 20 from damaging the second image.

In the above-described embodiment, the electronic device 1 includes the second camera 60 provided behind the display 20 for capturing an infrared image as a second image. However, the second camera 60 is not confined to an infrared camera. The second camera 60 may include a camera for capturing the second image outside the visible spectrum, and a filter may be provided in front of the camera to cut light in the visible spectrum and pass light outside the visible spectrum. For example, the second camera 60 may include an ultraviolet (UV) camera for capturing the second image in the UV spectrum, and a UV pass filter may be provided in front of the UV camera instead of the IR pass filter 40 to cut light in the visible spectrum and pass light in the UV spectrum. In this case, the electronic device 1 can provide a UV image of a user showing a sun-damaged area of skin of the user.

In the above-described embodiment, the processor 101 processes the first image and the second image to obtain the gaze-corrected image by the image processing procedure as shown FIG. 6. However, the image processing procedure and the output image are not confined to the above-mentioned procedure and output image. For example, the processor 101 may reduce noise in the first image based on the second image. When the first camera 50 captures the first image in a low-light scene, the first image is likely to include much digital noise. In such a case, the processor 101 can reduce noise in the first image by combining the first image with the second image that is captured in the infrared spectrum and includes less noise. Additionally, the processor may reduce at least one of stains, speckles and wrinkles on the face in the first image based on the face in the second image. Since the depth of light penetration in the epidermis is proportional to the wavelength of the incident light, the number of stains, speckles and wrinkles on the face in the second image captured in the infrared spectrum is less than the number of stains, speckles and wrinkles on the face in the first image captured in the visible spectrum. Thus, the processor can reduce such an unpreferable part on the face in the first image by combining the first image with the second image.

Furthermore, the processor may process the first image and the second image to add blur effects or background blur to the first image by using the first camera 50 and the second camera 60 as a stereo camera. For example, the processor estimates depth information based on a parallax between the first image and the second image, and processes the first image based on the depth information to obtain a background-blurred image. In particular, because the distance between the first camera 50 and the second camera is farther than the inter-camera distance of a stereo camera provided on an upper edge of a display screen of an electronic device, the processor 101 according to the present disclosure can obtain further depth information with more accuracy than the depth information obtained by the stereo camera.

In the above-described embodiment, the electronic device 1 includes the second camera 60 provided behind the display 20. However, sensors provided behind the display 20 are not confined to the second camera 60. For example, when the IR pass filter is provided on the back surface of the display 20, IR related devices can be provided behind the IR pass filter, such as a structured light pattern projector and a proximity sensor. FIG. 8 is a front view illustrating an electronic device 1B according to another embodiment of the present disclosure and FIG. 9 is a schematic cross-sectional view illustrating the electronic device 1B taken along line B-B of FIG. 8. As shown in FIGS. 8 and 9, a structured light pattern projector 70 and a proximity sensor 80 are provided behind the IR pass filter 40 and emit light in the infrared spectrum. Furthermore, the number of the IR related devices provided behind the IR pass filter 40 is not confined as shown in FIGS. 8 and 9, one IR related device or more than one IR related device can be provided behind the IR pass filter 40. For example, either of a structured light pattern projector and a proximity sensor may be provided behind the IR pass filter 40. Thus, the electronic device 1 can secure a larger display area comparing to an electronic device that includes such sensors outside a display screen of the electronic device.

The foregoing describes some example embodiments for explanatory purposes. Although the foregoing discussion has presented specific embodiments, persons skilled in the art will recognize that changes may be made in form and detail without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. This detailed description, therefore, is not to be taken in a limiting sense, and the scope of the invention is defined only by the included claims, along with the full range of equivalents to which such claims are entitled.

Claims

An electronic device comprising:

a light-transmissive display provided on a front surface of the electronic device;

a first camera provided on the front surface and outside a display screen of the display, and configured to capture a first image in the visible spectrum;

a filter covering at least a part of a back surface of the display and configured to cut light in the visible spectrum and pass light outside the visible spectrum;

a second camera provided behind the filter, and configured to capture a second image with light outside the visible spectrum passed through the display and the filter; and

a processor configured to process the first image and second image.
The electronic device according to claim 1, further comprising

a mirror provided on the back surface of the display to reflect light in the visible spectrum and including an aperture, wherein

the aperture is provided in front of the filter and configured to pass light both in and outside the visible spectrum;

the second camera configured to capture the second image with light outside the visible spectrum passed through the display, the aperture, and the filter.
The electronic device according to claim1, wherein

the filter covers the back surface of the display to reflect light in the visible spectrum emitted from the display.
The electronic device according to claim 1, wherein

the first camera is configured to capture an image of a face as the first image,

the second camera is configured to capture an image of the face as the second image; and

the processor is configured to correct a gaze of the face in the first image based on a gaze of the face in the second image.
The electronic device according to claim 1, wherein

the processor is configured to reduce noise in the first image based on the second image.
The electronic device according to claim 1, wherein

the first camera is configured to capture an image of a face as the first image,

the second camera is configured to capture an image of the face in the infrared spectrum as the second image; and

the processor is configured to reduce at least one of stains, speckles and wrinkles on the face in the first image based on the face in the second image.
The electronic device according to claim 1, wherein

the processor is configured to estimate depth information based on a parallax between the first image and the second image, and to process the first image based on the depth information.
The electronic device according to claim 1, further comprising

at least one IR related device provided behind the filter and configured to emit light in the infrared spectrum.
The electronic device according to claim 8, wherein

the at least one IR related device includes at least either of a structured light pattern projector and a proximity sensor.
An image processing method for an electronic device including a light-transmissive display provided on a front surface of the electronic device, a first camera provided on the front surface and outside a display screen of the display, a filter covering at least a part of a back surface of the display and configured to cut light in the visible spectrum and pass light outside the visible spectrum, and a second camera provided behind the filter, the method comprising:

capturing a first image in the visible spectrum by the first camera;

capturing a second image with light outside the visible spectrum passed through the display and the filter by the second camera; and

processing the first image and second image.