WO2020237441A1

WO2020237441A1 - User equipment and method of oblique view correction

Info

Publication number: WO2020237441A1
Application number: PCT/CN2019/088417
Authority: WO
Inventors: Hirotake Cho
Original assignee: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date: 2019-05-24
Filing date: 2019-05-24
Publication date: 2020-12-03
Also published as: CN113826376A; JP2022533975A; JP7346594B2; CN113826376B

Abstract

A user equipment (UE) and a method of oblique view correction are provided. The method of oblique view correction includes capturing a color image, an infrared image, and a depth image by an image sensing module, estimating plane parameters from the depth image, calculating focal distance data from the depth image, capturing partially focused images at these focal distances from the focal distance data by the image sensing module, and cutting focused image data from the partially focused images and composing these focused image data to form a wholly focused image..

Description

USER EQUIPMENT AND METHOD OF OBLIQUE VIEW CORRECTION

BACKGROUND OF DISCLOSURE

1. Field of Disclosure

The present disclosure relates to the field of image processing technologies, and more particularly, to a user equipment (UE) , and a method of oblique view correction.

2. Description of Related Art

Please refer to FIG. 1; in current technology, if a user takes a picture of an oblique oriented plane surface having a clear contour using a user equipment 1, such as a cell phone, the user equipment 1 will correct an image of the oblique oriented plane surface to a shape without distortion. However, if the oblique oriented plane surface has an unclear contour, or only a part is in a shooting area 2, or the oblique oriented planer surface has a horizontal width greater than a width of the shooting area 2, the user cannot obtain a single focused image of a whole target without perspective distortion.

US patent no. 6449004B1 discloses an electronic camera with oblique view correction. It discloses an electronic camera with an image pickup device for photoelectrically picking up a light image of an object to generate image data. An oblique angle information provider is configured for providing information on an oblique angle between a sensing surface of the image pickup device and a surface of the object. A distance detector is configured for detecting a distance to the object. A corrector is configured for correcting, based on the provided oblique angle information, and a detected distance, to generate image data, so as to produce a pseudo object image whose surface resides on a plane parallel with the sensing surface of the image pickup device.

US patent no. US7365301B2 discloses a three-dimensional shape detecting device, an image capturing device, and a three-dimensional shape detecting program. It discloses a three-dimensional shape detecting device comprising projection means which projects pattern light, image capturing means which captures a pattern light projection image of a subject on which the pattern light is projected, and a three-dimensional shape calculation means which calculates a three-dimensional shape of the subject based on a locus of the pattern light extracted from the pattern light projection image.

US patent application no. US7711259B2 discloses a method and an apparatus for increasing depth of field for an imager. It discloses that the imager captures a plurality of images at respective different focus positions, and combines the images into one image and sharpens the one image. In an alternative exemplary embodiment, a single image is captured while the focus positions change during image capture, and the resulting image is sharpened.

European patent application no. EP0908847A2 discloses an image synthesis apparatus and an image synthesis method. It discloses that an image synthesis apparatus employs a stored image information to generate coordinate transformation parameters that are used to set a positional relationship for selected images, changes the generated coordinate transformation parameters by using as a reference position for an arbitrary image, provides resultant coordinate transformation parameters as an image synthesis information, and synthesizes the images in accordance with the image synthesis information.

There is no technology to obtain a single image without perspective distortion by a camera with a field of view narrower than a subject.

It would still be a need to provide a user equipment, and a method of oblique view correction for users being able to obtain a single image without perspective distortion.

SUMMARY

An object of the present disclosure is to propose a user equipment (UE) , and a method of oblique view correction for users able to obtain a single image without perspective distortion.

In a first aspect of the present disclosure, a user equipment (UE) includes an image sensing module and a processor coupled to the image sensing module. The processor is configured to control the image sensing module to capture a color image, an infrared (IR) image and a depth image, estimate plane parameters from the depth image, calculate focal distance data from the depth image, control the image sensing module to capture partially focused images at focal distances from the focal distance data, and cut focused image data from the partially focused images and compose these focused image data to form a wholly focused image.

In the embodiment of the present disclosure, the processor is configured to adjust the wholly focused image to a non-perspective image.

In the embodiment of the present disclosure, the method of adjusting the wholly focused image to a non-perspective image includes steps of estimating coordinate data of four corners of the wholly focused image on perspective coordinate axes calculated from the depth image and dragging the wholly focused image to form a non-perspective image on real world coordinate axes.

In the embodiment of the present disclosure, the processor is configured to compose several of the non-perspective images to form a single image.

In the embodiment of the present disclosure, the UE further includes a display module, and the processor is configured to set a trimming candidate frame on the single image shown on the display module.

In the embodiment of the present disclosure, the method of estimating plane parameters from the depth image includes a step of estimating a normal vector of a plane from the depth image.

In the embodiment of the present disclosure, the UE further includes an inertial measurement unit (IMU) . The method of estimating plane parameters from the depth image further includes a step of estimating a perspective vertical coordinate axis and a perspective horizontal coordinate axis from data of IMU.

In the embodiment of the present disclosure, the method of calculating focal distance data from the depth image includes a step of determining several focal distances so that areas of depth of field at these focal distances are overlapping to cover whole of the color image.

In the embodiment of the present disclosure, the method of calculating focal distance data from the depth image further includes a step of determining whether each area of depth of field has texture.

In the embodiment of the present disclosure, the image sensing module includes a camera module for sensing color images, and a depth sensing module for sensing depth images.

In the embodiment of the present disclosure, the image sensing module further includes an image processor configured to control the camera module and the depth sensing module.

In the embodiment of the present disclosure, the camera module includes a lens module, an image sensor, an image sensor driver, a focus and an optical image stabilization (OIS) driver, a focus and OIS actuator and a gyro sensor. The image sensor driver is configured to control the image sensor to capture images. The focus and OIS driver is configured to control the focus and OIS actuator to focus the lens module and to move the lens module for compensating hand shock of human. The gyro sensor is configured for providing motion data to the focus and OIS driver.

In the embodiment of the present disclosure, the depth sensing module includes a projector, a lens, a range sensor, and a range sensor driver. The range sensor driver is configured to control the projector to project dot matrix pulse light, and control the range sensor to capture a reflecting dot matrix image focusing by the lens.

In the embodiment of the present disclosure, the UE further includes a memory configured to record programs, the image data, the plane parameters and a translation matrix.

In the embodiment of the present disclosure, the depth image includes point cloud data.

In the embodiment of the present disclosure, the UE further includes an input module configured to receive a human instruction, a codec configured to compress and decompress multimedia data, a speaker and a microphone connected to the codec, a wireless communication module configured to transmit and receive messages, and a global navigation satellite system (GNSS) module configured to provide positioning information.

In a second aspect of the present disclosure, a method of oblique view correction includes capturing a color image, an infrared (IR) image and a depth image by an image sensing module, estimating plane parameters from the depth image, calculating focal distance data from the depth image, capturing partially focused images at these focal distances from the focal distance data by the image sensing module, and cutting focused image data from the partially focused images and composing these focused image data to form a wholly focused image.

In the embodiment of the present disclosure, the method of oblique view correction further includes a step of adjusting the wholly focused image to a non-perspective image.

In the embodiment of the present disclosure, the step of adjusting the wholly focused image to a non-perspective image further includes steps of estimating coordinate data of four corners of the wholly focused image on perspective coordinate axes calculated from the depth image and dragging the wholly focused image to form a non-perspective image on real world coordinate axes.

In the embodiment of the present disclosure, the method of oblique view correction further includes a step of composing several of the non-perspective images to form a single image.

In the embodiment of the present disclosure, the method of oblique view correction further includes a step of setting a trimming candidate frame on the single image shown on a display module.

In the embodiment of the present disclosure, the step of estimating plane parameters from the depth image includes a step of estimating a normal vector of a plane from the depth image.

In the embodiment of the present disclosure, the step of estimating plane parameters from the depth image further includes a step of estimating a perspective vertical coordinate axis and a perspective horizontal coordinate axis from data of IMU.

In the embodiment of the present disclosure, the step of calculating focal distance data from the depth image includes a step of determining several focal distances so that areas of depth of field at these focal distances are overlapping to cover whole of the color image.

In the embodiment of the present disclosure, the step of calculating focal distance data from the depth image further includes a step of determining whether each area of depth of field has texture.

In the embodiment of the present disclosure, the method of oblique view correction further includes a step of providing a memory configured to record programs, image data, plane parameters and a translation matrix.

In the embodiment of the present disclosure, the method of oblique view correction further includes a step of providing an input module configured to receive a human instruction, a codec configured to compress and decompress multimedia data, a speaker and a microphone connected to the codec, a wireless communication module configured to transmit and receive messages, and a global navigation satellite system (GNSS) module configured to provide positioning information.

Therefore, embodiments of the present invention provide a user equipment (UE) , and a method of oblique view correction for users able to obtain a single image without perspective distortion.

BRIEF DESCRIPTION OF DRAWINGS

In order to more clearly illustrate the embodiments of the present disclosure or related art, the following figures will be described in the embodiments are briefly introduced. It is obvious that the drawings are merely some embodiments of the present disclosure, a person having ordinary skill in this field can obtain other figures according to these figures without paying the premise.

FIG. 1 is a schematic application diagram of a prior art user equipment.

FIG. 2 is a schematic diagram of a user equipment (UE) according to an embodiment of the present disclosure.

FIG. 3 is a flowchart illustrating method of oblique view correction according to an embodiment of the present disclosure.

FIG. 4 is a flowchart illustrating steps of estimating plane parameters from a depth image according to an embodiment of the present disclosure.

FIG. 5 is a flowchart illustrating steps of calculating focal distance data from the depth image according to an embodiment of the present disclosure.

FIG. 6 is a flowchart illustrating steps of adjusting a wholly focused image to a non-perspective image according to an embodiment of the present disclosure.

FIG. 7 is a schematic diagram of a step of capturing a color image, an infrared (IR) image, and a depth image according to an embodiment of the present disclosure.

FIG. 8 is a schematic diagram of steps of estimating plane parameters from the depth image according to an embodiment of the present disclosure.

FIG. 9 is a schematic diagram of a step of calculating focal distance data from the depth image according to an embodiment of the present disclosure.

FIG. 10 is a schematic diagram of describing relations between a focus position, a focus distance, depth of field (DOF) , and an area of DOF according to an embodiment of the present disclosure.

FIG. 11 is a schematic diagram of a step of capturing partially focused images at focal distances from the focal distance data according to an embodiment of the present disclosure.

FIG. 12 is a schematic diagram of a step of cutting focused image data from the partially focused images and composing these focused image data to form a wholly focused image according to an embodiment of the present disclosure.

FIG. 13 is a schematic diagram of a step of adjusting the wholly focused image to a non-perspective image according to an embodiment of the present disclosure.

FIG. 14 is a schematic diagram of a step of composing several of the non-perspective images to form a single image according to an embodiment of the present disclosure

FIG. 15 is a schematic diagram of a step of setting a trimming candidate frame on the single image shown on a display module according to an embodiment of the present disclosure

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure are described in detail with the technical matters, structural features, achieved objects, and effects with reference to the accompanying drawings as follows. Specifically, the terminologies in the embodiments of the present disclosure are merely for describing the purpose of the certain embodiment, but not to limit the disclosure.

Please refer to FIGs. 2, and 3; in some embodiments, a user equipment (UE) 100 includes an image sensing module 10, and a processor 20 coupled to the image sensing module 10. The processor 20 is configured to control the image sensing module 10 to capture a color image C_I, an infrared (IR) image IR_I, and a depth image D_I, estimate plane parameters from the depth image D_I, calculate focal distance data from the depth image D_I, control the image sensing module 10 to capture partially focused images PF_I at these focal distances from the focal distance data, and cut focused image data from the partially focused images PF_I and compose these focused image data to form a wholly focused image WF_I.

In detail, please refer to FIG. 7. The processor 20 of the UE 100 is configured to control the image sensing module 10 to capture a color image C_I, an infrared (IR) image IR_I, and a depth image D_I for a left side of a target, and capture a color image C_I’, an infrared (IR) image IR_I’, and a depth image D_I’ a right side of a target for example.

In some embodiments, please refer to FIGs. 4, and 8; the method of estimating plane parameters from the depth image D_I includes a step of estimating a normal vector N_V of a plane from the depth image D_I.

In detail, please refer to FIG. 8; the UE 100 estimates a normal vector N_V of a plane from the depth image D_I, and a normal vector N_V’ of a plane from the depth image D_I’.

In some embodiments, please refer to FIGs. 2, 4, and 8; the UE further includes an inertial measurement unit (IMU) 40. The method of estimating plane parameters from the depth image D_I further includes a step of estimating a perspective vertical coordinate axis PV_CA, and a perspective horizontal coordinate axis PH_CA from data of IMU 40.

In detail, please refer to FIG. 8; the UE 100 estimates a perspective vertical coordinate axis PV_CA, and a perspective horizontal coordinate axis PH_CA of depth image D_I from data of IMU 40 and estimates a perspective vertical coordinate axis PV_CA’, and a perspective horizontal coordinate axis PH_CA’ of depth image D_I’ from data of IMU 40.

In some embodiments, please refer to FIGs. 5, and 9; the method of calculating focal distance data from the depth image D_I includes a step of determining several focal distances FD_1 to FD_4 so that areas of depth of field DF_A1 to DF_A4 corresponding to these focal distances FD_1 to FD_4 are overlapping to cover whole of the color image C_I.

In detail, please refer to FIGs. 9, and 10; a focus position F_1 of UE 100 has a focus distance FD_1, and depth of field DF_1. An intersection area of depth of field DF_1 on a target is an area of depth of field DF_A1. The area of depth of field DF_A1 can be calculated out of the depth image D_I data. Meanwhile, a focus position F_2 of UE 100 has a focus distance FD_2, and depth of field DF_2. An intersection area of depth of field DF_2 on the target is an area of depth of field DF_A2. A focus position F_3 of UE 100 has a focus distance FD_3, and depth of field DF_3. An intersection area of depth of field DF_3 on the target is an area of depth of field DF_A3. A focus position F_4 of UE 100 has a focus distance FD_4, and depth of field DF_4. An intersection area of depth of field DF_4 on the target is an area of depth of field DF_A4.

In detail, please refer to FIGs. 5, 9, and 10; the UE 100 determines several focal positions F_1 to F_4 so that areas of depth of field DF_A1 to DF_A4 corresponding to these focal positions F_1 to F_4 are overlapping to cover whole of the color image C_I. The UE 100 determines several focal positions F_1’ to F_4’ so that areas of depth of field DF_A1’ to DF_A4’ corresponding to these focal positions F_1’ to F_4’ are overlapping to cover whole of the color image C_I’.

In some embodiments, please refer to FIG. 5; the method of calculating focal distance data from the depth image D_I further includes a step of determining whether each area of depth of field DF_A1 to DF_A4 has texture.

In detail, please refer to FIGs. 3, 11, and 12; the processor 20 controls the image sensing module 10 to capture partially focused images PF_I2 at the focal distances FD_2 from the focal distance data, to capture partially focused images PF_I3 at the focal distances FD_3 from the focal distance data, cuts focused image data from the partially focused images PF_I2, and focused image data from the partially focused images PF_I3, and composes these focused image data to form a wholly focused image WF_I. The processor 20 controls the image sensing module 10 to capture partially focused images PF_I2’, to capture partially focused images PF_I3’, cuts focused image data from the partially focused images PF_I2’, and focused image data from the partially focused images PF_I3’, and composes these focused image data to form a wholly focused image WF_I’.

In some embodiments, please refer to FIG. 3; the processor 20 is configured to adjust the wholly focused image WF_I to a non-perspective image NP_I. In some embodiments, please refer to FIGs. 3, 6, and 13; the method of adjusting the wholly focused image WF_I to a non-perspective image NP_I includes steps of estimating coordinate data of four corners C1 to C4 of the wholly focused image WF_I on perspective coordinate axes P_CA calculated from the depth image D_I, and dragging the wholly focused image WF_I to form a non-perspective image NP_I on real world coordinate axes R_CA.

In detail, the UE 100 estimates coordinate data of four corners C1 to C4 of the wholly focused image WF_I on perspective coordinate axes P_CA calculated from the depth image D_I and then drags the wholly focused image WF_I to form a non-perspective image NP_I on real world coordinate axes R_CA. In detail, the UE 100 provides a translation matrix from the perspective coordinate axes P_CA to the real world coordinate axes R_CA. The wholly focused image WF_I is translated to the non-perspective image NP_I by multiplying data of the wholly focused image WF_I with the translation matrix. Meanwhile, the UE 100 estimates coordinate data of four corners C1’ to C4’ of the wholly focused image WF_I’ on perspective coordinate axes P_CA’ calculated from the depth image D_I’ and then drags the wholly focused image WF_I ‘to form a non-perspective image NP_I’ on real world coordinate axes R_CA.

In some embodiments, please refer to FIG. 14; the processor 20 is configured to compose several of the non-perspective images NP_I to form a single image S_I.

In detail, the processor 20 is configured to compose non-perspective images NP_I, and NP_I’ to form a single image S_I.

In some embodiments, please refer to FIG. 15; the UE 100 further includes a display module 30, and the processor 20 is configured to set a trimming candidate frame TC_F on the single image S_I shown on the display module 30. In some embodiments, please refer to FIG. 2; the image sensing module 10 includes a camera module 11 for sensing color images C_I, and a depth sensing module 12 for sensing depth images D_I. In some embodiments, please refer to FIG. 2; the image sensing module 10 further includes an image processor 13 configured to control the camera module 11, and the depth sensing module 12. In some embodiments, please refer to FIG. 2; the camera module 11 includes a lens module 111, an image sensor 112, an image sensor driver 113, a focus and an optical image stabilization (OIS) driver 114, a focus and OIS actuator 115, and a gyro sensor 116. The image sensor driver 113 is configured to control the image sensor 112 to capture images. The focus and OIS driver 114 is configured to control the focus and OIS actuator 115 to focus the lens module 111, and to move the lens module 111 for compensating hand shock of human. The gyro sensor 116 is configured for providing motion data to the focus and OIS driver 114.

In some embodiments, please refer to FIG. 2; the depth sensing module 12 includes a projector 124, a lens 121, a range sensor 122, and a range sensor driver 123. The range sensor driver 123 is configured to control the projector 124 to project dot matrix pulse light, and control the range sensor 122 to capture a reflecting dot matrix image focusing by the lens 121. In some embodiments, please refer to FIG. 2; the UE 100 further includes a memory 50 configured to record programs, image data, the plane parameters, and the translation matrix. In some embodiments, the depth image D_I includes point cloud data. In some embodiments, please refer to FIG. 2; the UE 100 further includes an input module 60 configured to receive a human instruction, a codec 70 configured to compress and decompress multimedia data, a speaker 80, and a microphone 90 connected to the codec 70, a wireless communication module 91 configured to transmit and receive messages, and a global navigation satellite system (GNSS) module 92 configured to provide positioning information.

Further, please refer to FIG. 3; in some embodiments, a method of oblique view correction includes: at block S100, capturing a color image C_I, an infrared (IR) image IR_I, and a depth image D_I by an image sensing module 10, at block S200, estimating plane parameters from the depth image D_I, at block 300, calculating focal distance data from the depth image D_I, at block 400, capturing partially focused images PF_I at these focal distances FD_1 to FD_4 from the focal distance data by the image sensing module 10, and at block 500, cutting focused image data from the partially focused images PF_I and composing these focused image data to form a wholly focused image WF_I.

In detail, please refer to FIGs. 2, and 7; the processor 20 of the UE 100 is configured to control the image sensing module 10 to capture a color image C_I, an infrared (IR) image IR_I, and a depth image D_I for a left side of a target, and capture a color image C_I’, an infrared (IR) image IR_I’, and a depth image D_I’ a right side of a target for example.

In some embodiments, please refer to FIGs. 4, and 8; the step of estimating plane parameters from the depth image at block S200 includes a step of: at block S210, estimating a normal vector N_V of a plane from the depth image D_I.

In some embodiments, please refer to FIGs. 2, 4, and 8; the step of estimating plane parameters from the depth image at block S200 further includes a step of: at block S220, estimating a perspective vertical coordinate axis PV_CA, and a perspective horizontal coordinate axis PH_CA from data of IMU 40.

In some embodiments, please refer to FIGs. 5, and 9; the step of calculating focal distance data from the depth image at block S300 includes a step of: at block S310, determining several focal distances FD_1 to FD_4 so that areas of depth of field DF_A1 to DF_A4 corresponding to these focal distances FD_1 to FD_4 are overlapping to cover whole of the color image C_I.

In some embodiments, the step of calculating focal distance data from the depth image at block S300 further includes a step of: at block S320, determining whether each area of depth of field DF_A1 to DF_A4 has texture.

In some embodiments, please refer to FIG. 3; the method of oblique view correction further includes a step of: at block S600, adjusting the wholly focused image WF_I to a non-perspective image NP_I. In some embodiments, please refer to FIGs. 3, 6, and 13; the step of adjusting the wholly focused image WF_I to a non-perspective image NP_I at block S600 further includes steps of: at block S610, estimating coordinate data of four corners C1 to C4 of the wholly focused image WF_I on perspective coordinate axes P_CA calculated from the depth image D_I, and at block S620, dragging the wholly focused image WF_I to form a non-perspective image NP_I on real world coordinate axes R_CA.

In detail, the UE 100 estimates coordinate data of four corners C1 to C4 of the wholly focused image WF_I on perspective coordinate axes P_CA calculated from the depth image D_I and then drags the wholly focused image WF_I to form a non-perspective image NP_I on real world coordinate axes R_CA. In detail, the UE 100 provides a translation matrix from the perspective coordinate axes P_CA to the real world coordinate axes R_CA. The wholly focused image WF_I is translated to the non-perspective image NP_I by multiplying data of the wholly focused image WF_I with the translation matrix. Meanwhile, the UE 100 estimates coordinate data of four corners C1’ to C4’ of the wholly focused image WF_I’ on perspective coordinate axes P_CA’ calculated from the depth image D_I’ and then drags the wholly focused image WF_I ‘to form a non-perspective image NP_I’ on real world coordinate axes R_CA. In some embodiments, please refer to FIG. 14; the method of oblique view correction further includes a step of: at block S700, composing several of the non-perspective images NP_I to form a single image S_I.

In some embodiments, please refer to FIG. 15; the method of oblique view correction further includes a step of: at block S800, setting a trimming candidate frame TC_F on the single image S_I shown on a display module 30. In some embodiments, please refer to FIG. 2; the image sensing module 10 includes a camera module 11 for sensing color images C_I, and a depth sensing module 12 for sensing depth images D_I. In some embodiments, please refer to FIG. 2; the image sensing module 10 further includes an image processor 13 configured to control the camera module 11, and the depth sensing module 12.

In some embodiments, please refer to FIG. 2; the camera module 11 includes a lens module 111, an image sensor 112, an image sensor driver 113, a focus and an optical image stabilization (OIS) driver 114, a focus and OIS actuator 115, and a gyro sensor 116. The image sensor driver 113 is configured to control the image sensor 112 to capture images. The focus and OIS driver 114 is configured to control the focus and OIS actuator 115 to focus the lens module 111, and to move the lens module 111 for compensating hand shock of human. The gyro sensor 116 is configured for providing motion data to the focus and OIS driver 114. In some embodiments, please refer to FIG. 2; the depth sensing module 12 includes a projector 124, a lens 121, a range sensor 122, and a range sensor driver 123. The range sensor driver 123 is configured to control the projector 124 to project dot matrix pulse light, and control the range sensor 122 to capture a reflecting dot matrix image focusing by the lens 121.

In some embodiments, please refer to FIG. 2; the method of oblique view correction further includes a step of providing a memory 50 configured to record programs, image data, plane parameters, and a translation matrix. In some embodiments, the depth image D_I includes point cloud data. In some embodiments, please refer to FIG. 2; the method of oblique view correction further includes a step of providing an input module 60 configured to receive a human instruction, a codec 70 configured to compress and decompress multimedia data, a speaker 80, and a microphone 90 connected to the codec 70, a wireless communication module 91 configured to transmit and receive messages, and a global navigation satellite system (GNSS) module 92 configured to provide positioning information

Benefits of the method of oblique view correction include: 1. providing a single, wholly focused image without perspective distortion. 2. providing a single picture of a target object with a horizontal width greater than a width of a shooting area of a camera.

In the embodiment of the present disclosure, the UE, and the method of oblique view correction are provided. The method of an image sensor communication of the UE includes capturing a color image, an infrared image, and a depth image by an image sensing module, estimating plane parameters from the depth image, calculating focal distance data from the depth image, capturing partially focused images at these focal distances from the focal distance data by the image sensing module, and cutting focused image data from the partially focused images and composing these focused image data to form a wholly focused image, so as to provide a single focused image without perspective distortion.

A person having ordinary skill in the art understands that each of the units, algorithm, and steps described and disclosed in the embodiments of the present disclosure are realized using electronic hardware or combinations of software for computers and electronic hardware. Whether the functions run in hardware or software depends on the condition of application and design requirement for a technical plan. A person having ordinary skill in the art can use different ways to realize the function for each specific application while such realizations should not go beyond the scope of the present disclosure. It is understood by a person having ordinary skill in the art that he/she can refer to the working processes of the system, device, and unit in the above-mentioned embodiment since the working processes of the above-mentioned system, device, and unit are basically the same. For easy description and simplicity, these working processes will not be detailed. It is understood that the disclosed system, device, and method in the embodiments of the present disclosure can be realized with other ways. The above-mentioned embodiments are exemplary only. The division of the units is merely based on logical functions while other divisions exist in realization. It is possible that a plurality of units or components are combined or integrated in another system. It is also possible that some characteristics are omitted or skipped. On the other hand, the displayed or discussed mutual coupling, direct coupling, or communicative coupling operate through some ports, devices, or units whether indirectly or communicatively by ways of electrical, mechanical, or other kinds of forms. The units as separating components for explanation are or are not physically separated. The units for display are or are not physical units, that is, located in one place or distributed on a plurality of network units. Some or all of the units are used according to the purposes of the embodiments. Moreover, each of the functional units in each of the embodiments can be integrated in one processing unit, physically independent, or integrated in one processing unit with two or more than two units. If the software function unit is realized and used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical plan proposed by the present disclosure can be essentially or partially realized as the form of a software product. Or, one part of the technical plan beneficial to the conventional technology can be realized as the form of a software product. The software product in the computer is stored in a storage medium, including a plurality of commands for a computational device (such as a personal computer, a server, or a network device) to run all or some of the steps disclosed by the embodiments of the present disclosure. The storage medium includes a USB disk, a mobile hard disk, a read-only memory (ROM) , a random-access memory (RAM) , a floppy disk, or other kinds of media capable of storing program codes.

While the present disclosure has been described in connection with what is considered the most practical and preferred embodiments, it is understood that the present disclosure is not limited to the disclosed embodiments but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the appended claims.

Claims

A user equipment (UE) , comprising:

an image sensing module; and

a processor coupled to the image sensing module,

wherein the processor is configured to:

control the image sensing module to capture a color image, an infrared (IR) image, and a depth image;

estimate plane parameters from the depth image;

calculate focal distance data from the depth image;

control the image sensing module to capture partially focused images at these focal distances from the focal distance data; and

cut focused image data from the partially focused images and compose these focused image data to form a wholly focused image.
The UE of claim 1, wherein the processor is configured to adjust the wholly focused image to a non-perspective image.
The UE of claim 2, wherein to adjust the wholly focused image to a non-perspective image comprises to estimate coordinate data of four corners of the wholly focused image on perspective coordinate axes calculated from the depth image, and to drag the wholly focused image to form a non-perspective image on real world coordinate axes.
The UE of claim 2, wherein the processor is configured to compose several of the non-perspective images to form a single image.
The UE of claim 4 further comprising a display module, and the processor is configured to set a trimming candidate frame on the single image shown on the display module.
The UE of claim 1, wherein to estimate plane parameters from the depth image comprises to estimate a normal vector of a plane from the depth image.
The UE of claim 6, further comprising an inertial measurement unit (IMU) , wherein to estimate plane parameters from the depth image further comprises to estimate a perspective vertical coordinate axis and a perspective horizontal coordinate axis from data of IMU.
The UE of claim 1, wherein to calculate focal distance data from the depth image comprises to determine several focal distances so that areas of depth of field at these focal distances are overlapping to cover whole of the color image.
The UE of claim 8, wherein to calculate focal distance data from the depth image further comprises to determine whether each area of depth of field has a texture.
The UE of any one of claims 1 to 9, wherein the image sensing module comprises a camera module for sensing color images, and a depth sensing module for sensing depth images.
The UE of claim 10, wherein the image sensing module further comprises an image processor configured to control the camera module, and the depth sensing module.
The UE of claim 10, wherein the camera module comprises a lens module, an image sensor, an image sensor driver, a focus and an optical image stabilization (OIS) driver, a focus and OIS actuator, and a gyro sensor, the image sensor driver is configured to control the image sensor to capture images, the focus and OIS driver is configured to control the focus and OIS actuator to focus the lens module, and to move the lens module for compensating hand shock of human, and the gyro sensor is configured for providing motion data to the focus and OIS driver.
The UE of claim 10, wherein the depth sensing module comprises a projector, a lens, a range sensor, and a range sensor driver, the range sensor driver is configured to control the projector to project dot matrix pulse light, and control the range sensor to capture a reflecting dot matrix image focusing by the lens.
The UE of any one of claims 1 to 13, further comprising a memory configured to record programs, image data, plane parameters, and a translation matrix.
The UE of any one of claims 1 to 14, wherein the depth image comprises point cloud data.
The UE of any one of claims 1 to 15, further comprising an input module configured to receive a human instruction, a codec configured to compress and decompress multimedia data, a speaker and a microphone connected to the codec, a wireless communication module configured to transmit and receive messages, and a global navigation satellite system (GNSS) module configured to provide positioning information.
A method of oblique view correction, comprising:

capturing a color image, an infrared (IR) image, and a depth image by an image sensing module;

estimating plane parameters from the depth image;

calculating focal distance data from the depth image;

capturing partially focused images at these focal distances from the focal distance data by the image sensing module; and

cutting focused image data from the partially focused images, and composing these focused image data to form a wholly focused image.
The method of oblique view correction of claim 17, further comprising a step of adjusting the wholly focused image to a non-perspective image.
The method of oblique view correction of claim 18, wherein the step of adjusting the wholly focused image to a non-perspective image further comprises steps of estimating coordinate data of four corners of the wholly focused image on perspective coordinate axes calculated from the depth image and dragging the wholly focused image to form a non-perspective image on real world coordinate axes.
The method of oblique view correction of claim 18, further comprising a step of composing several of the non-perspective images to form a single image.
The method of oblique view correction of claim 18, further comprising a step of setting a trimming candidate frame on the single image shown on a display module.
The method of oblique view correction of claim 17, wherein the step of estimating plane parameters from the depth image comprises a step of estimating a normal vector of a plane from the depth image.
The method of oblique view correction of claim 17, wherein the step of estimating plane parameters from the depth image comprises further comprises a step of estimating a perspective vertical coordinate axis, and a perspective horizontal coordinate axis from data of IMU.
The method of oblique view correction of claim 17, wherein the step of calculating focal distance data from the depth image comprises a step of determining several focal distances so that areas of depth of field at these focal distances are overlapping to cover whole of the color image.
The method of oblique view correction of claim 24, wherein the step of calculating focal distance data from the depth image further comprises a step of determining whether each area of depth of field has texture.
The method of oblique view correction of any one of claims 17 to 25, wherein the image sensing module comprises a camera module for sensing color images, and a depth sensing module for sensing depth images.
The method of oblique view correction of claim 26, wherein the image sensing module further comprises an image processor configured to control the camera module, and the depth sensing module.
The method of oblique view correction of claim 26, wherein the camera module comprises a lens module, an image sensor, an image sensor driver, a focus and an optical image stabilization (OIS) driver, a focus and OIS actuator, and a gyro sensor, the image sensor driver is configured to control the image sensor to capture images, the focus and OIS driver is configured to control the focus and OIS actuator to focus the lens module and to move the lens module for compensating hand shock of humans, and the gyro sensor is configured for providing motion data to the focus and OIS driver.
The method of oblique view correction of claim 26, wherein the depth sensing module comprises a projector, a lens, a range sensor, and a range sensor driver, the range sensor driver is configured to control the projector to project dot matrix pulse light, and control the range sensor to capture a reflecting dot matrix image focusing by the lens.
The method of oblique view correction of any one of claims 17 to 29, further comprising a step of providing a memory configured to record programs, the image data, the plane parameters, and a translation matrix.
The method of oblique view correction of any one of claims 17 to 30, wherein the depth image comprises point cloud data.
The method of oblique view correction of any one of claims 17 to 31, further comprising a step of providing an input module configured to receive a human instruction, a codec configured to compress and decompress multimedia data, a speaker and a microphone connected to the codec, a wireless communication module configured to transmit and receive messages, and a global navigation satellite system (GNSS) module configured to provide positioning information.