JP7346594B2 - User equipment and strabismus correction method - Google Patents

Description

The present disclosure relates to the field of image processing technology, and more specifically to user equipment (UE) and strabismus correction methods.

Referring to FIG. 1, in the current technology, when a user uses a user equipment 1, such as a mobile phone, to take a picture of an oriented plane surface with sharp contours, the user equipment 1 Correct the image to a shape without distortion. However, if the outline of the obliquely oriented plane is unclear, or only a part of it is within the imaging area 2, or if the obliquely oriented plane has a width larger than the width of the imaging area 2, the user can A single focused image without perspective distortion cannot be obtained.

US Pat. No. 6,449,004 discloses an electronic camera with oblique view correction functionality. An electronic camera is disclosed that includes an image sensor that photoelectrically captures an optical image of a subject to generate image data. The tilt angle information providing device provides information regarding the tilt angle between the sensing surface of the image sensor and the front of the subject. A distance measuring device measures the distance to the subject. A corrector corrects the provided tilt angle information and the measured distance to generate image data, thereby generating a pseudo subject image whose front is in a plane parallel to the sensing surface of the image sensor.

US Pat. No. 7,365,301 discloses a three-dimensional shape detection device, an image capturing device, and a three-dimensional shape detection program. a projection means for projecting pattern light; an image photographing means for photographing a pattern light projection image of the subject onto which the pattern light is projected; and a three-dimensional shape of the subject based on the locus of the pattern light extracted from the pattern light projection image. A three-dimensional shape detection device including a three-dimensional shape calculation means for calculating the three-dimensional shape is disclosed.

US Pat. No. 7,711,259 discloses a method and apparatus for increasing the depth of field of an imager. It is disclosed that an imaging unit photographs a plurality of images at different focal positions, and combines these images into one image for sharpening. In another embodiment, a single image is captured while the focus position changes during image capture, and the resulting image is sharpened.

European Patent Application No. 0908847 discloses an image composition device and method. The image synthesis device generates coordinate transformation parameters for setting the positional relationship of the selected images using the stored image information, and changes the generated coordinate transformation parameters as a reference position of an arbitrary image, It is disclosed that the obtained coordinate transformation parameters are provided as image synthesis information and images are synthesized according to the image synthesis information.

There is no technique for obtaining a single image without perspective distortion using a camera whose field of view is narrower than the subject.

There is still a need to provide user equipment (UE) and strabismus correction methods that allow a user to obtain a single image without perspective distortion.

The purpose of the present disclosure is to propose a user equipment (UE) and strabismus correction method that allows a user to obtain a single image without perspective distortion.

In a first aspect of the disclosure, a user equipment (UE) includes an image sensing module and a processor connected to the image sensing module. The processor controls the image sensing module to take a color image, an infrared (IR) image, and a depth image, estimates a plane parameter from the depth image, calculates focal length data from the depth image, and captures a color image, an infrared (IR) image, and a depth image; controlling a sensing module to capture a partially focused image at a focal length from the focal length data; cropping focused image data from the partially focused image; The in-focus image data is synthesized into a completely focused image (wholly focused image).

In embodiments of the present disclosure, the processor adjusts the fully focused image to a non-perspective image.

In an embodiment of the present disclosure, the method for adjusting the fully focused image to a non-perspective image includes estimating coordinate data of four corners of the fully focused image on perspective coordinate axes calculated from the depth image. and dragging the fully focused image to form a non-perspective image in real world coordinate axes.

In an embodiment of the present disclosure, the processor combines several of the non-fluoroscopic images into a single image.

In an embodiment of the present disclosure, the UE further includes a display module, and the processor sets a trimming candidate frame on the single image displayed on the display module.

In an embodiment of the present disclosure, the method for estimating plane parameters from the depth image includes estimating a normal vector of a plane from the depth image.

In embodiments of the present disclosure, the UE further includes an inertial measurement unit (IMU). The method for estimating plane parameters from the depth image further includes estimating perspective vertical coordinate axes and perspective horizontal coordinate axes from data of the IMU.

In an embodiment of the present disclosure, the method for calculating focal length data from the depth image includes the steps of: calculating focal length data from the depth image such that depth of field regions corresponding to focal lengths overlap to cover the entire color image; including determining several focal lengths.

In an embodiment of the present disclosure, the method for calculating focal length data from the depth image further includes determining whether each region of depth of field has texture.

In an embodiment of the present disclosure, the image sensing module includes a camera module that senses a color image and a depth sensing module that senses a depth image.

In embodiments of the present disclosure, the image sensing module further includes an image processor that controls the camera module and the depth sensing module.

In embodiments of the present disclosure, the camera module includes a lens module, an image sensor, an image sensor driver, a focusing and optical image stabilization (OIS) driver, a focusing and OIS actuator, and a gyro sensor. . The image sensor driver controls the image sensor to capture an image. The focus and OIS driver controls the focus and OIS actuator to focus the lens module and move the lens module to compensate for human hand shock. The gyro sensor provides motion data to the focus and OIS driver.

In an embodiment of the present disclosure, the depth sensing module includes a projector, a lens, a range sensor, and a range sensor driver. The range sensor driver controls the projector to project dot matrix pulsed light, and controls the range sensor to capture a reflected dot matrix image focused by the lens.

In embodiments of the present disclosure, the UE further includes a memory for storing programs, image data, plane parameters, and translation matrices.

In an embodiment of the present disclosure, the depth image includes point cloud data.

In an embodiment of the present disclosure, the UE includes an input module that accepts human instructions, a codec that compresses and decompresses multimedia data, a speaker and a microphone connected to the codec, and a wireless communication module that sends and receives messages. , a Global Navigation Satellite System (GNSS) module that provides positioning information.

In a second aspect of the present disclosure, a strabismus correction method includes the steps of: capturing a color image, an infrared (IR) image, and a depth image by an image sensing module; estimating a plane parameter from the depth image; and estimating a plane parameter from the depth image. calculating focal length data from the focal length data; capturing partially focused images at these focal lengths from the focal length data by the image sensing module; and cropping focused image data from the partially focused images; compositing the focused image data into a fully focused image.

In an embodiment of the present disclosure, the strabismus correction method further includes adjusting the perfectly focused image to a non-fluoroscopic image.

In an embodiment of the present disclosure, the step of adjusting the perfectly focused image to a non-perspective image includes the step of estimating coordinate data of four corners of the fully focused image in the perspective coordinate axis calculated from the depth image; and dragging the focused image to form a non-perspective image in real world coordinate axes.

In an embodiment of the present disclosure, the strabismus correction method further includes combining several of the non-perspective images into a single image.

In an embodiment of the present disclosure, the strabismus correction method further includes the step of setting a cropping candidate frame in the single image displayed on a display module.

In an embodiment of the present disclosure, estimating plane parameters from the depth image includes estimating a normal vector of the plane from the depth image.

In embodiments of the present disclosure, estimating plane parameters from the depth image further includes estimating perspective vertical coordinate axes and perspective horizontal coordinate axes from data of an IMU.

In an embodiment of the present disclosure, the step of calculating focal length data from the depth image includes calculating focal length data from several focal lengths such that regions of depth of field corresponding to the focal lengths overlap and cover the entire color image. including the step of determining.

In embodiments of the present disclosure, calculating focal length data from the depth image further includes determining whether each region of depth of field has texture.

In an embodiment of the present disclosure, the image sensing module includes a camera module that senses a color image and a depth sensing module that senses a depth image.

In embodiments of the present disclosure, the image sensing module further includes an image processor that controls the camera module and the depth sensing module.

In embodiments of the present disclosure, the camera module includes a lens module, an image sensor, an image sensor driver, a focusing and optical image stabilization (OIS) driver, a focusing and OIS actuator, and a gyro sensor. . The image sensor driver controls the image sensor to capture an image. The focus and OIS driver controls the focus and OIS actuator to focus the lens module and move the lens module to compensate for human hand shake. The gyro sensor provides motion data to the focus and OIS driver.

In an embodiment of the present disclosure, the depth sensing module includes a projector, a lens, a range sensor, and a range sensor driver. The range sensor driver controls the projector to project dot matrix pulsed light, and controls the range sensor to capture a reflected dot matrix image focused by the lens.

In an embodiment of the present disclosure, the strabismus correction method further includes providing a memory for storing a program, image data, plane parameters, and a transformation matrix.

In an embodiment of the present disclosure, the depth image includes point cloud data.

In an embodiment of the present disclosure, the strabismus correction method includes an input module that receives instructions from a human, a codec that compresses and decompresses multimedia data, a speaker and a microphone connected to the codec, and wireless communication that sends and receives messages. and a Global Navigation Satellite System (GNSS) module that provides positioning information.

Accordingly, embodiments of the present invention provide a user equipment (UE) and strabismus correction method that allows a user to obtain a single image without perspective distortion.

BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly describe embodiments of the present disclosure or the prior art, the following drawings that are illustrated in the embodiments will be briefly introduced. It is clear that the drawings are only some embodiments of the disclosure and that a person skilled in the art can derive other drawings based on these drawings without making assumptions.

1 is a schematic application diagram of a user equipment of the prior art; FIG. 1 is a schematic diagram of a user equipment (UE) according to an embodiment of the present disclosure; FIG. 3 is a flowchart illustrating a strabismus correction method according to an embodiment of the present disclosure. 5 is a flowchart illustrating steps for estimating plane parameters from a depth image according to an embodiment of the present disclosure. 5 is a flowchart illustrating steps for calculating focal length data from a depth image according to an embodiment of the present disclosure. 3 is a flowchart illustrating steps for adjusting a fully focused image to a non-fluoroscopic image according to an embodiment of the present disclosure. 1 is a schematic diagram of capturing color images, infrared (IR) images, and depth images according to embodiments of the present disclosure; FIG. 3 is a schematic diagram of estimating plane parameters from a depth image according to an embodiment of the present disclosure; FIG. 3 is a schematic diagram of calculating focal length data from a depth image according to an embodiment of the present disclosure; FIG. FIG. 2 is a schematic diagram illustrating the relationship between a focal position, a focal length, a depth of field (DOF), and a region of the DOF according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram of capturing a partially focused image at a focal length from focal length data according to an embodiment of the present disclosure; 2 is a schematic diagram of the steps of cropping focused image data from a partially focused image and combining the focused image data into a fully focused image according to an embodiment of the present disclosure; FIG. FIG. 3 is a schematic diagram of adjusting a fully focused image to a non-transparent image according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram of combining several non-fluoroscopic images into a single image according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram of setting a cropping candidate frame on a single image displayed on a display module according to an embodiment of the present disclosure.

Embodiments of the present disclosure will be described in detail with technical matters, structural features, objectives and effects achieved, with reference to the accompanying drawings below. In particular, the terms used in the embodiments of the present disclosure are only used to describe the purpose of the particular embodiment, and are not intended to limit the present disclosure.

Referring to FIGS. 2 and 3, in some embodiments, user equipment (UE) 100 includes an image sensing module 10 and a processor 20 coupled to image sensing module 10. The processor 20 controls the image sensing module 10 to capture a color image C_I, an infrared (IR) image IR_I, and a depth image D_I, estimate plane parameters from the depth image D_I, and calculate focal length data from the depth image D_I. , controls the image sensing module 10 to capture partially focused images PF_I at these focal lengths from the focal length data, cuts out focused image data from the partially focused images PF_I, and converts these focused image data into fully focused images. Combine with image WF_I.

In detail, referring to FIG. 7, the processor 20 of the UE 100 controls the image sensing module 10 to capture, for example, a color image C_I, an infrared (IR) image IR_I, and a depth image D_I of the left side of the subject, and A color image C_I', an infrared (IR) image IR_I', and a depth image D_I' on the right side of the image are taken.

In some embodiments, with reference to FIGS. 4 and 8, a method of estimating plane parameters from a depth image D_I includes estimating a plane normal vector N_V from a depth image D_I.

Specifically, referring to FIG. 8, the UE 100 estimates a plane normal vector N_V from the depth image D_I, and estimates a plane normal vector N_V' from the depth image D_I'.

In some embodiments, referring to FIGS. 2, 4, and 8, the UE further includes an inertial measurement unit (IMU) 40. The method for estimating plane parameters from the depth image D_I further includes the step of estimating a perspective vertical coordinate axis PV_CA and a perspective horizontal coordinate axis PH_CA from data of the IMU 40.

Specifically, referring to FIG. 8, the UE 100 estimates the perspective vertical coordinate axis PV_CA and the perspective horizontal coordinate axis PH_CA of the depth image D_I from the data of the IMU 40, and estimates the perspective vertical coordinate axis PV_CA' and the perspective horizontal coordinate axis PV_CA' of the depth image D_I' from the data of the IMU 40. Estimate the horizontal coordinate axis PH_CA'.

In some embodiments, with reference to FIGS. 5 and 9, a method for calculating focal length data from a depth image D_I includes a method for calculating focal length data from a depth image D_I such that depth of field regions DF_A1 to DF_A4 corresponding to focal lengths FD_1 to FD_4 overlap and color It includes determining several focal lengths FD_1 to FD_4 to cover the entire image C_I.

Specifically, referring to FIGS. 9 and 10, the focal position F_1 of the UE 100 has a focal length FD_1 and a depth of field DF_1. The intersection area of the depth of field DF_1 on the subject is the depth of field area DF_A1. The depth of field area DF_A1 can be calculated from the data of the depth image D_I. On the other hand, the focal position F_2 of the UE 100 has a focal length FD_2 and a depth of field DF_2. The intersection area of depth of field DF_2 on the subject is depth of field area DF_A2. The focal position F_3 of the UE 100 has a focal length FD_3 and a depth of field DF_3. The intersection area of depth of field DF_3 on the subject is depth of field area DF_A3. The focal position F_4 of the UE 100 has a focal length FD_4 and a depth of field DF_4. The intersection area of depth of field DF_4 on the subject is depth of field area DF_A4.

Specifically, referring to FIGS. 5, 9, and 10, the UE 100 performs the following operations so that the depth of field regions DF_A1 to DF_A4 corresponding to the focal positions F_1 to F_4 overlap to cover the entire color image C_I. Determine several focal positions F_1 to F_4. The UE 100 determines several focal positions F_1' to F_4' so that the depth of field regions DF_A1' to DF_A4' corresponding to the focal positions F_1 to F_4 overlap and cover the entire color image C_I'. .

In some embodiments, referring to FIG. 5, the method of calculating focal length data from a depth image D_I further includes determining whether each of the depth of field regions DF_A1-DF_A4 has a texture. .

In particular, with reference to FIGS. 3, 11 and 12, the processor 20 controls the image sensing module 10 to capture a partially focused image PF_I2 at a focal length FD_2 from the focal length data; A partially focused image PF_I3 is captured at a focal length FD_3, focused image data is cut from the partially focused image PF_I2, focused image data is cut from the partially focused image PF_I3, and these focused image data are converted into a completely focused image. Combine into WF_I. The processor 20 controls the image sensing module 10 to capture a partially focused image PF_I2', capture a partially focused image PF_I3', cut out focused image data from the partially focused image PF_I2', and perform a partially focused image PF_I2'. Focused image data is cut out from image PF_I3', and these focused image data are combined into a completely focused image WF_I'.

In some embodiments, referring to FIG. 3, processor 20 adjusts the fully focused image WF_I to a non-transparent image NP_I. In some embodiments, with reference to FIGS. 3, 6, and 13, a method for adjusting a fully focused image WF_I to a non-perspective image NP_I includes adjusting a fully focused image WF_I to a non-perspective image NP_I in a perspective coordinate axis P_CA calculated from a depth image D_I. The method includes a step of estimating the coordinate data of the four corners C1 to C4 of WF_I, and a step of dragging the fully focused image WF_I to form a non-perspective image NP_I in the real world coordinate axis R_CA.

Specifically, the UE 100 estimates the coordinate data of the four corners C1 to C4 of the fully focused image WF_I in the perspective coordinate axis P_CA calculated from the depth image D_I, and then drags the fully focused image WF_I to the real world coordinate axis R_CA. A non-transparent image NP_I is formed. In detail, the UE 100 provides a transformation matrix from the perspective coordinate axis P_CA to the real world coordinate axis R_CA. The fully focused image WF_I is transformed into a non-perspective image NP_I by multiplying the data of the fully focused image WF_I by a transformation matrix. On the other hand, the UE 100 estimates the coordinate data of the four corners C1' to C4' of the fully focused image WF_I' in the perspective coordinate axis P_CA' calculated from the depth image D_I', and then drags the fully focused image WF_I' to execute the coordinate data. A non-perspective image NP_I' is formed on the world coordinate axis R_CA.

In some embodiments, referring to FIG. 14, processor 20 combines several non-perspective images NP_I into a single image S_I.

In detail, the processor 20 combines the non-transparent images NP_I, NP_I' into a single image S_I.

In some embodiments, referring to FIG. 15, the UE 100 further includes a display module 30, and the processor 20 sets a cropping candidate frame TC_F to the single image S_I displayed on the display module 30. In some embodiments, referring to FIG. 2, the image sensing module 10 includes a camera module 11 that senses a color image C_I and a depth sensing module 12 that senses a depth image D_I. In some embodiments, referring to FIG. 2, image sensing module 10 further includes an image processor 13 that controls camera module 11 and depth sensing module 12. In some embodiments, referring to FIG. 2, the camera module 11 includes a lens module 111, an image sensor 112, an image sensor driver 113, a focusing and optical image stabilization (OIS) driver 114, and a focusing and optical image stabilization (OIS) driver 114. , an OIS actuator 115 , and a gyro sensor 116 . The image sensor driver 113 controls the image sensor 112 to capture an image. A focus and OIS driver 114 controls a focus and OIS actuator 115 to focus the lens module 111 and move the lens module 111 to compensate for human hand shake. Gyro sensor 116 provides motion data to focus and OIS driver 114.

In some embodiments, referring to FIG. 2, depth sensing module 12 includes a projector 124, a lens 121, a range sensor 122, and a range sensor driver 123. The range sensor driver 123 controls the projector 124 to project dot matrix pulsed light, and controls the range sensor 122 to capture a reflected dot matrix image focused by the lens 121. In some embodiments, referring to FIG. 2, UE 100 further includes memory 50 for storing programs, image data, plane parameters, and transformation matrices. In some embodiments, depth image D_I includes point cloud data. In some embodiments, referring to FIG. 2, the UE 100 includes an input module 60 for accepting human instructions, a codec 70 for compressing and decompressing multimedia data, and a speaker 80 and a microphone 90 connected to the codec 70. , a wireless communication module 91 for transmitting and receiving messages, and a Global Navigation Satellite System (GNSS) module 92 for providing positioning information.

Also referring to FIG. 3, in some embodiments, the strabismus correction method includes, in block S100, capturing a color image C_I, an infrared (IR) image IR_I, and a depth image D_I by the image sensing module 10; S200, estimating plane parameters from the depth image D_I; block S300, calculating focal length data from the depth image D_I; and block S400, calculating these focal lengths from the focal length data by the image sensing module 10. A step of photographing a partially focused image PF_I in FD_1 to FD_4, and a step of cutting out focused image data from the partially focused image PF_I and combining these focused image data into a completely focused image WF_I in block S500. include.

In detail, referring to FIGS. 2 and 7, the processor 20 of the UE 100 controls the image sensing module 10 to capture, for example, a color image C_I, an infrared (IR) image IR_I, and a depth image D_I of the left side of the subject. Then, a color image C_I', an infrared (IR) image IR_I', and a depth image D_I' of the right side of the subject are photographed.

In some embodiments, with reference to FIGS. 4 and 8, estimating plane parameters from the depth image D_I in block S200 includes estimating the plane normal vector N_V from the depth image D_I in block S210. including.

Specifically, referring to FIG. 8, the UE 100 estimates a plane normal vector N_V from the depth image D_I, and estimates a plane normal vector N_V' from the depth image D_I'.

In some embodiments, with reference to FIGS. 2, 4, and 8, estimating plane parameters from the depth image in block S200 includes determining a perspective vertical coordinate axis PV_CA and a perspective horizontal coordinate axis PH_CA from data of IMU 40 in block S220. The method further includes the step of estimating.

Specifically, referring to FIG. 8, the UE 100 estimates the perspective vertical coordinate axis PV_CA and the perspective horizontal coordinate axis PH_CA of the depth image D_I from the data of the IMU 40, and estimates the perspective vertical coordinate axis PV_CA' and the perspective horizontal coordinate axis PV_CA' of the depth image D_I' from the data of the IMU 40. Estimate the horizontal coordinate axis PH_CA'.

In some embodiments, with reference to FIGS. 5 and 9, in block S300, calculating focal length data from the depth image includes, in block S310, calculating a region of depth of field corresponding to focal lengths FD_1 to FD_4. It includes determining several focal lengths FD_1 to FD_4 such that DF_A1 to DF_A4 overlap to cover the entire color image C_I.

Specifically, referring to FIGS. 9 and 10, the focal position F_1 of the UE 100 has a focal length FD_1 and a depth of field DF_1. The intersection area of depth of field DF_1 on the subject is depth of field area DF_A1. The depth of field area DF_A1 can be calculated from the data of the depth image D_I. On the other hand, the focal position F_2 of the UE 100 has a focal length FD_2 and a depth of field DF_2. The intersection area of depth of field DF_2 on the subject is depth of field area DF_A2. The focal position F_3 of the UE 100 has a focal length FD_3 and a depth of field DF_3. The intersection area of depth of field DF_3 on the subject is depth of field area DF_A3. The focal position F_4 of the UE 100 has a focal length FD_4 and a depth of field DF_4. The intersection area of depth of field DF_4 on the subject is depth of field area DF_A4.

Specifically, referring to FIGS. 5, 9, and 10, the UE 100 performs the following operations so that the depth of field regions DF_A1 to DF_A4 corresponding to the focal positions F_1 to F_4 overlap to cover the entire color image C_I. Determine several focal positions F_1 to F_4. The UE 100 sets several focal positions F_1' to F_4' so that the depth of field regions DF_A1' to DF_A4' corresponding to the focal positions F_1' to F_4' overlap and cover the entire color image C_I'. decide.

In some embodiments, calculating focal length data from the depth image in block S300 further comprises determining whether each of the depth of field regions DF_A1-DF_A4 has a texture in block S320. include.

In particular, with reference to FIGS. 3, 11 and 12, the processor 20 controls the image sensing module 10 to capture a partially focused image PF_I2 at a focal length FD_2 from the focal length data; A partially focused image PF_I3 is captured at a focal length FD_3, focused image data is cut from the partially focused image PF_I2, focused image data is cut from the partially focused image PF_I3, and these focused image data are converted into a completely focused image. Combine into WF_I. The processor 20 controls the image sensing module 10 to capture a partially focused image PF_I2', capture a partially focused image PF_I3', cut out focused image data from the partially focused image PF_I2', and perform a partially focused image PF_I2'. Focused image data is cut out from image PF_I3', and these focused image data are combined into a completely focused image WF_I'.

In some embodiments, referring to FIG. 3, the strabismus correction method further includes adjusting the fully focused image WF_I to a non-perspective image NP_I at block S600. In some embodiments, with reference to FIGS. 3, 6, and 13, adjusting the fully focused image WF_I to a non-perspective image NP_I in block S600 includes adjusting the fully focused image WF_I to a non-perspective image NP_I in block S610. estimating the coordinate data of the four corners C1 to C4 of the fully focused image WF_I on the perspective coordinate axis P_CA, and forming a non-perspective image NP_I on the real world coordinate axis R_CA by dragging the fully focused image WF_I in block S620. further including.

Specifically, the UE 100 estimates the coordinate data of the four corners C1 to C4 of the fully focused image WF_I in the perspective coordinate axis P_CA calculated from the depth image D_I, and then drags the fully focused image WF_I to the real world coordinate axis R_CA. A non-transparent image NP_I is formed. In detail, the UE 100 provides a transformation matrix from the perspective coordinate axis P_CA to the real world coordinate axis R_CA. The fully focused image WF_I is converted to a non-perspective image NP_I by multiplying the data of the fully focused image WF_I by a transformation matrix. On the other hand, the UE 100 estimates the coordinate data of the four corners C1' to C4' of the fully focused image WF_I' in the perspective coordinate axis P_CA' calculated from the depth image D_I', and then drags the fully focused image WF_I' to execute the coordinate data. A non-perspective image NP_I' is formed on the world coordinate axis R_CA. In some embodiments, referring to FIG. 14, the strabismus correction method further includes combining several non-perspective images NP_I into a single image S_I at block S700.

In detail, the processor 20 combines the non-transparent images NP_I, NP_I' into a single image S_I.

In some embodiments, referring to FIG. 15, the strabismus correction method further includes setting a cropping candidate frame TC_F in the single image S_I displayed on the display module 30 in block S800. In some embodiments, referring to FIG. 2, the image sensing module 10 includes a camera module 11 that senses a color image C_I and a depth sensing module 12 that senses a depth image D_I. In some embodiments, referring to FIG. 2, image sensing module 10 further includes an image processor 13 that controls camera module 11 and depth sensing module 12.

In some embodiments, referring to FIG. 2, the camera module 11 includes a lens module 111, an image sensor 112, an image sensor driver 113, a focusing and optical image stabilization (OIS) driver 114, and a focusing and optical image stabilization (OIS) driver 114. , an OIS actuator 115 , and a gyro sensor 116 . The image sensor driver 113 controls the image sensor 112 to capture an image. A focus and OIS driver 114 controls a focus and OIS actuator 115 to focus the lens module 111 and move the lens module 111 to compensate for human hand shake. Gyro sensor 116 provides motion data to focus and OIS driver 114. In some embodiments, referring to FIG. 2, depth sensing module 12 includes a projector 124, a lens 121, a range sensor 122, and a range sensor driver 123. The range sensor driver 123 controls the projector 124 to project dot matrix pulsed light, and controls the range sensor 122 to capture a reflected dot matrix image focused by the lens 121.

In some embodiments, referring to FIG. 2, the strabismus correction method further includes providing a memory 50 for storing programs, image data, plane parameters, and transformation matrices. In some embodiments, depth image D_I includes point cloud data. In some embodiments, referring to FIG. 2, the strabismus correction method includes an input module 60 for accepting human instructions, a codec 70 for compressing and decompressing multimedia data, and a speaker 80 and a microphone connected to the codec 70. 90, a wireless communication module 91 for transmitting and receiving messages, and a Global Navigation Satellite System (GNSS) module 92 for providing positioning information.

The advantages of the strabismus correction method include the following two points.
1. Provides a single perfectly focused image with no perspective distortion.
2. To provide a single image of a subject having a horizontal width larger than the width of a photographing area of a camera.

In embodiments of the present disclosure, a UE and strabismus correction method are provided. In order to provide a single focused image without perspective distortion, the image sensor communication method of the UE includes the steps of capturing a color image, an infrared image and a depth image by an image sensing module, and estimating plane parameters from the depth image. calculating focal length data from the depth image; capturing partially focused images at these focal lengths from the focal length data by an image sensing module; and cropping focused image data from the partially focused image. , combining these focused image data into a fully focused image.

It will be understood by those skilled in the art that each of the units, algorithms and steps described and disclosed in the embodiments of the present disclosure may be implemented using electronic hardware or a combination of computer software and electronic hardware. . Whether a function is performed by hardware or software depends on the application conditions and design requirements of the technical proposal. Those skilled in the art may use different methods to implement the functionality for each particular application, but such implementation shall not go beyond the scope of this disclosure. It will be understood by those skilled in the art that reference can be made to the operating processes of the systems, devices and units in the embodiments described above, since the operating processes of the systems, devices and units described above are basically the same. For convenience and brevity, these operational processes will not be described in detail. It is understood that the disclosed systems, apparatus, and methods in embodiments of the present disclosure can be implemented in other ways. The embodiments described above are exemplary. The division of units is based solely on logical function; other divisions are possible. Multiple units or components may be combined or integrated into other systems. Some features may be omitted or skipped. On the other hand, mutual, direct or communicative couplings shown or discussed may include several ports, devices or units, whether indirect or communicative in the form of electrical, mechanical or other types. Works through. Units described as separate components may or may not be physically separate. The display unit may or may not be a physical unit, that is, it may be located at one location or may be distributed over multiple network units. Some or all units may be used according to the purpose of the embodiment. Furthermore, each functional unit in each embodiment may be integrated into one processing unit, may be physically independent, or two or more units may be integrated into one processing unit. When a software functional unit is implemented, used and sold as a product, it can be stored on a computer readable storage medium. Based on this understanding, the technical solution proposed by this disclosure can be realized essentially or partially in the form of a software product. Alternatively, some of the technical solutions useful in the prior art can be implemented in the form of software products. A computer software product is stored on a storage medium that includes a plurality of instructions that cause a computing device (such as a personal computer, server, or network device) to perform all or some of the steps disclosed in embodiments of the present disclosure. The storage medium includes a USB disk, mobile hard disk, read only memory (ROM), random access memory (RAM), floppy disk, or other types of media that can store program codes.

This disclosure has been described in connection with what is considered to be the most practical and preferred embodiment. However, the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements that may be made without departing from the scope of the broadest interpretation of the appended claims. That is understood.

Claims (12)

an image sensing module;
a processor connected to the image sensing module, the user equipment comprising:
The processor includes:
controlling the image sensing module to take a number of mutually corresponding color images, infrared (IR) images and depth images at different photographing angles ;
estimating plane parameters from the depth image;
calculating focal length data from the same depth image in estimating plane parameters from the depth image ;
controlling the image sensing module to take partially focused images at these focal lengths from the focal length data;
Cutting out focused image data from the partially focused image and composing these focused image data into a fully focused image ;
adjusting the fully focused image to a non-fluoroscopic image;
User equipment, characterized in that it combines several said non-fluoroscopic images at different shooting angles into a single image . Adjusting the perfectly focused image to a non-fluorescent image includes estimating the coordinate data of the four corners of the perfectly focused image in the perspective coordinate axis calculated from the depth image, and dragging the perfectly focused image. and forming a non-perspective image in real-world coordinate axes . The user equipment according to claim 1 , wherein the user equipment further includes a display module, and wherein the processor sets a cropping candidate frame on the single image displayed on the display module. Estimating plane parameters from the depth image includes estimating a normal vector of the plane from the depth image,
The user equipment further includes an inertial measurement unit (IMU), and estimating plane parameters from the depth image further includes estimating a perspective vertical coordinate axis and a perspective horizontal coordinate axis from data of the IMU. , a user equipment according to claim 1. Calculating focal length data from the same depth image in estimating plane parameters from the depth image means that the depth of field regions corresponding to the focal lengths of the color image corresponding to the depth image overlap. including determining several focal lengths to cover the entire
Calculating focal length data from the same depth image in estimating plane parameters from the depth image determines whether each region of the depth of field in the color image has a texture of a subject . User equipment according to claim 1, further comprising: capturing several color images, infrared (IR) images and depth images corresponding to each other at different shooting angles by the image sensing module;
estimating plane parameters from the depth image;
calculating focal length data from the same depth image in the step of estimating plane parameters from the depth image ;
taking partially focused images at these focal lengths from the focal length data by the image sensing module;
cutting out focused image data from the partially focused image and combining the focused image data into a fully focused image ;
adjusting the fully focused image to a non-fluorescent image;
A method for correcting strabismus, comprising the step of combining several of the non-fluoroscopic images taken at different photographing angles into a single image . The step of adjusting the perfectly focused image to a non-perspective image includes the steps of estimating coordinate data of the four corners of the fully focused image in the perspective coordinate axis calculated from the depth image, and dragging the fully focused image. 7. The strabismus correction method according to claim 6 , further comprising the step of forming a non-perspective image in real-world coordinate axes. 7. The strabismus correction method according to claim 6 , further comprising the step of setting a cropping candidate frame in the single image displayed on the display module. 7. The strabismus correction method according to claim 6 , wherein the step of estimating plane parameters from the depth image includes the step of estimating a normal vector of a plane from the depth image. 7. The strabismus correction method according to claim 6 , wherein the step of estimating plane parameters from the depth image further includes the step of estimating perspective vertical coordinate axes and perspective horizontal coordinate axes from IMU data. In the step of estimating plane parameters from the depth image, the step of calculating focal length data from the same depth image includes the step of calculating the focal length data from the same depth image, in which the depth of field regions corresponding to the focal lengths of the color image corresponding to the depth image overlap. Strabismus correction method according to claim 6 , characterized in that it includes the step of determining several focal lengths so as to cover the whole. In the step of estimating plane parameters from the depth image, the step of calculating focal length data from the same depth image is a step of determining whether each region of the depth of field in the color image has the texture of the subject . The strabismus correction method according to claim 11 , further comprising:

Images