TW202420241A - Image processing method and electronic device - Google Patents

Abstract

An image processing method includes the following steps. A plurality of facial landmarks are analyzed from a face frame. A feature width is calculated according to the facial landmarks, and a head pose is analyzed according to the facial landmarks. The head pose is utilized to update the feature width to generate an updated width. A scale ratio of the updated width to an initial width is calculated. An object distance of a virtual camera is controlled according to the scale ratio. A two-dimensional image is captured from a virtual scene according to the object distance of the virtual camera.

Description

Image processing method and electronic device

This case relates to an image processing method, and more particularly to a virtual scene image processing method and an electronic device.

In today's environment, people are being asked to work remotely at a much higher rate. This has made many practitioners realize the benefits of remote collaboration. There are many reasons why collaborative settings are becoming popular, one of which is that collaborators no longer need to travel for hours or days to meet in person, they can communicate and collaborate reliably in the virtual world.

However, in some cases, collaborators must have specific equipment to effectively collaborate with others in the virtual world, such as a virtual reality head-mounted display that tracks head movements. These head-mounted displays are often paired with a set of joysticks equipped with tracking sensors.

Furthermore, virtual applications usually require quite powerful computing equipment to ensure comfort. Due to the above issues, there is still a big gap between collaborators joining from the real world and the virtual world.

Therefore, how to reduce the need for high-end equipment to analyze head posture and ensure that collaborators can work together in an easy and immersive collaborative system is an important issue in this field.

The present disclosure document provides an image processing method, which includes the following steps. Analyze multiple facial feature points of a facial frame. Calculate feature width based on the facial feature points, and analyze head pose based on the facial feature points. Update the feature width based on the head pose to generate an updated width. Calculate the scaling ratio of the updated width relative to the initial width. Control the shooting distance of a virtual camera in a virtual scene based on the scaling ratio. Sample a two-dimensional image from the virtual scene based on the shooting distance of the virtual camera.

The present disclosure document provides an electronic device. The electronic device includes an image sensor, a processor and a display. The image sensor is used to capture images. The processor is electrically coupled to the image sensor and the display, and the processor is used to perform the following steps. Analyze multiple facial feature points of a facial frame in an image. Calculate feature width based on the facial feature points, and analyze head posture based on the facial feature points. Update feature width based on the head posture to generate an updated width. Calculate the scaling ratio of the updated width relative to the initial width. Control the shooting distance of a virtual camera in a virtual scene based on the scaling ratio. Sample a two-dimensional image from the virtual scene based on the shooting distance of the virtual camera. The display is used to display two-dimensional images.

In summary, the image processing method and electronic device of the present invention controls the shooting distance of the virtual camera by analyzing the facial position in the image, and converts the three-dimensional scene into a two-dimensional image according to the field of view of the virtual camera, thereby providing users with an immersive experience in video conferencing, self-portrait images, landscape presentations or other interactive image processing technologies.

The following is a detailed description of the embodiments with the attached diagrams, but the embodiments provided are not intended to limit the scope of the disclosure, and the description of the structure operation is not intended to limit its execution order. Any device with equal functions produced by the re-combination of components is within the scope of the disclosure. In addition, the diagrams are for illustration purposes only and are not drawn according to the original size. For ease of understanding, the same or similar components in the following description will be marked with the same symbols.

The terms used throughout the specification and claims generally have the ordinary meanings of each term used in the art, in the context of this disclosure and in the specific context, unless otherwise specified.

In addition, the terms "include", "including", "have", "contain", etc. used in this article are open terms, which means "including but not limited to". In addition, "and/or" used in this article includes any one or more items in the relevant enumerated items and all combinations thereof.

In this article, when an element is referred to as "coupled" or "coupled", it may refer to "electrically coupled" or "electrically coupled". "Coupled" or "coupled" may also be used to indicate the coordinated operation or interaction between two or more elements. In addition, although the terms "first", "second", etc. are used in this article to describe different elements, the terms are only used to distinguish between elements or operations described with the same technical terms.

Please refer to FIG. 1 and FIG. 2. FIG. 1 is a schematic diagram of an electronic device 100 capturing an image IMG according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram of an electronic device 100 according to some embodiments of the present disclosure. In some embodiments, the electronic device 100 can be implemented by a personal computer, a tablet, a smart phone, or other electronic devices with image sensing and computing functions. In some embodiments, the electronic device 100 includes an image sensor 110, a processor 120, a memory device 130, and a display 140.

In some embodiments, the present invention utilizes widely deployed webcams to obtain video data from remote collaborators and associates the local user's head movements with the virtual world through the electronic device 100 to enhance the immersive experience.

To achieve this, we use a facial feature detection neural network model to estimate head pose from webcam images of the local user. The estimated head pose is linked to the motion of the virtual camera to reduce the gap between real-world and virtual events in real time, resulting in a more immersive user experience without the need for additional capture devices or image sensors.

In other words, a laptop, mobile phone, tablet, computer or other electronic device with computing, display and image sensing functions used to render a virtual world in a three-dimensional scene for collaboration can be used as a portal or window to the virtual world.

In some embodiments, the electronic device 100 of the present invention is used as an interface to interact with the virtual world, thereby reducing the need to wear a head device for sensing head posture, thereby realizing a three-dimensional scene viewing experience on the display of a general electronic device (e.g., a traditional laptop, mobile phone, computer) without the need to configure a head-mounted device (e.g., a head-mounted display) with virtual reality (Virtual Reality) or augmented reality (Augmented Reality) functions.

In some embodiments, the electronic device 100 of the present invention may also be used in conjunction with virtual reality or augmented reality technology. Therefore, the present invention is not limited thereto.

The processor 120 may be a central processing unit, a microprocessor, a graphics processor, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other hardware devices suitable for fetching or executing instructions stored in the memory device 130. The memory device 130 may be implemented by an electrical, magnetic, optical memory device, or other storage device for storing instructions or data.

The memory device 130 may be implemented by a volatile memory or a non-volatile memory. In some embodiments, the memory device 130 may be implemented by a random access memory (RAM), a dynamic random access memory (DRAM), a magnetoresistive random access memory (MRAM), a phase-change random access memory (PCRAM), or other storage devices. The memory device 130 is used to store data or instructions for the processor 120 to access and operate.

The image sensor 110 may be implemented by a CMOS (Complementary Metal Oxide Semiconductor; CMOS) image sensor, a CCD (Charge-Coupled Device; CCD) image sensor, or other light sensing components or light sensing devices. The image sensor 110 is electrically coupled to the processor 120 .

The image sensor 110 of the electronic device 100 is used to capture an image IMG.

At this time, if the user enters the field of view FOV of the image sensor 110, the electronic device 100 uses a neural network to detect a facial frame Ff from the image IMG captured by the image sensor 110, and the electronic device 100 analyzes the depth change of the user's face along the axis Ay and the displacement along the vertical plane perpendicular to the axis Ay in the image IMG, and adjusts the display image of the display 140 according to the depth change of the user's face and the displacement in the two-dimensional plane. How the processor 120 analyzes the depth change and displacement of the user's face and adjusts the display image of the display 140 will be described in detail in the subsequent embodiments.

Please refer to Figures 1 to 10B together. Figure 3 is a flow chart of an image processing method 200 illustrated in accordance with some embodiments of the present disclosure. Figure 4 is a schematic diagram of a convolutional neural network CNN illustrated in accordance with some embodiments of the present disclosure. Figure 5 is a schematic diagram of the specifications of the convolutional neural network CNN in Figure 4 illustrated in accordance with some embodiments of the present disclosure. Figure 6 is a schematic diagram of a user's head posture HP illustrated in accordance with some embodiments of the present disclosure. Figure 7 is a schematic diagram of facial feature points FPs in Figure 4 illustrated in accordance with some embodiments of the present disclosure. Figure 8 is a schematic diagram of a user's face center position _fc and feature width _fw illustrated in accordance with some embodiments of the present disclosure. FIG9A to FIG9B are schematic diagrams of a characteristic width _fw and an updated width _fw ' of a user's face according to some embodiments of the present disclosure. FIG10A and FIG10B are schematic diagrams of a virtual camera in a virtual scene according to some embodiments of the present disclosure.

As shown in FIG. 3 , the image processing method 200 includes steps S210 to S270 , wherein step S210 is performed by the image sensor 110 , step S270 is performed by the display 140 , and steps S220 to S260 are performed by the processor 120 .

In step S210, the image sensor 110 captures a two-dimensional image. The image sensor 110 of the electronic device 100 captures a two-dimensional image IMG, as shown in FIG. 1. At this time, if the user enters the field of view of the image sensor 110, the electronic device 100 detects a face frame Ff from the image IMG captured by the image sensor 110 using a neural network.

In step S220, the facial feature points FPs of the facial frame Ff in the image IMG are analyzed. In some embodiments, the processor 120 uses a facial feature neural network model CNN to analyze the facial feature points FPs of the facial frame Ff in the image IMG. In some embodiments, the facial feature neural network model CNN is implemented by a convolutional neural network architecture, and the convolutional neural network architecture includes 8 convolutional layers cov_1~cov_8, 2 fully connected layers dense_1~dense_2, and 4 pooling layers pool_1~pool_4. In some embodiments, the input of the convolutional neural network architecture is the image IMG. Furthermore, for a facial frame Ff, the convolutional neural network architecture outputs the (x, z) coordinates of 68 facial feature points FPs, as shown in FIGS. 4 , 5 and 7 .

In step S230, the feature width _fw , the center position _fc, and the head posture HP in the face frame Ff are analyzed based on the facial feature points FPs. In some embodiments, the center position _fc is implemented by the center point between the eyes, and the feature width _fw is implemented by the pupil distance. In other embodiments, the feature width _fw can be implemented by calculating the inner width of the eyes, the outer width of the eyes, the width of the palpebral fissure, the width of the face, the width of the mandible, the width of the mouth, or the width of the nose in the face frame Ff from other parts of the facial feature points FPs. Therefore, the present invention is not limited thereto.

For example, if the facial center position fc is implemented by the center between the eyes and the facial feature width _fw is implemented by the pupil distance, the processor 120 extracts the inner and outer corner positions _p43 and _p46 of the left eye and the inner and outer corner positions _p37 and _p40 of the right eye from the facial feature points FPs.

The processor 120 averages the inner and outer corner positions of the left eye p ₄₃ and p ₄₆ as the left eye position p _l , and the processor 120 averages the inner and outer corner positions of the right eye p ₃₇ and p ₄₀ as the right eye position p _r , as shown in the following formula.

The processor 120 averages the left eye position p _l and the right eye position _pr as the center position fc of the face, and the processor 120 calculates the difference between the left eye position p _l and the right eye position _pr as the feature width f _w , as shown in the following formula.

Furthermore, the processor 120 calculates the displacement _{df of the center position fc in} the two-dimensional plane and the scaling ratio _rf of the feature width _fw , as shown in the following formula.

In the above formula, f _c,0 represents the initial value of the center position, and f _w,0 represents the initial value of the face width.

Thus, if the user approaches the electronic device 100, the user's face will be magnified in the image IMG captured by the image sensor 110. On the other hand, if the user moves away from the electronic device 100, the user's face will be reduced in the image IMG captured by the image sensor 110. Therefore, the change in the depth/distance between the user and the electronic device 100 can be analyzed and determined through the scaling ratio _rf of the feature width _fw in the face frame Ff.

In some cases, if the user does not approach the electronic device 100, there is only a yaw of the head posture HP about the vertical axis (e.g., Az axis). At this time, since the user's face captured by the image sensor 110 is a profile, the characteristic distance fw calculated based on the distance between the two eyes will be reduced.

Therefore, in step S240, the feature width _fw is updated according to the head posture HP to generate an updated width _fw ', so as to use the updated width _fw ' to determine the change of the depth/distance between the user and the electronic device 100 with better accuracy.

Specifically, the processor 120 uses a neural network to analyze the facial direction FD of the user's head posture HP from the vector changes between the facial feature points FPs. Furthermore, the yaw of the user's head posture HP on the vertical axis is converted into an angle θ of the facial direction FD relative to the axial direction Ay on the horizontal plane, wherein the angle θ is in the range of 90 degrees to -90 degrees.

The processor 120 corrects/updates the feature width _fw in the face frame Ff according to the tilt angle θ of the user's head posture HP on the vertical axis (eg, Az axis) to generate an updated width _fw '. The updated width _fw ' can be expressed by the following formula.

Thus, the updated width _fw ' will be used in subsequent calculations to more accurately determine the distance from the electronic device 100 to the user along the axis Ay. The processor 120 calculates the scaling ratio of the updated width _fw ' relative to the initial width fw _,0 , which is represented by _rf '. The scaling ratio _rf ' is shown in the following formula.

In step S243, the shooting distance of the virtual camera VCA in the virtual scene is controlled according to the zoom ratio r _f '. If the zoom ratio r _f ' becomes larger (or greater than 1), it means that the user's face is relatively close to the electronic device 100. The processor 120 controls the virtual camera VCA to approach the fixed point VP, so that the field of view of the virtual camera VCA is reduced, so as to focus on a local area in the virtual scene VIR, as shown in FIG. 10A. On the other hand, if the zoom ratio r _f 'becomes smaller (or less than 1), it means that the user's face is relatively far away from the electronic device 100, and the processor 120 controls the virtual camera VCA to move away from the fixed point VP to expand the field of view of the virtual camera VCA, as shown in FIG. 10B.

It is worth noting that the virtual scene VIR can be implemented by a three-dimensional virtual scene. In other embodiments, the virtual scene VIR is implemented by a two-dimensional virtual scene. Therefore, the present invention is not limited thereto.

In some embodiments, the processor 120 may perform a difference calculation with smoothing on the scaling ratio r _f ′, and control the shooting distance of the virtual camera VCA in the virtual scene VIR according to the calculation result.

The lateral movement position and rotational posture of the virtual camera VCA are determined according to the center position f _c in the face frame Ff, which will be described in detail in steps S252 and S253.

For a better understanding, please refer to FIG. 1 to FIG. 3, FIG. 8, and FIG. 11A to FIG. 11C are schematic diagrams showing the center position _fc of the user's face relative to the displacement df of the electronic device 100 _mapped to the position and displacement of the virtual camera VCA in the virtual scene VIR according to some embodiments of the present disclosure.

In step S252, the displacement _df of the center position _fc relative to the initial position _fc,0 is calculated. In some embodiments, the setting of the initial position _fc,0 can be completed in the setting stage. For example, after the electronic device 100 enters the setting stage, it prompts the user to prepare to set the initial position _fc,0. After the user adjusts the distance between his face and the electronic device 100 to the most suitable distance, the electronic device 100 captures the current picture and calculates the center position of the face therein as the initial position _fc,0 , as shown in FIG. 11A.

In step S253, the viewing angle of the virtual camera VCA in the virtual scene VIR is controlled according to the displacement _df . The processor 120 moves the virtual camera VCA on the curved surface established based on the fixed point VP according to the displacement to adjust the viewing angle of the virtual camera VCA in the virtual scene VIR.

For example, if the center position _fc in the face frame Ff moves along the direction Dxp, the virtual camera VCA moves along the direction Dxp and rotates the viewing angle counterclockwise relative to the fixed point VP, thereby adjusting the viewing angle of the virtual camera VCA in the virtual scene VIR, as shown in FIG. 11B .

On the other hand, if the center position _fc in the face frame Ff moves along the direction Dxn, the virtual camera VCA moves along the direction Dxn and rotates the viewing angle clockwise relative to the fixed point VP, thereby adjusting the viewing angle of the virtual camera VCA in the virtual scene VIR, as shown in FIG. 11C .

It is worth noting that the processor 120 may perform a difference calculation with smoothing on the displacement _df of the center position _fc , and control the viewing angle of the virtual camera VCA in the virtual scene VIR according to the result of the calculation.

In the embodiments of FIG. 11B and FIG. 11C , the center position f _c of the face moves along the axis Ax. In some embodiments, the center position f _c of the face may move simultaneously in the vertical plane formed by the axes Ax and Az. The relationship between the displacement of the center position f _c of the face in the axis Az and the shooting angle of the virtual camera VCA is similar to the relationship between the displacement d _f of the center position f _c of the face in the axis Ax and the shooting angle of the virtual camera VCA, so it is not repeated.

In step S260, the virtual camera VCA samples a 2D image from the virtual scene VIR. The processor 120 samples a 2D image from the virtual scene VIR using a 3D-2D rendering engine that converts a 3D scene into a 2D image according to the shooting distance and viewing angle of the virtual camera VCA.

In step S270, the two-dimensional image sampled from the virtual scene VIR is displayed by the display 140. In some embodiments, when the field of view of the virtual camera VCA is reduced, the image captured by the virtual camera VCA can be enlarged to a size corresponding to the screen of the display 140. In this way, the user can pay more attention to the local area. In some embodiments, when the field of view of the virtual camera VCA is enlarged, the image captured by the virtual camera VCA can be reduced to a size corresponding to the screen of the display 140, so as to pay attention to the global area.

Please refer to FIG. 12A to FIG. 12C . FIG. 12A to FIG. 12C are schematic diagrams showing the movement of the user in the image IMG captured by the image sensor 110 and the image IMG_DIS of the display 140 according to some embodiments of the present disclosure.

As shown in FIG. 12A , when the user moves to the right in the real environment (since the front camera shoots and the image is mirrored, the user moves to the left in the image IMG), the virtual camera VCA moves to the right and rotates counterclockwise to capture the image in the left space.

As shown in FIG. 12B , when the user has no displacement relative to the initial position f _c,0 in the real environment, the virtual camera VCA captures the image of the central space.

As shown in FIG. 12C , when the user moves to the left in the real environment (since the front camera shoots the image and the user moves to the right in the image IMG), the virtual camera VCA moves to the left and rotates clockwise to capture the image in the right space.

In summary, the electronic device 100 and the image processing method 200 of the present invention control the shooting distance and the viewing angle of the virtual camera VCA by analyzing the facial position in the two-dimensional image, and convert the three-dimensional scene into a two-dimensional image according to the field of view of the virtual camera VCA, thereby providing the user with an immersive experience in video conferencing, self-portrait images, scene presentation or other interactive image processing technologies.

In order to make the above and other objects, features, advantages and embodiments of the present disclosure more clearly understandable, the attached symbols are described as follows: 100: electronic device 110: image sensor 120: processor 130: memory device 140: display 200: image processing method IMG, IMG_DIS: image Ff: face frame Ax, Ay, Az: axial FOV: field of view S210, S220, S230, S241, S242, S243, S252, S253, S260, S270: step CNN: facial feature neural network model FPs: facial feature point HP: head posture _p37 : right eye outer corner position _p40 : right eye inner corner position _p43 : left eye inner corner position _p46 : left eye outer corner position _pr : right eye position p _l : left eye position f _c : center position f _w : feature width d _r : displacement f _w ': updated width FD: face direction VP: fixed point VIR: virtual scene VCA: virtual camera Dxp, Dxn: direction

In order to make the above and other purposes, features, advantages and embodiments of the present disclosure more clearly understandable, the attached figures are described as follows: Figure 1 is a schematic diagram of an electronic device shooting an image according to some embodiments of the present disclosure. Figure 2 is a schematic diagram of an electronic device according to some embodiments of the present disclosure. Figure 3 is a flow chart of an image processing method according to some embodiments of the present disclosure. Figure 4 is a schematic diagram of a convolutional neural network according to some embodiments of the present disclosure. Figure 5 is a schematic diagram of the specifications of the convolutional neural network in Figure 4 according to some embodiments of the present disclosure. Figure 6 is a schematic diagram of a user's head posture according to some embodiments of the present disclosure. Figure 7 is a schematic diagram of facial feature points in Figure 4 according to some embodiments of the present disclosure. FIG. 8 is a schematic diagram of the center position and feature width of the user's face according to some embodiments of the present disclosure. FIG. 9A to FIG. 9B are schematic diagrams of the feature width and updated width of the user's face according to some embodiments of the present disclosure. FIG. 10A and FIG. 10B are schematic diagrams of a virtual camera in a virtual scene according to some embodiments of the present disclosure. FIG. 11A to FIG. 11C are schematic diagrams of the position and displacement of the virtual camera in the virtual scene relative to the displacement of the electronic device according to some embodiments of the present disclosure. FIG. 12A to FIG. 12C are schematic diagrams of the movement of the user in the image captured by the image sensor and the image of the display according to some embodiments of the present disclosure.

Domestic storage information (please note in the order of storage institution, date, and number) None Foreign storage information (please note in the order of storage country, institution, date, and number) None

100: Electronic devices

110: Image sensor

120: Processor

130: Memory device

140: Display

IMG: Image

Ff: Face frame

Claims (10)

An image processing method comprises: analyzing a plurality of facial feature points of a face frame; calculating a feature width based on the facial feature points, and analyzing a head pose based on the facial feature points; updating the feature width to generate an updated width based on the head pose; calculating a scaling ratio of the updated width relative to an initial width; controlling a shooting distance of a virtual camera in a virtual scene based on the scaling ratio; and sampling a two-dimensional image from the virtual scene based on the shooting distance of the virtual camera. The image processing method as described in claim 1, wherein the feature width is the inner width of the eyes, the pupil distance, the outer width of the eyes, the width of the palpebral fissure, the width of the face, the width of the mandible, the width of the mouth or the width of the nose in the face frame. The image processing method as described in claim 1 further comprises: Calculating a rotation angle of a facial orientation direction in the head posture relative to an axis on a horizontal plane; and Updating the feature width according to the rotation angle between the facial orientation direction and the axis to generate the updated width. The image processing method as described in claim 1 further comprises: According to the scaling ratio of the updated width relative to the initial width, the virtual camera is moved closer to or farther away from a fixed point to adjust the shooting distance of the virtual camera in the virtual scene. The image processing method as described in claim 1, wherein the step of adjusting the shooting distance of the virtual camera further comprises: If the zoom ratio decreases, the shooting distance of the virtual camera in the virtual scene is lengthened; or If the zoom ratio increases, the shooting distance of the virtual camera in the virtual scene is shortened. The image processing method as described in claim 1 comprises: Analyzing a center position between the facial feature points and the feature width; Calculating a displacement between the center position and an initial position; Based on the displacement, controlling the viewing angle of the virtual camera in the virtual scene; and Based on the viewing angle of the virtual camera and the shooting distance, sampling the two-dimensional image from the virtual scene. The image processing method as described in claim 6 further comprises: According to the displacement between the center position and the initial position, the virtual camera is moved on a curved surface relative to a fixed point to adjust the viewing angle of the virtual camera in the virtual scene. An image processing method as described in claim 6, wherein the center position is the center of the two eyes in the face frame, and wherein the feature width is the distance between the two eyes, wherein the control method further comprises: Extracting the inner and outer corner positions of the left eye and the inner and outer corner positions of the right eye from the facial feature points; Averaging the inner and outer corner positions of the left eye as a left eye position; Averaging the inner and outer corner positions of the right eye as a right eye position; and Averaging the left eye position and the right eye position as the center position of the two eyes. The image processing method as described in claim 8 further includes: Calculating the difference between the left eye position and the right eye position as the feature width. An electronic device includes: an image sensor for capturing an image; a processor electrically coupled to the image sensor for: analyzing a plurality of facial feature points of a face frame in the image; calculating a feature width based on the facial feature points, and analyzing a head posture based on the facial feature points; updating the feature width to generate an updated width based on the head posture; calculating a scaling ratio of the updated width relative to an initial width; controlling a shooting distance of a virtual camera in a virtual scene based on the scaling ratio; and sampling a two-dimensional image from the virtual scene based on the shooting distance of the virtual camera; and A display, electrically coupled to the processor, for displaying the two-dimensional image.

Images