CN117971035A

CN117971035A - Virtual mirror testing method and device based on 2D mapping

Info

Publication number: CN117971035A
Application number: CN202211351460.XA
Authority: CN
Inventors: 刘晓潇; 田羽慧; 韩道林; 李广琴; 马琳杰; 孙浩
Original assignee: Hisense Group Holding Co Ltd
Current assignee: Hisense Group Holding Co Ltd
Priority date: 2022-10-31
Filing date: 2022-10-31
Publication date: 2024-05-03

Abstract

The embodiment of the application provides a virtual mirror testing method and device based on 2D mapping, which relate to the technical field of image processing and comprise the following steps: acquiring coordinates of a facial key point from a facial image of a user; acquiring coordinates of key points of glasses from glasses images of glasses which are needed to be worn virtually by a user; determining a perspective change matrix and a rotation matrix according to the coordinates of the facial key points and the coordinates of the glasses key points, wherein the perspective change matrix is associated with a pitch angle and/or a yaw angle of the face of the user in the facial image, and the rotation matrix is associated with a rolling angle of the face of the user; updating coordinates of key points of the glasses according to the perspective change matrix and the rotation matrix; fusing the facial image and the glasses image according to the updated coordinates of the glasses key points; wherein, the glasses image is a 2D image. In the method, the virtual glasses try-on is realized based on the 2D mapping method, so that a complex flow of glasses modeling is omitted, and the operation and maintenance cost is reduced.

Description

Virtual mirror testing method and device based on 2D mapping

Technical Field

The application relates to the technical field of image processing, in particular to a virtual mirror testing method and device based on 2D mapping.

Background

At present, along with the development of economy and popularization of cultural education, people pay attention to the quality of life and pay more and more attention to the health care of eyes. Glasses have become "equipment" closely related to people's life and are becoming more and more important, because of being able to make up for the vision deficiency, and also being able to play a role in modifying the window of mind, expressing fashion personality and aesthetic orientation. And with the rapid development of the economy in China, the continuous improvement of national income and consumption level, the reduction of myopia, the increase of myopia population, the improvement of social aging degree and the enhancement of health care consciousness of domestic consumers, the promotion of the market of huge spectacles in China and great growth potential. At present, the number of people wearing glasses is up to 65%, and the competition of the glasses market is more and more vigorous due to the increase of the demands of people on the glasses.

Along with the development of virtual reality technology, virtual try-on glasses can be realized on line, and the time cost of going to off-line shops is saved. However, the prior art is a virtual try-on glasses implemented by a 3D modeling form, and the 3D modeling process is complex and expensive.

In view of the foregoing, there is a need for a method of virtual try-on glasses that can reduce costs.

Disclosure of Invention

The embodiment of the application provides a virtual test mirror method, a device, computing equipment and a computer readable storage medium based on 2D mapping, which meet the three-dimensional effect of virtual test belt glasses and reduce the cost.

In a first aspect, the present application provides a virtual mirror test method based on 2D mapping, the method comprising:

Acquiring coordinates of a facial key point from a facial image of a user; acquiring coordinates of key points of glasses from glasses images of glasses which are needed to be worn virtually by a user; determining a perspective change matrix and a rotation matrix according to the coordinates of the facial key points and the coordinates of the glasses key points, wherein the perspective change matrix is associated with a pitch angle and/or a yaw angle of the face of the user in the facial image, and the rotation matrix is associated with a rolling angle of the face of the user; updating coordinates of key points of the glasses according to the perspective change matrix and the rotation matrix; fusing the facial image and the glasses image according to the updated coordinates of the glasses key points; wherein, the glasses image is a 2D image.

According to the technical scheme, the coordinates of the key points of the face image and the coordinates of the key points of the glasses in the glasses image are obtained firstly, then the perspective change matrix and the rotation matrix are determined based on the obtained coordinates, then the coordinates of the key points of the glasses image are updated through the perspective change matrix and the rotation matrix, and finally the updated coordinates of the key points of the glasses image and the coordinates of the key points of the face image are fused to obtain the 2D virtual try-on chart. The three-dimensional effect of the virtual glasses try-on is achieved through the calculation mode of the 2D picture coordinate matrix, and the operation and maintenance cost is effectively reduced.

In one possible design, the facial key points include key points corresponding to left outer corners of eyes, right outer corners of eyes and nose bridges of the user respectively; the glasses key points comprise glasses frame key points, wherein the glasses frame key points comprise key points corresponding to four corners of the glasses frame; determining a perspective change matrix according to the coordinates of the facial key points and the coordinates of the glasses key points, wherein the method comprises the following steps of: determining the offset according to coordinates of key points corresponding to the left outer canthus, the right outer canthus and the nose bridge of the user respectively; determining coordinates of key points corresponding to the four corners of the glasses frame after offset according to the offset and the coordinates of key points corresponding to the four corners of the glasses frame respectively; and determining a perspective change matrix corresponding to the glasses frame according to coordinates of key points corresponding to the four corners of the glasses frame after the offset.

In the technical scheme, the key points of the glasses image comprise key points corresponding to four corners of the glasses frame respectively; the key points of the face comprise key points corresponding to the left outer canthus, the right outer canthus and the nose bridge respectively. And determining the coordinate offset based on the key points of the face, and finally determining the perspective change matrix according to the coordinates of the key points corresponding to the shifted glasses frame, wherein each step is obtained based on coordinate calculation, thereby being beneficial to improving the accuracy of perspective change.

In one possible design, the facial key points include key points corresponding to left outer corners and right outer corners of the user respectively; the eyeglass keypoints comprise the center keypoints of the eyeglass frame; determining a rotation matrix according to coordinates of the facial key points and coordinates of the glasses key points, and determining a rotation angle according to coordinates of the key points corresponding to the left outer eye corners and the right outer eye corners of the user respectively; the rotation matrix is determined based on the rotation angle and the center key point of the eyeglass frame.

In the above technical solution, the rotation angle is determined according to the coordinates of the key points corresponding to the left outer corner of the eye and the right outer corner of the eye of the user, and the rotation matrix is determined based on the rotation angle and the key point of the center of the orbit. The determination of the rotation matrix is beneficial to the fact that when the user performs angle transformation in operations such as turning, the glasses can also be transformed along with the face of the user, so that the effect of the virtual test mirror is more stereoscopic and real.

In one possible design, updating coordinates of the eyeglass frame keypoints according to the perspective change matrix and the rotation matrix includes: performing perspective change on the coordinates of the glasses frame in the glasses key points according to the perspective change matrix to obtain the coordinates of the glasses frame key points after the perspective change; and rotating the coordinates of the spectacle frame key points after the perspective change according to the rotation matrix to obtain the coordinates of the spectacle frame key points after the rotation.

In the above technical scheme, after the key points of the glasses frame pass through the perspective change matrix and the rotation matrix, the coordinates of the key points of the glasses frame can be changed, the key points of the glasses frame after the perspective change matrix and the rotation matrix need to be updated, and the key points of the glasses frame obtained after updating can be more matched with the face of the user.

In one possible design, the facial key points include the left outer corner of the eyes and the right outer corner of the eyes of the user; the glasses key points comprise key points corresponding to two sides of the glasses frame respectively; before rotating the coordinates of the key points of the glasses after the perspective change according to the rotation matrix, the method further comprises: determining the width of the face of the user according to the left outer canthus and the right outer canthus of the user; determining the width of the glasses according to the key points corresponding to the two sides of the glasses frame; determining a scaling ratio according to the width of the face of the user and the width of the glasses; according to the scaling, the size of the glasses image is adjusted so that the glasses image is matched with the face image; according to the rotation matrix, rotating coordinates of the key points of the glasses frame after perspective change, including: acquiring coordinates of key points of the glasses after the sizes of the glasses images are adjusted; and rotating coordinates of the key points of the glasses frame after the size of the glasses image is adjusted according to the rotation matrix.

In the technical scheme, the scaling of the glasses is determined by determining the width of the face of the user and the width of the glasses, and the size of the glasses image is determined after the scaling is determined, so that the matching effect of the glasses image and the face image is better; after the scaling of the glasses frame is determined, the coordinates of key points of the glasses frame need to be updated based on the rotation matrix, so that the glasses frame moves along with the rotation of the face, and the tracking effect of the glasses is better.

In one possible design, the method further comprises, after obtaining the coordinates of the pivotal points of the eyeglass frame after rotation:

Determining a perspective change matrix corresponding to the glasses legs according to the coordinates of the facial key points, the coordinates of the glasses leg key points in the glasses key points and the perspective change matrix corresponding to the glasses frame;

and performing perspective change on coordinates of the key points of the glasses legs according to the perspective change matrix corresponding to the glasses legs, and obtaining the coordinates of the key points of the glasses legs after the perspective change.

According to the technical scheme, after the updated key point coordinates of the glasses frame are obtained after rotation, the perspective change matrix corresponding to the glasses legs is determined according to the coordinates of the facial key points, the coordinates of the glasses leg key points in the glasses key points and the perspective change matrix corresponding to the glasses frame, and then the coordinates of the glasses leg key points after perspective change can be obtained based on the perspective change matrix corresponding to the glasses legs, so that the position relationship between the glasses legs and the glasses frame can be effectively connected, the position of the glasses legs can be changed continuously according to the position of the glasses frame, and the three-dimensional effect presentation is facilitated.

In one possible design, the facial key points comprise key points corresponding to left ears and right ears respectively; according to the coordinates of the facial key points, the coordinates of the glasses leg key points in the glasses key points and the perspective change matrix corresponding to the glasses frame, determining the perspective change matrix corresponding to the glasses leg comprises the following steps: acquiring coordinates of key points corresponding to two sides of the rotated glasses frame respectively; determining perspective transformation coordinates according to coordinates of key points corresponding to two sides of the rotated glasses frame and coordinates of key points corresponding to the left ear and the right ear respectively; and determining a perspective transformation matrix corresponding to the glasses legs according to the perspective change coordinates.

In the above technical solution, how to determine the perspective change matrix of the glasses leg is specifically introduced. The method mainly comprises the step of determining a perspective change matrix corresponding to the glasses legs according to coordinates in the face key points, coordinates of the glasses leg key points, the perspective change matrix corresponding to the glasses frame and coordinates of the glasses leg key points. Wherein the coordinates of the temples are updated coordinates after rotation. The method that coordinates of glasses legs reach perspective effect is determined according to the updated left side of the glasses frame, so that the stereoscopic effect of the glasses is improved, and the glasses are more realistic.

In one possible design, fusing the face image and the eyeglass image according to coordinates of key points of the eyeglass frame after updating includes: determining a facial region of interest from the facial image according to the updated coordinates of the spectacle frame key points; acquiring single-channel transparency of the glasses image, and acquiring multi-channel transparency of the glasses image and multi-channel transparency of a region of interest of the face according to the single-channel transparency of the glasses image; and fusing the face image and the glasses image according to the multi-channel transparency of the glasses image and the multi-channel transparency of the region of interest of the face to obtain the 2D virtual try-on map.

According to the technical scheme, the updated coordinates of the key points of the glasses frame are updated after perspective, rotation and scaling, the interested region in the face image is obtained based on the updated coordinates, and the multichannel transparency of the glasses image and the multichannel transparency of the interested region of the face are obtained according to the single-channel transparency of the glasses image, so that the basis is provided for fusion of the subsequent glasses image and the face image. The calculation of the transparency ensures the integration of the facial image and the eyeglass image, and allows light to pass through the eyeglass.

In one possible design, the merging of the face image and the glasses image according to the multi-channel transparency of the glasses image and the multi-channel transparency of the region of interest of the face comprises: determining a product of the multi-channel transparency of the eyeglass image and the foreground color level; determining the product of the multi-channel transparency of the region of interest of the face and the background color level; and fusing the face image and the glasses image according to the product of the multichannel transparency and the foreground color level of the glasses image and the product of the multichannel transparency and the background color level of the face region of interest.

According to the technical scheme, the multi-channel transparency of the glasses image and the multi-channel transparency of the region of interest of the face are used for calculation, and the image fusion is achieved mainly through adding the product of the multi-channel transparency of the face image and the foreground color level and the product of the multi-channel transparency of the glasses image and the foreground color level, so that the original image is replaced by the fixed position.

In one possible design, determining a facial region of interest from a facial image based on coordinates of updated eyeglass keypoints, comprises: taking the coordinates of the nose bridge of the user in the facial key points as the coordinates of the central points of the glasses; determining the size of the glasses image according to the updated coordinates of the glasses key points; a facial region of interest is determined from the facial image based on the size of the eyeglass image and the eyeglass center point coordinates.

According to the technical scheme, the area of interest of the face is determined mainly through the size of the glasses image and the coordinates of the center points of the glasses, the area of interest of the face can be actually understood to be the position area of the face, and the confirmation based on the key points is more accurate.

In a second aspect, the present application provides a virtual mirror device based on 2D mapping, the device comprising:

an acquisition unit configured to acquire coordinates of a face key point from a face image of a user;

The acquisition unit is also used for acquiring coordinates of key points of the glasses from the glasses images of the glasses which the user needs to wear virtually;

A determining unit, configured to determine a perspective change matrix and a rotation matrix according to coordinates of the facial key points and coordinates of the glasses key points, where the perspective change matrix is associated with a pitch angle and/or a yaw angle of a face of the user in the facial image, and the rotation matrix is associated with a roll angle of the face of the user;

The updating unit is used for updating the coordinates of the key points of the glasses according to the perspective change matrix and the rotation matrix;

The fusion unit is used for fusing the facial image and the glasses image according to the updated coordinates of the glasses key points; wherein, the glasses image is a 2D image.

In one possible design, the determining unit is specifically configured to determine the offset according to coordinates of key points corresponding to the left outer corner of the eye, the right outer corner of the eye, and the bridge of the nose of the user; determining coordinates of key points corresponding to the four corners of the glasses frame after offset according to the offset and the coordinates of key points corresponding to the four corners of the glasses frame respectively; and determining a perspective change matrix corresponding to the glasses frame according to coordinates of key points corresponding to the four corners of the glasses frame after the offset.

In one possible design, the determining unit is specifically configured to determine the rotation angle according to coordinates of key points corresponding to the left outer corner of the eye and the right outer corner of the eye of the user; the rotation matrix is determined based on the rotation angle and the center key point of the eyeglass frame.

In one possible design, the updating unit is specifically configured to perform perspective change on coordinates of a lens frame key point in the lens key points according to the perspective change matrix, so as to obtain coordinates of the lens frame key point after the perspective change; and rotating the coordinates of the spectacle frame key points after the perspective change according to the rotation matrix to obtain the coordinates of the spectacle frame key points after the rotation.

In a possible design, the determining unit is further configured to determine a width of the face of the user based on the left outer corner of the eye and the right outer corner of the eye of the user; determining the width of the glasses according to the key points corresponding to the two sides of the glasses frame; determining a scaling ratio according to the width of the face of the user and the width of the glasses; the updating unit is also used for adjusting the size of the glasses image according to the scaling so as to enable the glasses image to be matched with the face image; the acquisition unit is specifically used for acquiring coordinates of key points of the glasses after the size of the glasses image is adjusted; and rotating coordinates of the key points of the glasses frame after the size of the glasses image is adjusted according to the rotation matrix.

In one possible design, the determining unit is specifically further configured to determine a perspective change matrix corresponding to the glasses leg according to coordinates of the face key points, coordinates of the glasses leg key points in the glasses key points, and a perspective change matrix corresponding to the glasses frame; and performing perspective change on coordinates of the key points of the glasses legs according to the perspective change matrix corresponding to the glasses legs, and obtaining the coordinates of the key points of the glasses legs after the perspective change.

In one possible design, the determining unit is specifically configured to include key points corresponding to the left ear and the right ear in the face key points; according to the coordinates of the facial key points, the coordinates of the glasses leg key points in the glasses key points and the perspective change matrix corresponding to the glasses frame, determining the perspective change matrix corresponding to the glasses leg comprises the following steps: acquiring coordinates of key points corresponding to two sides of the rotated glasses frame respectively; determining perspective transformation coordinates according to coordinates of key points corresponding to two sides of the rotated glasses frame and coordinates of key points corresponding to the left ear and the right ear respectively; and determining a perspective transformation matrix corresponding to the glasses legs according to the perspective change coordinates.

In one possible design, the fusion unit is specifically configured to determine a region of interest of the face from the face image according to coordinates of the updated key points of the glasses; acquiring single-channel transparency of the glasses image, and acquiring multi-channel transparency of the glasses image and multi-channel transparency of a region of interest of the face according to the single-channel transparency of the glasses image; and fusing the face image and the glasses image according to the multi-channel transparency of the glasses image and the multi-channel transparency of the region of interest of the face to obtain the 2D virtual try-on map.

In one possible design, the fusion unit is specifically configured to determine a product of the multi-channel transparency of the eyeglass image and the foreground color level; determining the product of the multi-channel transparency of the region of interest of the face and the background color level; and fusing the face image and the glasses image according to the product of the multichannel transparency and the foreground color level of the glasses image and the product of the multichannel transparency and the background color level of the face region of interest.

In a possible design, the determining unit is specifically configured to use the coordinates of the bridge of the nose of the user in the facial key points as glasses center point coordinates; determining the size of an eyeglass image according to the updated coordinates of the eyeglass key points; a facial region of interest is determined from the facial image based on the size of the eyeglass image and the eyeglass center point coordinates.

The beneficial effects of the virtual mirror device based on 2D mapping provided in the second aspect may be referred to as beneficial effects brought by embodiments of each possible design of the first aspect, which are not described herein.

In a third aspect, the present application also provides a computing device comprising: a memory and a processor; the memory is used for storing program instructions; the processor is configured to invoke program instructions in the memory to cause the electronic device to perform the method of any of the possible designs of the first aspect described above.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computing device to perform a method of any one of the possible designs of the first aspect described above when the program is run on the computing device.

In addition, the technical effects caused by any implementation manner of the third aspect to the fourth aspect may refer to the technical effects caused by different implementation manners of the first aspect, which are not described herein.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1a is a schematic view of a scene provided in an embodiment of the present application;

FIG. 1b is a schematic diagram of a mobile phone to which the present application is applicable;

FIG. 1c is a block diagram of a software architecture of an electronic device according to an embodiment of the present application;

FIG. 2 is a flow chart of a virtual mirror test method based on a 2D map according to an embodiment of the present application;

FIG. 3a is a schematic view of a rotation angle according to an embodiment of the present application;

FIG. 3b is a schematic perspective view of a pair of virtual glasses based on a 2D map according to an embodiment of the present application;

FIG. 4 is a flowchart of a perspective matrix determining method according to an embodiment of the present application;

FIG. 5 is a flowchart of a scaling of an eyeglass image according to an embodiment of the present application;

FIG. 6 is a flow chart of a face image and eyeglass image fusion according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a virtual mirror device according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a computing device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

At present, along with the development of technology, people pay attention to the quality of life and pay more attention to the health care of eyes. Glasses have become "equipment" closely related to people's life and are becoming more and more important, because of being able to make up for the vision deficiency, and also being able to play a role in modifying the window of mind, expressing fashion personality and aesthetic orientation. According to incomplete statistics, 65% of users can not leave the glasses at present. The eye technology combines with the virtual reality technology to bring a virtual glasses test strip method, but the manufacturing cost of the current virtual glasses test strip method is higher.

Therefore, in view of the above problems, the present application provides a new virtual mirror test method for satisfying the stereoscopic effect of virtual test strip glasses while reducing the cost.

Referring to fig. 1a, fig. 1a is a schematic view of a scenario provided in an embodiment of the present application, where the scenario includes at least an electronic device 101 and a user 102.

The electronic device 101 is configured to acquire a facial image key point of the user 102 and fuse the facial image key point with the acquired glasses image key point in the electronic device 101, so as to achieve a virtual try-on effect.

Specifically, the electronic device 101 obtains, from the cloud (or the server), a glasses image of glasses that the user needs to wear virtually, and extracts a glasses image information key point from the glasses image. The electronic device 101 photographs the face of the user 102 to obtain a face image, and further extracts the face image information key points from the face image. The electronic device 101 updates the glasses image information key points through perspective, rotation, scaling and other processes on the glasses image information key points according to the face image information key points, and fuses the updated glasses image information key points with the face image information key points to obtain the 2D virtual glasses wearing view.

For example, the electronic device 101 is a mobile phone, and a schematic structure of the mobile phone is shown in fig. 1 b.

As shown in fig. 1B, the cellular phone may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (un IVERSA L SER IA L bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identity module (subscr iber IDENT IFICAT ion modu le, SIM) card interface 195, and the like.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (app l icat ion processor, AP), a modem processor, a graphics processor (graph ics process ing un it, GPU), an image signal processor (IMAGE S IGNA L processor, ISP), a controller, a memory, a video codec, a digital signal processor (D IGITA L S IGNA L processor, DSP), a baseband processor, and/or a neural network processor (neura l-network process ing un it, NPU), among others. Wherein the different processing units may be separate devices or may be integrated in one or more processors. The controller can be a neural center and a command center of the mobile phone. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution. A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses may be avoided and the latency of the processor 110 may be reduced, thereby improving the efficiency of the system.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect to a charger to charge the phone, or may be used to transfer data between the phone and peripheral devices. The charge management module 140 is configured to receive a charge input from a charger. The power management module 141 is configured to be connected to the battery 142, and the power management module 141 receives input from the battery 142 and/or the charge management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like.

The wireless communication function of the mobile phone can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor and the like. The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the handset may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc. applied to a cell phone. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noi SE AMP L IFIER, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (Wi reless loca l area networks, WLAN) applied to a cell phone (e.g., wireless fidelity (WI RE LESS FIDE L ITY, wi-Fi) network), bluetooth (b l uetooth, BT), global navigation satellite system (globa l navigat ion SATE L L ITE SYSTEM, GNSS), frequency modulation (frequency modu l at ion, FM), near field communication (NEAR FIE LD communication), infrared technology (I R), etc. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, the antenna 1 and the mobile communication module 150 of the handset are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the handset can communicate with a network and other devices through wireless communication technology. The wireless communication techniques may include the Global System for Mobile communications (globa l system for mobi le commun icat ions, GSM), general packet radio service (GENERA L PACKET RAD io service, GPRS), code division multiple access (code d ivi s ion mu LT IP LE ACCESS, CDMA), wideband code division multiple access (wideband code d ivi s ion mu LT IP LE ACCESS, WCDMA), time division code division multiple access (t time-d ivi s ion code d ivi s ion mu LT IP LE ACCESS, TD-SCDMA), long term evolution (long term evo l ut ion, LTE), BT, GNSS, WLAN, NFC, FM, and/or I R techniques, among others. The GNSS may include a global satellite positioning system (globa l pos it ion ING SYSTEM, GPS), a global navigation satellite system (globa lnavigat ion SATE L L ITE SYSTEM, GLONASS), a beidou satellite navigation system (beidou navigat ion SATE L L ITE SYSTEM, BDS), a quasi zenith satellite system (quas i-zen ITH SATE L L ITE SYSTEM, QZSS) and/or a satellite based augmentation system (SATE L L ITE based augmentat ion systems, SBAS).

The display 194 is used to display a display interface of an application or the like. The display 194 includes a display panel. The display panel may employ a liquid crystal display (l iqu ID CRYSTA L D I SP L AY, LCD), an organic light emitting diode (organ ic l ight-EMITT ING D iode, OLED), an active matrix organic light emitting diode or active matrix organic light emitting diode (act ive-matr ix organ IC L IGHT EMITT ING D iode, AMOLED), a flexible light emitting diode (f lex l ight-EMITT ING D iode, FLED), mini led, microLed, micro-oLed, quantum dot L IGHT EMITT ING D iodes, QLED, or the like. In some embodiments, the handset may include 1 or N display screens 194, N being a positive integer greater than 1. As can be seen from the foregoing examples, the display 194 in this embodiment of the present application may be a curved screen (i.e. the screen may cover a side portion of the mobile phone, such as the left and rear side portions of the mobile phone, as well as the main screen), or a folded screen (i.e. the display may be unfolded or folded together according to the use requirement).

The camera 193 is used to capture still images or video. In some embodiments, camera 193 may include at least one camera, such as a front camera and a rear camera.

The internal memory 121 may be used to store computer-executable program code that includes instructions. The processor 110 executes various functional applications of the cellular phone and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an operating system, and software code of at least one application program (e.g., an aiqi application, a WeChat application, etc.), etc. The data storage area may store data (e.g., images, video, etc.) generated during use of the handset, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), etc.

The external memory interface 120 may be used to connect to an external memory card, such as a Micro SD card, to extend the memory capabilities of the handset. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as pictures and videos are stored in an external memory card.

The handset may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playing, recording, etc.

The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The gyro sensor 180B may be used to determine the motion gesture of the handset. In some embodiments, the angular velocity of the handset about three coordinate axes (i.e., x-axis, y-axis, and z-axis) may be determined by gyro sensor 180B.

The gyro sensor 180B may be used for photographing anti-shake. The air pressure sensor 180C is used to measure air pressure. In some embodiments, the handset calculates altitude from the barometric pressure value measured by barometric pressure sensor 180C, aiding in positioning and navigation. The magnetic sensor 180D includes a hall sensor. The mobile phone can detect the opening and closing of the flip leather sheath by using the magnetic sensor 180D. In some embodiments, when the handset is a flip phone, the handset may detect the opening and closing of the flip based on the magnetic sensor 180D. And then according to the detected opening and closing state of the leather sheath or the opening and closing state of the flip, the characteristics of automatic unlocking of the flip and the like are set. The acceleration sensor 180E can detect the magnitude of acceleration of the mobile phone in various directions (typically three coordinate axes). The magnitude and direction of gravity can be detected when the mobile phone is stationary. The electronic equipment gesture recognition method can also be used for recognizing the gesture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

A distance sensor 180F for measuring a distance. The cell phone may measure distance by infrared or laser. In some embodiments, the cell phone may range using the distance sensor 180F to achieve fast focus. The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone emits infrared light outwards through the light emitting diode. The mobile phone uses a photodiode to detect infrared reflected light from nearby objects, and when sufficient reflected light is detected, it can be determined that objects are nearby the mobile phone; when insufficient reflected light is detected, the handset may determine that there is no object in the vicinity of the handset. The mobile phone can use the proximity light sensor 180G to detect that the user holds the mobile phone close to the ear for communication, so as to automatically extinguish the screen to achieve the purpose of saving electricity. The proximity light sensor 180G may also be used in holster mode, pocket mode to automatically unlock and lock the screen.

The ambient light sensor 180L is used to sense ambient light level. The cell phone can adaptively adjust the brightness of the display 194 according to the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust white balance when taking a photograph. The ambient light sensor 180L may also cooperate with the proximity light sensor 180G to detect if the phone is in a pocket to prevent false touches. The fingerprint sensor 180H is used to collect a fingerprint. The mobile phone can realize fingerprint unlocking, access to an application lock, fingerprint photographing, fingerprint incoming call answering and the like by utilizing the collected fingerprint characteristics.

The temperature sensor 180J is for detecting temperature. In some embodiments, the handset performs a temperature processing strategy using the temperature detected by temperature sensor 180J. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, the handset performs a reduction in the performance of a processor located near temperature sensor 180J in order to reduce power consumption to implement thermal protection. In other embodiments, when the temperature is below another threshold, the cell phone heats the battery 142 to avoid the low temperature causing the cell phone to be abnormally shut down. In other embodiments, when the temperature is below a further threshold, the handset performs boosting of the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperatures.

The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may be disposed on the surface of the mobile phone at a different location than the display 194.

The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, bone conduction sensor 180M may acquire a vibration signal of a human vocal tract vibrating bone pieces. The bone conduction sensor 180M may also contact the pulse of the human body to receive the blood pressure pulsation signal.

The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The handset may receive key inputs, generating key signal inputs related to user settings and function controls of the handset. The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc. The SIM card interface 195 is used to connect a SIM card. The SIM card may be contacted and separated from the handset by inserting into the SIM card interface 195 or extracting from the SIM card interface 195.

It will be appreciated that the components shown in fig. 1b are not to be construed as a specific limitation on the cell phone, and that the cell phone may include more or fewer components than shown in fig. 1b, or may combine certain components, or split certain components, or may have a different arrangement of components. In the following embodiments, a mobile phone shown in fig. 1b is taken as an example to describe the present invention.

Fig. 1c is a block diagram of a software structure of an electronic device according to the present application. As shown in fig. 1c, the software structure of the electronic device may be a hierarchical architecture, for example, the software may be divided into several layers, each layer having a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer (may be simply referred to as an "application layer"), an application framework layer (FWK), a An Zhuoyun line (Android runt ime) and a system library layer (may also be described as a system runtime layer), and a kernel layer.

At least one application program is operated in the application program layer, and the application programs can be a Window program (Window) of an operating system, a system setting program, a contact program, a short message program, a clock program, a camera program and the like; applications developed by third party developers, such as instant messaging programs (e.g., weChat, QQ, etc.), photo beautification programs (e.g., graphic shows, beauty cameras, etc.), game programs, etc., are also possible. Of course, in the implementation, the application packages in the application layer are not limited to the above examples, and may actually include other application packages, which are not limited by the embodiment of the present application.

The application framework layer provides an application programming interface (app l icat ion programming interface, API) and programming framework for the application of the application layer. The application framework layer may include some predefined functions. The application framework layer corresponds to a processing center that determines to have the application in the application layer perform the corresponding operation. As shown in FIG. 1c, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, an activity manager, and the like. Of course, the application framework layer may also include other devices in implementations, and embodiments of the present application are not limited in this regard.

Wherein the window manager is used for managing window programs. The window manager can acquire the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

Acquire data and make the data accessible to applications. The data may be used by content providers to store and receive calls including video, images, audio, make and receive calls, browse history and bookmarks, phone books, and the like.

The view system may include visual controls, such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view displaying text and a view displaying a picture.

The telephony manager may be used to provide communication functions for the handset. Such as the management of call status (including on, hung-up, etc.).

The resource manager provides various resources for the application program, such as localization strings, icons, pictures, layout files, video files, and the like.

The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction. Such as notification manager is used to inform that the download is complete, message alerts, etc. The notification manager may also be a notification in the form of a chart or scroll bar text that appears on the system top status bar, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in a status bar, a prompt tone is emitted, the electronic device vibrates, and an indicator light blinks, etc.

The activity manager may be used to manage lifecycle of the respective applications as well as general navigation rollback functions, such as controlling exit of the applications (including switching a currently displayed user interface in the display screen to the system desktop), opening or backing (including switching a currently displayed user interface in the display screen to a previous user interface of the currently displayed user interface), etc.

Android runt ime include core libraries and virtual machines. Android runt ime is responsible for scheduling and management of the android system.

The core library consists of two parts: one part is a function which needs to be called by java language, and the other part is a core library of android. The application layer and the application framework layer run in a virtual machine. The virtual machine executes java files of the application program layer and the application program framework layer as binary files. The virtual machine is used for executing the functions of object life cycle management, stack management, thread management, security and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: a surface manager (surface manager), a media library (med ia l ibrar ies), a three-dimensional graphics processing library (e.g., openGL ES), a 2D graphics engine (e.g., SGL), etc. The surface manager is used for managing the display subsystem and providing fusion of 2D and 3D layers for a plurality of application programs. Media libraries support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio and video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing. In addition, the system library can also comprise a form detection module used for identifying the physical form of the electronic equipment. For example, the form detection module may be configured to determine a physical form of the electronic device according to sensor data uploaded by various sensors in the hardware layer. The physical form may include a folded form, an unfolded form, a semi-folded form unfolded at an angle, and the like. The system library may further include a coordinate mapping module configured to record a first position coordinate of a first display area/control of the side screen and a second position coordinate of a second display area/control on the main screen/sub-screen, and construct a mapping relationship between the first position coordinate and the second position coordinate.

Based on the schematic diagram of fig. 1a, a specific method flowchart is shown in fig. 2, and fig. 2 is a flowchart of a virtual mirror test method based on 2D mapping according to an embodiment of the present application, where steps in the flowchart may be specifically executed by the electronic device 101 in fig. 1 a. The process comprises the following steps:

S201, acquiring coordinates of facial key points from a facial image of a user.

In one specific implementation, a color camera in the computing device acquires a face image of a user in real time by using the color camera, and further acquires coordinates of facial key points from the face image of the user, where the acquiring key points may specifically be acquiring 468-point 3D facial key points by using a media pipeline application (MEIDAPIPE).

The facial key points include left lateral corner coordinates (eye_left_point) and right lateral corner coordinates (eye_ r ight _point), left lateral root coordinates (face_left) and right lateral root coordinates (face_ r ight), and a position coordinate point (gl asses _center) where the nose bridge is located.

That is, the computing device collects the left and right lateral eye corner coordinates, the left lateral ear root coordinates, the lateral ear root coordinates, and the position coordinate point of the nose bridge in the face image of the user in real time.

S202, acquiring coordinates of key points of glasses from glasses images of glasses which are needed to be worn virtually by a user.

In one possible approach, the computing device may be connected to a server from which the eyeglass images of the eyeglasses are obtained. Optionally, the computing device sends a request to the server, where the request includes an identifier of glasses that need to be worn virtually, and the server sends a glasses image to the computing device.

The computing device obtains an eyeglass image, and performs a matting process on the eyeglass frame and the eyeglass legs in the eyeglass image to obtain an image area of the eyeglass front-side eyeglass frame and an image area of the left and right eyeglass legs. Subsequently, the computing device marks the keypoint information of the eyeglass frame in an image area of the eyeglass front eyeglass frame, wherein the keypoint information of the eyeglass frame comprises the keypoint coordinates of both sides of the eyeglass; the computing device marks the glasses frame's key point information in the glasses leg's picture area, which includes the glasses frame's left and right side breakpoint coordinates and the glasses leg's junction center position coordinates. And extracting coordinates of key points of the glasses according to the glasses images of the glasses.

S203, determining a perspective change matrix and a rotation matrix according to the coordinates of the facial key points and the coordinates of the glasses key points, wherein the perspective change matrix is related to a pitch (pitch) angle and/or a yaw (yaw) angle of the face of the user in the facial image, and the rotation matrix is related to a roll (ro l l) angle of the face of the user.

Wherein a schematic representation of pitch (pitch), yaw (yaw), roll (ro l) is shown with reference to fig. 3 a. In pose estimation, three concepts are often mentioned, namely pitch (pitch) angle, yaw (yaw) angle, roll (ro l l) angle. Pose estimation is a representation of the orientation of an object in three-dimensional space. The object pose is generally described with the earth as the reference frame (standard coordinate system). The world coordinate system is converted into a scene coordinate system. Pitch (pitch) is the angle of rotation about the X-axis, meaning the angle of the line of sight from the horizontal, viewed from below, is called the elevation angle. The angle between the line of sight and the horizontal is called the depression angle, which is simply understood as the head-up and head-down colloquially understood as the pitch angle in fig. 3 a. The angle of yaw (yaw) angle rotation around the Y axis means the angle of the actual heading from the planned heading, and a simple understanding is to turn the face left and right on the horizontal as in fig. 3 a. The roll (ro l l) angle is the angle of rotation about the Z axis, as is simply understood by the straight forward rotation of the face of FIG. 3 a.

Perspective change, meaning "line of sight penetration"; referring to fig. 3b, fig. 3b is a perspective schematic view of a virtual glasses based on a 2D map according to an embodiment of the present application, and the size of the virtual glasses is also changed according to the position change, for example, the right eye of the user wearing the glasses is close to the camera, and the picture taken visually is that the glasses frame of the right eye of the user is larger than the glasses frame of the left eye. According to the specific flowchart, as shown in fig. 4, fig. 4 is a flowchart of a method for determining a perspective matrix according to an embodiment of the present application, which includes:

S401, determining a perspective change matrix corresponding to the glasses frame according to coordinates of the facial key points and coordinates of the glasses frame key points;

The facial key points comprise key points corresponding to the left outer corner of the eye, the right outer corner of the eye and the bridge of the nose of the user respectively; the glasses frame key points comprise key point coordinate points corresponding to four corners of the glasses frame respectively, namely pos1[ (0, 0), (x_fg, y_fg), (0, y_fg) ]. The realization of the perspective matrix for the glasses image requires the perspective treatment of the glasses frame and the glasses legs respectively.

The perspective change processing of the glasses frame is mainly determined by the coordinates of the key points of the face and the coordinates of the key points of the glasses frame, and the specific determination mode is as follows:

in it_center＝(x_fg/2,y_fg/2)

left_di s=abs (left outer eye angular position [0] -glasses wearing position [0 ])

R ight _di s=abs (right outer eye angular position [0] -glasses wearing position [0 ])

d iff＝abs(r ight_d i s-left_d i s)/image.shape[1]*gl asses.shape[1]

Wherein, the center coordinates of the in it_center glasses frame image, that is, the position coordinates of the nose bridge, x_fg and y_fg are the length and width of the glasses frame image, left_dis and r ight _dis are the distance between the left outer corner coordinates and the right outer corner coordinates in the x direction in the face key point, abs is the rule in the formula, d iff is the offset, image.

Secondly, according to the offset and coordinates of key points corresponding to the four corners of the glasses frame, determining the coordinates of key points corresponding to the four corners of the glasses frame after offset, wherein a specific calculation formula is as follows:

The perspective change matrix is calculated according to four-point coordinate carry-in functions corresponding to left rotation (r ight _dis > = left_dis) and right rotation (r ight _dis < = left_dis) according to the difference of head rotation. The perspective change matrix is calculated according to four-point coordinate carry-in functions corresponding to left rotation (r ight _dis > = left_dis) and right rotation (r ight _dis < = l eft _dis) according to the difference of head rotation. The specific corresponding four-point coordinate formula is as follows:

pos2[(d iff,y_fg/x_fg*d iff),(x_fg,0),(x_fg,y_fg),(d iffy_fg-y_fg/x_fg

*d iff)]

pos2[(0,0),(x_fg-d iff,y_fg/x_fg*d iff),(x_fg-d iff,y_fg-y_fg/x_fg*d iff),

(0，y_fg)]

Where d iff is the frame image offset, pos2 is the four-point coordinate corresponding to the left turn, and pos2 is the four-point coordinate corresponding to the right turn.

And finally, determining a perspective change matrix corresponding to the glasses frame according to coordinates of key points corresponding to the four corners of the glasses frame after the offset. Illustratively, the perspective change matrix is a 3*3 matrix, and the perspective matrix of the glasses frame obtained based on the above change maps the original image coordinates (X, Y, 1) to the new view plane corresponding coordinates (X, Y, Z). The calculation formula is as follows:

Wherein, For the resulting perspective change matrix and a ₁₁…a₃₃ as a constant, X ' Y ' Z ' is the representation of the resulting deformation of X, Y, Z calculated, as defined by X, Y, Z.

S402, determining a perspective change matrix corresponding to the glasses legs according to the coordinates of the facial key points, the coordinates of the glasses leg key points and the perspective change matrix corresponding to the glasses frame.

The calculation of the glasses leg perspective matrix is similar to the change of the glasses frame perspective matrix, and the corresponding perspective matrix of the glasses legs is finally determined mainly through the coordinates of the glasses leg key points in the facial key points and the glasses image key points and the glasses frame perspective matrix. The facial key points comprise key points corresponding to the left ear and the right ear of the user respectively; the glasses leg key points comprise coordinates of the glasses leg key points to determine a perspective change matrix corresponding to the glasses legs. The specific calculation formula is as follows:

dis_x=1.5 x (glass_left_new_x-left ear position coordinate_x)/rate

long＝int((x_leg-d i s)/2)

Dis_y=abs (glass_left_new [1] -left ear position coordinates [1 ])/rate/2

leg_pos2＝[(long,0),(x_leg-long,d i s_y),(x_leg-long,y_leg+d i s_y),(long,y_leg

-d i s_y)]

leg_pos2＝[(long,d i s_y),(x_leg-long,0),(x_leg-long,y_leg),(long,y_leg-d i s_y)]

Where dis_x is the difference in x-axis direction, glass_left_new is the key point of the glasses frame, long is the perspective coordinate transformation distance in x-direction, dis_x is the coordinate transformation of the perspective transformation in x-direction, dis_y is the coordinate transformation of the perspective transformation in y-direction, leg_pos2 is the glasses frame boundary position higher than the auricular root coordinate, leg_pos2' is the glasses frame boundary position lower than the auricular root coordinate, x_leg and y_leg represent the length and width of the original image of the glasses leg, and rate is the bias angle.

Optionally, the scaling of the eyeglass image needs to be determined before the determination of the rotation matrix is performed. Fig. 5 is a flowchart of a scaling procedure of an eyeglass image according to an embodiment of the present application, including:

S501, determining the width of the face of the user according to the left outer canthus and the right outer canthus of the user;

specifically, the width of the face of the user can be obtained from the coordinate information of the left outer corner and the right outer corner of the face image.

S502, determining the width of the glasses frame according to key points corresponding to two sides of the glasses frame;

And obtaining corresponding key points after perspective change, namely glass_left_new_ts and glass_ r ight _new_ts, according to the perspective change matrix, and obtaining the width of the glasses frame after perspective change based on the key points. The specific formula is as follows:

gl ass_long＝gl ass_r ight_new_ts[0]-gl ass_left_new_ts[0]

Wherein glass_long is the width of the glasses frame after the perspective change.

S503, determining a scaling ratio according to the width of the face of the user and the width of the glasses;

The width of the glasses frame is taken as the size of the glasses image. Determining a scaling ratio according to the width of the face of the user and the size of the glasses image; wherein, the formula of the scaling is as follows:

rate = size of glasses image/glass_long

Where rate is the scale, glass_long is the width of the glasses frame after the perspective change.

S504, adjusting the size of the glasses image according to the scaling ratio so as to enable the glasses image to be matched with the face image.

Since the size of the eye image and the eye distance have a direct relationship, the eye image can be obtained by calculating the distance between the eye key points, and the distance between the outer corners of the eyes is 1.6. Knowing the scaling may be used to obtain the scaled eyeglass frame image gl asses _ res ize as a function. Meanwhile, the position coordinates of the zoomed key points need to be updated, and the position coordinates of the key points after specific zooming are expressed as follows:

gl ass_left_new_rs＝(gl ass_left_new_ts[0]*rate,gl ass_left_new_ts[1]*rate)

gl ass_r ight_new_rs＝(gl ass_r ight_new_ts[0]*rate,gl ass_r ight_new_ts[1]*rate)

gl ass_center_new_rs＝(gl ass_center_ts[0]*rate,gl ass_center_ts[1]*rate)

Wherein, glass_left_new_rs and glass_ r ight _new_rs and glass_center_new_rs are scaled key point position coordinates, rate is scaling, glass_left_new_ts and glass_ r ight _new_ts are key points at two sides of the glasses after perspective, and glass_center_ts is a key point of the center of the glasses after perspective.

A rotation matrix, which in the present application is understood to vary in roll angle. The rotation matrix is also established based on the coordinates of the facial key points and the coordinates of the eyeglass key points, including the center key point of the eyeglass frame. Specifically, first, according to coordinates of key points corresponding to the left outer corner and the right outer corner of the user, determining a rotation angle; the specific formula is as follows:

angle= actan [ (left outer eye angular position_y-left outer eye angular position_x)/(left outer eye angular position_x-left outer eye angular position_x) ]

Wherein angle is the rotation angle.

Next, a rotation matrix is determined based on the rotation angle and the center key point of the eyeglass frame.

The rotation angle is obtained according to the above formula, and the rotation matrix is obtained by a function of the coordinates of the key points of the glasses. The specific formula is as follows:

matRotat ion＝cv2.getRotat ionMatr ix2D((width/2,height/2),angle,1)

Wherein matRotat ion is a rotation matrix, getRotat ionMatr ix2D is a function rule, width and height are scaled image sizes, and angle is a rotation angle.

Illustratively, the temple scale is determined based on the frame scale, and further, the position coordinates of the temple key are updated based on the temple scale. The rotation angle of the glasses frame is used as the rotation angle of the glasses legs, the rotation change of the glasses legs is completed according to the rotation angle of the glasses legs, and then the position coordinates of key points of the glasses legs are updated.

For example, the scaling and the rotation matrix are not established in sequence, in the application, the related coordinate points are updated by the scaling, the rotation matrix is calculated based on the updated coordinate points after the scaling, or the rotation matrix is calculated first, and the scaling calculation is performed, which is not limited herein.

S204, updating coordinates of key points of the glasses according to the perspective change matrix and the rotation matrix;

In one possible mode, firstly, carrying out perspective change on coordinates of the glasses key points according to a perspective change matrix to obtain coordinates of the glasses key points after the perspective change; specifically, key points corresponding to perspective change, namely, glass_left_new_ts and glass_ r ight _new_ts, are obtained according to the perspective change matrix and key points glass_left and glass_ r ight on two sides of the frame marked by the original image. Secondly, rotating coordinates of the glasses key points after perspective change according to the rotation matrix to obtain coordinates of the glasses key points after rotation, and specifically calculating key points glass_left_new_ts and glass_ r ight _new_ts corresponding to perspective change and position coordinate points of glasses wearing according to the rotation matrix to correspond to face images acquired in real time to obtain updated glasses image key points glass_left_new and glass_ r ight _new.

S205, fusing the facial image and the glasses image according to the updated coordinates of the glasses key points; wherein, the glasses image is a 2D image.

Referring to fig. 6, fig. 6 is a flowchart of a face image and glasses image fusion according to an embodiment of the present application, including:

S601, determining a facial region of interest from a facial image according to coordinates of updated glasses key points;

And determining a region of interest of the face on the face image according to the size of the glasses frame image and the central coordinate point of the wearing glasses, wherein the region of interest of the face is a region roi required to finish the fusion of the glasses frame. The specific formula is as follows:

Roi=face_img [ max (0, int (eyeglass wearing position_y-h/2)): int (eyeglass wearing position_y+h/2), max (0, int (eyeglass wearing position_x-w/2)): int (eyeglass wearing position_x+w/2) ]

Where w is wide, h is height, x is length, int is a calculation rule, max represents the maximum value, face_img is the face shape.

S602, acquiring single-image transparency of the glasses image, and acquiring face image transparency and glasses image transparency according to the single-image transparency of the glasses image;

The glasses frame image after image transformation is split according to the channels, and the image color value over l ay_img of the first three channels and the transparency over l ay_mask of the fourth glasses frame image are obtained. The transparency background_mask and the transparency over l ay_mask of the face image are obtained through the fourth glasses frame image transparency calculation, and the specific formulas are as follows:

background_mask＝gray2bgr(255-over l ay_mask)

over l ay_mask＝gray2bgr(over l ay_mask)

wherein, gray2bgr is a parameter.

S603, multiplying, adding, processing and fusing according to the transparency of the face image and the transparency of the glasses image to obtain the 2D virtual try-on chart.

And (3) carrying out normalization adding processing after obtaining the transparency of the face image and the transparency of the glasses image, wherein the normalization adding processing is to multiply the transparency of the face image by the fusion area and multiply the transparency of the glasses image by the color value over l ay img of the image, and finally obtaining the fused 2D virtual try-on image. The specific formula is as follows:

a l pha*roi*(1/255.0)*background_mask*(1/255.0)+beta*over l ay_img

*(1/255.0)*over l ay_mask*(1/255.0)

Based on the same technical concept, as shown in fig. 7, fig. 7 is a schematic structural diagram of a virtual mirror device according to an embodiment of the present application.

An acquisition unit 701 for acquiring coordinates of a face key point from a face image of a user;

The acquiring unit 701 is further configured to acquire coordinates of key points of glasses from glasses images of glasses that the user needs to wear virtually;

A determining unit 702, configured to determine a perspective change matrix and a rotation matrix according to coordinates of the facial key points and coordinates of the glasses key points, where the perspective change matrix is associated with a pitch angle and/or a yaw angle of a face of the user in the facial image, and the rotation matrix is associated with a roll angle of the face of the user;

An updating unit 703, configured to update coordinates of the key points of the glasses according to the perspective change matrix and the rotation matrix;

a fusion unit 704, configured to fuse the face image and the glasses image according to the updated coordinates of the glasses key points; wherein, the glasses image is a 2D image.

In an alternative embodiment, the determining unit 702 is specifically configured to determine a perspective change matrix corresponding to the glasses frame according to the coordinates of the facial key points and the coordinates of the glasses frame key points; and determining the perspective change matrix corresponding to the glasses legs according to the coordinates of the facial key points, the coordinates of the glasses leg key points and the perspective change matrix corresponding to the glasses frame.

In an alternative embodiment, the determining unit 702 is specifically configured to determine the offset according to coordinates of key points corresponding to the left outer corner of the eye, the right outer corner of the eye, and the bridge of the nose of the user, respectively; determining coordinates of key points corresponding to the four corners of the glasses frame after offset according to the offset and the coordinates of key points corresponding to the four corners of the glasses frame respectively; and determining a perspective change matrix corresponding to the glasses frame according to coordinates of key points corresponding to the four corners of the glasses frame after the offset.

In an alternative embodiment, the determining unit 702 is specifically configured to determine the rotation angle according to coordinates of key points corresponding to the left outer corner of the eye and the right outer corner of the eye of the user; the rotation matrix is determined based on the rotation angle and the center key point of the eyeglass frame.

In an optional implementation manner, the updating unit 703 is specifically configured to perform perspective change on coordinates of the glasses key points according to the perspective change matrix, so as to obtain coordinates of the glasses key points after the perspective change; and rotating the coordinates of the eyeglass key points after perspective change according to the rotation matrix to obtain the coordinates of the eyeglass key points after rotation.

In an alternative embodiment, the determining unit 702 is further specifically configured to determine a width of the face of the user according to the left outer corner of the eye and the right outer corner of the eye of the user; determining the width of the glasses according to the key points corresponding to the two sides of the glasses frame; determining a scaling ratio according to the width of the face of the user and the width of the glasses; the eyeglass image is resized according to the scaling so that the eyeglass image matches the face image.

In an alternative embodiment, the determining unit 702 is further specifically configured to determine a facial region of interest from the facial image according to the coordinates of the updated glasses key points; the acquiring unit 701 is specifically configured to acquire a single-image transparency of the glasses image, and acquire a face image transparency and a glasses image transparency according to the single-image transparency of the glasses image; the fusion unit 704 is specifically configured to perform multiplication, addition, processing and fusion according to the transparency of the face image and the transparency of the glasses image to obtain the 2D virtual fitting chart.

Based on the same technical concept, the present application further provides a schematic structural diagram of a computing device, and as shown in fig. 8, an apparatus 800 provided by an embodiment of the present application, where the apparatus shown in fig. 8 may be an implementation manner of a hardware circuit of the apparatus shown in fig. 8. The apparatus may be adapted to perform the functions of the first terminal device or the second terminal device in the above-described method embodiments in the flowcharts shown above.

For ease of illustration, fig. 8 shows only the main components of the device.

Memory 830 is coupled to processor 820. The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units, or modules, which may be in electrical, mechanical, or other forms for information interaction between the devices, units, or modules. At least one of the memories 830 may be included in the processor 820.

In embodiments of the present application, the communication interface may be a transceiver, a circuit, a bus, a module, or other type of communication interface. In the embodiment of the application, when the communication interface is a transceiver, the transceiver can comprise an independent receiver and an independent transmitter; a transceiver integrating the transceiving function, or a communication interface, is also possible.

The apparatus 800 may also include a communication line 840. Wherein the communication interface 810, the processor 820 and the memory 830 may be interconnected by a communication line 840; the communication line 840 may be a peripheral component interconnect standard (PER IPHERA lcomponent interconnect, PCI) bus, or an extended industry standard architecture (extended industry STANDARD ARCH itecture, EISA) bus, or the like. The communication lines 840 may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 8, but not only one bus or one type of bus.

Based on the same technical concept, the embodiment of the application also provides a computer readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the method for performing the virtual mirror test based on the 2D mapping is performed.

Based on the same technical idea, the present application provides a computer program product comprising a computer program or instructions which, when executed by a communication device, implement the above-mentioned method of 2D map-based virtual mirror.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. The virtual mirror testing method based on the 2D mapping is characterized by comprising the following steps of:

acquiring coordinates of a facial key point from a facial image of a user;

Acquiring coordinates of key points of glasses from glasses images of glasses which the user needs to wear virtually;

Determining a perspective change matrix and a rotation matrix according to the coordinates of the facial key points and the coordinates of the glasses key points, wherein the perspective change matrix is associated with a pitch angle and/or a yaw angle of the face of the user in the facial image, and the rotation matrix is associated with a rolling angle of the face of the user;

updating coordinates of the key points of the glasses according to the perspective change matrix and the rotation matrix;

Fusing the face image and the glasses image according to the updated coordinates of the glasses key points; wherein the eyeglass image is a 2D image.

2. The method of claim 1, wherein the facial keypoints comprise keypoints corresponding to left-side outer canthus, right-side outer canthus, and nose bridge of the user, respectively;

The glasses key points comprise glasses frame key points, wherein the glasses frame key points comprise key points corresponding to four angles of the glasses frame;

The determining a perspective change matrix according to the coordinates of the facial key points and the coordinates of the glasses key points comprises the following steps:

Determining offset according to coordinates of key points corresponding to the left outer canthus, the right outer canthus and the nose bridge of the user respectively;

Determining coordinates of key points corresponding to the four corners of the glasses frame after offset according to the offset and the coordinates of key points corresponding to the four corners of the glasses frame respectively;

and determining a perspective change matrix corresponding to the glasses frame according to coordinates of key points corresponding to the four corners of the glasses frame after the offset.

3. The method of claim 1, wherein the facial keypoints comprise keypoints corresponding to left-side outer corners of eyes and right-side outer corners of eyes of the user, respectively;

the eyeglass keypoints comprise center keypoints of an eyeglass frame;

The determining a rotation matrix according to the coordinates of the facial key points and the coordinates of the glasses key points comprises:

determining a rotation angle according to coordinates of key points corresponding to the left outer canthus and the right outer canthus of the user respectively;

And determining the rotation matrix according to the rotation angle and the central key point of the glasses frame.

4. The method of claim 1, wherein updating the coordinates of the eyeglass keypoints based on the perspective change matrix and the rotation matrix comprises:

performing perspective change on coordinates of the glasses frame key points in the glasses key points according to the perspective change matrix to obtain coordinates of the glasses frame key points after the perspective change;

And rotating the coordinates of the glasses frame key points after perspective change according to the rotation matrix to obtain the coordinates of the glasses frame key points after rotation.

5. The method of claim 4, wherein the facial key points comprise left outer corners of eyes, right outer corners of eyes of the user;

the glasses frame key points comprise key points corresponding to two sides of the glasses frame respectively;

Before the coordinates of the key points of the glasses after the perspective change are rotated according to the rotation matrix, the method further comprises:

Determining the width of the face of the user according to the left outer canthus and the right outer canthus of the user;

Determining the width of the glasses according to the key points corresponding to the two sides of the glasses frame respectively;

determining the scaling according to the width of the face of the user and the width of the glasses;

According to the scaling, the size of the glasses image is adjusted so that the glasses image is matched with the face image;

the rotating the coordinates of the key points of the glasses frame after the perspective change according to the rotation matrix comprises the following steps:

Acquiring coordinates of the key points of the glasses after the size of the glasses image is adjusted;

and rotating coordinates of the glasses frame key points after the size of the glasses images is adjusted according to the rotation matrix.

6. The method of claim 4, wherein the obtaining coordinates of the pivotal point of the eyeglass frame after rotation further comprises:

and carrying out perspective change on the coordinates of the key points of the glasses legs according to the perspective change matrix corresponding to the glasses legs, and obtaining the coordinates of the key points of the glasses legs after the perspective change.

7. The method of claim 6, wherein the left ear and the right ear of the facial keypoints correspond to respective keypoints; the determining the perspective change matrix corresponding to the glasses legs according to the coordinates of the facial key points, the coordinates of the glasses leg key points and the perspective change matrix corresponding to the glasses frame comprises the following steps:

The glasses key points comprise key points corresponding to two sides of the glasses frame respectively according to the coordinates of the glasses key points after the corresponding rotation of the glasses frame;

according to the key points corresponding to the two sides of the glasses frame respectively, the key points corresponding to the left ear and the right ear respectively, and determining coordinates after perspective change corresponding to the glasses legs;

And determining a perspective change matrix corresponding to the glasses legs according to the coordinates after the perspective change corresponding to the glasses legs.

8. The method of claim 1, wherein the fusing the face image and the eyeglass image according to the updated coordinates of the eyeglass keypoints comprises:

Determining a facial region of interest from the facial image according to the updated coordinates of the eyeglass key points;

Acquiring the single-channel transparency of the eyeglass image, and acquiring the multi-channel transparency of the eyeglass image and the multi-channel transparency of the region of interest of the face according to the single-channel transparency of the eyeglass image;

and fusing the face image and the glasses image according to the multi-channel transparency of the glasses image and the multi-channel transparency of the region of interest of the face to obtain a 2D virtual try-on view.

9. The method of claim 7, wherein the fusing the face image and the eyewear image according to the multi-channel transparency of the eyewear image and the multi-channel transparency of the face region of interest comprises:

determining a product of a multi-channel transparency and a foreground color level of the eyeglass image;

Determining a product of the multi-channel transparency of the facial region of interest and a background color level;

and fusing the face image and the glasses image according to the product of the multichannel transparency and the foreground color level of the glasses image and the product of the multichannel transparency and the background color level of the region of interest of the face.

10. The method of claim 7, wherein the determining a facial region of interest from the facial image based on the updated coordinates of the eyeglass keypoints comprises:

Taking the coordinates of the nose bridge of the user in the facial image key points as the coordinates of the central points of the glasses;

Determining the size of the glasses image according to the updated coordinates of the glasses key points;

And determining a region of interest of the face from the face image according to the size of the eyeglass image and the coordinates of the center point of the eyeglass.