CN114983338A

CN114983338A - Skin detection method and electronic equipment

Info

Publication number: CN114983338A
Application number: CN202110231643.7A
Authority: CN
Inventors: 丁欣; 胡宏伟; 郜文美; 卢曰万; 姜永涛
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-03-02
Filing date: 2021-03-02
Publication date: 2022-09-02
Also published as: WO2022184084A1

Abstract

The embodiment of the application provides a skin detection method and electronic equipment, and relates to the technical field of electronics. The specific scheme is as follows: the electronic equipment automatically acquires N initial images of a user, screens skin measuring images from the N initial images, and displays skin measuring results based on the skin measuring images, wherein the initial images comprise face images, and the skin measuring images are initial images of which the features meet skin measuring conditions. The embodiment of the application is used for a skin detection process.

Description

Skin detection method and electronic equipment

Technical Field

The embodiment of the application relates to the technical field of electronics, in particular to a skin detection method and electronic equipment.

Background

It is good for people to enjoy the heart. As the standard of living increases, more and more people are willing to invest more time and money for the sake of beauty. Among them, the skin detection technology is a technology that can satisfy the demand of people for beauty. The skin detection technology is a technology for acquiring detection results regarding various features of the skin by acquiring face data using a sensor or taking a face image of a user and then performing skin measurement analysis based on the face data or the face image. The skin detection technology can be used for obtaining the detection results of skin characteristics such as facial pores, color spots, acnes, fine lines and the like, and can provide corresponding skin care suggestions for the user based on the skin detection results, so that the user is helped to better care the skin and improve the skin state.

Currently, a commonly used skin detection method is to acquire a facial image of a user through an electronic device, such as a mobile phone, and further analyze the acquired facial image according to a preset algorithm in the electronic device to obtain a skin detection result. In the process of acquiring the facial image of the user, in order to obtain a high-quality facial image, the user needs to strictly cooperate in the process of acquiring the facial image, for example, the face of the user and a camera of the electronic device are in a position, a distance or an angle in a proper range, and meanwhile, it may also need to ensure that the surrounding light condition meets a preset requirement. Even if the collected face image does not conform to the preset algorithm, the user is required to cooperate to take a picture again. Therefore, the user interaction flow in the existing skin detection process is complex, and the user experience is poor.

Disclosure of Invention

The embodiment of the application provides a skin detection method and electronic equipment, wherein the electronic equipment can actively acquire a plurality of initial images including face images of a user within a preset time period, screen skin measurement images from the plurality of initial images, perform skin measurement analysis based on the skin measurement images, and further display skin measurement results. In this scheme, electronic equipment can obtain initial image under the condition that the user does not have the perception, from the screening suitable survey skin image and survey the skin analysis, and the whole skin process of surveying does not need user's strict cooperation, can be when guaranteeing to survey the accuracy and the stability of skin result, can reduce the interaction between survey skin in-process user and the electronic equipment for the user obtains surveying the skin result more conveniently, promotes user experience.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

in one aspect, an embodiment of the present application provides a skin detection method. The method comprises the following steps: the electronic equipment automatically acquires N initial images of a user, wherein N is an integer greater than 1, and the initial images comprise face images; screening out a skin measuring image from the N initial images, wherein the skin measuring image is an initial image in which the characteristics of the face image meet skin measuring conditions, and the characteristics of the face image comprise: one or more of pitch attitude, angle, brightness, occlusion, proportion in the initial image, expression and definition; and displaying the skin measuring result based on the skin measuring image.

In the scheme, the electronic equipment can actively acquire the initial image, screen out the skin measuring image which accords with the skin measuring condition, and then carry out skin measuring analysis based on the skin measuring image. In the skin measuring method, a user does not need to actively initiate skin measurement and strictly cooperate in the skin measuring process, so that interaction between the user and electronic equipment in the skin measuring process can be reduced, the user can obtain a skin measuring result more conveniently, and user experience is improved. Meanwhile, the electronic equipment actively acquires an initial image, screens out a skin measuring image and performs mansion analysis, and can acquire a denser skin measuring result, so that the skin state of a user is effectively tracked and monitored, and skin care suggestions are better provided for the user.

In one possible design, the skin test image includes a first skin test image and a second skin test image; screening skin measuring images from N initial images, which comprises the following steps: for each of the N initial images: if the first characteristic of the face image in the initial image meets the skin measuring requirement and the second characteristic of the face image in the initial image meets the skin measuring requirement, taking the initial image as a first skin measuring image; if the first feature of the face image in the initial image meets the skin measurement requirement, the second feature of the first partial image of the face image in the initial image meets the skin measurement requirement, and the second feature of the second partial image of the face image in the initial image does not meet the skin measurement requirement, taking the initial image as a second skin measurement image; the first partial image and the second partial image are different partial images in the initial image; the first feature includes: one or more of pitch attitude, angle, expression, and proportion in the initial image; the second feature includes: brightness, occlusion, sharpness.

In this scenario, the skin test image may include a first skin test image and a second skin test image. The first skin measuring image is an initial image in which the whole face image meets the skin measuring requirement, and the second skin measuring image is an initial image in which the local image in the face image meets the skin measuring requirement.

In another possible design, if the skin measurement image includes a second skin measurement image, the second skin measurement image corresponds to a first label, and the first label is used to indicate that a first partial image in the second skin measurement image is used for skin measurement analysis; displaying a skin test result based on the skin test image, including: acquiring a local skin measurement result of the first local image based on the first label corresponding to the second skin measurement image; and displaying the skin measuring result based on the local skin measuring result.

In this scheme, if the skin measurement image is the second skin measurement image, the electronic device obtains a skin measurement result based on the partial image that meets the skin measurement requirement.

In another possible design, the displaying the skin test result based on the local skin test result includes: if the local skin measurement result comprises the local skin measurement result of all local images in the face image, displaying the skin measurement result based on the local skin measurement result; and if the local skin measurement result comprises the local skin measurement result of a part of local images in the face image, displaying the skin measurement result based on the local skin measurement result and the historical skin measurement result.

In this way, the electronic device may display the skin measurement results in combination with the historical skin measurement results without collecting the local skin measurement results for all local images.

In another possible design, if the skin measurement image includes a first skin measurement image, the first skin measurement image corresponds to a second label, and the second label is used for indicating that the whole face image in the first skin measurement image is used for skin measurement analysis; displaying a skin test result based on the skin test image, including: acquiring a skin measurement result of the face image of the first skin measurement image based on a second label corresponding to the first skin measurement image; and displaying the skin test result.

In this scheme, if the skin measurement image is the first skin measurement image, the electronic device obtains a skin measurement result based on the whole face image.

In another possible design, the method further includes: and if no skin measuring image is screened from the N initial images, displaying a skin measuring result based on the historical skin measuring result.

In this scheme, if there is no skin measurement image that meets the skin measurement requirement, the skin measurement result of this time can be predicted based on the historical skin measurement result.

In another possible design, the automatically acquiring N initial images of the user includes: starting a camera within a preset time period and collecting a preview image; when a face image is detected in the preview image, a camera is adopted to shoot and obtain N initial images.

Therefore, the electronic equipment can actively shoot the initial image within a preset time period, so that the interaction between a user and the electronic equipment is reduced, and the user experience is improved.

In another possible design, when a face image is detected in the preview image, capturing N initial images includes: when a face image is detected in the preview image, the focusing point of the camera is adjusted according to the positions of eyes in the face image, and N initial images are obtained by shooting through the camera after the focusing point is adjusted.

In another possible design, the method further includes: and if the skin measuring result is obtained within the preset time period, the camera is not started.

Therefore, the electronic equipment does not need to repeatedly detect the skin for many times, and the computing power and the storage space can be saved.

In another possible design, the electronic device is a smart mirror.

Because the user usually uses the mirror to carry out activities such as daily skin care every day, the electronic equipment can be an intelligent mirror, so that the living habits of the user are better utilized, the skin measurement result is intensively obtained, the skin state of the user is more effectively detected, and a better skin care suggestion is given.

In another possible design, during the skin detection, the method further includes: and outputting prompt information, wherein the prompt information is used for prompting the skin measuring process to a user.

In this way, the user can intuitively understand the current stage of skin detection.

In another aspect, an embodiment of the present application provides an electronic device, including: a camera; one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions, which when executed by the electronic device, cause the electronic device to perform the skin detection method provided by the embodiments of the present application.

In another aspect, the present application provides a computer-readable storage medium, which includes computer instructions, when the computer instructions are executed on a computer, the computer executes the skin detection method provided by the present application.

In another aspect, the present application provides a computer program product, which when run on a computer, causes the computer to execute the skin detection method provided by the present application.

For the advantageous effects of the other aspects, reference may be made to the description of the advantageous effects of the method aspects, which is not repeated herein.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 2 is a schematic structural diagram of an intelligent mirror provided in an embodiment of the present application;

fig. 3 is a flowchart of a skin detection method according to an embodiment of the present application;

fig. 4 is a flowchart of an image acquiring process according to an embodiment of the present application;

FIG. 5 is a flowchart of an image screening process provided in an embodiment of the present application;

fig. 6 is a schematic diagram of a partial image of a face image according to an embodiment of the present application;

fig. 7 is a flowchart of a skin test analysis process provided in an embodiment of the present application;

fig. 8 is a schematic diagram of a region of interest for skin detection according to an embodiment of the present application;

FIG. 9 is an interface diagram provided by an embodiment of the present application;

fig. 10 is a scene diagram provided in an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present embodiment, "a plurality" means two or more unless otherwise specified.

With the increasing demand for beauty, more and more people regularly conduct skin tests, which provide quantitative skin test results to better care the skin. Therefore, the demand of users for skin detection technology is increasing.

Currently, a user may perform skin detection in a variety of ways. For example, the user may perform skin detection through a professional skin detector, or through a skin detection Application (APP) in a cell phone.

In one prior art technique, a user may perform a skin test with a professional skin test meter (e.g., a visia skin test meter). When the skin detector is used for detecting skin, a user needs to put the head on the fixed bracket, keep the front face posture of the face, have no shielding and do not move stably, then the skin detector starts different illumination modes according to a preset program, and the user's face images are shot in sequence in different illumination modes. Furthermore, the skin detector analyzes the shot facial image by adopting a preset algorithm, and a skin detection result is displayed on a computer connected with the skin detector.

In another prior art, the user may also perform skin detection through a cell phone or smart mirror. Taking the example that the user takes a picture through the mobile phone to perform the skin detection, the mobile phone usually performs the skin detection through the installed skin detection APP (for example, hua is "love skin" APP). In this case, the user needs to perform a photographing process according to an instruction of the skin measurement APP, for example, move the mobile phone or the face according to an instruction such as APP voice, text, or icon, so that the face of the user is at a proper position, distance, angle of the mobile phone camera, and the light is proper. At the moment, the mobile phone collects the facial image of the user, the shot facial image is analyzed by adopting an algorithm preset in the skin detection APP, and a skin detection result is displayed.

In the above prior art, no matter whether a skin detection instrument is used or a mobile phone or a smart mirror is used to collect a facial image of a user for skin detection, the following problems exist:

on the one hand, in order to obtain a higher quality face image, the user needs to be strictly coordinated throughout the skin measurement process. For example, the user needs to fix the head in a proper posture, distance, light, and the like; or, the user needs to prompt the mobile phone or the face according to the APP. In addition, if the quality of the shot image does not meet the requirement of the preset algorithm, the user needs to cooperate with the face image to be collected again. In the skin detection process, the interaction flow between the user and electronic equipment such as a skin detector or a mobile phone is complex and tedious, and the user experience is poor.

On the other hand, in the whole skin measuring process, the user is required to actively initiate the skin measuring process. That is, in the skin measurement process of the prior art, the electronic device may start skin detection only when the user initiates a skin measurement procedure, for example, turns on the skin measurement APP or turns on the skin measurement mode of the skin detector. Therefore, if the frequency of skin measurement initiated by the user is less, the skin measurement result is less, and the skin measurement period is longer, so that effective skin state tracking monitoring cannot be realized, and further, a perfect skin care suggestion cannot be provided for the user.

The embodiment of the application provides a skin detection method, which can be applied to electronic equipment, can actively acquire a plurality of initial images including face images of a user within a preset time period without strict cooperation and active initiation of the user, actively screen out skin measurement images from the plurality of initial images, perform skin measurement analysis based on the skin measurement images, and further display a skin measurement result. The whole skin measuring process does not need active initiation and strict cooperation of a user, so that the skin measuring process can be carried out under the condition that the user does not sense the skin measuring process, and the user experience is improved.

In addition, after the electronic equipment acquires a plurality of initial images, the skin measuring images meeting the skin measuring requirements can be actively screened out, and the final skin measuring results and skin care suggestions are displayed based on the skin measuring images and/or historical skin measuring results, so that the accuracy and stability of the skin measuring results are ensured.

For example, the electronic device in the embodiment of the present application may be a smart mirror, a mobile phone, a tablet computer, a wearable device (e.g., a smart watch), an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, a super-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), or other mobile terminals, or may be a professional camera or other devices.

Fig. 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a Central Processing Unit (CPU), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), among others. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be, among other things, a neural center and a command center of the electronic device 100. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.

The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be used to monitor parameters such as battery capacity, battery cycle count, battery state of health (leakage, impedance), etc. In other embodiments, the power management module 141 may be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 100.

The wireless communication module 160 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into electromagnetic waves through the antenna 2 to radiate the electromagnetic waves.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1. In an embodiment of the present application, the display screen 194 may be used to display prompt information and skin test results.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a user takes a picture, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, an optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and converting the electric signal into an image visible to the naked eye. The ISP can also carry out algorithm optimization on noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The data storage area may store data (such as audio data, historical skin test results, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

In an embodiment of the present application, the processor 110 may screen the acquired initial image for skin measurement by executing instructions stored in the internal memory 121, and display the final skin measurement result to the user based on the skin measurement image and/or the historical skin measurement result.

The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light to the outside through the light emitting diode. The electronic device 100 detects infrared reflected light from nearby objects using a photodiode. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 may determine that there are no objects near the electronic device 100. The electronic device 100 may detect whether the electronic device 100 has an occlusion using the proximity light sensor 180G, and if so, may further determine whether the occlusion is a face of the user.

The ambient light sensor 180L is used to sense the ambient light level. Electronic device 100 may adaptively adjust the brightness of display screen 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust the white balance when taking a picture.

The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch screen. The touch sensor 180K is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type. Visual output associated with the touch operation may be provided via the display screen 194. In other embodiments, the touch sensor 180K may be disposed on a surface of the electronic device 100, different from the position of the display screen 194.

It is to be understood that the illustrated structure of the embodiment of the present application does not specifically limit the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In the embodiment of the present application, the camera 193 may acquire a plurality of initial images within a preset time period; the processor 110 screens out a skin measurement image meeting the skin measurement requirement from the collected multiple initial images by operating the instruction stored in the internal memory 121, and then performs skin measurement information based on the skin measurement image and the stored historical skin measurement result. The final display screen 194 displays the skin test results and skin care advice.

Fig. 2 shows a schematic view of a smart mirror having the structure shown in fig. 1. Because the user usually uses the mirror to perform activities such as skin care in the morning and evening, the intelligent mirror can collect facial images of the user when the user performs daily activities such as skin care.

As shown in fig. 2, the smart mirror 200 may include a camera 201, a mirror 202, a display screen 203, a proximity light sensor 204, a bracket 205, a switch button 206, a base 207, an LED annular light strip 208, and the like.

Wherein camera 201 may be camera 193 of fig. 1, display 203 may be display 194 of fig. 1, and proximity light sensor 204 may be proximity light sensor 180G of fig. 1.

The mirror 202 may be made of flat glass, mirror stainless steel, aluminum, etc. for displaying the image of the object or user in front of the mirror 202. The material and shape of the mirror surface 202 are not limited in the embodiments of the present application.

The display 203 may be embedded in any position of the mirror 202 or may be disposed outside the mirror 202. If the display 203 is embedded in the mirror 202, the display 203 may not display any information as part of the mirror 202, which is convenient for the user to view the face image in the mirror 202. The display screen 203 may respond to the user's instruction or after the skin measurement is finished, and then display the obtained skin measurement result.

The proximity light sensor 204 may be disposed on the mirror 202 or disposed outside the mirror 202, which is not limited in this embodiment.

The switch button 206 may be a physical button or a touch button, and the application is not limited to a specific position of the switch button 206.

The support 205 and the base 207 are used for supporting the mirror plate 202, and the shape and material of the support 205 and the base 206 are not limited in the embodiment of the present application. In some embodiments, the stand 205 and base 206 may also be foldable.

The LED annular light band 208 surrounds the outside of the mirror surface 202 for supplementing light, so that the user can better view the face image presented in the mirror surface during the dressing or skin care process.

In some embodiments, a switch button 206 may be used to control the turning on or off of the LED annular band of light.

In other embodiments, the smart mirror may not include the LED annular light strip 208.

In some embodiments, the switch button 206 may not be included in the smart mirror, which performs the function of the switch button in response to a voice indication of the user.

It can be understood that the intelligent mirror 200 provided by the embodiment of the present application is based on the structure of the electronic device 100 shown in fig. 1. The illustrated structure of the embodiment of the present application does not specifically limit the smart mirror 200.

The skin detection method provided by the embodiment of the present application will be explained below by taking the intelligent mirror with the above structure as an example. As shown in fig. 3, the method may include:

301. the smart mirror automatically acquires N initial images of the user.

In the embodiment of the application, the intelligent mirror can open the camera within a preset time period, and then N initial images of a user are automatically acquired. In the process, the intelligent mirror does not need to be actively started and strictly matched by a user when acquiring the initial image of the user, and when the user sits in front of the mirror and starts daily skin care or dressing, the intelligent mirror can automatically acquire the initial image of the user through the camera, so that the initial image is acquired under the condition that the user does not sense the initial image, and subsequent skin measurement analysis is performed.

The term "no perception" in the embodiment of the present application means that the user does not need to actively initiate skin measurement and strictly cooperate in the skin measurement process, and the user can conveniently obtain N initial images of the user by the intelligent mirror in the development of daily activities (for example, daily skin care).

In the embodiment of the application, N is an integer larger than 1. For example, N may be 10.

In some embodiments, the smart mirror may be preset with N. In other embodiments, the user can set N by himself or herself according to needs. The embodiments of the present application do not limit this.

In the embodiment of the present application, as shown in fig. 4, the intelligent mirror automatically acquires N initial images of the user, including the following steps:

401. the intelligent mirror acquires a preset time period and current system time, and determines whether the current system time is within the preset time.

In some embodiments, the user may set the preset time period according to his/her own habits of daily skin care, dressing, etc. in using the mirror. For example, if the user generally performs skin care between 8-10 am and 8-10 pm, the preset time periods may be previously set to 8:00-10:00 and 20:00-22: 00. In this way, the intelligent mirror automatically acquires the facial image of the user and performs skin detection within a preset time period set by the user.

In other embodiments, the user may also set the detection period based on his or her own habits of using mirrors for daily skin care, dressing, etc. For example, if the user typically applies skin care once in the morning and once in the evening, the test period may be set to test once every 12 hours. In this way, the intelligent mirror can periodically acquire facial images of the user and perform skin detection according to the set detection period.

The embodiment of the application does not limit the specific duration of the preset time period or the detection period, and the user can set or adjust the time period or the detection period according to requirements.

In the embodiment of the application, the intelligent mirror can also acquire the current system time according to the clock of the intelligent mirror, and determine whether the initial image should be acquired currently or not based on the current system time.

In some embodiments, the smart mirror periodically acquires the current system time.

After the intelligent mirror acquires the preset time period and the current system time, the intelligent mirror can determine whether the current system time is within the preset time period by comparing the current system time with the set preset time period.

If the current system time is within the preset time period, the intelligent mirror performs the following step 402.

If the current system time is not within the preset time period, the intelligent mirror can not start the camera. Since the intelligent mirror may periodically acquire the current system time, in some embodiments, the intelligent mirror may perform the following step 402 when the current system time acquired later is within a preset time period.

402. The intelligent mirror determines whether a skin test result within a preset time period already exists.

After the intelligent mirror determines that the current system time is within the preset time period, the intelligent mirror may further determine whether there has been a skin test result within the preset time period. It is also contemplated that the smart mirror may determine whether a skin test has been performed within a preset time period.

If the skin measurement result in the preset time period does not exist, the intelligent mirror executes the following step 403 to perform the subsequent process.

If the skin detection result in the preset time period exists, the intelligent mirror does not repeatedly carry out skin detection. That is, the smart mirror does not need to turn on the camera.

It can be considered that the intelligent mirror performs skin detection at most once in the preset time period, so that repeated skin detection in the preset time period is avoided, and calculation power and storage space are saved.

403. When the intelligent mirror determines that shielding exists, the camera is started, a preview image is collected, and whether a face image exists in the preview image is determined.

When the intelligent mirror determines that skin detection has not been performed within a preset time period, the intelligent mirror may further determine whether occlusion exists.

In some embodiments, the smart mirror determines whether occlusion exists by proximity to a light sensor. For example, the smart mirror determines whether or not there is occlusion within 1s by reading data that is close to the light sensor.

In the embodiment of the application, if the intelligent mirror does not detect the shielding, the camera does not need to be started, and the process is finished.

If the intelligent mirror detects the occlusion, it indicates that a person or an object exists in front of the intelligent mirror within a preset time period. At the moment, the intelligent mirror can open the camera to collect the preview image, and then whether the human face image exists can be judged based on the collected preview image.

In some embodiments, the intelligent mirror may detect the presence of a face image in the preview image through an existing neural network model for face detection. The embodiment of the application does not limit the neural network model for face detection.

In some embodiments, the smart mirror may determine whether the user is within a preset distance of the smart mirror based further on the preview image. For example, the intelligent mirror can determine the distance from the user corresponding to the face to the intelligent mirror according to the proportion of the face image in the preview image.

If the distance between the user and the intelligent mirror exceeds the preset distance, the face image in the preview image is possibly too small to be suitable for subsequent skin measurement analysis.

In other embodiments, if there is no skin measurement result within a preset time period, the intelligent mirror can directly start the camera to collect a preview image without prompting the user. That is, the smart mirror may actively capture the preview image within a preset time period.

404. If the preview image contains a human face image, the intelligent mirror adjusts the focusing point of the camera and shoots N initial images.

If the intelligent mirror detects a face image in the preview image, adjusting the focus of the camera according to the positions of eyes in the face image, and shooting N initial images by using the camera after the focus is adjusted.

Since the initial image is captured when the face image exists in the preview image, the face image is included in the initial image.

In some embodiments, if the intelligent mirror determines that the distance between the user and the intelligent mirror is within a preset distance, the intelligent mirror adjusts the focus of the camera to capture an initial image. Like this, can ensure that the user just shoots initial image when being nearer apart from intelligent mirror, avoid shooing initial image when the distance between user and the intelligent mirror is far away, and the initial image that leads to in the face undersize, can't carry out follow-up survey the problem of skin processing.

In some embodiments, the user can set the photographing duration according to the requirement. For example, the photographing time may be 5s, and if the photographing time is longer than the photographing time after the photographing is completed, the camera is turned off; if the photographing time is not exceeded, it is continuously determined whether N initial images have been photographed. If N initial images are shot, closing the camera; if the N initial images are not shot, the intelligent mirror collects the preview images again, and whether the face images exist in the preview images is determined.

In some embodiments, the smart mirror turns off the camera after taking the N initial images.

405. And if the human face image does not exist in the preview image and the human face image is not detected continuously within the preset time, closing the camera.

If the intelligent mirror does not detect the face image in the preview image, it can be shown that the blocking is not caused by the user facing the mirror for skin care or dressing.

In the embodiment of the application, if the camera of the intelligent mirror detects the face images continuously according to the preset frequency and the preview images collected within the preset time, it indicates that the user does not face the mirror to perform activities such as skin care or dressing, and the camera is closed by the intelligent mirror.

In some embodiments, the above steps 401-405 may all be done in the background, without presenting the user with a process.

In other embodiments, the intelligent mirror may prompt the user for the currently executed content while performing steps 401-405 above. For example, the intelligent mirror may present currently executed content to the user through a screen display, a voice prompt, a signal light, or the like in the process of acquiring the initial image. For example, "a face image is detected", "the first image has been captured", or "N images have been captured", and the like.

Through the steps 401 to 405, the intelligent mirror can acquire N initial images including the face image. The intelligent mirror can automatically acquire initial images in daily activities facing the mirror, such as normal skin care or dressing of a user. In the process of collecting the initial image, the user does not need to actively initiate the process of collecting the image, and does not need to adjust the sitting posture, the light and the direction to cooperate with the intelligent mirror to collect the initial image, that is, the intelligent mirror can collect the initial image under the condition that the user does not have the perception, so that the initial image can be conveniently collected, and the user experience is improved.

After acquiring N initial images of the user, the smart mirror may perform image screening, and execute the following steps 302.

302. The intelligent mirror screens out skin measuring images from the N initial images, wherein the skin measuring images are the initial images of which the characteristics of the face images meet skin measuring conditions.

After the intelligent mirror acquires N initial images of a user, the initial images which accord with skin measuring conditions can be screened out from the N initial images to be used as skin measuring images, and then skin measuring analysis is carried out according to the skin measuring images.

In the embodiment of the application, the intelligent mirror automatically screens the skin measuring image from N initial images. The skin test image may include a first skin test image or a second skin test image. The whole face image in the first skin measurement image is used for skin measurement analysis, and the local image in the face image in the second skin measurement image is used for skin measurement analysis.

In the embodiment of the present application, for example, as shown in fig. 5, the intelligent mirror may screen out a skin measurement image from N initial images, and may include the following steps:

501. and aiming at each initial image in the N initial images, the intelligent mirror judges whether the first characteristic of the face image in each initial image meets the skin measuring requirement.

In some embodiments, the intelligent mirror may cache the acquired initial images, and after acquiring the preset N initial images, the intelligent mirror may read the initial images one by one, and determine whether the first feature of the face image in each initial image meets the skin measurement requirement.

In other embodiments, the intelligent mirror may also directly determine whether the first feature of the face image in the acquired initial image meets the skin measurement requirement when each initial image is acquired.

If the intelligent mirror determines that the first feature of the face image in an initial image meets the skin measurement requirement, the intelligent mirror performs the following step 502.

And if the intelligent mirror determines that the first characteristic of the face image in the initial image does not meet the skin measuring requirement, the process is ended. In some embodiments, the smart mirror may also record that the initial image does not meet skin measurement requirements.

In the embodiment of the present application, the first feature includes: pitch attitude, angle, expression, proportion in the initial image, etc.

The pitch attitude refers to a state in which the user lowers or raises his head. It can be considered that the judgment of whether the pitching attitude meets the skin measurement requirement means that whether the angle of lowering or raising the head of the user is in a proper range is judged.

In some embodiments, the pitch pose may be determined based on the proportion of the three eyes in the face image.

Wherein the angle refers to an angle at which the face of the user is inclined left and right. It can be considered that the judgment of whether the angle meets the skin measurement requirement refers to judgment of whether the left-leaning angle or the right-leaning angle of the head of the user is in a proper range.

In some embodiments, the angle may be determined from the left-right symmetry of the face image.

Wherein, the expression refers to the expression of the user in the face image. It can be understood that if the user's face has some expressions, the facial image may not reflect the real state of the user's skin, for example, when the user frows, the number of fine lines in the facial image increases, thereby interfering with the subsequent skin measurement analysis.

In some embodiments, the expression may be determined according to a neural network model.

Wherein, the proportion in the initial image refers to the size of the face image in the initial image. It can be considered that the judgment of whether the ratio in the initial image meets the skin measurement requirement means that whether the distance between the user and the intelligent mirror is within an appropriate range.

In some embodiments, the size of the face image in the initial image can be calculated according to the proportion of the face image in the initial image, and the distance between the user and the intelligent mirror can be obtained.

In some embodiments, if the first characteristic comprises a plurality of characteristics, the intelligent mirror may sequentially determine whether each characteristic meets the skin test requirements. For example, if the first features include pitch attitude, angle, expression and proportion in the initial image, the intelligent mirror can sequentially judge whether each feature meets the skin measurement requirement, and all the features meet the skin measurement requirement, and then the next step is performed; and once one feature does not meet the skin measurement requirement, the initial image does not meet the skin measurement requirement, and the initial image is directly screened out after judging whether the subsequent features meet the skin measurement requirement.

It should be noted that, the order in which the intelligent mirror sequentially determines whether each feature meets the skin test requirement is not limited in the embodiment of the present application.

In other embodiments, if the first characteristic includes multiple characteristics, the intelligent mirror may also determine whether the multiple characteristics meet the skin test requirement. And screening out the initial image once one of the plurality of characteristics does not meet the skin measuring requirement.

It is understood that the more features included in the first feature, the more accurate the skin test result obtained based on the skin test image obtained by the screening.

502. The intelligent mirror judges whether the second characteristics of the face image in the initial image meet the skin measuring requirements or not.

Under the condition that the first characteristic of the face image in the initial image meets the skin measuring requirement, the intelligent mirror can continuously judge whether the second characteristic of the face image in the initial image meets the skin measuring requirement.

The second feature of the face image in the initial image refers to the second feature of the entire face image in the initial image. That is, the second feature of the face image represents the second feature of the entire face image.

In embodiments of the application, the second feature comprises one or more of a brightness, an occlusion, a sharpness, etc.

Judging whether the second characteristics of the face image in the initial image meet the skin measuring requirements or not refers to judging whether the overall characteristics of the face image such as brightness, shielding and definition meet the skin measuring requirements or not.

In some embodiments, the brightness may be determined from the gray scale of the face image. Wherein the larger the gradation value, the larger the luminance.

In some embodiments, occlusion may be determined based on an existing model, which is not described herein.

In some embodiments, sharpness may be determined based on sharpness of the face image. Wherein the greater the sharpness value, the greater the sharpness.

In some embodiments, if the second features include multiple features, the intelligent mirror may sequentially determine whether each feature meets the skin test requirements.

In other embodiments, if the second characteristic comprises a plurality of characteristics, the intelligent mirror can simultaneously judge whether each characteristic meets the skin test requirement.

The embodiment of the present application does not limit this, and does not limit the order of sequentially judging whether each feature meets the requirement.

503. And when the second characteristics of the face image in the initial image meet the skin measuring requirements, the intelligent mirror takes the initial image as a first skin measuring image.

And if the second characteristic of the face image in the initial image meets the skin measurement requirement, the whole face image meets the skin measurement requirement. It can be considered that the whole face image in the initial image can be used for subsequent skin measurement analysis.

In the embodiment of the application, the first feature of the face image in the initial image meets the skin measurement requirement, and the second feature meets the skin measurement requirement, so that the initial image can be used as a first skin measurement image for subsequent skin measurement analysis.

In some embodiments, when the intelligent mirror obtains the first skin measurement image, the first skin measurement image is directly adopted to perform subsequent skin measurement analysis, and whether other initial images meet the skin measurement requirements is not judged. In this case, the speed of skin detection can be increased.

In other embodiments, the intelligent mirror may cache a first skin measurement image meeting a skin measurement requirement, and when a new first skin measurement image is obtained, determine whether an existing first skin measurement image exists, and if so, the intelligent mirror may select an image with a higher score as the first skin measurement image for skin measurement analysis according to a composite score of a second feature of the face image in the cached first skin measurement image and a composite score of a second feature of the face image in the new first skin measurement image. Therefore, the first skin measuring image is the image with the highest quality, and the accuracy of the subsequent skin measuring analysis can be improved.

In some embodiments, the skin measurement algorithm in the intelligent mirror may perform a skin measurement analysis based on only one first skin measurement image. For example, the first skin image may be the first skin image or the first skin image with the highest quality.

In other embodiments, the intelligent mirror may cache all first skin images that meet the skin test requirements, and the skin test algorithm in the intelligent mirror performs the skin test analysis based on all the cached first skin images. That is, the first skin measurement image may be a plurality of images.

In some embodiments, the first skin test image may further correspond to a second label (also referred to as an overall label), where the second label is used to indicate that the whole face image in the first skin test image is used for skin test analysis.

504. When the second characteristics of the face image in the initial image do not meet the skin measuring requirements, the intelligent mirror judges whether the second characteristics of the local image meet the skin measuring requirements or not according to the local image of the face image in the initial image.

If the second feature of the face image in the initial image does not meet the skin measurement requirement, that is, the second feature of the whole face image does not meet the skin measurement requirement, the intelligent mirror can judge whether the second feature of each local image of the face image meets the skin measurement requirement.

If the second feature of a certain partial image of the face image in the initial image meets the skin measurement requirement, the intelligent mirror executes the following step 505.

And if the second characteristic of a certain local image of the face image in the initial image does not accord with the skin measuring requirement, ending the process. In some embodiments, the smart mirror may also record that the initial image does not meet skin test requirements.

In the embodiment of the present application, as shown in fig. 6, the face image may be divided into three regions, that is, a forehead region, a left face region, and a right face region, where each region corresponds to one local image. That is, the face image may be divided into a forehead partial image, a left face partial image, and a right face partial image. The intelligent mirror can judge whether the second characteristics of each partial image in the three partial images meet the skin measuring requirement or not.

It is understood that the division of the face image into the three partial images is only an example, and the division manner of the partial images of the face image and the number of the partial images are not limited in the embodiment of the present application.

In some embodiments, the intelligent mirror may sequentially determine whether the second characteristic of each partial image meets the skin test requirement.

In other embodiments, the intelligent mirror may also determine whether the second characteristic of each partial image meets the skin test requirement.

In the embodiment of the present application, the determining whether the second feature of the local image meets the skin measurement requirement refers to determining whether the brightness, the shielding, the definition, and the like of the local image meet the skin measurement requirement.

Illustratively, the intelligent mirror can divide the face image into three partial images, namely a forehead partial image, a left face partial image and a right face partial image, and respectively judge whether the brightness, the shielding and the definition of each partial image meet the skin measuring requirements.

It can be understood that, because the intelligent mirror determines whether the second feature of the partial image of the face image meets the skin measurement requirement only when the second feature of the whole face image does not meet the skin measurement requirement, the second feature of at least one partial image in the face image does not meet the skin measurement requirement.

505. The intelligent mirror takes the initial image as a second skin measurement image.

Under the condition that the second characteristic of a certain local image in the initial image meets the skin measuring requirement, the intelligent mirror can take the initial image as a second skin measuring image, and the local image in the second skin measuring image can be used for skin measuring analysis.

In this embodiment, the second skin test image may correspond to a first label (also referred to as a local label), where the first label is used to indicate a local image for skin test analysis in the second skin test image.

For example, if the first label corresponding to the second skin measurement image is a forehead label, it indicates that the forehead local image of the face image in the second skin measurement image meets the skin measurement requirement, and the forehead local image may be used for subsequent skin measurement analysis.

In some embodiments, when the intelligent mirror obtains the second skin measurement image in which the first partial image meets the skin measurement requirement, it may be no longer determined whether the first partial image of the face image in the subsequent initial image meets the skin measurement requirement, but it is only necessary to determine whether other partial images of the face image in the subsequent initial image except the first partial image meet the skin measurement requirement.

In other embodiments, when the intelligent mirror obtains a second skin test image in which the first partial image meets the skin test requirement, the intelligent mirror may cache the second skin test image and the corresponding first label, and when a new first partial image meeting the skin test requirement is obtained, determine whether the existing first partial image exists, and if so, the intelligent mirror may select, as the second skin test image to be finally used for skin test analysis, the second skin test image in which the first partial image with a higher score is located according to the comprehensive score of the second feature of the first partial image in the cached second skin test image and the comprehensive score of the second feature of the new first partial image. In this way, since the quality of the first local image in the second skin image used for skin measurement analysis is high, the accuracy of subsequent skin measurement analysis can be improved.

In some embodiments, the skin measurement algorithm in the intelligent mirror may perform the skin measurement analysis based only on at most one local image for each local correspondence. For example, the second skin-test image used by the intelligent mirror for skin-test analysis only includes the initial image with the highest quality of the forehead partial image, the initial image with the highest quality of the left face partial image, and the initial image with the highest quality of the right face partial image.

In other embodiments, the intelligent mirror may buffer all initial images for which one or more local images meet the skin test requirements as second skin test images, and the skin test algorithm in the intelligent mirror performs the skin test analysis based on the second skin test images and the respective corresponding first labels.

In some embodiments, steps 501-505 described above may all be done in the background, without presenting the user with processing.

In other embodiments, the intelligent mirror may prompt the user for the current execution when performing steps 501-505, described above. For example, the intelligent mirror can present currently executed contents to the user through screen display, voice prompt or signal light and the like in the process of screening the initial image. For example, "the initial image 1 as a whole is qualified", "the forehead partial image of the initial image 2 is qualified", and the like.

Through the steps 501 to 505, the intelligent mirror can automatically screen out skin measurement images from the acquired N initial images, and then perform subsequent skin measurement analysis based on the skin measurement images. Because the intelligent mirror automatically filters the initial image, the user is not required to initiate image filtering, and therefore the intelligent mirror can automatically continue to filter the skin measuring image under the condition that the user does not sense the skin measuring image after the initial image is automatically acquired, so that subsequent skin measuring analysis is performed, interaction processes between the user and the intelligent mirror in the skin measuring process are reduced, and user experience is improved.

After the skin measurement image is screened out by the intelligent mirror, skin measurement analysis can be automatically performed based on the skin measurement image to obtain a skin measurement result, namely the content of the following step 303 is executed.

303. The intelligent mirror carries out skin measurement analysis based on the skin measurement image and displays the skin measurement result.

After the skin measuring image is screened from the initial image by the intelligent mirror, skin measuring analysis can be automatically carried out based on the skin measuring image, and a skin measuring result is displayed.

In some embodiments, if the first skin test image exists, the intelligent mirror performs a skin test analysis based on the first skin test image to obtain a skin test result.

In some embodiments, if the first skin measurement image does not exist and the second skin measurement image exists, the intelligent mirror performs skin measurement analysis based on the second skin measurement image to obtain a skin measurement result.

In other embodiments, the intelligent mirror performs skin measurement analysis based on the first skin measurement image and the second skin measurement image, and takes a skin measurement result with a better result as a final skin measurement result.

In the embodiment of the application, as shown in fig. 7, the intelligent mirror performs skin measurement analysis based on the skin measurement image, and displays a skin measurement result, including the following steps:

701. the smart mirror determines whether a skin test image is present.

After step 302, the smart mirror may read the skin images one by one. In this process, the smart mirror can determine whether a skin image is present.

If the smart mirror determines that a skin test image is present, the smart mirror performs the following step 702.

If the intelligent mirror determines that no skin measuring image exists, the intelligent mirror indicates that a proper skin measuring image is not screened from the initial image. For example, if the light of the environment where the user is located is poor, so that the brightness of the whole or part of the face image in the shot initial image does not meet the requirement, the initial image obtained at this time cannot be used as the skin measurement image.

In the event that the smart mirror determines that no skin measurement images are present, the smart mirror may obtain a predicted skin measurement based on the historical skin measurements and display the predicted skin measurement.

In some embodiments, the smart mirror may obtain a predicted skin test result based on the historical skin test results according to a linear regression or history fitting method, or the like. The method for obtaining the predicted skin measurement result by the intelligent mirror based on the historical skin measurement result is not limited in the embodiment of the application.

In some embodiments, the historical skin test result may be stored in an internal memory or in a cloud, which is not limited in this application.

702. The intelligent mirror determines whether the skin test image is a first skin test image.

In this embodiment of the application, the intelligent mirror may determine whether the skin test image is the first skin test image according to whether the second label exists in the skin test image.

If the second label (namely the integral label) exists in the skin measuring image, the intelligent mirror determines that the skin measuring image is the first skin measuring image.

In other embodiments, if the second label is not present in the skin test image, and no other label (e.g., the first label) is present in the skin test image, the smart mirror may also determine that the skin test image is the first skin test image.

703. And if the intelligent mirror determines that the skin measuring image is a first skin measuring image, acquiring a skin measuring result of the face image of the first skin measuring image based on the first skin measuring image.

In this embodiment of the application, if the intelligent mirror determines that the skin measurement image is the first skin measurement image, it indicates that the skin measurement result may be obtained based on the whole face image in the first skin measurement image.

In some embodiments, the intelligent mirror may extract a region of interest (ROI) of each detection item based on the face image in the first skin detection image, and input the ROI of each detection item into a neural network model (e.g., a classification network) for skin detection.

The neural network model used for skin detection is not limited in the embodiments of the present application.

Illustratively, fig. 8 shows ROIs of respective detection items. As shown in fig. 8 (a), the pore, stain detection ROI is generally the cheek; as shown in fig. 8 (b), the fine grain detection ROI is generally forehead and under-eye; as shown in (c) of fig. 8, the blackhead detection ROI is generally a nose; as shown in fig. 8 (d), the pouch, black eye detection ROI is generally around the eye; as shown in fig. 8 (e), pox, gloss detection ROIs are generally forehead, nose, cheek and chin.

704. If the intelligent mirror determines that the skin measuring image is not the first skin measuring image but the second skin measuring image, the local skin measuring result of the corresponding local image is obtained based on the first label corresponding to the second skin measuring image, and then the skin measuring result is obtained.

Because the skin measurement image comprises the first skin measurement image and the second skin measurement image, if the skin measurement image exists and is not the first skin measurement image, the skin measurement image can be indicated as the second skin measurement image.

In an embodiment of the application, the smart mirror may determine a local image for skin measurement analysis based on the first label corresponding to the second skin measurement image. For example, if the first label is a forehead label, it indicates that the forehead partial image of the face image in the second skin measurement image can be used for skin measurement analysis.

In some embodiments, the smart mirror may extract the ROI in the corresponding partial image based on the first label. For example, if the first label is a forehead label, the smart mirror may extract a forehead ROI for fine lines, acne, gloss. And then, the ROI in the local image is input into a neural network model (such as a classification network) to carry out skin detection, and a local skin detection result is obtained. And then, the intelligent mirror can normalize all local skin measurement results when all skin measurement images are read, so that a final skin measurement result is obtained.

In other embodiments, the intelligent mirror may extract ROIs in the corresponding local images based on the first label, normalize all the extracted ROIs when all the skin measurement images are read, and input the normalized ROIs into a neural network model (e.g., a classification network) for skin detection, thereby obtaining a final skin measurement result.

In this embodiment of the application, before the intelligent mirror acquires the final skin measurement result based on the second skin measurement image, the method may further include: the intelligent mirror determines whether the partial images of the face image are collected.

If the intelligent mirror set is aligned with all the partial images of the face image, for example, the intelligent mirror set is aligned with the forehead partial image, the left face partial image and the right face partial image, it is indicated that the intelligent mirror can obtain a complete skin measurement result of the face image through all the partial images. That is, when the partial images of the face image are collected, the final skin measurement result may be considered as the skin measurement result of the entire face image.

In some embodiments, if the intelligent mirror does not collect all the partial images of the face image, a final skin measurement result is obtained based on the partial skin measurement result and the historical skin measurement result.

If the intelligent mirror does not collect all local images of the face image, the skin measuring result of the local area which does not collect the proper image can be obtained by predicting the historical skin measuring result. For example, the intelligent mirror only acquires the local skin measurement result of the left face local image and the local skin measurement result of the right face local image, and does not acquire the local skin measurement result of the forehead local image, so that the intelligent mirror can obtain the predicted skin measurement result of the forehead local image based on the historical skin measurement result, and further obtain the final skin measurement result by combining the local skin measurement result of the left face local image and the local skin measurement result of the right face local image.

In some embodiments, the smart mirror may also correct the final skin test results based on historical skin test results.

705. The intelligent mirror displays the skin test result.

In some embodiments, after the smart mirror obtains the skin measurement result, the skin measurement result may be displayed, where the skin measurement result includes the current skin state of the user. For example, pores, pox, blackheads or fine lines, etc.

In other embodiments, the intelligent mirror may further perform comparative analysis on the current skin measurement result and a historical skin measurement result, and finally display the skin measurement result to the user, where the skin measurement result further includes a comparative analysis result and a skin care suggestion.

Illustratively, fig. 9 shows an interface displaying the results of a skin test. Fig. 9 (a) and (b) show the current skin glossiness and dark eye circle state of the user, and the result of comparing and analyzing the glossiness in the current skin test result with the glossiness in the historical skin test result, respectively. Fig. 9 (c) and (d) provide the user with skin care advice given based on the current skin state.

In some embodiments, the smart mirror may present the skin measurement results to the user via the display screen after obtaining the skin measurement results.

In other embodiments, the intelligent mirror presents the skin measurement result to the user in response to the user triggering the operation of displaying the skin measurement result after obtaining the skin measurement result. For example, after daily skin care results, the user may click on a particular area of the mirror and proceed to display the skin test results. The operation of triggering and displaying the skin measurement result may be touch operation, click operation, voice indication, gesture indication, or the like, which is not limited in the embodiment of the present application.

In still other embodiments, the smart mirror may also present the skin measurement results to the user without the display screen, e.g., the smart mirror may also present the skin measurement results to the user by voice; or the intelligent mirror can also directly send the skin measurement result to personal electronic equipment such as a mobile phone of the user and the like, so that the user can check the skin measurement result.

In some embodiments, steps 701-705 above may all be done in the background, without presenting the user with processing.

In other embodiments, the intelligent mirror may prompt the user for the currently executed content while performing steps 701-705 above. For example, the intelligent mirror may present currently executed content to the user through a screen display, a voice prompt, a signal light, or the like during the skin test analysis. For example, "sweat pores are being analyzed," "stain is being analyzed," and so forth.

Through the steps 701 to 705, the intelligent mirror can perform skin measurement analysis based on the skin measurement image, and display the skin measurement result. Because the intelligent mirror automatically carries out skin measurement and analysis, the interaction flow between a user and the intelligent mirror in the skin measurement process can be reduced, and the user experience is improved.

Fig. 10 shows a scene diagram in an embodiment of the present application. As shown in FIG. 10, the user performs daily skin care activities for a predetermined period of time, such as 8:00-10:00 a.m. When a user faces the intelligent mirror to care skin, the intelligent mirror can actively acquire facial images of the user, automatically screen the facial images to obtain skin measurement images, and perform skin measurement analysis based on the skin measurement images. At the end of the skin test analysis or in response to a user operation to view the skin test results, the smart mirror presents the skin test results to the user.

In the scene shown in fig. 10, the user only needs to care skin daily according to own habits, does not need to actively initiate skin measurement or coordinate with skin measurement to adjust own posture, and the like, and the intelligent mirror can automatically complete the whole skin measurement process under the condition that the user does not sense the skin.

According to the content, in the skin detection method provided by the embodiment of the application, a user does not need to actively initiate skin measurement and does not need to strictly cooperate in the skin measurement process, so that the interaction between the user and an intelligent mirror in the skin measurement process can be reduced, the user can obtain a skin measurement result more conveniently, and the user experience is improved.

Meanwhile, the intelligent mirror can periodically and intensively acquire the facial images of the user and perform skin measurement analysis based on the habit of daily use of the mirror by the user (such as daily skin care time), so that the skin detection method based on the scheme can acquire more intensive skin measurement results, effectively track and monitor the skin state of the user and better provide skin care suggestions for the user.

In addition, after the intelligent mirror automatically acquires a plurality of initial images, the initial images can be screened according to the screening method and the skin measurement analysis method provided by the scheme, and a skin measurement result is obtained based on the screened skin measurement images and/or historical skin measurement results, so that the accuracy and stability of the skin measurement result can be ensured.

It will be appreciated that in order to implement the above-described functions, the electronic device comprises corresponding hardware and/or software modules for performing the respective functions. The present application is capable of being implemented in hardware or a combination of hardware and computer software in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed in hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, with the embodiment described in connection with the particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In this embodiment, the electronic device may be divided into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be implemented in the form of hardware. It should be noted that the division of the modules in this embodiment is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

In the case of dividing each functional module by corresponding functions, fig. 11 shows a possible composition diagram of the electronic device 1100 involved in the above embodiment, and as shown in fig. 11, the electronic device 1100 may include: an image acquisition unit 1101, an image filtering unit 1102, a skin test analysis unit 1103, and a result output unit 1104.

Among other things, image acquisition unit 1101 may be used to enable electronic device 1100 to perform steps 301, 401 through 405, etc., described above, and/or other processes for the techniques described herein.

Image screening unit 1102 may be used to enable electronic device 1100 to perform steps 302, 501-505, etc., described above, and/or other processes for the techniques described herein.

The skin analysis unit 1103 may be used to support the electronic device 1100 in performing the above-described steps 303, 701 to 704, etc., and/or other processes for the techniques described herein.

The result output unit 1104 may be used to enable the electronic device 1100 to perform the above-described steps 705, etc., and/or other processes for the techniques described herein.

In some embodiments, the image acquisition unit 1101 may correspond to an image acquisition module (or a sensorless skin start-stop control and image acquisition module); the image filtering unit 1102 may correspond to an image filtering module; the skin test analysis unit 1103 may correspond to a single/multiple image skin test algorithm analysis module; the result output unit 1104 may correspond to a result output module.

It should be noted that all relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

The electronic device 1100 provided by the embodiment is used for executing the skin detection method, so that the same effects as the implementation method can be achieved.

Where an integrated unit is employed, the electronic device 1100 may include a processing module, a memory module, and a communication module. The processing module may be configured to control and manage actions of the electronic device 1100, and for example, may be configured to support the electronic device 1100 to perform the steps performed by the image acquisition unit 1101, the image filtering unit 1102, the skin test analysis unit 1103, and the result output unit 1104. The memory module may be used to support the electronic device 1100 for storing program codes and data, etc. A communications module may be used to support communications of electronic device 1100 with other devices, such as with wireless access devices.

The processing module may be a processor or a controller. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. A processor may also be a combination of computing functions, e.g., a combination comprising one or more microprocessors, Digital Signal Processing (DSP) and microprocessors, or the like. The storage module may be a memory. The communication module may specifically be a radio frequency circuit, a bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

Embodiments of the present application also provide an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors for storing computer program code comprising computer instructions which, when executed by the one or more processors, cause the electronic device to perform the associated method steps described above to implement the skin detection method in the embodiments described above.

Embodiments of the present application further provide a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are executed on an electronic device, the electronic device is caused to execute the above related method steps to implement the skin detection method in the above embodiments.

Embodiments of the present application further provide a computer program product, which when run on a computer, causes the computer to execute the relevant steps described above, so as to implement the skin detection method performed by the electronic device in the above embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip can execute the skin detection method executed by the electronic device in the above-mentioned method embodiments.

The electronic device, the computer-readable storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device, the computer-readable storage medium, the computer program product, or the chip may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the above embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the above functional modules is used as an example, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application, or portions of the technical solutions that substantially contribute to the prior art, or all or portions of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A skin detection method, applied to an electronic device, the method comprising:

automatically acquiring N initial images of a user, wherein N is an integer greater than 1, and the initial images comprise face images;

screening out a skin measuring image from the N initial images, wherein the skin measuring image is an initial image of which the characteristics of the face image meet skin measuring conditions, and the characteristics of the face image comprise: one or more of pitch attitude, angle, brightness, occlusion, proportion in the initial image, expression, and sharpness;

and displaying a skin measuring result based on the skin measuring image.

2. The method of claim 1, wherein the skin test image comprises a first skin test image and a second skin test image;

the screening of the skin measuring image from the N initial images comprises the following steps:

for each of the N initial images:

if the first feature of the face image in the initial image meets the skin measurement requirement and the second feature of the face image in the initial image meets the skin measurement requirement, taking the initial image as the first skin measurement image;

if the first feature of the face image in the initial image meets the skin measurement requirement, the second feature of the first partial image of the face image in the initial image meets the skin measurement requirement, and the second feature of the second partial image of the face image in the initial image does not meet the skin measurement requirement, taking the initial image as the second skin measurement image; the first partial image and the second partial image are different partial images in the initial image;

wherein the first feature comprises: one or more of pitch attitude, angle, expression, proportion in the initial image;

the second feature includes: brightness, occlusion, sharpness.

3. The method of claim 2, wherein if the skin test image comprises the second skin test image, the second skin test image corresponds to a first label indicating that the first partial image in the second skin test image is used for skin test analysis;

the displaying of the skin test result based on the skin test image comprises:

acquiring a local skin measurement result of the first local image based on a first label corresponding to the second skin measurement image;

and displaying the skin test result based on the local skin test result.

4. The method of claim 3, wherein displaying the skin test result based on the local skin test result comprises:

if the local skin measurement result comprises the local skin measurement result of all local images in the face image, displaying the skin measurement result based on the local skin measurement result;

and if the local skin measurement result comprises a local skin measurement result of a part of local images in the face image, displaying the skin measurement result based on the local skin measurement result and the historical skin measurement result.

5. The method according to any one of claims 2 to 4, wherein if the skin test image comprises the first skin test image, the first skin test image corresponds to a second label, and the second label is used for indicating that the whole face image in the first skin test image is used for skin test analysis;

the displaying of the skin test result based on the skin test image comprises:

acquiring a skin measurement result of the face image of the first skin measurement image based on a second label corresponding to the first skin measurement image;

and displaying the skin test result.

6. The method according to any one of claims 1-5, further comprising:

and if the skin measuring image is not screened from the N initial images, displaying the skin measuring result based on the historical skin measuring result.

7. The method according to any one of claims 1-6, wherein said automatically acquiring N initial images of a user comprises:

starting a camera within a preset time period and collecting a preview image;

and when the face image is detected in the preview image, shooting by adopting the camera to obtain the N initial images.

8. The method according to claim 7, wherein the capturing and acquiring the N initial images when the face image is detected in the preview image comprises:

and when the face image is detected in the preview image, adjusting the focusing point of the camera according to the positions of eyes in the face image, and shooting by adopting the camera after the focusing point is adjusted to obtain the N initial images.

9. The method of any one of claims 1-8, wherein the electronic device is a smart mirror.

10. The method of any one of claims 1-9, wherein during skin detection, the method further comprises:

and outputting prompt information, wherein the prompt information is used for prompting the skin measuring process for a user.

11. An electronic device, comprising:

a camera;

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the electronic device, cause the electronic device to perform the skin detection method of any one of claims 1-10.

12. A computer-readable storage medium comprising computer instructions which, when executed on a computer, cause the computer to perform the skin detection method of any one of claims 1-10.

13. A computer program product, characterized in that it, when run on a computer, causes the computer to carry out the skin detection method according to any one of claims 1-10.