WO2023011348A1

WO2023011348A1 - Detection method and electronic device

Info

Publication number: WO2023011348A1
Application number: PCT/CN2022/108950
Authority: WO
Inventors: 赵琳; 胡宏伟; 郜文美; 卢曰万; 丁欣; 周一丹
Original assignee: 华为技术有限公司
Priority date: 2021-07-31
Filing date: 2022-07-29
Publication date: 2023-02-09
Also published as: CN115700841A

Abstract

The present application provides a detection method and an electronic device, relating to the technical field of terminals. The electronic device has an ultraviolet camera and collects an ultraviolet image and a color image by means of the ultraviolet camera and an RGB camera, respectively, then matches a first region identified in the ultraviolet image to the corresponding color image, thereby improving the user experience. Furthermore, by means of collecting ultraviolet images at different angles for combined analysis of sunscreen cream application information, the accuracy of detection is increased, and meticulous guidance for sun exposure can be provided to the user.

Description

Detection method and electronic equipment

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on July 31, 2021, with application number 202110877247.1, and the title of the invention is "Detection Method and Electronic Equipment", the entire contents of which are incorporated in this application by reference.

technical field

The present application relates to the field of electronic technology, in particular to a detection method and electronic equipment.

Background technique

At present, more users are keen on the prevention of skin problems. Skin problems can include blackheads, spots, wrinkles, etc. Among them, there are many skin problems that can be caused by ultraviolet radiation. In this case, it is particularly important to take some physical or chemical methods to reduce the damage of ultraviolet rays. In chemical sun protection measures, users can apply sunscreen on the parts that need sun protection, so that part of the ultraviolet rays can be absorbed and the damage of ultraviolet rays to the parts where the sunscreen is applied can be reduced.

However, in some scenarios, users often forget to apply sunscreen, or even if the user applies sunscreen, it may not be applied evenly, and the user will still be exposed to strong ultraviolet rays, and a series of skin problems of the user have not been effectively treated. solution. To this end, an embodiment of the present application provides a detection method. The detection method can, but is not limited to, detect sunscreen application.

Contents of the invention

The detection method and electronic equipment provided by the present application, the electronic equipment has an ultraviolet camera, respectively collects ultraviolet images and color images through the ultraviolet camera and the RGB camera, and matches the first region in the identified ultraviolet image to the corresponding color image, improving User experience.

In order to achieve the above object, the application adopts the following technical solutions:

In the first aspect, the present application provides a detection method, which is applied to an electronic device or a device (such as a chip) capable of realizing or matching the functions of an electronic device. Taking the electronic device as an example, the electronic device includes a first camera and a second camera ; Wherein, the first camera is a color camera, and the second camera is an ultraviolet camera, and the method includes:

The first image is collected by the first camera, and the second image is collected by the second camera, wherein the color image includes the first content.

Wherein, the first content may be the object to be detected collected by the color image. The object to be detected is an object to be detected for application of the sunscreen. The objects to be detected include parts of the human body, such as the face, hands, arms and other parts that need sun protection. Objects to be detected may also include other objects, for example, objects such as vehicles that need to be isolated from ultraviolet rays for sun protection.

Based on the fact that the UV camera can capture ultraviolet information and that sunscreens can absorb ultraviolet rays, the electronic device can determine the application of sunscreen based on the ultraviolet information recorded in the ultraviolet image, so as to determine the area related to the application of sunscreen on the ultraviolet image, such as first area. Afterwards, the electronic device determines a second area corresponding to the first area on the color image, and displays a user interface, the user interface includes first content and first information for indicating the second area, that is, marking sunscreen in the color image Apply to the second area where the condition is relevant. In this way, the user can observe the second area related to the application of sunscreen through the color image. Since the imaging details of the color image are richer, the user can clearly and accurately judge the application of sunscreen according to the second area marked in the color image, thereby improving the user experience. Moreover, users can take sun protection measures according to the application of sunscreen, such as reapplying sunscreen. It can be seen that through the technical solutions of the embodiments of the present application, it is possible to provide refined guidance to users on sun protection and reduce the probability of users being sunburned.

In a possible design of the first aspect, the first content includes a first face; the second image includes a second face, and the first face and the second face belong to the same user;

Determining the first region on the second image specifically includes:

On the second face of the second image, determine the first region;

Determining a second area corresponding to the first area on the first image specifically includes:

On the first face of the first image, a second area corresponding to the first area is determined.

In this way, it is possible to determine the area related to sunscreen application on the human face of the ultraviolet image, and match this area to the corresponding area on the human face of the color image, so that the user can understand according to the corresponding area of the human face of the color image The application of sunscreen, the visual effect is more intuitive and easy for users to understand.

In a possible design of the first aspect, the first area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.

That is to say, the electronic device can accurately detect one or more of the above-mentioned areas of the ultraviolet image, so that it can provide users with rich sun protection reminders and provide refined guidance for users on sun protection.

In a possible design of the first aspect, after the first image is collected by the first camera and the second image is collected by the second camera, the method further includes:

collecting a third image through the first camera, and collecting a fourth image through the second camera when the third image satisfies the second condition;

determining a third region on the fourth image;

According to the first area, determining a second area corresponding to the first area on the first image includes:

determining a second area corresponding to the first area and the third area on the first image according to the first area and the third area;

The second condition includes: the third image includes a human face, and the angle of the human face included in the third image relative to the electronic device is within a preset angle range.

In this solution, the joint analysis of sunscreen application information by collecting ultraviolet images from different angles can improve the accuracy of detection and provide refined guidance to users on sunscreen.

In a possible design of the first aspect, collecting the first image through the first camera and collecting the second image through the second camera includes: at the same time, collecting the first image through the first camera and collecting the second image through the second camera second image.

Among other things, UV images can be used to detect sunscreen application. Color images can be used to detect faces, and can also be used to identify the angle of a face relative to an electronic device.

Since the angle of the user's face relative to the electronic device is fixed at the same time, the first camera and the second camera capture images synchronously, which can make the angle of the face relative to the electronic device in the images captured by the first camera and the second camera same. In this way, the electronic device can deduce the angle of the human face in the ultraviolet image relative to the electronic device according to the angle of the human face in the color image relative to the electronic device, so as to determine the usability of the ultraviolet image.

In a possible design of the first aspect, the first image is collected by the first camera, and the second image is collected by the second camera, including:

collecting a fifth image through the first camera;

When the fifth image satisfies the first condition, capturing the first image through the first camera and capturing the second image through the second camera;

The first condition includes: the fifth image includes a human face.

In this detection scheme, the electronic device first collects a color image through a color camera, and when it is determined that the color image includes a human face, that is, there is an object to be detected for sunscreen application, the electronic device then collects an ultraviolet image through the ultraviolet camera (for detecting sunscreen application), and collect color images through a color camera to prevent power consumption problems caused by turning on the UV camera earlier.

In a possible design of the first aspect, the electronic device further includes an ultraviolet supplementary light, and collecting the second image through the second camera includes: turning on the ultraviolet supplementary light, and collecting the second image through the second camera. In this linkage setting method, since the ultraviolet camera and the ultraviolet supplementary light can be turned on in linkage, the complexity of realizing the acquisition of ultraviolet images can be reduced.

In a possible design of the first aspect, the method further includes:

Determine the ambient UV intensity;

Capture a second image through a second camera, including:

When the ambient ultraviolet intensity is less than the third threshold, the ultraviolet supplementary light is turned on, and the second image is collected by the second camera.

That is to say, the ultraviolet supplementary light is turned on only when the ambient ultraviolet intensity is low, so the damage of the ultraviolet supplementary light to the human body can be reduced.

In a possible design of the first aspect, after the first image is captured by the first camera, after the second image is captured by the second camera, and before the fourth image is captured by the second camera, the method further includes: displaying the second information, and the second information is used to prompt the user to adjust the angle of the human face relative to the electronic device. By prompting the user to adjust the angle, the electronic device can collect ultraviolet images from multiple angles, and then conduct joint analysis of sunscreen application information through ultraviolet images from different angles, improving the accuracy of detection.

In a possible design of the first aspect, determining the first area on the second face of the second image includes:

determining a fourth region on the second face of the second image;

The first area is determined in the fourth area of the second image based on the pixel values of the pixels contained in the fourth area.

In a possible design of the first aspect, determining the first area in the fourth area of the second image according to the pixel values of the pixels contained in the fourth area includes:

According to the pixel values of the pixels contained in the fourth area, a plurality of target sub-areas are determined in the fourth area, and the target sub-area whose area is greater than or equal to the fourth threshold value among the plurality of target sub-areas is determined as the first area, and the target sub-area The third condition is met, and the third condition includes: the pixel mean value of the target sub-region is less than the fifth threshold, and the pixel value of the pixels contained in the target sub-region is less than the sixth threshold.

Using this solution, the mobile phone can exclude small areas such as moles from the candidate areas for sunscreen application, thereby reducing the probability of areas such as moles being misjudged as areas for applying sunscreen, and improving detection accuracy.

In a possible design of the first aspect, before determining the first region on the second image, the method further includes:

Based on the first human face in the first image, a second human face in the second image is determined.

In this scheme, it is considered that color images have richer imaging details and higher quality than ultraviolet images. Therefore, the mobile phone can accurately identify the face in the color image, and match the corresponding area of the recognized face to the ultraviolet image.

In a possible design of the first aspect, the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.

In a possible design of the first aspect, the fourth area does not include areas corresponding to eyes, eyebrows, and mouth in the human face. The fourth zone is usually the one that doesn't need sunscreen. In this scheme, the fourth area is determined from the face of the ultraviolet image, and the first area is determined in the fourth area. Since the pixel values of areas such as eyes, eyebrows, and mouth can not be calculated, the amount of calculation can be reduced, and the detection efficiency of the electronic device can be improved.

In a possible design of the first aspect, the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area. In this way, the user can know the accurate sunscreen application situation through the first information displayed on the electronic device, which is convenient for sunscreen guidance.

In a second aspect, an electronic device is provided, and the electronic device includes a processor, a display screen, a first camera, and a second camera; wherein, the first camera is a color camera, and the second camera is an ultraviolet camera;

The first camera is used to collect the first image;

The second camera is used to collect a second image, wherein the first image contains the first content;

a processor, configured to determine a first area on the second image; determine a second area corresponding to the first area on the first image according to the first area;

The display screen is used to display a user interface, the user interface includes first content and first information, and the first information is used to indicate the second area.

In a possible design of the second aspect, the first content includes a first face; the second image includes a second face, and the first face and the second face belong to the same user;

Determining the first region on the second image specifically includes:

On the second face of the second image, determine the first region;

In a possible design of the second aspect, the first area is any one or more of the following areas: the area where sunscreen is applied, the area where sunscreen is not applied, the area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.

In a possible design of the second aspect, the first camera is also used to collect a third image, and the second camera is used to collect a fourth image when the third image satisfies the second condition;

a processor configured to determine a third region on the fourth image;

In a possible design of the second aspect, the first camera is used to collect the first image, and the second camera is used to collect the second image, including: at the same moment, the first camera collects the first image and the second The camera collects the second image.

In a possible design of the second aspect, the first camera is used to collect the first image, and the second camera is used to collect the second image, including:

The first camera is used to collect the fifth image; when the fifth image satisfies the first condition, the first image is collected, and the second camera is used to collect the second image;

The first condition includes: the fifth image includes a human face.

In a possible design of the second aspect, the electronic device further includes an ultraviolet fill light, and the second camera is used to capture the second image, including: turning on the ultraviolet fill light, and the second camera captures the second image.

In a possible design of the second aspect, the processor is also used to determine the ambient ultraviolet intensity;

The second camera is used to collect the second image, including:

When the ambient ultraviolet intensity is less than the third threshold, the ultraviolet supplementary light is turned on, and the second camera collects the second image.

In a possible design of the second aspect, the display screen is further used to display second information, and the second information is used to prompt the user to adjust the angle of the human face relative to the electronic device.

In a possible design of the second aspect, determining the first area on the second face of the second image includes:

determining a fourth region on the second face of the second image;

In a possible design of the second aspect, determining the first area in the fourth area of the second image according to the pixel values of the pixels included in the fourth area includes:

In a possible design of the second aspect, the processor is further configured to determine a second human face in the second image according to the first human face in the first image.

In a possible design of the second aspect, the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.

In a possible design of the second aspect, the fourth area does not include areas corresponding to eyes, eyebrows, and mouth in the human face. The fourth zone is usually the one that doesn't need sunscreen.

In a possible design of the second aspect, the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area.

In a third aspect, the present application provides an electronic device, which has a function of implementing the detection method in any one of the foregoing aspects and any possible implementation manner. This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a fourth aspect, the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are run on an electronic device, the electronic device executes any one of any aspect and any one of its possible implementations. detection method.

In a fifth aspect, the present application provides a computer program product, which, when the computer program product is run on an electronic device, causes the electronic device to execute any detection method according to any aspect and any possible implementation manner thereof.

In a sixth aspect, a circuit system is provided, and the circuit system includes a processing circuit, and the processing circuit is configured to execute the detection method in any one of the above aspects and any possible implementation manners thereof.

In the seventh aspect, the embodiment of the present application provides a chip system, including at least one processor and at least one interface circuit. When executing the instructions, at least one processor executes the detection method in any one of the above aspects and any possible implementation manners.

Description of drawings

Fig. 1 is a schematic diagram of the principle of ultraviolet imaging provided by the embodiment of the present application;

2A-2E are schematic diagrams of the electronic equipment provided by the embodiment of the present application;

FIG. 2F is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of the software architecture of the electronic device provided by the embodiment of the present application;

Fig. 4 is the interface diagram provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of a method for recognizing face angles provided by an embodiment of the present application;

FIG. 6 is an interface diagram provided by the embodiment of the present application;

Fig. 7 is a schematic diagram of the method flow provided by the embodiment of the present application;

Figure 8 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application;

Figure 9 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application;

Figure 10 is an interface diagram provided by the embodiment of the present application;

Figure 11 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application;

Fig. 12A is a schematic diagram of the flow of the detection method provided by the embodiment of the present application;

Fig. 12B is a schematic diagram of the detection method flow provided by the embodiment of the present application;

Figure 13 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application;

Figure 14 is a schematic diagram of the flow of the detection method provided by the embodiment of the present application;

Fig. 15 is a schematic diagram of the training and use of the classifier provided by the embodiment of the present application;

Fig. 16 is a schematic diagram of the flow of the detection method provided by the embodiment of the present application;

Figure 17 is a schematic diagram of the detection method flow provided by the embodiment of the present application;

Figure 18 is a schematic diagram of the detection method flow provided by the embodiment of the present application;

Figure 19 and Figure 20 are schematic diagrams of interfaces provided by the embodiment of this application;

Figure 21 is a schematic diagram of the device provided by the embodiment of the present application;

FIG. 22 is a schematic diagram of a chip system provided by an embodiment of the present application.

Detailed ways

First, some technical terms involved in the embodiments of the present application are introduced.

Ultraviolet Imaging: The wavelengths of ultraviolet light are not in the visible light range, therefore, to the human eye, ultraviolet light is invisible. In some schemes, ultraviolet light can be used as a light source, and an ultraviolet imaging system can be used for imaging. This technology of capturing ultraviolet light and imaging it with an ultraviolet imaging system can be called ultraviolet imaging technology. With ultraviolet imaging, users can observe the ultraviolet image formed by the object under ultraviolet irradiation.

The ultraviolet imaging system includes an ultraviolet camera module. For example, the ultraviolet camera module can be a camera capable of capturing ultraviolet rays, which is called an ultraviolet camera. As shown in Figure 1, the ultraviolet camera can include components such as a lens and an image sensor. The lens can be made of materials such as but not limited to quartz glass, which can transmit ultraviolet rays. Image sensors record information about the light that passes through the lens.

In the embodiment of the present application, taking into account the characteristics of the ultraviolet imaging system, the ultraviolet imaging system can be applied to skin detection (including but not limited to sunscreen application detection), so as to detect the skin's absorption or reflection of ultraviolet rays.

As shown in Figure 1, when ultraviolet rays irradiate different parts of a character, they can be reflected or absorbed by human body parts. For example, ultraviolet rays will be reflected on parts such as the neck where no sunscreen is applied, and the reflected ultraviolet rays can pass through the ultraviolet lens, and then record ultraviolet information on the image sensor. Correspondingly, the pixel value of the imaging area of the neck and other parts is relatively high. In the area of the face where sunscreen is applied, the sunscreen will protect the skin and absorb part of the ultraviolet rays, so that no ultraviolet rays or less ultraviolet rays are reflected into the ultraviolet lens, and the image sensor will not record or record less. Apply sunscreen on the face The light information of the part. Correspondingly, the pixel value of the imaging area of the sunscreen-applied part of the face is relatively low, for example, the imaging area 1 of the sunscreen-applied face shown in FIG. 1 is black.

It can be understood that when other conditions are the same, the thicker the sunscreen is applied on a certain part, the stronger the absorption ability of ultraviolet rays is usually, then the imaging pixel value of this part in the ultraviolet image is usually lower (the darker the color). Conversely, the thinner the sunscreen is applied on a certain part, the weaker its ability to absorb ultraviolet rays, and the imaging pixel value of this part in the ultraviolet image is usually higher (the lighter the color). Still taking Figure 1 as an example, compared with Area 1, the application of sunscreen in Area 2 is thinner, so the imaging pixel value of Area 2 in the ultraviolet image is higher and the color is lighter.

The embodiment of the present application provides a detection method, which can be applied in an electronic device, and the electronic device is equipped with an ultraviolet supplementary light and an ultraviolet camera. The electronic device in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, a smart cosmetic mirror (such as shown in Figure 2A), etc. The mobile phone in the embodiment of the present application can be a folding screen mobile phone or For mobile phones with non-folding screens, the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

For example, the layout of the cameras on the electronic device 100 can be referred to FIG. 2B , where the front of the electronic device 100 is the plane where the display screen 194 is located. As shown in (a) of FIG. 2B , the camera 1931 is located on the front of the electronic device 100 , so the camera is a front camera. As shown in (b) of FIG. 2B , the camera 1932 is located on the back of the electronic device 100 , and the camera is a rear camera. Cameras can include UV cameras and color cameras. Color cameras include but are not limited to red, green, blue (RGB) cameras.

Optionally, the solution of the embodiment of the present application may be applied to an electronic device 100 with a folding screen (that is, the display screen can be folded) having multiple display screens.

In some embodiments, the folding screen may be a flexible folding screen. Wherein, the flexible folding screen includes a folding shaft made of flexible material. Part or all of the flexible folding screen is made of flexible materials. For example: only the foldable part (such as the folding shaft) of the flexible folding screen is made of flexible material, and the other parts are made of rigid material; or, all of the flexible folding screen is made of flexible material. The folding screen can be folded along the folding axis to form at least two sub-screens.

Exemplarily, (c) in FIG. 2B shows a folding screen electronic device 100 . In response to the user's operation, as shown in (d) in FIG. 2B , the foldable screen is folded inwardly (or outwardly) along the folding edge, so that the foldable screen forms at least two sub-screens (for example, A sub-screen and B sub-screen). ). Optionally, as shown in (e) of FIG. 2B , there is a display screen (such as a C screen) on the outer side of the fold. If the electronic device 100 is provided with a camera on the surface where the C-screen is located. Then, in the unfolded scene of the electronic device 100 as shown in (c) of FIG. 2B , the camera on the C-screen is on the back of the electronic device 100 and can be regarded as a rear camera. In the scene where the electronic device 100 is folded as shown in (e) of FIG. 2B , the camera on the C-screen becomes the front of the electronic device 100 and can be regarded as a front camera. That is to say, the front camera and the rear camera in this application do not limit the nature of the camera itself, but are only an illustration of a positional relationship.

Taking the folding screen mobile phone in Fig. 2B as an example, the ultraviolet camera can be set on the C screen (by digging a hole or setting up an under-screen camera, etc.). In this way, the user can collect the user's facial image through the camera, and check the application of sunscreen through the C screen. The ultraviolet camera can also be set on the A screen (or B screen), so that the user can collect facial images through the camera, and check the application of sunscreen through the A screen (or B screen).

Optionally, the folding screen of the folding screen electronic device may form multiple (such as 2, 3, etc.) sub-screens. Exemplarily, the flexible folding screen shown in (1) in FIG. 2C may include a folding line 030 and a folding line 031 . After being folded longitudinally along the folding line 030, a sub-screen 032, a sub-screen 033, and a sub-screen 034 as shown in (2) of FIG. 2C can be formed.

Optionally, the screen arrangement of the folding screen electronic device may be a top-bottom screen arrangement such as that shown in (d) in FIG. 2B, or may also be a left-right screen arrangement such as that shown in (1) or (2) in FIG. 2D wait. The embodiment of the present application does not limit the screen arrangement of the folding screen electronic device. The flexible folding screen shown in (1) in FIG. 2D, after being folded laterally along the folding line 040, can form a sub-screen 041 and a sub-screen 042 as shown in (2) in FIG. 2D. For example, the folding screen device may be a mobile phone in a folded state (refer to the state shown in (1) of FIG. 2E ), and may be a tablet computer in an unfolded state.

In the embodiment of the present application, the folding screen may be a single-sided screen (that is, only one side can display a user interface) or a double-sided screen (that is, both opposite sides can display a user interface).

For a single-sided folding screen, when the folding screen is turned toward the side that can display the user interface (that is, the front of the single-sided folding screen), it can be called forward folding; when the folding screen is turned toward the side that can display the user interface Folding is performed on the opposite side (ie, the back of the single-sided folding screen), which can be called reverse folding. For example, (2) in FIG. 2D represents a schematic diagram of forward folding, and (2) in FIG. 2E represents a schematic diagram of reverse folding. A folding screen device can determine whether it is currently being folded forward or backward.

Exemplarily, FIG. 2F shows a schematic structural diagram of the electronic device 100 . The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments of the present application, the processor 110 is configured to call the ultraviolet camera to capture ultraviolet images when an operation triggering the detection is detected, and detect the application of sunscreen according to the ultraviolet images. Afterwards, the processor 110 may control to prompt the user about the application of sunscreen through the display screen. Among them, the application of sunscreen includes but is not limited to whether the user applies sunscreen, the location where the sunscreen is applied, the location where the sunscreen is not applied, whether the sunscreen is applied evenly at the location where the sunscreen is applied, the location where the sunscreen is not applied evenly, and the locations where the sunscreen is not applied evenly. Information on how thick to apply sunscreen and recommended information on sunscreen.

As a possible implementation manner, the processor 110 controls the ultraviolet camera to capture ultraviolet images, which may be to control the ultraviolet camera to capture multiple ultraviolet images from different angles. Afterwards, the processor 110 can detect the application of sunscreen according to multiple ultraviolet images from different angles. In this way, the detection error caused by the reflective area in the part-angle ultraviolet image can be offset, and the accuracy of the detection result can be improved.

As a possible implementation, the processor 110 controls to remind the user of the application of sunscreen through the display screen, which may be to mark different facial areas in the color image, such as marking the position where the sunscreen is not applied evenly in the color image, etc. The color image can record clearer skin details, help users clearly identify the application of sunscreen, and guide users to take more detailed sun protection measures. For example, users can finely reapply according to the application of sunscreen.

The charging management module 140 is configured to receive a charging input from a charger.

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input of the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the display screen 194 , the camera 193 and so on.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (bluetooth, BT) and other wireless communication functions applied on the electronic device 100. solution.

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the optical signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 . For example, the ISP can control the photosensitive element to expose and take pictures according to the shooting parameters.

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals.

In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1. Wherein, the camera 193 may be located in the edge area of the electronic device, and may be an under-screen camera, a liftable camera, or a hole-digging camera. The camera 193 may include a rear camera, and may also include a front camera. The embodiment of the present application does not limit the specific position and shape of the camera 193 .

In the embodiment of the present application, the camera 193 includes a common camera and an ultraviolet camera.

Wherein, the ordinary camera includes a color camera. The color camera may be, for example but not limited to, an RGB camera.

In the embodiment of the present application, the color camera can be used to collect images, and the collected images can be used to determine whether there is a human face waiting to be detected. Captured images can also be used to flag sunscreen application. For example, a phone could mark in a color image where sunscreen was not applied evenly.

The ultraviolet camera can be used to take ultraviolet images, and the electronic device 100 can detect the application of sunscreen according to the taken ultraviolet images. The ultraviolet camera can also be used to capture ultraviolet rays in the current environment. Specifically, the image sensor (such as CMOS) in the ultraviolet camera captures the ultraviolet rays in the current environment. The electronic device 100 can determine the intensity of ultraviolet rays in the current environment according to the captured ultraviolet rays.

In the embodiment of the present application, the electronic device 100 may further include an ultraviolet supplementary light. The ultraviolet supplementary light can emit ultraviolet rays and can be used as an imaging light source for ultraviolet cameras. As a possible implementation, in order to avoid excessive ultraviolet irradiation of the user by the ultraviolet supplementary light, only when the ambient ultraviolet intensity is lower than a certain threshold, that is, the ambient ultraviolet intensity cannot reach the ultraviolet intensity required by the ultraviolet camera for imaging, Turn on the UV light. When the ambient ultraviolet intensity is higher than the threshold, the electronic device does not turn on the ultraviolet supplementary light.

Wherein, the electronic device 100 acquires the ambient ultraviolet intensity by detecting the ultraviolet intensity through its own sensor, or acquires the ambient ultraviolet intensity from the cloud side. The embodiment of the present application does not limit the manner in which the electronic device 100 acquires the intensity of ambient ultraviolet rays.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

In some embodiments, the NPU uses image recognition technology to identify whether the image collected by the camera 193 contains a face image.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio data into an analog audio electrical signal output, and is also used to convert an analog audio electrical signal input into digital audio data. The audio module 170 may include an analog/digital converter and a digital/analog converter. For example, the audio module 170 is used to convert the analog audio electrical signal output by the microphone 170C into digital audio data. The audio module 170 may also be used to encode and decode audio data. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Speaker 170A, also called "horn", is used to convert analog audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert analog audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to receive the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into analog audio electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C. Wherein, the microphone 170C may be a built-in component of the electronic device 100 , or may be an external accessory of the electronic device 100 .

In some embodiments, the electronic device 100 may include one or more microphones 170C, where each microphone or multiple microphones cooperate to collect sound signals from various directions, and convert the collected sound signals into analog audio electrical signals. It can also achieve noise reduction, identify the source of sound, or directional recording functions.

The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

Among them, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

The distance sensor 180F is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Touch sensor 180K, also known as "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it.

It can be understood that, the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention uses a layered architecture as an example to illustrate the software structure of the electronic device 100 .

FIG. 3 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the operating system of the electronic device (such as the system to be detected) is divided into four layers, which are respectively the kernel layer, the hardware abstract layer (hardware abstract layer, HAL), the application program framework layer, from bottom to top. and the application layer.

The kernel layer is the layer between hardware and software. The kernel layer includes at least camera driver, audio driver, display driver, and sensor driver. Sensor drivers include but are not limited to image sensor drivers and acoustic sensor drivers.

In some embodiments, such as in a face recognition scenario, a kernel layer such as a camera driver is invoked to turn on the camera. For another example, the image sensor is called by the image sensor driver to complete the image acquisition.

The hardware abstract layer (hardware abstract layer, HAL) is located between the kernel layer and the application framework layer, and is used to define the interface for driving the hardware implementation of the application program, and convert the value implemented by the driving hardware into the software implementation programming language. For example, identify the value driven by the camera, convert it into a software program language and upload it to the application framework layer, and then call the corresponding function.

In some embodiments, the HAL can upload the face image collected by the camera 193 to the application framework layer for further processing.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer obtains the original input event from the kernel layer via the HAL, and identifies the control corresponding to the input event. The application framework layer includes some predefined functions.

As shown in Figure 3, the application framework layer can include view system, phone manager, resource manager, notification manager, window manager, etc.

In some embodiments, the application framework layer includes a first module. The first module is used to call the first camera to collect the first image through the camera driver, and call the second camera to collect the second image through the camera driver. Afterwards, a first area is determined on the second image, and a second area corresponding to the first area is determined on the first image according to the first area. Afterwards, the first module may invoke a display driver to display a user interface, where the user interface includes the first face and first information, and the first information is used to indicate the second area.

Optionally, the first module can also be arranged in other layers, and the first module can also be divided into more sub-modules. Each sub-module is used to perform a corresponding function.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100 . For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

The application layer can consist of a series of application packages.

As shown in Figure 3, the application package may include applications such as camera, video, call, WLAN, music, short message, Bluetooth, map, calendar, gallery, and navigation.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The following will take the electronic device as an example of a mobile phone with a structure such as that shown in Figure 2F and Figure 3. In some embodiments, when the mobile phone detects that the user has opened an application such as a camera or a mirror, the sunscreen of the embodiment of the present application can be activated. Frost smear detection method. Or, after the mobile phone detects that the user has input a preset instruction to the mobile phone, the sunscreen application detection method according to the embodiment of the present application can be activated. The embodiment of the present application does not limit the manner of enabling the sunscreen application detection function. As follows, the technical solutions of the embodiments of the present application are introduced by taking the method of turning on the sunscreen application detection method by turning on the sunscreen detection switch in the mirror application as an example.

Optionally, the user can instruct the mobile phone to start the mirror application and display the shooting preview interface by means of touch operation, button operation, air gesture operation or voice operation. Exemplarily, the mobile phone displays the main interface as shown in (a) in Figure 4. When an operation such as clicking the mirror application icon is detected by the user, the mobile phone starts and starts the mirror application, and calls a common camera (such as an RGB camera) to collect images to display the shooting preview interface shown in (b) in FIG. 4 . Or, when detecting the voice instruction operation of the user opening the mirror application, the mobile phone starts the mirror application, and calls a common camera to capture images, and obtains the shooting preview interface shown in (b) in FIG. 4 . The shooting preview interface may include one or more controls for realizing different functions. For example, it includes a setting control 401 and a sun protection detection control 402 . Wherein, the setting control 401 can be used to set the functions of the mirror application. The sun protection detection control 402 can be used to enable the sun protection detection function of the embodiment of the present application.

Optionally, the shooting preview interface may further include other controls. For example, controls that can be used to turn on or off the filter function, controls that can be used to turn on or off the flash function, controls that can be used to watch beauty tutorials, controls to control taking pictures, controls to adjust focus, and controls to adjust screen brightness.

In some embodiments, the mobile phone detects, for example, that the user clicks on the sunscreen detection control 402 on the shooting preview interface, and then starts the sunscreen application detection function. When the sunscreen application detection function is turned on, the mobile phone can detect the application of sunscreen. As a possible implementation, considering that the ultraviolet camera (an example of the second camera) can capture the imaging characteristics of ultraviolet rays and the sunscreen has the characteristics of absorbing ultraviolet rays, the mobile phone can call the ultraviolet camera to collect one or more ultraviolet images, The application of sunscreen is detected based on the one or more ultraviolet images.

As a possible implementation, after the mobile phone is triggered to enable the sun protection detection function, for example, after the user clicks on the sun protection detection control 402 shown in (b) of Figure 4, the mobile phone can continue to call the color camera (an example of the first camera) Acquire color images.

In some embodiments, if the image captured by the color camera (an example of the fifth image) is detected to meet the first condition, for example, there is a human face in the image, the mobile phone can automatically call the ultraviolet camera to collect the ultraviolet image. Acquire color images. Wherein, the ultraviolet camera and the color camera can collect images synchronously or alternately.

In some embodiments, considering the side effects of ultraviolet rays on the human body and the imaging effect of the ultraviolet camera, the mobile phone can be equipped with an ultraviolet supplementary light, and the mobile phone can detect and judge whether to turn on the ultraviolet supplementary light when collecting ultraviolet images based on the intensity of ambient ultraviolet rays. As a possible implementation, if the color camera detects that there is a human face in the captured image (an example of the fifth image), the mobile phone can call the ultraviolet camera to capture ambient ultraviolet rays, and determine the intensity of ambient ultraviolet rays according to the captured ultraviolet rays. In some cases, if the current ultraviolet intensity of the environment is lower than the third threshold, indicating that the ultraviolet intensity of the environment cannot reach the required intensity for imaging by the ultraviolet camera, the mobile phone turns on the ultraviolet fill light. Call the UV camera to collect a preset number of UV images, and call the color camera to collect color images. In other cases, if the current ultraviolet intensity of the environment is higher than the third threshold, the mobile phone calls the ultraviolet camera (the ultraviolet fill light may not be turned on) to collect a preset number of ultraviolet images. Optionally, after the preset number of ultraviolet images are collected, the ultraviolet camera is turned off, so as to reduce the power consumption caused by the ultraviolet camera.

It should be noted that after turning on the ultraviolet camera, the mobile phone may not turn off the color camera. Wherein, the image collected by the color camera can be displayed in the preview interface for the user to watch. The images captured by the UV camera can be used to determine the application of sunscreen.

Wherein, when the ultraviolet supplementary light is turned on to supplement the light source intensity of the ultraviolet camera, in order to reduce the ultraviolet irradiation of the ultraviolet supplementary light to the user, it is necessary to turn off the ultraviolet supplementary light in time. In the embodiment of the present application, turning off the ultraviolet supplementary light may be: automatically turning off the ultraviolet supplementary light after detecting that the collection of a preset number of ultraviolet images is completed. Alternatively, when it is detected that the UV supplementary light is turned on for t1, the UV supplementary light is automatically turned off. Alternatively, it may also be determined whether to turn off the ultraviolet supplementary light in combination with the turn-on duration and whether the preset number of ultraviolet images has been collected. For example, after it is detected that the UV supplementary light is turned on for t1 and a preset number of ultraviolet images are collected, the UV supplementary light is automatically turned off.

In some other embodiments, the ultraviolet supplementary light can be set to be linked with the ultraviolet camera. Under this setting, if there is a human face in the image detected by the ordinary camera, the UV camera and the linked UV fill light can be turned on, and a preset number of UV images can be collected through the UV camera. In this linkage setting method, since the ultraviolet camera and the ultraviolet supplementary light can be turned on in linkage, the complexity of realizing the acquisition of ultraviolet images can be reduced.

In some other embodiments, the mobile phone turns on the ultraviolet camera, which can also be triggered by the user's operation. For example, a control for triggering the ultraviolet camera to collect ultraviolet images is set in the user interface, and when a user's preset operation on the control is detected, the mobile phone can turn on the ultraviolet camera to collect ultraviolet images. Optionally, the color camera captures color images synchronously.

Afterwards, the mobile phone can detect the application of sunscreen based on the collected ultraviolet images. After detecting the application of sunscreen, the mobile phone prompts the user to apply the information of the sunscreen. Among them, the sunscreen application information includes but is not limited to any one or more of the following information: whether the user applies sunscreen, the location where the sunscreen is applied, the location where the sunscreen is not applied, whether the sunscreen is applied evenly, and whether the sunscreen is not applied evenly location, information on the thickness of sunscreen application in each area, and sunscreen recommendation information.

Exemplarily, as shown in (b) of Figure 4, the mobile phone detects the user's click operation on the sunscreen detection control 402, then detects the application of sunscreen, and can present the prompt box shown in (c) of Figure 4 to the user 403.

Through the sunscreen application detection scheme of the embodiment of the present application, the user can intuitively determine the precise position where the sunscreen is not applied uniformly or thinly applied from the mark of the mobile phone, so that the sunscreen can be reapplied according to the precise position, and the user can be fine-tuned. sun protection guidelines.

Optionally, in order to improve the accuracy of the detection results, it is also possible to collect ultraviolet images at a specific angle, and determine the application of sunscreen according to the ultraviolet images at a specific angle. For example, the mobile phone can collect the ultraviolet image of the front face, and determine the application of sunscreen according to the ultraviolet image of the front face. Optionally, the mobile phone can prompt the user to "align the front face with the mobile phone" before collecting the ultraviolet image of the front face, so that the user can align the front face with the mobile phone under the prompt.

Optionally, considering that in some scenes, some parts of the face have problems such as reflections in the collected ultraviolet images at a certain angle, resulting in inaccurate judgments when judging the application of sunscreen from the ultraviolet image at this angle, In the embodiment of the present application, it is possible to collect facial ultraviolet images from different angles, and judge the application of sunscreen according to the facial ultraviolet images from different angles. That is to say, the above-mentioned preset number of ultraviolet images collected by the ultraviolet camera can be multiple ultraviolet images, and the multiple ultraviolet images are images collected when the user's face turns to different angles, and the multiple ultraviolet images include people from different angles. face image information. Different UV images can correspond to different facial angles. The angle here may be the angle of the user's face relative to the mobile phone.

In this way, for a certain part of the face, there can be image information from different angles, and according to the image information of the different angles of the part, the application of sunscreen on the part can be judged more accurately.

As a possible implementation manner, the mobile phone may determine the angle of the human face in the image relative to the mobile phone based on the positional relationship between some key points of human faces detected in the image. Among them, the key points of the face used to detect the relationship such as angle and position can be the key point P1 of the left face, the key point P2 near the nose wing, and the key point P3 of the right face. The position relationship ratio between the key points=(P2-P1)/( P3-P2). The mobile phone can obtain the angle of the collected image according to the relationship between the positional ratio of the key points and the actual left-right rotation angle. Wherein, P2-P1 represents the distance between P2 and P1, and P3-P2 represents the distance between P3 and P2. Exemplarily, when the image collected by the mobile phone is the image 1401 shown in (a) of FIG. 5 , the mobile phone calculates the ratio of the positional relationship of the facial key points, and determines that the ratio corresponds to the frontal face angle. When the image collected by the mobile phone is the image 1402 shown in (b) of Figure 5, the mobile phone calculates the positional relationship ratio of the key points of the face, and determines that the ratio corresponds to the user's face turning 30 degrees to the right. Wherein, the

images

1401 and 1402 may be color images or ultraviolet images.

As a possible implementation, the RGB camera and the UV camera can work synchronously, which means that at the same moment, the mobile phone calls the RGB camera and the UV camera at the same time. Moreover, the mobile phone can recognize the key points of the face of the RGB image collected by the RGB camera, and judge the angle of the face in the RGB image to the mobile phone according to the key points of the face of the RGB image. The angle of the face relative to the mobile phone in the simultaneously acquired UV image.

As another possible implementation, the RGB camera may not work synchronously with the UV camera. In this case, the mobile phone can determine the key points of the face in the RGB image, and identify the key points of the face in the ultraviolet image according to the image matching algorithm and the key points of the face in the RGB image. Afterwards, the mobile phone can determine the angle of the human face in the ultraviolet image relative to the mobile phone according to the key points of the human face in the ultraviolet image.

Optionally, the mobile phone collects multiple facial ultraviolet images from multiple angles, which may be during the process of the user actively turning the head (turning the head left or right), and the mobile phone collects multiple facial ultraviolet images from multiple angles . Exemplarily, the user opens the mirror application, and instructs the mobile phone to enable the sun protection detection function by, for example, clicking on the sun protection detection control 402 shown in (b) of FIG. At the same time as the image, the RGB image is collected by the RGB camera. Optionally, the mobile phone can calculate the angle of the human face relative to the mobile phone in each RGB image, and select the target RGB image whose angle meets the preset conditions from multiple RGB images according to the calculated angle, and the corresponding ultraviolet light of the target RGB image The image can be used as a UV image to calculate sunscreen application. The angle conforming to the preset condition may mean that the angle of the human face in the RGB image relative to the mobile phone is within the first preset angle range of left rotation, or within the second preset angle range of right rotation, or within the angle range of facing the mobile phone . Exemplarily, taking the first preset angle and the second preset angle as 10-30 degrees, and the face is facing the mobile phone, the angle of the face relative to the mobile phone is between 5 degrees to the left and 5 degrees to the right as an example , the mobile phone has 10 RGB images, and in one RGB image, the angle of the face relative to the mobile phone is 25 degrees to the left, in one RGB image, the angle of the face relative to the mobile phone is 20 degrees to the right, and in one RGB image The angle of the human face relative to the mobile phone is rotated 25 degrees to the right, and the angle of the human face relative to the mobile phone is rotated 2 degrees in one RGB image. Then, the mobile phone regards these 4 RGB images as the target RGB images that meet the preset conditions , the UV images corresponding to these 4 RGB images can be used as the UV images used to calculate the application of sunscreen.

Of course, the first preset angle, the second preset angle, and the facing angle can be flexibly set separately. The preset conditions may also be set separately, which is not limited in this embodiment of the present application.

Exemplarily, the mobile phone collects 10 RGB images (RGB images 1-10) and 10 UV images (UV images 1-10), wherein the RGB image L (L is a positive integer, L is greater than or equal to 1, and less than or equal to 10) It is collected synchronously with the ultraviolet image L. Among them, if the RGB image 3-RGB image 5 is within the range of 30 degrees of left rotation to 30 degrees of right rotation, the mobile phone can use the UV image 3-5 corresponding to the RGB image 3-RGB image 5 as the UV used to calculate the application of sunscreen. image.

Or, optionally, the mobile phone collects multiple facial ultraviolet images from multiple angles, or the mobile phone prompts the user to turn the head, and the mobile phone collects facial ultraviolet images from different angles during the process of turning the head under the prompt of the mobile phone. image. When the mobile phone detects the user's operation such as clicking the sun protection detection control 402 shown in (b) of FIG. 4 , the mobile phone starts the sun protection detection function. Optionally, as shown in (a) of FIG. 6 , the mobile phone may prompt the user to face the mobile phone in the preview interface, and collect RGB images through RGB images, and collect ultraviolet images through ultraviolet cameras. The mobile phone can judge the angle of the face in the RGB image relative to the mobile phone according to the RGB image. After determining that the face is facing the mobile phone, the RGB image of the front face (an example of the third image) and the ultraviolet image of the front face (face When the angle relative to the mobile phone satisfies the second condition, that is, meets the angle range of the frontal face), the mobile phone can continue to collect RGB images from other angles and ultraviolet images from other angles. Optionally, after determining that the RGB image of the front face and the ultraviolet image of the front face have been collected, the mobile phone may also present a prompt box 405 (an example of the second information) shown in (b) of FIG. 6 in the preview interface, to prompt the user to turn their head to the left. The mobile phone collects an RGB image (the fourth image) through the RGB image, and collects an ultraviolet image (the sixth image) through the ultraviolet camera. When the mobile phone determines that the face angle satisfies a certain left-turn angle based on the collected RGB images, the mobile phone can continue to collect RGB images from other angles and UV images from other angles. Later, optionally, after determining that the RGB image of the left side face and the ultraviolet image of the left side face have been collected, the mobile phone can also present a prompt box 406 shown in (c) of FIG. Turn your head to the right. Optionally, the image presented on the preview interface may be a color image. When the user turns the head, the mobile phone can call the ultraviolet camera to collect multiple ultraviolet images and call the color camera to collect multiple color images. Among them, the color image can be used to present the preview interface and assist in determining the angle of the face in the ultraviolet image relative to the mobile phone, and multiple ultraviolet images can be used to determine the application of sunscreen. Afterwards, the mobile phone can present a prompt box such as that shown in (d) of Figure 6, so as to remind the user of the application of sunscreen and provide effective and detailed guidance to the user on sun protection.

Certainly, the interface for the mobile phone to prompt the user to turn the head is not limited to the interface shown in FIG. 6 . For example, directly prompt the user "Please turn your head left or right to collect multi-angle face images".

It should be noted that the frontal face image mentioned in the embodiment of the present application may be an image within a certain angle deviation range, and is not limited to an image in which the face is strictly facing the mobile phone.

Optionally, the mobile phone presents the prompt box shown in (a) of FIG. 6 , which may be to present the prompt box 405 immediately after detecting the user's operation such as clicking the sun protection detection control 402 shown in FIG. 4 (b) . It may also be that after the user's operation such as clicking on the sun protection detection control 402 is detected, the prompt box 405 is presented when the user does not detect turning the head within a preset period of time (which can be set flexibly). The embodiment of the present application does not limit the timing at which the mobile phone prompts the user to turn the head.

As follows, still taking the sunscreen detection function enabled in the mirror application as an example, the specific implementation method of detecting sunscreen application based on ultraviolet images is described in detail.

First, introduce the process of detecting whether sunscreen is applied. As shown in Figure 7, the process includes the following steps:

S101. The mobile phone recognizes the face image in the RGB image.

As a possible implementation, it is considered that RGB images have richer imaging details and higher quality than ultraviolet images. Therefore, the mobile phone can recognize the face in the RGB image and match the corresponding area of the recognized face to the UV image.

Wherein, the RGB image may be collected by calling the RGB camera of the mobile phone after the user opens the mirror application, and the timing of collecting the RGB image is not limited. For example, it may be that when the ultraviolet camera collects the ultraviolet image, the RGB camera collects the RGB image synchronously, or the RGB camera and the ultraviolet camera collect alternately. In the embodiment of the present application, the synchronous acquisition of the RGB camera and the ultraviolet camera after the ultraviolet camera is turned on is used as an example for illustration, but the technical solution of the embodiment of the present application is not limited thereto.

As a possible implementation, the mobile phone recognizes the face image in the RGB image based on the face recognition algorithm. Optionally, the mobile phone can extract facial feature points from the recognized face image, and mark eyes, eyebrows, mouth and other facial parts in the face image that usually do not apply sunscreen according to the extracted facial feature points. Later, when the mobile phone calculates the application of sunscreen, it does not need to calculate the eyes, eyebrows and other areas of the face image. In this way, the calculation amount of the mobile phone can be reduced. Exemplarily, as shown in (a) of Figure 8, the mobile phone extracts the facial feature points in the face image, and as shown in (b) of Figure 8, marks the facial feature points in the face image, such as eyes and mouth The area surrounded by their respective feature points.

It should be noted that in FIG. 8, the outline surrounding the imaging area such as the eye is taken as a box, and the box area is marked as an example. In other embodiments, the outline surrounding the imaging area such as the eye can also be other The shape, such as a circle, can also be, for example, a shape that fits the contours of the edges of imaging areas such as eyes.

In the embodiment of the present application, the eyes, lips and other areas that do not need to be applied with sunscreen can be removed from the face image to obtain the fourth area. The follow-up, that is, the face area processed in the subsequent steps is to remove the eyes, lips, eyebrows, etc. and other areas (fourth area). For example, the mobile phone may determine the first area in the fourth area of the second image according to the pixel values of the pixels included in the fourth area, without performing traversal processing on pixels outside the fourth area.

Alternatively, this part of the area is not eliminated, but only this part of the area is marked, and subsequently, the mobile phone can choose not to process the pixels of this part of the area according to the mark made.

S102. The mobile phone acquires the human face (an example of the second human face) in the ultraviolet image (an example of the second image) according to the human face (the first human face) in the RGB image (an example of the first image).

Wherein, the first human face and the second human face belong to the same user.

As a possible implementation manner, after the mobile phone acquires the human face image in the RGB image, the mobile phone may acquire the human face image in the ultraviolet image through, for example, an image matching algorithm. Exemplarily, the image matching algorithm includes but not limited to a scale-invariant feature transform (scale-invariant feature transform, SITF) algorithm. In the SITF algorithm, the mobile phone can extract the key points of the face image in the RGB image. Key points can be, but are not limited to, points that will not disappear due to factors such as illumination, scale, and rotation, such as the lowest point of the chin, the highest point of the nose, and so on. The mobile phone can also obtain the feature vector of each key point of the face, and locate each key point in the ultraviolet image according to the feature vector of each key point of the face in the RGB image. In this way, the mobile phone can determine the face image according to the key points of the face in the ultraviolet image.

Taking the location of the lowest point of the chin and the highest point of the alar in the ultraviolet image as an example, the mobile phone obtains the eigenvector 1 of the lowest point of the chin and the eigenvector 2 of the highest point of the alar in the RGB image, and finds the eigenvectors in the ultraviolet image. The pixel close to eigenvector 1 is used as the lower edge point of the chin, and the pixel point whose eigenvector is closest to eigenvector 2 is found in the ultraviolet image as the highest point of the alar.

Exemplarily, as shown in (b) and (c) of FIG. 8 , the mobile phone may recognize the face image in the ultraviolet image according to the face image in the RGB image. Among them, the mobile phone can also identify the positions of the eyes and lips in the ultraviolet image according to the positions of the eyes and lips in the RGB image. Subsequently, the processing of the positions of the eyes and lips can be different from other areas of the face.

In the embodiment of the present application, optionally, after acquiring the human face image in the ultraviolet image, the mobile phone may also execute S103.

It should be noted that the above-mentioned embodiment is mainly described by taking the color camera as an RGB camera, and the collected color image is an RGB image as an example. The color camera in the embodiment of the present application can also be other types of cameras. Correspondingly, the collected The color image can be of other types.

S103. The mobile phone preprocesses the face image in the ultraviolet image.

Wherein, the preprocessing includes but not limited to filtering processing, so as to eliminate the influence of noise in the image. Filtering may include, but is not limited to, mean filtering, median filtering, Gaussian filtering, and the like.

In some embodiments, the mobile phone may perform preprocessing on all regions of the face image in the ultraviolet image. Or, in order to reduce the amount of preprocessing calculations, the mobile phone can also preprocess the areas in the face image except the eyes, lips, etc. (that is, areas that usually do not need to apply sunscreen), that is, areas other than the marked area.

S104. The mobile phone recognizes the application of sunscreen based on the face image in the ultraviolet image.

In the embodiment of the present application, the mobile phone may determine the first area on the second face of the ultraviolet image (second image) according to the image features of the ultraviolet image, and determine the application of sunscreen in the first area.

Wherein, the first area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is unevenly applied, an area where sunscreen is evenly applied, and the thickness of sunscreen applied is greater than In the area of the first threshold, the area where the thickness of the sunscreen application is smaller than the second threshold. Image features include but are not limited to one or more of the following features: the average pixel value of the image area, the maximum pixel value, the minimum pixel value, the difference between the pixel values of adjacent pixels in the image area, the color of each pixel in the image area, the image Regional contrast, histogram, histogram of oriented gradients (HOG), standard deviation, color scale, mean square deviation, variance. The color value of a pixel includes the pixel value of each channel of RGB.

The principle of determining the first region according to the image features of the ultraviolet image is introduced as follows.

First, the principle of determining the sunscreen application area based on the pixel value in the UV image feature is introduced. As mentioned above, sunscreen can absorb ultraviolet rays, so in the ultraviolet image, the brightness of the area where sunscreen is applied is lower, and the corresponding pixel value is usually lower, and the brightness of the area without sunscreen is higher, corresponding to Pixel values are usually higher. Based on this principle, the mobile phone can detect the pixel value of the face image, and use the area with a pixel value higher than the threshold as the area without sun protection, and the area with the pixel value less than or equal to the threshold as the area with sun protection.

In the embodiment of the present application, an area whose pixel value is higher than the pixel threshold may be referred to as a highlighted area. Areas with pixel values less than or equal to the threshold can be called non-highlight areas.

As a possible implementation manner, a pixel window with a size of W1*W2 may be set. The face image in the ultraviolet image can be traversed through the sliding window. Every time the window slides to a position, the average pixel value ˉF of the face image in the window area is counted once, and the window area of ˉF<T1 is divided into non-highlight areas , divide the window area of ˉF>T1 into the highlighted area.

Exemplarily, still referring to (c) of FIG. 8 , the mobile phone detects that the imaging region in the ultraviolet image whose pixel average value is higher than the threshold (lighter in color) is the region 408a, then the region 408a can be regarded by the mobile phone as a place where no sunscreen is applied The corresponding imaging area of the site. On the contrary, the imaging area 407a whose pixel mean value is smaller than the threshold (darker color) is regarded by the mobile phone as the imaging area corresponding to the part where the sunscreen is applied.

Alternatively, the mobile phone may iterate over the pixel values of each pixel, classify pixels with pixel values greater than the threshold into highlighted areas, and classify pixels with pixel values less than or equal to the threshold into non-highlight areas.

It can be understood that when other conditions are the same, the thicker the sunscreen is applied on a certain part, the stronger the absorption ability of ultraviolet rays is usually, then the imaging pixel value of this part in the ultraviolet image is usually lower (the darker the color). Conversely, the thinner the sunscreen is applied on a certain part, the weaker its ability to absorb ultraviolet light, and the higher the imaging pixel value (lighter the color) of this part in the ultraviolet image. Based on this principle, the mobile phone can also determine the uniformity and thickness of the sunscreen application based on the pixel values included in the ultraviolet image. Taking Figure 1 as an example, compared with Area 1, the application of sunscreen in Area 2 is thinner, so the imaging pixel value of Area 2 in the ultraviolet image is higher and the color is lighter.

After the mobile phone determines the application of sunscreen (such as the area where the sunscreen is applied and the area where the sunscreen is not applied), the mobile phone can prompt the user for the application of the sunscreen. In the embodiment of the present application, the application of sunscreen can be marked in a color image or a grayscale image. Figure 8 describes the process of labeling sunscreen application in color images, and Figure 9 describes the process of labeling sunscreen application in grayscale images. A detailed description will be given below in conjunction with FIG. 8 and FIG. 9 . Wherein, the implementation process of (a) and (b) in FIG. 9 can refer to (a) and (b) in FIG. 8 , which will not be repeated here.

In some embodiments, the mobile phone may mark prompt information (first information) in the face (first face) of the color image. Wherein, the color image may be collected by the mobile phone calling the color camera after the user opens the mirror application. The embodiment of the present application does not limit the timing of collecting color images. For example, it may be that when the ultraviolet camera collects the ultraviolet image, the color camera collects the color image synchronously, or the color camera and the ultraviolet camera collect alternately.

As a possible implementation, the mobile phone can use an image matching algorithm to determine the second area corresponding to the first area on the face of the color image, that is, match the first area in the ultraviolet image to the second region of the color image. That is, sunscreen reminders can be marked in color images. Exemplarily, the mobile phone determines that the area where sunscreen is applied is 407a (an example of the first area) according to the ultraviolet image shown in (c) of FIG. 407a' (an example of the second area) of the RGB image shown in (d), and mark the area 407a' where the sunscreen is applied in the RGB image. It is understood that the mobile phone can also match the unapplied sunscreen area 408a (another example of the first area) of the UV image to the area 408a' of the RGB image, and mark the area 408a' (another example of the second area) in the RGB image. example).

Wherein, the second area in the color image is the area corresponding to the first area in the ultraviolet image. The second area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, an area where the thickness of sunscreen is greater than the first threshold, and where sunscreen is applied Regions with a thickness less than the second threshold.

Optionally, different colors may be used to mark different application conditions of the sunscreen so that the user can clearly identify the application conditions of the sunscreen. Optionally, in order to distinguish different areas more easily, the difference between colors used to represent different application situations may be relatively large. For example, areas with sunscreen applied are marked with a green outline (an example of the first message) to indicate where sunscreen is applied, and areas without sunscreen are marked with a red outline to indicate where no sunscreen is applied . As another example, mark only the places where no sunscreen is applied, or, mark only the places where sunscreen is applied. Labeling sunscreen application information in color images makes it easier for users to understand these labeling information. Moreover, color images can often record more detailed skin conditions, which is more conducive to guiding users to reapply sunscreen.

Optionally, the mobile phone may present a prompt text on a corresponding area of the face in the color image. For example, near the area where sunscreen is applied, the user is prompted that this area is the area where sunscreen is applied.

Optionally, the mobile phone may hide the prompt text, and when it is detected that the user clicks on a corresponding area, the mobile phone presents the prompt text. For example, when it is detected that the user clicks on the area where sunscreen is applied, the mobile phone then prompts the user that this area is the area where sunscreen is applied.

In this embodiment, by marking contour lines of different colors in the color image, automatically marking prompt text, and marking prompt text based on user operations, the user can be explicitly or implicitly prompted about the application of sunscreen.

Optionally, the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area. Exemplarily, as shown in FIG. 4 , the mobile phone can also explicitly prompt the thickness and uniformity of the sunscreen in the form of text and the like. Or, when the user taps the area surrounded by the dotted frame, the mobile phone displays text prompting the uniformity and thickness of the sunscreen, etc.

In other embodiments, after the mobile phone determines the application of sunscreen according to the ultraviolet image, it may also mark the application of sunscreen in the grayscale image. The grayscale image may be an image obtained by processing an ultraviolet image.

As a possible implementation, the ultraviolet image can be processed to increase the difference in pixel difference between the sunscreen-applied area and the non-sunscreen-applied area. For example, the color of the area with sunscreen in the ultraviolet image can be darkened, and the color of the area without sunscreen in the ultraviolet image can be lightened, so that the area with sunscreen in the ultraviolet image and the area without sunscreen The contrast of color depth deepens.

Optionally, the grayscale image used to mark the application of sunscreen may be a binary image. That is to say, the ultraviolet image can be processed to obtain the corresponding binary image, and the application of sunscreen can be marked on the binary image. Among them, the binary image, that is, the pixel value of the pixel in the image, has two situations. Optionally, in order to increase the pixel value difference between the smeared area and the unsmeared area, the difference between the two pixel values of the binary image may be set to be greater than a certain threshold. Exemplarily, the pixel value of any pixel in the binary image may be 255 (indicating white), or 0 (indicating black).

Taking the binary image including black pixels and white pixels as an example, as a possible implementation, the mobile phone calculates the binary image corresponding to the ultraviolet image, which can be implemented as follows: the mobile phone traverses the pixels included in the face image in the ultraviolet image, and for traversal For pixels whose pixel value is greater than the threshold value, the pixel value of the pixel at the corresponding position in the binary image is set to white by the mobile phone (the pixel value of the white pixel can be 255, for example), and for the pixel value traversed that is less than or equal to the threshold value Pixel, the pixel value of the corresponding position of the pixel in the binary image is set by the mobile phone as black (the pixel value of the black pixel may be 0, for example). Exemplarily, the mobile phone traverses each pixel included in the ultraviolet face image as shown in (c) of FIG. is less than the binarization threshold, then, the shadow area 407a of the ultraviolet image is in the corresponding position in the binary image, that is, the pixel value of the pixel included in the area 407a "is set to black. The pixel value of the pixel included in the white area 408a is greater than the threshold ( Corresponding to the area where sunscreen may not be applied), then, the white area 408a of the ultraviolet image is at the corresponding position in the binary image, that is, the pixel value of the pixel included in the area 408a' is set to white.

It should be noted that the embodiment of the present application does not limit the specific implementation of calculating the binary image by the mobile phone and the threshold used when calculating the binary image.

Exemplarily, as shown in (c) of FIG. 9 , it is assumed that the shaded area 407a in the ultraviolet image represents the area where sunscreen is applied, wherein the higher the density of the oblique lines in the shadow, the thicker the sunscreen is applied (the corresponding area's The lower the pixel value), the lower the density of the slash, which means the thinner the sunscreen is applied (the higher the pixel value of the corresponding area). When the user observes with the naked eye, it may be difficult to distinguish some areas with relatively similar pixel values. For example, the pixel values of the area where the sunscreen is thinly applied are relatively close to the pixel values of the area where no sunscreen is applied, and the user may apply sunscreen. area misjudged as an area where no sunscreen was applied, and it was impossible to clearly distinguish the application of sunscreen. To improve the discrimination between applied and unapplied areas, the phone can mark whether sunscreen is applied or not through binary images. Exemplarily, the mobile phone may calculate a binary image corresponding to the ultraviolet image shown in (c) of FIG. 9 , and the binary image is shown in (d) of FIG. 9 . It can be seen from (d) of FIG. 9 that the difference between the two pixel values in the binary image is relatively large. In this way, the color contrast between the area where sunscreen is applied and the area where sunscreen is not applied is deepened, and the user can more easily and intuitively distinguish which parts are applied with sunscreen and which parts are not.

In some other embodiments, marking the application of sunscreen in the grayscale image may also be directly marking the application of sunscreen in the ultraviolet image. For example, for areas in the UV image where sunscreen is not applied, mark the area with a shade or a dashed box.

In some other embodiments, as shown in FIG. 10 , the mobile phone may provide two option controls of a grayscale view and a color view. Mark sunscreen application on the color image when it detects an action from the user indicating use of the color view, for example, when the phone detects that the user taps a color view control. When detecting the user's operation indicating to use the grayscale view, for example, when the mobile phone detects that the user clicks on the grayscale view control, a grayscale image (such as a binary image) is presented, and the application of sunscreen is marked by the grayscale image.

Considering that in some cases, there may be interference factors affecting the detection. For example, in the ultraviolet image, the imaging features of moles on the face are similar to those of the parts where sunscreen is applied. Therefore, if the user has moles on the face, the moles may be misjudged as the parts where sunscreen is applied , which means that moles are likely to interfere with the test results of sunscreen. For another example, in the ultraviolet image, the imaging characteristics of the part where sunscreen is applied but the reflection is similar to the imaging characteristics of the part where no sunscreen is applied, and the pixel values are higher. Based on the characteristics of the reflective part, during the detection process, the part with sunscreen applied but the reflective part may be misjudged as the part without sunscreen.

Optionally, in order to reduce the misjudgment of the detection result, the recognition accuracy of the detection is further improved. For example, in order to reduce the probability of moles and other parts being misjudged as areas where sunscreen is applied, the mobile phone can exclude the imaging area of interference factors such as moles, that is, the interference area is not determined as the area where sunscreen is applied. Considering that when users apply sunscreen, the area where the sunscreen is usually applied will be larger. As a possible implementation, the mobile phone determines a plurality of target sub-regions in the fourth region according to the pixel values of the pixels contained in the fourth region (the region of the human face excluding eyes, eyebrows, and lips). Among the plurality of target sub-regions, a target sub-region whose area is greater than or equal to a fourth threshold is determined as the first region. The target sub-area satisfies a third condition, and the third condition includes but is not limited to any one or more of the following: the pixel mean value of the target sub-area is less than the first threshold, and the pixels contained in the target sub-area value is less than the second threshold. That is to say, for a non-highlight area in the ultraviolet image whose area is smaller than the area threshold, the mobile phone does not determine it as an area for applying sunscreen. For the non-highlight area in the UV image whose area is greater than or equal to the area threshold, the mobile phone determines it as the area where sunscreen is applied.

Exemplarily, as shown in (a), (b) and (c) of FIG. 11 , areas in the ultraviolet image where the pixel values are smaller than the pixel value threshold (darker in color) are 407a and 409a. Wherein, for the area 407a whose area is greater than or equal to the threshold, the mobile phone determines the corresponding part as the part where sunscreen is applied. For the area 409a whose area is smaller than the threshold, since it may be the image of interference factors, the mobile phone does not determine its corresponding part as the part where sunscreen is applied. Finally, the part where the sunscreen is applied calculated by the mobile phone is the part corresponding to the area 407a, and the part where the sunscreen is not applied calculated by the mobile phone is the part corresponding to the area 408a. Optionally, the mobile phone may display a prompt box as shown in (d) of FIG. 11 to prompt the user to reapply the parts where no sunscreen has been applied.

Optionally, in order to reduce the probability that the reflective parts are misjudged as parts without sunscreen, in the embodiment of the present application, the facial ultraviolet images from multiple angles can be processed, and according to the processing results of the multi-angle facial ultraviolet images Screen for reflective areas to identify areas where sunscreen is applied and areas where sunscreen is not. Specifically, as a possible implementation, referring to FIG. 12A, the above step S104 may include the following steps:

S201a, the mobile phone acquires the highlighted area 408a and the non-highlighted area 407a of the human face image in the ultraviolet image 901 .

As shown in Figure 13, the ultraviolet images 901-903 are three ultraviolet images collected by the mobile phone through the ultraviolet camera after the sun protection detection function is turned on. The ultraviolet images 901-903 meet the preset angle conditions, for example, the angle of the face in the image relative to the mobile phone is within the range of 30 degrees to the left and 30 degrees to the right. Wherein, the ultraviolet image 901 may be a reference image selected from multiple ultraviolet images from multiple angles. Optionally, the reference image is an image including more imaging details. In one example, the reference image may be an ultraviolet image from a frontal perspective. Of course, the reference image can be determined flexibly, and the manner of determining the reference image is not limited.

As a possible implementation, the mobile phone can determine the human face image in the ultraviolet image 901 according to the scheme of the above-mentioned embodiment, and can determine the human face image in areas other than the eyes and lips of the human face image (the area where sunscreen is usually not applied). Find highlighted areas and non-highlighted areas in order to reduce computational complexity.

As a possible implementation manner, for the ultraviolet image 901 , threshold segmentation may be performed according to the pixel value, so as to locate the highlighted region and the non-highlighted region in the ultraviolet image 901 . For example, if the threshold is set to T1 (can be set flexibly), the mobile phone can traverse the pixels in the image 901. For the pixels in the image 901, when the pixel value f(x, y)>T1, the mobile phone will use this pixel as the composition of the highlighted area On the contrary, when the pixel value f(x,y)<T1, the mobile phone regards this pixel as the constituent pixel of the non-highlight area. Optionally, the mobile phone can also calculate the relative position coordinates of the highlighted area relative to one or more specific parts of the eyes, nose, and lips, and the mobile phone can calculate the relative position coordinates of the non-highlight area relative to one or more of the eyes, nose, and lips. The relative location coordinates of a particular part.

Exemplarily, as shown in (a) of FIG. 13 , the mobile phone calculates that the highlighted area in the ultraviolet image 901 is 408a and the non-highlighted area 407a.

S202a, according to the highlighted area 408a of the ultraviolet image 901, acquire the target area 408b corresponding to the highlighted area 408a in the ultraviolet image 902, and acquire the target area 408c corresponding to the highlighted area 408a in the ultraviolet image 903.

As a possible implementation, an image matching algorithm can be used to match corresponding regions in ultraviolet images from different angles.

Exemplarily, still referring to FIG. 13 , according to the position of the highlighted area 408a in the ultraviolet image 901 and the image matching algorithm, the mobile phone can acquire the target area 408b corresponding to the highlighted area 408a in the ultraviolet image 902 . Among them, the positional relationship between the highlighted area 408a and specific parts such as nose and eyes in the ultraviolet image 901 and the positional relationship between the target area 408b and specific parts such as nose and eyes in the ultraviolet image 902 are consistent.

As another example, still referring to FIG. 13 , according to the position coordinates of the highlighted area 408a in the ultraviolet image 901 and the image matching algorithm, the mobile phone can acquire the target area 408c corresponding to the highlighted area 408a in the ultraviolet image 903 .

S203a. Determine whether the

target areas

408b, 408c include target non-highlight areas. If at least one target area includes a target non-highlight area, execute step S204a, and if none of the target areas includes a target non-highlight area, execute step S205a.

Taking the ultraviolet image 902 as an example, as shown in FIG. 13 , if the target area 408b of the ultraviolet image 902 includes a non-highlight area, it means that the highlighted area 408a in the ultraviolet image 901 probably includes a reflective area and an area without sunscreen. In this case, directly determining the highlighted area 408a as an area where no sunscreen is applied may lead to misjudgment. In order to avoid this kind of misjudgment, the following step S204a may be performed to remove a region that may cause a misjudgment from the highlighted region 408a, and the removed region may be replaced by a corresponding non-highlighted region. Conversely, if one or more target areas corresponding to the highlighted areas in the frontal ultraviolet image (such as the target area 408b shown in Figure 13(b) and the target area 408c shown in Figure 13(c)) do not include non-highlighted area, that is, all the areas of the UV image at all angles are highlighted areas, which means that the highlighted area 408a in the UV image of the frontal face is indeed caused by not applying sunscreen. Then, the following areas can be directly executed without adjusting the area to be detected. Step S205a.

As another example, still referring to FIG. 13 , for the target area 408c (included in the ultraviolet image 903 ) corresponding to the highlighted area 408a, the mobile phone detects whether it includes a non-highlighted area. After detection, the target area 408c includes a non-highlight area 411a.

S204a, according to the highlighted area 408a, the non-highlighted area 407a and the target non-highlighted area of the ultraviolet image 901, determine the area where sunscreen is applied and the area where sunscreen is not applied.

Exemplarily, as shown in FIG. 13 , the mobile phone can screen reflective parts according to ultraviolet images 901 , 902 , and 903 . Specifically, as shown in (a) of FIG. 13 , the pixel value of the shaded area 407a is smaller than the pixel value threshold, and the pixel value of the area 408a is greater than the pixel value threshold. The mobile phone may preliminarily determine the part corresponding to the area 408a as the part where sunscreen may not be applied according to the above method, and preliminarily determine the area 407a as the area where the sunscreen is applied. Wherein, the white pixels in the region 408a may be due to the fact that no sunscreen is applied to the part corresponding to the region 408a, or it may be because the part corresponding to the region 408a is not applied with sunscreen, and the other part is coated with sunscreen but reflects light. In order to further detect whether the area 408a really corresponds to the area without sunscreen, the mobile phone can be in the corresponding positions of the ultraviolet image 902 shown in Figure 13(b) and the ultraviolet image 903 shown in Figure 13(c), that is, in the

areas

408b and 408c Find if there is a non-highlighted region of interest. After searching, there is a target non-highlight area 410a in the area 408b, which indicates that the part corresponding to the target non-highlight area 410a is likely to be covered with sunscreen but reflects light at a frontal angle. The phone then determines area 410a as the area to apply sunscreen. Similarly, if there is a target non-highlight area 411a in the target area 408c of the ultraviolet image 903, then the mobile phone determines the area target non-highlight area 411a as the area where sunscreen is applied. In this way, the area where sunscreen is finally calculated by the mobile phone includes area 407a, area 410a and area 411a, and the area where no sunscreen is applied is the area in 408a except area 410a and area 411a. Optionally, (d) of FIG. 13 shows areas where sunscreen is applied and areas that are not applied. It should be noted that (d) of FIG. 13 only exemplarily shows the area where the sunscreen is applied and the area that is not applied, and the mobile phone may not generate the image shown in (d) of FIG. 13 during actual processing.

Optionally, the mobile phone may display the prompt box 801 shown in FIG. 13 , wherein the area enclosed by the dotted frame is the area where sunscreen is applied, and the area outside the dotted frame is the area where no sunscreen is applied.

S205a, according to the non-highlight area 407a and the highlighted area 408a of the image 901, determine the area where the sunscreen is applied and the area where the sunscreen is not applied.

In other cases, after the mobile phone preliminarily determines the candidate area (such as the area 408a shown in FIG. Areas not covered with sunscreen. Still taking Figure 13 as an example, after the mobile phone preliminarily determines the area 408a where no sunscreen is applied according to the ultraviolet image 901, it searches the ultraviolet image 902 and the ultraviolet image 903, if the corresponding position of the ultraviolet image 902 (area 408b) and the corresponding position of the ultraviolet image 903 There is no target non-highlight area in (408c), which means that there is no misjudgment problem caused by the reflection problem, and the area 408a is indeed an area where no sunscreen is applied. That is, the mobile phone finally determines that the area without sunscreen is 408a, and the area with sunscreen is 407a.

The embodiment of the present application also provides a detection method, which can determine the area where sunscreen is applied and the area where sunscreen is not applied. Referring to Figure 12B, the method includes:

S201b. The mobile phone acquires the highlighted area and/or the non-highlighted area of the face image in the multiple ultraviolet images.

For an ultraviolet image, the ultraviolet image may include highlighted regions and/or non-highlighted regions. The mobile phone can determine the non-highlight area in the ultraviolet image according to the highlighted area in the ultraviolet image, and similarly, the mobile phone can determine the non-highlight area according to the non-highlight area in the ultraviolet image. Based on this principle, step S201b can be implemented as: the mobile phone acquires the highlighted area of the face image in each of the multiple ultraviolet images, and can determine the non-highlighted area according to the highlighted area. Or, the mobile phone acquires the non-highlight area of the face image in each ultraviolet image, and can determine the highlight area according to the non-highlight area. Or, the mobile phone obtains the non-highlight area and the high light area of the face image in each ultraviolet image.

Exemplarily, still referring to FIG. 13 , after the sun protection detection function is turned on, the ultraviolet images collected by the channel ultraviolet camera of the mobile phone include ultraviolet images 901-903. The mobile phone can respectively determine the face images 1-3 in the ultraviolet images 901-903 according to the method of the above-mentioned embodiment, and respectively determine the highlighted area and the non-highlighted area in the face images 1-3. Exemplarily, in FIG. 13, the highlighted area of face image 1 in ultraviolet image 901 is 408a, the non-highlighted area is 407a, the highlighted area of human face image 2 in ultraviolet image 902 is 408b, and the

non-highlighted area

410a and 407b, the highlighted area of the face image in the ultraviolet image 903 is 408c, and the non-highlighted areas are 407c and 411a.

S202b. The mobile phone determines the area where the sunscreen is applied and the area where the sunscreen is not applied according to the highlighted area and/or the non-highlighted area.

Exemplarily, in the ultraviolet image 901, the highlighted area of the face image 1 is 408a, the non-highlighted area is 407a, the highlighted area of the human face image 2 in the ultraviolet image 902 is 408b, and the non-highlighted areas are 410a and 407b , the highlighted region of the face image in the ultraviolet image 903 is 408c, the non-highlighted region is 407c and 411a, the region 407a in the ultraviolet image corresponds to 407b in the ultraviolet image 902, and corresponds to 407c in the ultraviolet image 903, in the ultraviolet image The region 408a of , corresponds to 408b in the ultraviolet image 902 and corresponds to 408c in the ultraviolet image 903 . The area 411a in the ultraviolet image 903 does not have a corresponding relationship with the area 410a in the ultraviolet image 902, and the non-highlight areas corresponding to the

areas

410a and 411a do not exist in the ultraviolet image 901. The area where the mobile phone determines to apply sunscreen is shown in Figure 13 As shown in (d) 407a, 410a, 411a, the area without sunscreen is the area in 408a except 410a and 411a.

As another example, the mobile phone can obtain the non-highlight region 407a in the ultraviolet image 901, the

non-highlight regions

407b and 410a in the ultraviolet image 902, and the

non-highlight regions

407c and 411a in the ultraviolet image 903, and according to these non-highlight regions Highlight areas, identify highlighted areas, and identify areas where sunscreen is applied and areas where sunscreen is not applied.

As another example, the mobile phone can obtain the highlighted area 408a in the ultraviolet image 901, the highlighted area of the ultraviolet image 902, that is, the areas in the human face other than the eyes, eyebrows, mouth, 407b, and 410a, and the highlighted area in the ultraviolet image 903. Highlighted areas, that is, areas other than the eyes, eyebrows, mouth, 407c, and 411a of the face, and based on these highlighted areas, determine the non-highlighted areas, and determine the area where sunscreen is applied and the area where sunscreen is not applied .

The embodiment of the present application also provides a detection method, which can be used to detect the application uniformity of the sunscreen. As a possible implementation, see Figure 14. In the embodiment of the present application, multiple image features can be extracted from the area to be detected, and the feature vector composed of multiple image features can be input into the classifier, and the classifier will output the sun protection of the corresponding parts Evenness of cream application.

Wherein, the area to be detected may be an area where sunscreen is applied. Optionally, the manner of determining the application area of sunscreen may refer to corresponding embodiments such as FIG. 12A or FIG. 12B , which will not be repeated here.

Wherein, the feature vector includes a plurality of image features.

First, the training process of the classifier involved in the embodiment of the present application is introduced. The training process is shown in Figure 15. To train a classifier for identifying whether the sunscreen is applied evenly, N (N is a positive integer) samples are required. The samples are the feature vectors and their labels of image regions whose application uniformity is known ( Characterize whether the corresponding part of the image area is evenly coated with sunscreen), and train multiple samples to obtain a classifier.

Optionally, before training the classifier, data such as training feature vectors may be processed, such as smoothing and normalization. Among them, the normalization process can reduce the complexity of the algorithm. Smoothing can further include operations such as noise reduction and fitting to reduce the impact of statistical errors.

Optionally, in order to improve the recognition accuracy of the classifier, the classifier may be evaluated and tested. When the recognition rate of the classifier reaches a certain threshold, it indicates that the classifier has been trained. When the recognition rate of the classifier is low, the classifier can continue to be trained until the recognition accuracy of the classifier reaches a certain threshold.

Optionally, the training process of the classifier can be performed on the device side (such as a terminal such as a mobile phone) or on the cloud side (such as a server). Training can be offline training or online training. The embodiment of the present application does not limit the specific training method of the classifier. Subsequently, the trained classifier can output whether the sunscreen in the corresponding image area is evenly applied according to the input feature vector of the image area whose application uniformity is unknown.

Fig. 16 shows the flow of the method for identifying the uniformity of sunscreen application based on the classifier. As shown in Figure 16, the method includes:

S301. Divide the area to be detected into multiple sub-areas, and extract feature vectors of the multiple sub-areas respectively.

Wherein, the area to be detected is the area where sunscreen is applied. The specific implementation of determining the area to apply sunscreen can refer to the corresponding embodiments such as FIG. 12A or FIG. 12B , which will not be repeated here.

Exemplarily, as shown in Figure 17, the mobile phone divides the area to be detected (that is, the area where sunscreen is applied) into blocks, and the multiple image areas obtained are respectively Area 1-Area 3 shown in (b) of Figure 17, and respectively Extract feature vectors for regions 1-3. Suppose the eigenvector of region 1 is

The eigenvectors of region 2 are

The eigenvectors of region 3 are

Among them, f _ji is the ith (i is a positive integer) image feature of region j (j is 1, 2, 3). The three eigenvectors can be combined into the following eigenmatrix:

Among them, in the area to be detected shown in (a) of FIG. 17 , the boxes not marked with oblique lines, that is, the white boxes represent the pixels where no sunscreen is applied. Boxes marked with slashes represent pixels where sunscreen was applied. The density of the slanted lines can represent the high or low pixel value. In one example, the higher the slash density, the smaller the pixel value. Optionally, the pixel value is used to represent the thickness of the smear.

302. Input feature vectors of multiple regions into a classifier, and obtain a recognition result of the multiple regions by the classifier.

Exemplarily, still referring to FIG. 17, the mobile phone inputs the feature vectors of image region 1-region 3 into the classifier, that is,

The three feature vectors are input into the classifier respectively, or the three feature vectors are combined to form a feature matrix, and the feature matrix is input into the classifier. Afterwards, the classifier detects whether the sunscreen application in the image area 1-area 3 is uniform according to the input feature vector. After calculation, the classifier outputs the classification results of region 1-region 3: region 1<->1, region 2<->0, region 3<->0. Among them, 1 means that the sunscreen is applied unevenly, and 0 means that the sunscreen is applied evenly. That is, the application of the sunscreen in the image area 1 is uneven, and the application of the sunscreen in the image areas 2 and 3 is uniform.

S303. Determine whether the area of each area divided into the area to be detected is smaller than a threshold. If yes, execute step S304. If not, select the image area whose classification result is that the sunscreen is not evenly applied from the multiple areas divided into the area to be detected, update this part of the image area as the area to be detected, and jump to step S301.

It can be understood that for a finer detection of sunscreen application on the skin, the uniformity of application of the sunscreen within a smaller skin size can be detected. Therefore, a threshold can be set. When the area of the image area divided into the area to be detected is smaller than the threshold T3, it is considered that the detection granularity requirement of skin sunscreen application uniformity is met, and no further division of smaller image areas is required. Conversely, when the area of the image area divided into the area to be detected is larger than the threshold, it is necessary to divide the larger image area to form a smaller image area, and to perform smear uniformity detection on the smaller image area until The size and granularity of the image area detected by the smear uniformity meet the threshold, and step S304 is executed.

Exemplarily, after judging, assuming that as shown in Figure 17, the area of each image area in area 1-area 3 is greater than the threshold, then the mobile phone according to the classification results of area 1-area 3: area 1<->1 (smearing is not Uniform), area 2<->0 (smeared evenly), area 3<->0, select from area 1-area 3 the image area where the classification result is that the sunscreen is not evenly applied, that is, select area 1, and set area 1 Update to the region to be detected in subsequent steps.

After that, the mobile phone can jump to step S301 to divide the area to be detected (area 1) into multiple areas. As shown in FIG. 18, area 1 is divided into area 4-area 6. Moreover, the mobile phone can extract feature vectors of multiple regions 4-6, for example,

The eigenvectors of region 4-region 6 can also be combined into the following eigenmatrix:

It should be noted that, each time a smaller image area is divided, the number of small areas into which a large area is divided may be different or the same.

As a possible implementation, the mobile phone can divide smaller image areas according to M=N/iter. Among them, iter is a preset coefficient, M is the number of n+1 subdivisions, N is the number of nth subdivisions, iter, M, and N are all positive integers, and n is an integer greater than or equal to 1.

Exemplarily, when the image area is divided for the first time, the area to be detected may be divided into M=16 image areas J. Taking iter=2 as an example, when dividing the image area for the second time, N=16, iter=2, then, for the image area J formed by the first division, the image area J is divided into M=16/2=8 block image areas K. When dividing the image area for the third time, N=8, iter=2, then, for the image area K formed by the second division, the image area K is divided into M=8/2 block image areas, and so on. Of course, the mobile phone may divide the image area in other manners, which are not limited in this embodiment of the present application.

S304. According to the classification results of the plurality of image regions included in the region to be detected by the classifier, obtain the application uniformity of the sunscreen.

It should be noted that the area of the image area included in the area to be detected may be different. For example, the area to be detected includes area 1-area 3 shown in Figure 17, and area 1 may further include area 4-area 6 shown in Figure 18. Among them, the classification results of areas 1-3 are shown in Figure 17, and area 4-area The classification results of 6 are shown in Figure 18. It can be seen that area 4 is an area where sunscreen is applied unevenly, and areas where sunscreen is evenly applied are areas 2, 35, and 6.

As a possible implementation, the mobile phone calculates the pixel mean value of each image area included in the area to be detected, and judges the thickness of the sunscreen application in each image area according to the pixel mean value. By detecting the average value of pixels in the uniform area and the non-uniform area ˉF. As a possible design, the sunscreen application thickness can be divided into several levels according to the pixel mean value. For example, it is divided into several pixel value intervals [0, T4), [T4, T5), [T5, T6), [T6, T7), [T7, 255], if the pixel mean value ˉF<threshold T4 , it is determined that the sunscreen in this area is very thick (too thick). If T4≤ˉF<threshold T5, it is determined that the thickness of the sunscreen is moderate. If T5≤ˉF<threshold T6, it is determined that the sunscreen is thin. If T7 >ˉF≥T6, it is determined that the sunscreen is very thin (too thin), and if ˉF>T7, it is determined that no sunscreen is applied. Exemplarily, the mobile phone calculates the average pixel value of each image area in area 2, area 3, and area 4-area 6 shown in Figure 17, assuming that the pixel average values of areas 2, 3, 5, and 6 are T4≤ˉF< Threshold T5, it is determined that the thickness of the sunscreen in areas 2, 3, 5, and 6 is moderate, and the average pixel value of area 4 is F<T4, then the mobile phone can determine that the thickness of the sunscreen in this area is relatively thick. It can be seen that from the sunscreen application uniformity detection process, the mobile phone can determine the uneven application of sunscreen, and can also determine the thickness of the uneven application of sunscreen. In some embodiments, the mobile phone can also determine other information about sunscreen application through the detection process of uniformity of sunscreen application, such as the location where no sunscreen is applied, the thickness of the application where the sunscreen is evenly applied, and the like.

After the mobile phone determines the application of sunscreen, it can prompt the user for the application of sunscreen. Including, but not limited to, indicating the position where sunscreen is not applied, indicating the position where the sunscreen is applied unevenly, and indicating the thickness of the sunscreen in the designated area.

As a possible implementation, when the mobile phone prompts the application of sunscreen, the prompt information can be marked in the color image, and different application conditions can be marked with different colors and interface effects. Interface effects include, but are not limited to, flickering effects, glowing effects, and shadow effects. For example, if a certain area is an area where sunscreen is applied unevenly, and sub-area A of this area is an area where the application thickness is thin, then sub-area A is marked by the mobile phone with a red and flashing effect. In this way, the user's attention Concentrate on areas where sunscreen is thinner and reapply sunscreen to that area.

In some other embodiments, some interface effects can also be used to distinguish the application of sunscreen in the grayscale image. For example, in the grayscale image, there are all black pixel areas. The black pixel area 1 marks the flickering effect. The flickering effect can indicate that the sunscreen is applied thinly. The black pixel area 2 is not marked with the flickering effect. The unmarked flickering effect can indicate that the sunscreen is applied with a moderate thickness .

It should be noted that in the embodiment of the present application, pixel value intervals may be further divided according to actual application scenarios, and there is no limit to the thickness of the sunscreen represented by each pixel value interval.

Moreover, in the embodiment of the present application, the pixel value is between 0 and 255 as an example. In other embodiments, when more or less bits are used to represent the color, the pixel value can also have other values The scope is unified here.

Optionally, the colors and interface effects of different regions can be preset by the system or set by the user.

Optionally, the mobile phone can explain the meaning of the application of sunscreen represented by each color and interface effect in the form of text on the prompt interface. Alternatively, after the user clicks on the corresponding area, the mobile phone explains the meaning of the application of sunscreen represented by each color and interface effect in text or other forms.

The embodiment of the present application also provides a detection method, which determines the third area on the sixth image, and according to the first area and the third area in the above-mentioned embodiment, determines the difference between the first face and the first face in the first image. area and the second area corresponding to the third area. Wherein, the first image and the sixth image are images from different angles. The third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, an area where the thickness of sunscreen is greater than the first threshold, and where sunscreen is applied Regions with a thickness less than the second threshold.

Exemplarily, taking FIG. 13 as an example, the first image is an ultraviolet image 901, and the sixth image is an image 902. Assume that the first area determined by the mobile phone on the ultraviolet image 901 is the non-highlight area 407a, and the mobile phone is on the ultraviolet image 902 The determined third area is the

non-highlight area

407b and 410a, then, the mobile phone can determine the second area corresponding to 407b, 410a, and 407a on the ultraviolet image 901 according to the first area and the third area, that is, 407a and 410a. Therefore, the mobile phone can determine that the areas where the sunscreen is applied are 407a and 410a.

As another example, the mobile phone can also collect multiple images from different angles, for example, collect the ultraviolet images 901-903 shown in FIG. 13 , and the mobile phone can determine the application of sunscreen according to the areas in the three ultraviolet images.

The embodiment of the present application also provides a detection method, the mobile phone can obtain the current weather state, including but not limited to obtain the current ultraviolet intensity of the environment. In this way, the mobile phone can recommend information about sunscreen suitable for the current weather state to the user according to the current antenna state. Including but not limited to prompting the user with the sun protection factor (sun protection factor, SPF) of the sunscreen suitable for the current weather state, and the thickness of the sunscreen.

Generally, the intensity of ultraviolet rays can be divided into 5 levels (of course, it can be divided into more or less levels under different grading methods). Usually, the higher the level of ultraviolet intensity, the stronger the ultraviolet rays, and users need to take stronger sun protection measures. When the mobile phone obtains that the current ultraviolet intensity is strong, it can guide the user to protect himself from the sun by one or more of the following methods: recommending a sunscreen with a higher SPF to the user, and prompting the user to apply thicker sunscreen.

It should be noted that a sunscreen with a higher SPF usually has a stronger ability to absorb ultraviolet rays and is more irritating to the skin. Therefore, when the mobile phone recommends sunscreen to users, it does not recommend the sunscreen with the highest SPF to users. , but choose the appropriate sunscreen according to the outdoor ultraviolet intensity, so as to avoid recommending sunscreens with too high SPF to irritate the user's skin.

Optionally, the mobile phone can also detect the user's skin information, such as skin type information, whether it is sensitive skin, oily skin, mixed skin, etc. The mobile phone can recommend sunscreens suitable for the user's skin type to the user based on the user's skin information and other information (including but not limited to weather status information), such as recommending a suitable SPF and application thickness.

In the embodiment of the present application, the mobile phone obtains the weather status information from the cloud side (such as a server), or the mobile phone detects the weather status information by itself.

Taking the ultraviolet intensity obtained by the mobile phone as an example, the mobile phone can detect the current location information, and when it is currently indoors, the mobile phone can obtain the ultraviolet intensity from the server. When currently outdoors, the mobile phone can capture ultraviolet rays through the image sensor in the ultraviolet camera and judge the intensity of ultraviolet rays. For example, in the process of collecting ultraviolet images, the mobile phone judges the intensity of ultraviolet rays according to the ultraviolet rays captured by the ultraviolet camera.

Take the user’s mobile phone detection outdoors as an example. When the user uses the mobile phone’s camera or mirror to check his makeup outdoors, when it is detected that the user triggers the sun protection detection function (such as clicking the sun protection detection control), the mobile phone can call such as color camera Detect faces. After detecting a human face, the mobile phone can call the ultraviolet camera to collect ultraviolet images, and detect the application of sunscreen on the user's face according to the ultraviolet images. During this process, the mobile phone can determine the ambient ultraviolet intensity based on the ultraviolet rays captured by the ultraviolet camera. After that, the mobile phone can prompt the user with information about sunscreen application through the user interface (UI) according to the intensity of the ambient ultraviolet light and the application of sunscreen on the user's face, including but not limited to one or more of the following information: According to the strength of the sunscreen, it is recommended that the user use the SPF of the sunscreen, whether the sunscreen is applied evenly on the face, where the sunscreen is applied unevenly on the face, and where the sunscreen is not applied on the face. Exemplarily, as shown in FIG. 19, after the mobile phone detects the current ultraviolet intensity and the user's application of sunscreen, a prompt box 2401 is displayed, which is used to prompt the user for the current ultraviolet intensity, the position where sunscreen needs to be reapplied, and the User prompts for desired SPF. For another example, as shown in Figure 20, after the mobile phone detects the current ultraviolet intensity and the user's application of sunscreen, a prompt box is displayed to remind the user of the current ultraviolet intensity and to remind the user that the sunscreen is applied evenly on the face.

The embodiment of the present application also provides a detection method. When the user triggers the sun protection detection function, the mobile phone can detect whether there is a previous detection result within a preset period of time. If the detection has not been performed within a preset period of time, the mobile phone can detect the application of sunscreen and prompt the user for the application of sunscreen. If the test has been performed within a preset period of time, the phone can prompt the user with the result of the most recent sunscreen application test. That is to say, not in all cases, the mobile phone will re-detect, so that the calculation load of the mobile phone can be reduced.

Wherein, the preset time period may be related to the sunscreen action time of the sunscreen and the like. For example, assuming that the statistics show that sunscreen usually works for 3-4 hours, then there is usually no need to reapply sunscreen within 3-4 hours.

Exemplarily, after the mobile phone detects the user's operation such as clicking on the sunscreen detection control 402, if the mobile phone detects that the sunscreen detection process has been performed within 3 hours, it prompts the user that there is no need to reapply. Optionally, the mobile phone can also present the last detection result of the sunscreen to the user. Conversely, if the mobile phone detects that the sunscreen detection process has not been performed within 3 hours, it will prompt the user with the detection result.

For another example, when it is detected that the user triggers the sun protection detection function, the mobile phone can determine whether the detection is performed in the morning, if the detection is not performed in the morning, the detection is performed, and the detection result is prompted to the user. If the mobile phone detects that the detection has been carried out in the morning, and the sunscreen has been applied or reapplied in place, then the current time information will be obtained, and it will be judged whether the current time belongs to the preset time period for sunscreen detection (for example, 12:00-14:00 ). If the current time is not between 12:00-14:00, it means that the sunscreen applied in the morning has not expired, and there is no need to re-test. If the current time is between 12:00-14:00, it means that the sunscreen applied in the morning may be invalid. Then, the mobile phone will re-test and prompt the user with the result of this test.

It should be noted that some steps in the above method flow can also be replaced by other steps, or some steps can be added or subtracted. The specific steps included in the method flow are not limited in the embodiments of the present application.

Exemplarily, in some embodiments, a shortcut icon can also be set separately for the sun protection detection function in applications such as mirrors. After detecting the user's operation such as clicking the shortcut icon, the mobile phone can perform detection. Optionally, the mobile phone may prompt the user that sunscreen application is being detected. After the mobile phone detection is completed, the user can be prompted for the detection result. It can be seen that in this scenario, the operation that triggers the mobile phone to start the sunscreen detection process opens the shortcut application of sunscreen detection for the user.

In some embodiments, the user can also perform an operation such as long pressing the mirror application to trigger the mobile phone to pop up the mirror function box 2801 . The user can enable the sunscreen detection function by, for example, clicking on the sunscreen detection option, then the mobile phone will detect the application of sunscreen, and can prompt the user for the application of sunscreen.

As another example, detecting whether to apply sunscreen can also be implemented as: the mobile phone inputs the face image into a classifier for detecting whether to apply sunscreen, and then the mobile phone determines the area where sunscreen is applied and the area where sunscreen is not applied according to the classification result of the classifier area with sunscreen. For another example, the application of sunscreen in the embodiment of the present application can be determined according to the recognition result of the classifier. For the training and use process of the classifier used to detect whether to apply sunscreen, refer to the corresponding embodiment in FIG. 15 , which will not be repeated here.

In some embodiments, after the mobile phone prompts the user about the application of sunscreen, if the user has not exited the mirror application, the mobile phone detects that the user is still using the mirror application for reapplying through the RGB camera, and the mobile phone continues to detect the user's sunscreen application. Cream touch-up condition. For example, the mobile phone can turn on the ultraviolet camera to collect face images at a certain time interval, so as to detect the user's reapplying of sunscreen until the user's reapplied sunscreen is sufficient to resist the ultraviolet rays of the current environment.

Of course, after the mobile phone notifies the user of the application of sunscreen, the mobile phone may not actively detect the user's application of sunscreen, but trigger the detection process again after detecting the user's operation such as clicking the sunscreen detection control.

The interfaces listed in the embodiments of the present application are exemplary, and the embodiments of the present application do not limit the implementation forms of the interfaces. For example, in some interfaces, you can also prompt the location where no sunscreen is applied and the location where sunscreen is applied but not evenly applied. For another example, in the interface that prompts the user to apply sunscreen, both the color view and the grayscale view can be provided. For another example, within a preset period of time after it is detected that the user clicks on the sun protection detection control 402, the mobile phone may stop displaying the sun protection detection control 402, so that the screen has more display space to display other necessary controls.

It should be noted that the above-mentioned embodiments mainly use face detection as an example, but it should be understood that the detection method in the embodiment of the present application may also be applicable to various other objects requiring sun protection. Examples include, but are not limited to, human faces and other parts of the body. Alternatively, it can also be applied to non-human objects, for example, parts of a vehicle that need sun protection.

Moreover, the above embodiment mainly uses the RGB image to identify the human face image, and then searches for the human face image in the ultraviolet image according to the human face image in the RGB image as an example to describe the human face image recognition process. In some other embodiments, the ultraviolet image can also be directly processed using the face recognition algorithm, so as to recognize the human face image in the ultraviolet image and extract feature points such as eyes and lips. The embodiment of the present application does not limit the specific implementation manner of acquiring the human face image in the ultraviolet image.

In addition, the above-mentioned embodiment mainly uses three ultraviolet images from different angles to exclude the influence of reflective parts on sunscreen detection as an example. for other.

In the embodiment of the present application, the color camera and the ultraviolet camera collect synchronously, which may be collected at the same time, or collected within a preset time interval, may be collected first by the ultraviolet camera, and collected by the color camera within a preset time interval, or, The color camera collects first, and within a preset time interval, the ultraviolet camera collects. The embodiment of the present application does not limit the collection timing and collection conditions of the two.

The embodiment of the present application mainly uses the determination of the application of sunscreen based on the pixel value of the ultraviolet image as an example for illustration, and the application of sunscreen may also be determined based on other image features of the ultraviolet image. For example, based on the variance, standard deviation, maximum pixel value, and minimum pixel value in the image features, the uniformity and thickness of sunscreen application can be determined.

Some other embodiments of the present application provide an apparatus, and the apparatus may be the above-mentioned electronic device (such as a folding screen mobile phone). The apparatus may include: a display screen, memory and one or more processors. The display screen, memory and processor are coupled. The memory is used to store computer program code comprising computer instructions. When the processor executes the computer instructions, the electronic device can execute various functions or steps performed by the mobile phone in the foregoing method embodiments. For the structure of the electronic device, reference may be made to the electronic device shown in FIGS. 2A-2F .

Wherein, the core structure of the electronic device may be represented as the structure shown in FIG. 21 , and the core structure may include: a processing module 1301 , an input module 1302 , a storage module 1303 , and a display module 1304 .

The processing module 1301 may include at least one of a central processing unit (CPU), an application processor (Application Processor, AP) or a communication processor (Communication Processor, CP). The processing module 1301 may perform operations or data processing related to control and/or communication of at least one of other elements of the user electronic device. Specifically, the processing module 1301 can be configured to control the content displayed on the main screen according to a certain trigger condition. Or determine what is displayed on the screen according to preset rules. The processing module 1301 is also used to process the input instruction or data, and determine the display style according to the processed data.

The input module 1302 is configured to obtain instructions or data input by the user, and transmit the obtained instructions or data to other modules of the electronic device. Optionally, the input mode of the input module 1302 may include touch, gesture, screen approach, etc., and may also be voice input. For example, the input module may be a screen of an electronic device, acquires user input operations, generates input signals according to the acquired input operations, and transmits the input signals to the processing module 1301 . In the embodiment of the present application, the input module may be used to receive a sunscreen detection instruction input by a user, and/or perform other steps.

The collection module 1306 is configured to collect data and transmit the collected data to other modules of the electronic device. Optionally, the collection module 1306 may be a camera of the electronic device, and the camera may transmit the collected images to the processing module 1301, and/or perform other steps. Cameras include but are not limited to color cameras and ultraviolet cameras.

The storage module 1303 may include a volatile memory and/or a nonvolatile memory. The storage module is used to store at least one instruction or data related to other modules of the user terminal device. Specifically, the storage module can record images collected by the camera.

The display module 1304 may include, for example, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, an Organic Light Emitting Diode (OLED) display, a Micro Electro Mechanical System (MEMS) display or an electronic paper display. Used to display user-viewable content (eg, text, images, videos, icons, symbols, etc.). In the embodiment of the present application, the display module may be implemented as a display screen.

Optionally, the structure shown in FIG. 21 can also have a communication module 1305, which is used to support the electronic device to communicate with other electronic devices. For example, the communication module can be connected to a network via wireless communication or wired communication to communicate with other personal terminals or a network server. The wireless communication may employ at least one of cellular communication protocols, such as Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Communications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). Wireless communications may include, for example, short-range communications. The short-range communication may include at least one of wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), magnetic stripe transmission (MST), or GNSS.

The embodiment of the present application also provides a chip system, as shown in FIG. 22 , the chip system includes at least one processor 1401 and at least one interface circuit 1402 . The processor 1401 and the interface circuit 1402 may be interconnected through wires. For example, interface circuit 1402 may be used to receive signals from other devices, such as memory of an electronic device. As another example, the interface circuit 1402 may be used to send signals to other devices (such as the processor 1401). Exemplarily, the interface circuit 1402 can read instructions stored in the memory, and send the instructions to the processor 1401 . When the instructions are executed by the processor 1401, the electronic device may be made to execute various steps in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which is not specifically limited in this embodiment of the present application.

The embodiment of the present application also provides a computer storage medium, the computer storage medium includes computer instructions, and when the computer instructions are run on the above-mentioned electronic device, the electronic device is made to perform the various functions or steps performed by the mobile phone in the above-mentioned method embodiment .

The embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to execute the various functions or steps performed by the mobile phone in the above method embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be assigned by Completion of different functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk.

The above content is only the specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered within the protection scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

A detection method, wherein the method is applied to an electronic device, and the electronic device includes a first camera and a second camera; wherein the first camera is a color camera, and the second camera is an ultraviolet camera, The methods include:

collecting a first image through a first camera, and collecting a second image through a second camera, wherein the first image contains first content;

determining a first region on the second image;

determining a second area corresponding to the first area on the first image according to the first area;

A user interface is displayed, the user interface includes the first content and first information, and the first information is used to indicate the second area.
The method according to claim 1, wherein the first content contains a first human face; the second image contains a second human face, and the first human face and the second human face belong to the same user;

The determining the first region on the second image specifically includes:

determining the first region on the second face of the second image;

The determining the second area corresponding to the first area on the first image specifically includes:

On the first face of the first image, the second area corresponding to the first area is determined.
The method according to claim 1 or 2, wherein the first area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, and an area where sunscreen is applied unevenly , the area where the sunscreen application thickness is greater than the first threshold, and the area where the sunscreen application thickness is less than the second threshold.
The method according to any one of claims 1-3, wherein after the first image is captured by the first camera and the second image is captured by the second camera, the method further comprises:

collecting a third image through the first camera, and collecting a fourth image through the second camera when the third image satisfies a second condition;

determining a third region on the fourth image;

The determining a second area corresponding to the first area on the first image according to the first area includes:

determining the second area corresponding to the first area and the third area on the first image according to the first area and the third area;

The second condition includes: the third image includes a human face, and the angle of the human face included in the third image relative to the electronic device is within a preset angle range.
The method according to any one of claims 1-4, wherein capturing the first image through the first camera and capturing the second image through the second camera comprises: at the same moment, capturing the image through the first camera The first image and the second image are captured by the second camera.
The method according to any one of claims 1-5, wherein the collecting the first image through the first camera and collecting the second image through the second camera includes:

collecting a fifth image through the first camera;

When the fifth image satisfies the first condition, capturing the first image through the first camera and capturing the second image through the second camera;

The first condition includes: the fifth image includes a human face.
The method according to any one of claims 1-6, wherein the electronic device further includes an ultraviolet supplementary light, and collecting the second image through the second camera includes: turning on the ultraviolet supplementary light light, and collect the second image through the second camera.
The method according to claim 7, wherein the method further comprises:

Determine the ambient UV intensity;

The collecting the second image through the second camera includes:

When the ambient ultraviolet intensity is less than a third threshold, the ultraviolet supplementary light is turned on, and the second image is collected by the second camera.
The method according to any one of claims 4-8, characterized in that, after the first image is captured by the first camera, the second image is captured by the second camera, and the second image is captured by the second camera Before the four images, the method further includes: displaying second information, where the second information is used to prompt the user to adjust the angle of the human face relative to the electronic device.
The method according to any one of claims 2-9, wherein determining the first area on the second face of the second image comprises:

determining a fourth region on the second face of the second image;

The first region is determined in a fourth region of the second image based on pixel values of pixels contained in the fourth region.
The method according to claim 10, wherein determining the first region in the fourth region of the second image according to pixel values of pixels contained in the fourth region comprises:

According to the pixel values of the pixels contained in the fourth area, a plurality of target sub-areas are determined in the fourth area, and a target sub-area among the plurality of target sub-areas whose area is greater than or equal to a fourth threshold is determined as the target sub-area The first area, the target sub-area satisfies the third condition, the third condition includes: the pixel mean value of the target sub-area is less than the fifth threshold, and the pixel value of the pixels contained in the target sub-area is less than the sixth threshold .
The method according to any one of claims 2-10, wherein, before determining the first region on the second image, the method further comprises:

Determining a second human face in the second image based on the first human face in the first image.
The method according to claim 4, wherein the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is unevenly applied, sunscreen The cream is applied to areas with a thickness greater than the first threshold, and the sunscreen is applied to areas with a thickness less than the second threshold.
The method according to claim 10, wherein the fourth area does not include areas corresponding to eyes, eyebrows, and mouth in the human face.
The method according to any one of claims 1-14, wherein the first information is also used to indicate the uniformity of sunscreen application and/or the thickness and/or thickness of sunscreen application in the second area. Whether to apply sunscreen.
An electronic device, characterized in that it includes: a processor, a memory, the memory is coupled to the processor, the memory is used to store computer program codes, the computer program codes include computer instructions, when the processor Reading the computer instructions from the memory causes the electronic device to execute the method according to any one of claims 1-15.
A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, wherein when the instructions are run on an electronic device, the electronic device is made to perform any one of claims 1-15 the method described.
A computer program product containing instructions, characterized in that, when the computer program product is run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-15.